Compare commits

...
905 Commits
Author SHA1 Message Date
Ryan Houdek 512643d3d6 Docs: Update for release FEX-2502 2025-02-08 01:04:19 -08:00
Ryan Houdek b3a69af752 Merge pull request #4338 from Sonicadvance1/fix_vl_int16
FEXCore/vl64: Fixes int16 encoding
2025-02-07 19:08:28 -08:00
Ryan Houdek 64c0dc47a9 unittests/FEXCore: Fixes VL test and adds decode check
So when encoding we also test the decode path.
2025-02-07 15:30:16 -08:00
Ryan Houdek 923c323d6f FEXCore/vl64: Fixes int16 encoding
For some reason when I was writing the tests I got the byte order
incorrect. The type header needs to be in the first byte, not the second
byte.
2025-02-07 15:29:51 -08:00
Ryan Houdek ee47b5bbc9 Merge pull request #4331 from Sonicadvance1/fix_portable_fexserver
FEXServer: Fixes FEX_PORTABLE usage
2025-02-07 12:15:58 -08:00
LC e8cd655c84 Merge pull request #4330 from Sonicadvance1/hotblock_tso_32bit
InstcountCI: Adds a hotblock for 32-bit TSO testing
2025-02-07 14:56:19 -05:00
Ryan Houdek 9af52fb642 Merge pull request #4335 from Sonicadvance1/fix_portable_wine
WINE: Fixes FEX_PORTABLE usage
2025-02-06 17:49:59 -08:00
Ryan Houdek eaddd44d17 Merge pull request #4326 from bylaws/mbfast2
Frontend: Split blocks at jump target boundaries
2025-02-06 17:48:56 -08:00
Ryan Houdek 854e699589 WINE: Fixes FEX_PORTABLE usage
Completely didn't listen to FEX_PORTABLE. Necessary otherwise it can
read configs from some random locations when portable is enabled.
2025-02-06 15:36:27 -08:00
Ryan Houdek 20b00ecc9b Merge pull request #4332 from bylaws/ecdmsk
ARM64EC: Set EC_ENTRY_CPUAREA_REG at inline SMC dispatcher entry
2025-02-05 17:06:32 -08:00
Billy Laws 40662f947f OpcodeDispatcher: Only set mark _Break as setting RIP in the non-MB case
If we're starting a new block here then the newly started block won't have
set RIP and it is erroneous to set it.
2025-02-06 00:01:40 +00:00
Billy Laws 6e01934edc Frontend: Zero InstructionSize before decoding
Required for PeekByte to work correctly before decoding.
2025-02-06 00:01:17 +00:00
Billy Laws 151fc5e97f Frontend: Split blocks at jump target boundaries
With the prior approach, backwards jumps into existing blocks would
explore the overlapping part rather than splitting the block, generating
needless code and wasting time decoding. Similarly, the current block
wouldn't be split when it is extended to overlap with a pending jump target.

Solve this by tracking blocks in a sorted vector and splitting existing blocks
on jumps when appropriate, in order to avoid any possibility of overlapping
blocks, which would break the lookup, misaligned and zero instruction blocks
are disallowed.
2025-02-06 00:01:17 +00:00
Billy Laws 5f431dc776 Frontend: Track the current instruction start address 2025-02-06 00:01:17 +00:00
Billy Laws 5ec4d3125a ARM64EC: Set EC_ENTRY_CPUAREA_REG at inline SMC dispatcher entry 2025-02-06 00:00:38 +00:00
Ryan Houdek 8e511e7db4 FEXServer: Fixes FEX_PORTABLE usage
This was causing FEXServer to look in to global installed paths and
local paths for things when FEXServer was started.

Ensure it listens to FEX_PORTABLE so this doesn't occur.
This also requires us to scan both data directories and config
directories to find them.
2025-02-05 15:59:29 -08:00
Ryan Houdek 99b8046f03 InstcountCI: Adds a hotblock for 32-bit TSO testing 2025-02-05 13:44:51 -08:00
Ryan Houdek d39dea1ae3 Merge pull request #4325 from bylaws/ircopy2
FEXCore: Don't copy IR after compilation
2025-02-05 12:23:33 -08:00
LC e94643d5ca Merge pull request #4302 from Sonicadvance1/vl_jit_reconstruction
FEXCore/JIT: Encode the JITRIPReconstructionEntries using variable length integer
2025-02-05 14:19:15 -05:00
Ryan Houdek 1becbab0dc Merge pull request #4328 from Sonicadvance1/clang_thunks_default
CMake: Default enable clang thunk building
2025-02-04 12:00:53 -08:00
Ryan Houdek d49efb451e CMake: Default enable clang thunk building
We already mandate clang for building FEX and building thunks with clang
has been well tested since the PPA builder has been using it for a long
time.
2025-02-04 11:27:40 -08:00
Ryan Houdek d2a56ebd8c Merge pull request #4223 from Sonicadvance1/netstream_timeout
GdbServer: Implement new netstream that can be interrupted
2025-02-04 10:58:11 -08:00
Billy Laws 0e11a9b7ac FEXCore: Don't copy IR after compilation
This wastes a significant amount of time when the copies IR is promptly
thrown away after compilation anyway.
2025-02-04 16:52:16 +00:00
Billy Laws 68c77d12d4 FEXCore: Don't delete IRListView move constructor 2025-02-04 16:51:59 +00:00
Ryan Houdek dcfbc2f20e FEXCore/JIT: Encode the JITRIPReconstructionEntries using variable length integer
When #2722 implemented this initially and #4271 switched over to signed
int16_t there was assumptions made that int16_t was a reasonable
trade-off in encoding size versus needing to deal with 8-bit values
being too small in some cases.

In the common case we are almost always encoding 8-bit values because
instructions are typically linear (and less than 15-bytes in size), but
16-bit was chosen because optimizing JIT and multiple instructions that
don't cause exceptions can add up to larger than 8-bit.

Instead of hardcoding 16-bit values, implement a variable length integer
class where ~96.8% of values are 8-bit encoded, and the remaining 3.19% are encoded using 16-bit.
Due to some constraints that #4271 put in place, we can basically
guarantee currently that branch targets are within 16-bit. The VL class
does support 32-bit and 64-bit as well so if we change behaviour then
nothing needs to change.

Some stats when running Sonic Mania with multiblock enabled.
Encoded integers: 3,504,907
Encoded 8-bit:    3,393,095 (96.8%)
Encoded 16-bit:     111,812 (3.19%)
Encoded 32/64-bit:        0

Encoded Size:       3,615,181 bytes (3.44MiB)
Fixed encoded size: 7,007,604 bytes (6.68MiB)

Definitely worth using and saves the headache of large RIP/PC offsets
causing problems.
2025-02-03 11:54:52 -08:00
Ryan Houdek 0b29c99fed review 2025-02-03 11:54:12 -08:00
Ryan Houdek 18556a9f75 Netstream: Use a std::variant 2025-02-03 11:54:12 -08:00
Ryan Houdek 02d93782ba GdbServer: Implement new netstream that can be interrupted
A major limitation of iostream is that you can't have reads or writes
with a safe interrupt. Instead rewrite the interface with Linux ppoll so
that these can be safely interrupted with a signal and return early.
2025-02-03 11:54:12 -08:00
Ryan Houdek 62cfc26262 Merge pull request #4322 from Sonicadvance1/protect_first_page_altstack
SignalDelegator: Protect first page of the altstack
2025-02-03 11:53:45 -08:00
Ryan Houdek b01a6b94e7 SignalDelegator: Protect first page of the altstack
When the alt-stack gets overflown then it is hard to see what went wrong
since the TLS variable is no longer accessible.

Protect the first page that contains the TLS variable.

Fixes #4320
2025-02-02 23:21:03 -08:00
Ryan Houdek 713ebf1476 Merge pull request #4315 from neobrain/refactor_irdumper_const
IRDumper: Allow const RA data
2025-01-31 14:57:05 -08:00
Ryan Houdek 26e50efdb2 Merge pull request #4317 from neobrain/change_fexserver_close_timeout
FEXServer: Lower close timeout
2025-01-31 14:56:49 -08:00
Ryan Houdek c8928999bf Merge pull request #4316 from neobrain/fix_check_catch2_version
CMake: Check for compatible Catch2 versions
2025-01-31 14:56:36 -08:00
Ryan Houdek e1f378c6cf Merge pull request #4318 from neobrain/fix_allocator_format_string
Allocator: Fix format string
2025-01-31 14:56:22 -08:00
Tony Wasserka d10222329c Allocator: Fix format string 2025-01-31 15:27:44 +01:00
Tony Wasserka 176fa7ab1d Lower FEXServer close timeout 2025-01-31 15:24:47 +01:00
Tony Wasserka ebb7137839 IRDumper: Allow const RA data 2025-01-31 15:20:23 +01:00
Tony Wasserka d7092a1231 CMake: Check for compatible Catch2 versions 2025-01-31 15:15:35 +01:00
Ryan Houdek 55cbb0b340 Merge pull request #4313 from pmatos/StartupSleepname
Add option StartupSleepProcName
2025-01-30 07:44:46 -08:00
Ryan Houdek db7fb56e9d Merge pull request #4311 from pmatos/RevertPredRA
Revert "Enable RA of SVE Predicate Registers"
2025-01-30 07:41:51 -08:00
Ryan Houdek 2bed7440a8 Merge pull request #4309 from pmatos/3DnowSkip
Skip 3DNow tests with precision issues
2025-01-30 07:40:18 -08:00
Ryan Houdek 0019bdecef Merge pull request #4314 from neobrain/fix_tso_ldr_bitmask
Arm64: Fix bitmask used to match load/store instructions
2025-01-30 06:32:28 -08:00
Tony Wasserka 51d355da30 Arm64: Fix bitmask used to match load/store instructions
When multiple threads simultaneously SIGBUS on the same address, one of them
will perform the backpatching while the other will detect the backpatched
instruction sequence and hence report the SIGBUS as "handled".

This typo broke the instruction detection logic: The second thread would
assume the source of the SIGBUS was unrelated to TSO emulation and hence
report the signal as unhandled (generally triggering program abortion).

In practice, this problem did not manifest as FEX does not currently share
CodeBuffers between threads.
2025-01-30 14:56:17 +01:00
Paulo Matos 28170fd723 Add option StartupSleepProcName
Sleeps only if current process matches this name. Leave empty to sleep
StartupSleep seconds on all processes.
2025-01-30 10:16:35 +01:00
Paulo Matos 44c65c35c8 Revert "Enable RA of SVE Predicate Registers"
This reverts commit fcbf0de05a.

The initial user of this code has been re-implemented in  b148cc6c.
This is not needed any longer so we're removing it.
2025-01-29 11:56:19 +01:00
Paulo Matos d8f8daf48a Skip 3DNow tests with precision issues
Fixes #4280
2025-01-29 08:37:55 +01:00
Ryan Houdek b148cc6ca3 Merge pull request #4292 from pmatos/EnsurePredCacheReset2
Predicate cache alternative implementation
2025-01-28 18:41:52 -08:00
Ryan Houdek 2a4c169fff Merge pull request #4306 from Sonicadvance1/robust_zero_length_envp
FEXLoader/ELFCodeLoader: Be robust against zero length environment variables
2025-01-28 18:41:33 -08:00
Ryan Houdek c2c84e4bd8 FEXLoader/ELFCodeLoader: Be robust against zero length environment variables
For some reason steamwebhelper is setting a zero length environment
variable. This was causing an assert to be raised early as the web
helper was starting up.

Just stop trying to memcpy the zero length string, gets steamwebhelper
working in the steam beta client again
2025-01-28 16:52:55 -08:00
LC c2f8b5b1ba Merge pull request #4299 from Sonicadvance1/fix_48bit_wine
FEX: Allocate a VMA allocator when running on a 48-bit VA
2025-01-28 19:26:31 -05:00
Ryan Houdek 3a33f554a0 FEXCore/unittests: Adds a FlexBitSet test
To ensure correctness
2025-01-28 16:07:51 -08:00
Ryan Houdek 11ce97655b Allocator: Still need to return memory regions to frontend 2025-01-28 16:07:51 -08:00
Ryan Houdek dc866538d4 FEXCore/Allocator: Ensure small reservations aren't used
Anything less than three pages can't be used for FEX allocations due to
VMA implementation details. Plus we may have reduced a single page
reservation to zero with the prior ObjectAlloc size reservation.
2025-01-28 16:07:51 -08:00
Ryan Houdek 48ad9e9a87 Allocator/FlexBitSet: Fixes a rounding issue with small allocation regions
When small regions were being used for VMA allocations (less than 64
pages), this function was truncating the result to zero. Resulting in
incorrect `LiveVMARegion` size calculations. It would calculate that the
FlexBitSet consumes zero bits of space, even though it needs to use at
least 2, or 3 if we actually want to allocate anything from that
LiveVMARegion.

This was noticed in this PR because our VMA region tracking is being
used more, which has a more likely chance to have small VMA regions for
allocating from. Cause a 1page allocation to try and use a 3 page VMA
region for allocation, but failing because the FlexBitSet size wasn't
calculated correctly.

- Page layout:
- [0x0, 0x1000):    struct LiveVMARegion
- [0x1000, 0x2000): FlexBitSet<uint64_t> UsedPages
-  ^ This space wasn't allocated/mprotected due to the size not
   calculating correctly.
- [0x2000, 0x3000): Memory for allocation
2025-01-28 16:07:51 -08:00
Ryan Houdek c75778abeb FEX: Allocate a VMA allocator when running on a 48-bit VA
When running on a system with a 48-bit VA, if FEX does any allocations
between us reserving the upper 128TB and the application running, then
/technically/ we are intersecting with the application's memory region
in the lower 47-bits.

This didn't typically result in any problems due to how ASLR works, but
if we did any large allocations (like #4291 wants with 128MB VMA region)
then these typically get pushed higher in the VA space.

Again not usually a problem, but if you happen to be running an
application that is using MAP_FIXED with hardcoded addresses then this
can stomp over FEX-Emu memory causing problems.

This is what happens with Wine, it reserves the upper-32MB of its 47-bit
VA space, which is /highly/ likely to stomp on FEX memory. In-fact it
likely occurs all the time, we just got lucky with whatever it was
clobbering wasn't used at the time.

On 39-bit VA systems this isn't a problem because the mmap fails
outright with a warning message from WINE.

Because we are already reserving the upper 128TB of VA space, instead
just always enable our allocator and use the regions that were reserved.
We need to be a little bit careful to ensure we don't accidentally
allocate more memory post-reservation but that just requires a small
adjustment to our unique_ptr and constructor for the 64BitAllocator.

This means /all/ FEX-Emu allocations will be in the upper 128TB VA space
when running 64-bit applications on a 48-bit VA system. Which is kind of
nice.

Fixes WINE in #4291 when the allocator stats are bumped to 128MB per
process.
2025-01-28 16:07:51 -08:00
LC fb2a59a67f Merge pull request #4308 from Sonicadvance1/gpuvis_stack_mem
Profiler/GPUViz: Stop allocating memory
2025-01-28 18:11:33 -05:00
LC f4c92756fc Merge pull request #4307 from Sonicadvance1/fix_4296
InstCountCI: Hardcode xchg instructions
2025-01-28 00:02:46 -05:00
Ryan Houdek 657c27556c Profiler/GPUViz: Stop allocating memory
These parsing strings are tiny, less than 64 bytes all the time. Just
stack allocate the buffer. Makes it safer to use during extenuating
circumstances as well, like SIGBUS and SIGSEGV.
2025-01-27 18:49:15 -08:00
Ryan Houdek 42e68d8544 InstCountCI: Hardcode xchg instructions
Nasm between the versions of 2.16.03 and 2.15.05 starting changing the
operand order of the instruction on us. Hard code both operand encodings
to ensure coverage.

Fixes #4296
2025-01-27 18:29:58 -08:00
Ryan Houdek ae69c4d895 Merge pull request #4305 from Sonicadvance1/fix_gpuviz_typo
Profiler/GPUViz: Fixes typo in instant TraceObject
2025-01-27 14:15:09 -08:00
Ryan Houdek 5a0db4d812 Profiler/GPUViz: Fixes typo in instant TraceObject
String parsing was adding a newline but then we failed to use it. I
don't think it caused any issues considering how infrequent instant
profiler objects are used.
2025-01-27 12:59:21 -08:00
Paulo Matos ddd241fe39 instcount: Ensure predicate cache is reset when control flow leaves block 2025-01-27 20:12:48 +01:00
Paulo Matos 3dc7b8d90a asm_tests: Ensure predicate cache is reset when control flow leaves block 2025-01-27 20:12:48 +01:00
Paulo Matos 0bccb1ece5 Ensure predicate cache is reset when control flow leaves block
Whenever the control float leaves the block, it might clobber the
predicate register so we reset the cache whenever that happens.

Fixes #4264
2025-01-27 20:12:43 +01:00
LC bf1e319d90 Merge pull request #4303 from Sonicadvance1/missing_clang_format
CodeEmitter: Fixes clang_format
2025-01-27 01:08:18 -05:00
Ryan Houdek bc6ae7feb4 CodeEmitter: Fixes clang_format 2025-01-26 19:05:50 -08:00
Ryan Houdek 8d6a43d708 Merge pull request #4290 from Sonicadvance1/fix_v6.13
LinuxEmulation: Ensure syscall wrapper declaration has CpuStateFrame as the first argument
2025-01-23 13:55:42 -08:00
Ryan Houdek bd1bca2c3a Merge pull request #4298 from neobrain/fix_libfwd_wl_regression
Library Forwarding/wayland: Fix regression caused by erroneous format
2025-01-23 13:43:18 -08:00
Ryan Houdek 9858ab7388 LinuxEmulation: Ensure syscall wrapper declaration has CpuStateFrame as the first argument
Otherwise crashes occur.
2025-01-23 12:26:10 -08:00
Tony Wasserka 1f6b69573c CI fix 2025-01-23 19:26:00 +01:00
Tony Wasserka 1c8c5b77f1 Library Forwarding/wayland: Fix regression caused by erroneous format
Auto-formatting turned this into "libwayland - client", making FEX fail to
load the host-side equivalent of this library.
2025-01-23 19:20:01 +01:00
Ryan Houdek e9bd037cf9 Merge pull request #4297 from pmatos/upload-art
Update upload-artifact action to v4
2025-01-23 09:22:28 -08:00
Paulo Matos 6f8353ab28 Update upload-artifact action to v4 2025-01-23 16:47:15 +01:00
LC c25720429d Merge pull request #4294 from neobrain/refactor_codeemitter_cleanups
CodeEmitter: Various cleanups
2025-01-23 01:47:32 -05:00
Tony Wasserka 276e9aded3 Merge pull request #4295 from neobrain/fix_changelog_script
Scripts: Fix indentation of changelog items
2025-01-22 13:40:08 -05:00
Tony Wasserka b88ac3359d Scripts: Fix indentation of changelog items
GitHub's markdown parser requires at least 2 spaces to open a new level
of indentation.
2025-01-22 19:24:54 +01:00
Tony Wasserka 264f3be8b4 CodeEmitter: Remove unused Bind validation logic 2025-01-22 18:25:35 +01:00
Tony Wasserka 4282f96d35 CodeEmitter: Use inline constexpr constants over constexpr functions 2025-01-22 18:25:35 +01:00
Tony Wasserka 9cdd759fc1 CodeEmitter: Convert template specialization into function overload 2025-01-22 18:25:35 +01:00
Tony Wasserka 403e8f8702 CodeEmitter: Unify SingleUseForwardLabel and ForwardLabel 2025-01-22 18:25:35 +01:00
Ryan Houdek 2e989e4262 Merge pull request #4293 from pmatos/CleanupCode
NFC: Code cleanup
2025-01-22 09:14:52 -08:00
Paulo Matos 5666a352d4 NFC: Code cleanup
Removing unused declarations.
Cleaning up unused headers and empty lines.
Avoiding static analysis warnings on `const auto` defaulting to int.
2025-01-22 10:22:25 +01:00
Ryan Houdek 1aa8c6f996 Merge pull request #4284 from neobrain/refactor_autoformat_inl
CodeEmitter: Auto-format .inl headers
2025-01-21 13:24:53 -08:00
Tony Wasserka 9882f53613 Scripts: Add inl files to reformat.sh 2025-01-21 21:28:57 +01:00
Tony Wasserka 8760c593ec CodeEmitter: Reformat inl files 2025-01-21 21:28:33 +01:00
Tony Wasserka ad695bdd59 CodeEmitter: Allow inl headers to be processed by external tooling 2025-01-21 21:28:33 +01:00
Ryan Houdek adff4bb1d7 Merge pull request #4289 from pmatos/PassThroughFPRs
Pass through FPRs argument
2025-01-21 09:23:09 -08:00
Ryan Houdek 42c931cf22 Merge pull request #4288 from OFFTKP/sext
Fix slight inaccuracy in test 3_F7_05_2
2025-01-21 09:22:14 -08:00
Paulo Matos 5d44dea47c Pass through FPRs argument 2025-01-21 18:06:41 +01:00
Ryan Houdek 56c95e3b36 Merge pull request #4285 from neobrain/fix_ptso_offsets
Fix crashes in Paranoid TSO mode
2025-01-21 08:31:11 -08:00
Ryan Houdek 840f306a7d Merge pull request #4287 from neobrain/refactor_warn_fixes
Fix warnings about unused objects
2025-01-21 08:30:45 -08:00
Ryan Houdek 9def89d5f8 Merge pull request #4286 from neobrain/refactor_dont_assume
Drop assume-asserting logging macros
2025-01-21 08:30:22 -08:00
offtkp 84c2f93dab Sign extend into RDX 2025-01-21 15:31:34 +02:00
Tony Wasserka b30733e2a7 Fix warnings about unused objects 2025-01-21 12:28:21 +01:00
Tony Wasserka da58e6a597 Fix warnings about unused variables 2025-01-21 12:07:33 +01:00
Tony Wasserka 229e7c5b61 LogManager: Remove assuming assert macros
Placing optimization hints everywhere interferes with debugging of
RelWithDebInfo builds, since the debugger won't be able to reliably
inspect variables or control flow. These hints are better placed on an
individual basis after identifying bottlenecks in a profiler.
2025-01-21 12:01:33 +01:00
Tony Wasserka 26685143be Update code formatting for logging macros 2025-01-21 12:01:33 +01:00
Tony Wasserka e54b9237c6 Drop use of assume-asserting logging macros 2025-01-21 12:01:33 +01:00
LC ac1b6d9482 Merge pull request #4283 from Sonicadvance1/v6.13_syscalls
LinuxSyscalls: Update for new v6.13 syscalls
2025-01-20 20:29:14 -05:00
LC bb6e98a6fc Merge pull request #4282 from Sonicadvance1/v6.13_drm
IoctlEmulation/drm: Update for v6.13
2025-01-20 20:28:58 -05:00
Tony Wasserka 32c75f06b3 Arm64: Drop unnecessary nops in memcpy/memset 2025-01-20 17:58:54 +01:00
Tony Wasserka a5de2d1008 Context: More broadly enable TSO emulation in paranoid TSO mode
Previously, many games would fail to run due to accidentally disabling
TSO emulation in most instructions.
2025-01-20 17:58:54 +01:00
Tony Wasserka f841912c75 Arm64: Implement indirect memory addressing in paranoid TSO mode 2025-01-20 17:58:54 +01:00
Tony Wasserka 3b8c36882d Merge pull request #4270 from bylaws/crosspg
Frontend: Disallow cross-page branches in multiblock
2025-01-20 07:17:36 -05:00
Ryan Houdek fca4c7e6bf LinuxSyscalls: Update for new v6.13 syscalls
Just four new *at variants of the xattr syscalls.
This will also let us use the *at variants for the non-at versions but I
didn't implement that optimization because this is brand new.
2025-01-19 18:41:30 -08:00
Ryan Houdek 5ffc611d13 IoctlEmulation/drm: Update for v6.13 2025-01-19 17:51:49 -08:00
Ryan Houdek 130f02647b Externals/drm: Update to v6.13 2025-01-19 17:49:29 -08:00
Ryan Houdek 981eea6ade Merge pull request #4271 from bylaws/soff
CPUBackend: Make guest RIP reconstruction offsets signed
2025-01-17 14:35:25 -08:00
Ryan Houdek 3f788eb4a8 Merge pull request #4266 from pmatos/FSTOpt
x87 fst/fld optimization for different addrmodes
2025-01-17 13:52:56 -08:00
Billy Laws 486dc974c4 CPUBackend: Make guest RIP reconstruction offsets signed
With multiblock enabled, host code generated from guest code with a
lower address may be placed after host code generated from guest code
with a higher address in a multiblock. As each guest RIP reconstruction
entry is always relative to the one before it the offset needs to be
signed to allow this.
2025-01-17 21:49:43 +00:00
Billy Laws 3b1fbbc766 Frontend: Disallow cross-page branches in multiblock
This avoids both the generation of multiblocks that cover massive spans
of guest code, which causes issues for both context reconstruction
overflowing the RIP offset and attempting to decode branch targets
in unmapped memory regions.

Once support for querying mappings from the FEX frontend is in place this
limit could be increased if necessary, but this seems fine for now.
2025-01-17 21:41:58 +00:00
Ryan Houdek d01db8f293 Merge pull request #4278 from neobrain/refactor_reduce_vixl_options
CMake: Simplify vixl-related options
2025-01-15 14:14:24 -08:00
LC fd09ded049 Merge pull request #4277 from bylaws/wine
Windows: Fix wine check
2025-01-15 13:55:49 -05:00
LC 8e2b4a306d Merge pull request #4260 from Sonicadvance1/profile_win32
Profiler: Setup for usage on Windows
2025-01-15 13:54:54 -05:00
Tony Wasserka 1d58f38aa5 CMake: Clarify that ENABLE_VIXL_SIMULATOR won't work in production 2025-01-15 17:08:59 +01:00
Tony Wasserka 5ff9a83b07 CMake: Drop COMPILE_VIXL_DISASSEMBLER option 2025-01-15 17:04:54 +01:00
Billy Laws f5decb5f83 Windows: Fix overcommit size logic in the wine path 2025-01-14 20:27:10 +00:00
Billy Laws 11fc49a0f8 Windows: Fix wine check
This did not work before :)
2025-01-14 20:27:03 +00:00
Alyssa Rosenzweig 48c03d747a Merge pull request #4273 from bylaws/earlyend
Frontend: End multiblocks early after hitting 2 consecutive null bytes
2025-01-14 12:34:25 -05:00
Billy Laws 643750817a Frontend: End multiblocks early after hitting 2 consecutive null bytes
'add [rax], al' is almost never seen in actual code so the assumption
can be made that we are most likely trying to explore garbage code and
that this will never be hit. If it is then code will be generated at
that point (where Entrypoint == true).
2025-01-14 17:12:11 +00:00
LC a52dd71e44 Merge pull request #4276 from neobrain/fix_vixl_tests
CMake: Compile vixl if ENABLE_VIXL_DISASSEMBLER is set
2025-01-14 11:54:11 -05:00
Alyssa Rosenzweig 8c02bd43df Merge pull request #4269 from bylaws/jumpext
JIT: Avoid OOB EC bitmap checks in ExitFunction
2025-01-14 11:47:58 -05:00
Alyssa Rosenzweig f635a12129 Merge pull request #4272 from bylaws/declimit
Frontend: Stop all decoding once MaxInst/DecodeBufferSize is reached
2025-01-14 11:40:23 -05:00
Paulo Matos 8191c4905b instcountci: x87 fst/fld optimization for different addrmodes 2025-01-14 16:20:51 +01:00
Paulo Matos 58a034b79d asm_tests: x87 fst/fld optimization for different addrmodes 2025-01-14 16:20:47 +01:00
Paulo Matos 2d53867668 x87 fst/fld optimization for different addrmodes
Includes tests and instcountci files and tests.
When the x87 optimizations were implement, we missed
optimizing different addressing modes. This commit addresses this issue.

Discussed in #4252.
2025-01-14 16:20:33 +01:00
Tony Wasserka 8a57fc5838 CMake: Compile vixl if ENABLE_VIXL_DISASSEMBLER is set 2025-01-14 13:01:49 +01:00
Billy Laws 5481e6d79a Frontend: Stop all decoding once MaxInst/DecodeBufferSize is reached
Currently FinalInstruction causes only to the currently decoding block
to be terminated, but that is not enough as both MaxInst and
DefaultDecodedBufferSize are global limits that apply across all blocks
within a multiblock.
2025-01-12 21:27:53 +00:00
Billy Laws c852a58ee3 JIT: Avoid OOB EC bitmap checks in ExitFunction 2025-01-12 21:25:50 +00:00
Ryan Houdek 8cfc016b3f Merge pull request #4265 from pmatos/RevertPredCache
Revert pred cache
2025-01-10 12:25:38 -08:00
Ryan Houdek 8c94b782c6 Merge pull request #4263 from pmatos/patch-1
Print arg type f80Bit
2025-01-10 09:17:49 -08:00
Paulo Matos 1dce4919f2 instcountci: Revert "Cache predicate register generation from pattern" 2025-01-10 12:53:15 +01:00
Paulo Matos cbda688e29 Revert "Cache predicate register generation from pattern"
This reverts commit 72a4063651.

Caused #4264
2025-01-10 12:52:11 +01:00
Paulo Matos 159ed07e68 Print arg type f80Bit 2025-01-10 09:14:49 +01:00
LC a18b2d0e17 Merge pull request #4262 from Sonicadvance1/fix_fileleak
Windows/CRT: Fixes FD leak
2025-01-09 19:51:25 -05:00
Ryan Houdek 4c9adab58d Windows/CRT: Fixes FD leak
Noticed that the FEXCore config file was open forever.
2025-01-09 15:27:12 -08:00
Ryan Houdek 4c9f1b105d CRT/IO: Fixes sharing rules when writing is used
Fixes trace file opening since it needs to share with other users
opening the file for writing.
2025-01-09 14:25:48 -08:00
Ryan Houdek 2290353295 Wine: Ensure the profiler is initialized. 2025-01-09 14:25:48 -08:00
Ryan Houdek c16bf09310 Profiler: Setup for usage on Windows
This will get gpuviz working under Wine.
2025-01-09 14:25:48 -08:00
LC 90db9486ce Merge pull request #4259 from Sonicadvance1/fix_4121
FEXConfig: Fixes instcount not being editable by keyboard
2025-01-08 23:12:50 -05:00
Ryan Houdek b79faa6207 FEXConfig: Fixes instcount not being editable by keyboard
Fixes #4121
2025-01-08 16:43:34 -08:00
LC 2293d3067a Merge pull request #4258 from Sonicadvance1/libraries
cmake: Adds some missing STATIC qualifiers
2025-01-07 19:52:17 -05:00
LC de431f113e Merge pull request #4257 from Sonicadvance1/remove_dup_n2
CPUID: Remove duplicated ARM Neoverse-N2
2025-01-07 19:51:29 -05:00
Ryan Houdek 34e265a801 cmake: Adds some missing STATIC qualifiers
Noticed this as I was scrolling through some cmake. Usually this doesn't
matter as we declare `BUILD_SHARED_LIBS` as False/Off, but this can
technically be overridden even when we don't want to.

Updates the two definitions of `add_library` that was missing the static
qualifier to ensure they generate the code we want.
2025-01-07 16:01:34 -08:00
Ryan Houdek a668492fb7 CPUID: Remove duplicated ARM Neoverse-N2
This was declared twice in the list.
2025-01-07 15:32:38 -08:00
Ryan Houdek da069571f3 Docs: Update for release FEX-2501 2025-01-07 13:07:46 -08:00
Ryan Houdek d2bac45b49 Merge pull request #4256 from bylaws/crtd
Windows: Only deinit the thread CRT when destroying the current thread
2025-01-06 21:46:32 -08:00
LC 8913c59acc Merge pull request #4250 from Sonicadvance1/staticanalysis
Just a few things picked up from static analysis
2025-01-06 19:11:38 -05:00
LC c3261b4aeb Merge pull request #4249 from Sonicadvance1/log_bad_fork_flags
LinuxSyscalls: Log unhandled clone3 fork flags
2025-01-06 19:11:02 -05:00
LC c7fb95aec5 Merge pull request #4248 from Sonicadvance1/fix_cefsimple
LinuxSyscalls: Ensure CSIGNAL is merged back in to flags for clone2
2025-01-06 19:10:18 -05:00
Billy Laws c00cef6dc1 WOW64: Fix warning 2025-01-06 19:07:07 +00:00
Billy Laws 429ff94dc5 Windows: Only deinit the thread CRT when destroying the current thread
The thread termination callback can be called for other threads in the
process, not just the current one, in which case we cannot call DeinitCRT.
Deinitializing the CRT of another thread would be awkward so just skip that
and accept the small leak for now.
2025-01-06 19:07:07 +00:00
LC a6c67ca749 Merge pull request #4251 from Sonicadvance1/ir_numelements_to_elementsize
IR: Change convention from number of elements to elementsize
2025-01-04 18:39:24 -05:00
LC f51812a670 Merge pull request #4253 from Sonicadvance1/minor_f80_opt
x87StackOptimizationPass: Minor opt to f80 fchs and fabs
2025-01-04 06:42:29 -05:00
Ryan Houdek 686294f1c4 InstcountCI: Update 2025-01-03 13:49:40 -08:00
Ryan Houdek a47ed105e7 x87StackOptimizationPass: Minor opt to f80 fchs and fabs
It's faster to load the f80 sign mask from our named vector constants
than synthesizing the values. Changes a 4 instruction sequence to
synthesize to be 1 load.
2025-01-03 13:47:10 -08:00
Ryan Houdek b2d579a268 OpcodeDispatcher: Assert on invalid size to LoadRegCachePair
Coverity scan
2025-01-03 11:06:30 -08:00
Ryan Houdek eb1050092f OpcodeDispatcher: Assert on invalid size to SelectPairAddressMode
Coverity scan
2025-01-03 11:05:38 -08:00
Ryan Houdek b3794f5541 OpcodeDispatcher: FEX_UNREACHABLE in programming error case
Coverity scan
2025-01-03 11:05:38 -08:00
Ryan Houdek 1ecfa3253d IR: Change convention from number of elements to elementsize
The IR stores elementsize, where the json was wanting number of
elements. While the IR Emitter function declaration always wanted
element size. This was causing us to do a little dance from ElementSize
-> Number of elements -> ElementSize. Just pass the ElementSize directly
instead of this bogus little dance.
2025-01-03 11:01:03 -08:00
Ryan Houdek 5daf007b6a OpcodeDispatcher: FEX_UNREACHABLE in programming error case
Coverity scan
2025-01-03 10:34:22 -08:00
Ryan Houdek 8efa5febd0 LinuxSyscalls: Log unhandled clone3 fork flags
Make sure to pass the clone3 arguments all the way to the fork handler
so it can check the flags. Currently nothing I know of uses fork plus
the new clone3 flags, but it would be hard to see without any logging.
2025-01-03 09:03:47 -08:00
Ryan Houdek 5fee8028cd LinuxSyscalls: Ensure CSIGNAL is merged back in to flags for clone2
This fixes #4247
2025-01-03 08:34:44 -08:00
LC 6bc7a83c64 Merge pull request #4245 from Sonicadvance1/update_kernel_minspec
FEXLoader: Increase minimum kernel requirement from 5.0 to 5.15
2025-01-02 14:48:26 -05:00
LC e55b5d0d11 Merge pull request #4246 from Sonicadvance1/fix_typo
Linux: Fixes typo in removing RESOLVE_IN_ROOT flag
2025-01-02 14:46:53 -05:00
Ryan Houdek 19de7f2785 Linux: Fixes typo in removing RESOLVE_IN_ROOT flag 2025-01-02 10:18:07 -08:00
LC e32c5384ab Merge pull request #4243 from Sonicadvance1/fix_4155
FEXLoader: Enable early logs output to stderr
2025-01-01 14:23:51 -05:00
LC b391fe6b92 Merge pull request #4244 from Sonicadvance1/fix_4150
unittests/ASM: Fix incorrect instruction form test
2025-01-01 14:23:04 -05:00
Ryan Houdek 4cfb81156f FEXLoader: Increase minimum kernel requirement from 5.0 to 5.15
Brought up in #4225 where it had issues with Openat2 which was added in
5.8.

The main driving force around minimum kernel version requirement is that
the lowest kernel version in our CI is 5.15. A benefit to this choice is
that this is an LTS release, which is also what Ubuntu 22.04 is
shipping.

Once the single CI machine is fixed to ship something newer then the
next logical choice would be kernel 6.1 which is also LTS, but until
then just lift it to 5.15. This version was released in October 2021,
and is supported by the kernel developers until 2026. Our previous
minimum of 5.0 was released in March 2019, so a two year leap here.

This removes the openat2 workaround that was necessary to pass our CI
since it is no longer necessary.
2025-01-01 11:22:54 -08:00
Ryan Houdek 6121708e55 unittests/ASM: Fix incorrect instruction form test
This test was generating the wrong form of instruction. There's no way
to choose this form with nasm deliberately, so manually encode it.

Fixes #4150
2025-01-01 10:12:51 -08:00
Ryan Houdek 6ab214adea FEXLoader: Enable early logs output to stderr
Some early FEXServer startup log failures weren't getting printed
correctly. They were going through the LogManager but before FEXServer
setup, or even stderr/stdout logman setup. So they were just getting
written to -1 and failing.

Fixes #4155
2025-01-01 10:00:22 -08:00
LC 90b1ac4162 Merge pull request #4241 from Sonicadvance1/fix_h0f3a_rex_decode
OpcodeDispatcher: Fixes FEX's H0F3A table handling of REX.W
2025-01-01 11:55:08 -05:00
LC 3abe6c14a1 Merge pull request #4240 from Sonicadvance1/3dnow_modrm_sib_test
unittests: Adds a 3DNow! ModRM SIB encoding test
2025-01-01 11:53:11 -05:00
LC fc1b500eff Merge pull request #4242 from Sonicadvance1/missing_tests
unittests/ASM: Adds missing MMX PADDQ test
2025-01-01 11:52:15 -05:00
Ryan Houdek 5d47b9195b unittests/ASM: Adds missing MMX PADDQ test 2025-01-01 08:22:38 -08:00
Ryan Houdek a8272b74f6 unittests/ASM: Ensure REX.W prefixed instructions from H0F3A are tested
We just want to ensure these instructions are decoded, the regular tests
are ensuring that the behaviour is correct.
2025-01-01 08:22:19 -08:00
Ryan Houdek 12dc16780f OpcodeDispatcher: Fixes FEX's H0F3A table handling of REX.W
Most of this table ignores REX.W, but two encodings change behaviour
based on REX.W. These two encodings are PEXTRD/PEXTRQ and PINSRD/PINSRQ.

For every other instruction encoding, they will ignore REX.W, but FEX
was requiring that they didn't have REX.W encoding. I had special cased
this in the past by adding PALIGNR, but that didn't handle any of the
other instructions.

We can't just handle REX.W in the OpcodeDispatcher and remove the two
special cased instructions because these vector operations also interact
with instruction prefix 0x66 which changes the operating size to 16bit
with regular instructions.

So instead just generate all listings of instructions with REX.W being
zero and one and install handlers in all cases.
2025-01-01 08:22:19 -08:00
Ryan Houdek b8af569841 unittests: Adds a 3DNow! ModRM SIB encoding test
This codepath was unttested in our CI.
2025-01-01 08:21:56 -08:00
LC 8bee101795 Merge pull request #4232 from Sonicadvance1/disable_gvisor_tests
unittests/gvisor: Disable memfd tests
2025-01-01 08:38:36 -05:00
Ryan Houdek 2d66bc258a Merge pull request #4225 from asahilina/merged-rootfs
Support a merged RootFS (and a bunch of related fixes)
2024-12-31 17:29:06 -08:00
Ryan Houdek d2f86e49f7 Merge pull request #4237 from bylaws/fpfix
Fix float->int conversion overflow behaviour
2024-12-31 16:00:20 -08:00
Ryan Houdek d66cd16cfb Merge pull request #4230 from asahilina/thunks-build-sysroot
Library Forwarding: Allow reading standard library headers from a development x86 rootfs
2024-12-30 18:00:34 -08:00
Ryan Houdek 04e785e434 Merge pull request #4231 from Sonicadvance1/minor_div_opt
OpcodeDispatcher: Minor division improvement
2024-12-30 17:32:53 -08:00
Ryan Houdek 15a1a0f7d9 Merge pull request #4239 from bylaws/3dn
Frontend: Fix ModRM handling with 3DNow!
2024-12-30 17:31:58 -08:00
Billy Laws 0a58ce6134 Frontend: Fix ModRM handling with 3DNow! 2024-12-30 18:35:39 +00:00
Billy Laws 8f5607f0e8 Update InstCountCI 2024-12-30 01:07:36 +00:00
Billy Laws a21789d3d8 ASM_Tests: Test F2I conversion overflow behaviour 2024-12-30 00:47:00 +00:00
Billy Laws efd6e95059 OpcodeDispatcher: Match x86 overflow behaviour for F2I conversions
ARM behaviour here is to saturate on overflow or NaN inputs, whereas
X86 returns a sentinel value of 2^(bitsize-1), explicitly emulate this.
2024-12-30 00:42:55 +00:00
Billy Laws 9bdb1f4306 OpcodeDispatcher: Make narrowing implicit for F64->I32 conversions
This is always used, removing it avoids needing to handle unused codepaths.
2024-12-30 00:36:17 +00:00
Billy Laws ae4b7135d5 OpcodeDispatcher: Share AVX F2I/I2F code for 256-bit SVE 2024-12-30 00:29:31 +00:00
Tony Wasserka d503366816 Library Forwarding: Allow reading standard library headers from a development x86 rootfs 2024-12-24 19:41:29 +09:00
Ryan Houdek 0fe2827fcc unittests/gvisor: Disable memfd tests
This tests some bugged or changed behaviour. So we need to disable these
since our CI crosses kernel versions that hit both behaviour paths.
2024-12-22 03:11:08 -08:00
LC cd6722f77b Merge pull request #4229 from Sonicadvance1/more_lrcpc2_tests
InstCountCI: Adds more LRCPC2 tests that are missed
2024-12-20 22:57:06 -05:00
Ryan Houdek ffb745b662 InstCountCI: Update for divison improvements 2024-12-20 13:22:42 -08:00
Ryan Houdek bb10f25808 OpcodeDispatcher: Minor division improvement
No need to extract the subregisters out before operating on them since
the long division and long remainder IR operations correctly zero/sign
extend the incoming sources as necessary. Saves a couple of
instructions.
2024-12-20 13:20:52 -08:00
Ryan Houdek aa1076d12b InstCountCI: Adds more LRCPC2 tests that are missed
We weren't testing 64-bit variants, and we also weren't testing 8-bit
and 16-bit loadstores. Add some more to ensure we are hitting these.
2024-12-20 12:12:24 -08:00
Ryan Houdek 1e827ec7a6 Merge pull request #4227 from Sonicadvance1/fix_atomic_loadstore
ArchHelpers/Arm64: Fixes LDAPUR and STLUR backpatching
2024-12-20 11:46:13 -08:00
Asahi Lina 3fe2650787 FileManagement: Gate new openat2() codepaths on recent enough kernel 2024-12-21 00:52:12 +09:00
Asahi Lina 3e99e814bc FileManagement: Use openat2() with RESOLVE_IN_ROOT for RootFS open ops
This avoids having to do the symlink chasing in GetEmulatedFDPath, since
the kernel does it for us. On top of that, with a merged RootFS
setup, this will correctly handle symlinks from user directories into
the RootFS, fixing wine on Fedora.
2024-12-21 00:52:11 +09:00
Ryan Houdek 2019f8138e ArchHelpers/Arm64: Fixes LDAPUR and STLUR backpatching
The immediate offset masking was at the completely wrong offset when I
wrote these handlers. No idea how I managed to mess those up so badly.

Should fix at least some of the issues with #4216
2024-12-19 17:29:45 -08:00
LC e44d1f136b Merge pull request #4226 from alyssarosenzweig/instc/factorio
InstructionCountCI: add some hot blocks from Factorio
2024-12-19 15:52:59 -05:00
Alyssa Rosenzweig 09872402df InstructionCountCI: add some hot blocks from Factorio
Factorio hammers its drawSprite() function and ends up cpu bound under FEX.
Unfortunately, its hot blocks seem to be translated pretty optimally :-/

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-12-19 15:15:38 -05:00
Asahi Lina b078a41a02 FileManagement: Fix return val of readlink*
The wrappers handle errno, we just need to return -1 on errors.
2024-12-20 03:01:54 +09:00
Asahi Lina 3a5eeb5700 Syscalls: Fix multiple shebang handling issues
- Parse the shebang line properly (use FHU::ParseArgumentsFromString
  which is the same code the loader uses)
- Make native-interpreter shebang files work by deferring to the kernel
  in that case (previously, they'd get executed through the loader and
  it would choke on the architecture of the interpreter)
- Do not use the RootFS-prepended path when executing shebang files. The
  loader will prepend that anyway when looking it up, but it needs the
  bare guest path so it can pass it as an argument to the interpreter,
  which (since it's emulated) will do the lookup through the RootFS.
2024-12-20 03:01:54 +09:00
Asahi Lina 9433ae3405 Syscalls: Handle execve of native binaries with merged RootFS
With a merged RootFS, all binaries are executed through the RootFS. When
executing a binary that is actually a native binary, we want to do so
outside the RootFS. Handle this by stripping the RootFS prefix in that
case.
2024-12-20 01:58:12 +09:00
Asahi Lina 4658b24f9a FileManagement: Handle RootFS symlinks into RootFS properly
If a RootFS symlink links to an absolute path within the RootFS, we need
to strip the RootFS prefix. This would not normally happen with a plain
RootFS, but it can happen if /proc is mounted within the RootFS.
2024-12-20 00:41:07 +09:00
Asahi Lina 4e7d0e6be0 FileManagement: Fix path resolution for symlinks to the root
If there's a symlink to / within the RootFS, don't attempt to follow it,
since that will end up trying to look up the empty string within the
RootFS (which is not legal). Just return the symlink.
2024-12-20 00:41:07 +09:00
Asahi Lina 4ddd98708f FileManagement: Handle readlink /proc/self/fd/* properly
If the guest reads a RootFS path from /proc/self/fd/*, we should return
it with the RootFS prefix stripped.
2024-12-20 00:41:07 +09:00
Asahi Lina c161fd218c FileManagement: Simplify emulated file lookup
To locate whether a path is in the emulated list, EmulatedFDManager::OpenAt()
attemps to resolve the path. realpath() ends up calling readlinkat() on
every path component, which is a lot of syscalls for every open()
variant syscall. It also makes interaction with the rootfs complex and
error-prone.

There's a much easier way to do this: We just open the file without
emulation and check its real path via get_fdpath(). This is just one
readlink() syscall per open, instead of one per path component. If the
file turns out to be emulated (uncommon case), we swap out the fds.

This also decouples EmulatedFDManager from guest path resolution
entirely, so it will never fall out of sync with the RootFS logic.
2024-12-20 00:41:07 +09:00
LC 7e257cc268 Merge pull request #4222 from bylaws/fmtt
External: Update bundled libfmt
2024-12-18 19:54:55 -05:00
Ryan Houdek d8ef70280c Merge pull request #4221 from Sonicadvance1/threadmanager_footexplosions
ThreadManager: Add some sanity asserts
2024-12-18 11:30:19 -08:00
Billy Laws ec003281be External: Update bundled libfmt 2024-12-18 15:25:45 +00:00
Ryan Houdek e58f67b76c ThreadManager: Add some sanity asserts
These couple of functions have some footguns that I'm encountering while
rewriting gdbserver. Ensure that assertion builds capture the problems
2024-12-17 15:06:53 -08:00
LC 57178abcd2 Merge pull request #4220 from Sonicadvance1/expose_faultsafe
Linux/FaultSafeUserMemAccess: Break out fault safe handler
2024-12-16 17:02:06 -05:00
Ryan Houdek 73ca4f8314 Linux/FaultSafeUserMemAccess: Break out fault safe handler
This is going to get used by gdbserver soon for ensuring memory accesses
are fault safe, because it tries to read outside of correct memory
bounds at times.
2024-12-16 11:06:15 -08:00
LC 527752c25b Merge pull request #4218 from Sonicadvance1/fix_file_loading
Utils/FileLoading: Fix LoadFileImpl
2024-12-13 22:57:35 -05:00
Ryan Houdek 38fa866c91 Utils/FileLoading: Fix LoadFileImpl
It is not an error that pread returns /less/ than what was requested. In
fact it's very common for the Linux kernel to return less than the data
requested from procfs.

procfs keeps coming back to bite this function, previously it was fstat
returning size of 0 which it hit. Now it only feeds data as much as it
wants per loop. In particular /proc/self/maps would only read ~3k bytes
on my system, but not be complete.

To fully fix the issue, always make sure to keep reading until there is
either an error OR zero is reached!
2024-12-13 19:42:00 -08:00
Ryan Houdek c902b8807a Merge pull request #4215 from alyssarosenzweig/fix/constprop-zext
ConstProp: fix 32-bit masking behaviour
2024-12-13 17:33:30 -08:00
Alyssa Rosenzweig 4934c1fd94 InstCountCI: Update
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-12-13 10:44:56 -05:00
Billy Laws 766fbe3db3 unittests: Add a test for constprop size bugs
fails on main, fixed by this PR.
2024-12-13 10:44:56 -05:00
Alyssa Rosenzweig 29405f2690 ConstProp: fix 32-bit masking behaviour
if we want to replace a node with one of its sources, we need to zero extend if
the source is 64-bit and the destination is 32-bit.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-12-13 10:44:56 -05:00
Alyssa Rosenzweig 51f505acca ConstProp: drop some unused headers
ycm complained.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-12-13 10:44:56 -05:00
Alyssa Rosenzweig 77415538f7 OpcodeDispatcher: use 64-bit XOR for AF calc
we don't need masking and the masking gets in the way of constprop.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-12-13 10:44:56 -05:00
Alyssa Rosenzweig 9fb69ed206 Merge pull request #4209 from Sonicadvance1/tso_support_instcountci
InstCountCI: Implement support for TSO and LRCPC and add hot block that could be optimized
2024-12-13 09:33:45 -05:00
LC 735a4f90db Merge pull request #4212 from Sonicadvance1/fix_encoding
GdbServer: Fixes encoding of hex
2024-12-12 22:49:43 -05:00
Ryan Houdek 7ef8dc13ba GdbServer: Fixes encoding of hex
Just a typo accidentally prefixing 0x on the hex when it shouldn't.
2024-12-12 16:15:41 -08:00
Ryan Houdek 656477ec63 Merge pull request #4165 from bylaws/denuvo
Support inline self modifying code
2024-12-12 13:38:31 -08:00
Billy Laws d080180e85 ARM64EC: Process pending cross-process work on syscalls and exceptions
This is used to notify the JIT of e.g. memory writes by a debugger.
2024-12-12 21:28:37 +00:00
Billy Laws af1d2d6005 ARM64EC: Implement inline SMC support using context reconstruction
When an SMC trap happens: reconstruct the context before the SMC write
then compile the write as a single instruction block to reduce it to
regular SMC. SMC where the writing instruction is the instruction being
patched will hit the signal handler at most twice: the 1st will trigger
the write to be compiled as a single instuction block, the 2nd will
detect inline SMC of a single instruction block and then just take the
usual invalidate+reprotect+continue step, avoiding a potential infinite
loop of recompilation.
2024-12-12 21:28:37 +00:00
Billy Laws 90c1282f3a Dispatcher: Support forcing a temp single instr block on ARM64EC JIT entry 2024-12-12 21:28:37 +00:00
Billy Laws d5d7eec8b0 FEXCore: Expose an API to check if the current block represents a single
guest instruction

Single instruction blocks need to be treated specially when inline SMC
is detected, the frontend only needs to reprotect RWX and invalidate
caches then continue execution as side effects from the SMC shouldn't be
seen until the instruction executes.
2024-12-12 21:28:37 +00:00
Billy Laws 5337b9537d FEXCore: Expose an API to query intersection with the current block
Frontends need to detect this in order to handle SMC within the current
block (inline SMC) differently to regular SMC which can just reprotect
and continue.
2024-12-12 21:28:37 +00:00
Billy Laws e72c016230 Core: Split blocks on invalid instructions 2024-12-12 21:28:37 +00:00
Ryan Houdek 072cf4c5bd Merge pull request #4205 from Sonicadvance1/gdbserver_support_32bit
GdbServer: Support 32-bit context definitions
2024-12-12 12:55:24 -08:00
Ryan Houdek 27ededf47f Merge pull request #4206 from bylaws/smcim
Windows: Track RWX regions in mapped images
2024-12-12 12:53:13 -08:00
Ryan Houdek 82d7f9fdd7 GdbServer: Support 32-bit context definitions
Requires restructuring a couple of things, but nothing too crazy here.
2024-12-12 12:35:58 -08:00
Ryan Houdek d85153d6b3 GdbServer: Save off some signal information when it occurs
Enough for some state reconstruction that is missing
2024-12-12 12:14:55 -08:00
Ryan Houdek 6b698e6cd1 SignalDelegator: Make SpillSRA public
GdbServer wants to use it
2024-12-12 12:14:54 -08:00
Ryan Houdek 9475f79ec6 GdbServer: Save off SignalDelegator 2024-12-12 12:14:54 -08:00
Ryan Houdek f906c6a0f4 Merge pull request #4211 from asahilina/pthread-attr-memleak
Threads: Fix memory leak in joinable()
2024-12-12 12:13:17 -08:00
Ryan Houdek e88c92de57 Merge pull request #4161 from bylaws/tf
FEXCore: Emulate EFLAGS.TF
2024-12-12 11:51:53 -08:00
Asahi Lina 48ed906a7b Threads: Fix memory leak in joinable() 2024-12-13 04:47:41 +09:00
LC b03b02d2f2 Merge pull request #4210 from Sonicadvance1/add_missing_comment
IR/Passes: Adds missing comment that clang-format keeps complaining about locally
2024-12-11 18:51:57 -05:00
Ryan Houdek d00d476a0a IR/Passes: Adds missing comment that clang-format keeps complaining about locally
NFC
2024-12-11 15:03:28 -08:00
Ryan Houdek ac1e32994a InstCountCI: Adds hot block that doesn't generate optimal code 2024-12-11 15:03:02 -08:00
Ryan Houdek 800d447f3d InstCountCI: Add support for TSO and LRCPC1/2 2024-12-11 14:55:19 -08:00
LC 8111b7cc7f Merge pull request #4194 from Sonicadvance1/fcw_pc_instructions
FEXCore: Override x87 precision control when necessary
2024-12-10 17:24:55 -05:00
LC a86c922073 Merge pull request #4203 from Sonicadvance1/const_reconstruct
Context: Constify GPRs passed to ReconstructCompactedEFLAGS
2024-12-10 12:46:05 -05:00
LC 46fb8583bb Merge pull request #4204 from Sonicadvance1/gdbserver_vkill
GdbServer: Implement support for `$vKill`
2024-12-10 12:45:06 -05:00
Billy Laws 3487d120ec Windows: Treat PAGE_EXECUTE_WRITECOPY memory as RWX 2024-12-10 15:26:23 +00:00
Billy Laws 07394d6a6e Windows: Track RWX regions in mapped images
As section permissions are set on the unix side we don't get a
protection callback for them, workaround this by iterating over
the sections of all executables after mapping and tracking the RWX
ones.
2024-12-10 15:24:54 +00:00
Billy Laws 8d3204171c instcountci: update 2024-12-10 15:24:03 +00:00
Billy Laws 7641f722e9 unittests: Test TF 2024-12-10 15:20:47 +00:00
Billy Laws b51fa497c5 OpcodeDispatcher: Mask TF for pop ss instructions 2024-12-10 15:20:47 +00:00
Billy Laws 34722bed3d SignalDelegator: Clear TF when running signal handlers 2024-12-10 15:20:47 +00:00
Billy Laws 981c3009ee FEXCore: Emulate EFLAGS.TF
When set - either via POPF or a thread context operation - the trap flag
raises a single step exception after the execution of each instruction.
As e.g. a JUMP instruction with TF set will raise an exception at the
jump target. Handle this on the FEX side by storing both the flag itself
(in bit 0) and a 'block exceptions' flag (in bit 1, inverted). Each
generated block when TF is set is then forced to a single instruction
with logic to raise the exception at the start. Initially after setting
TF exceptions are blocked, then at the start of the block they are
unblocked so that after the instruction executes an exception is raised
at the start of the next block.
2024-12-10 15:20:47 +00:00
Ryan Houdek 38cf357d85 GdbServer: Implement support for $vKill
This is the command used when the `k` argument is passed to gdb. There
is nothing to do once this is received other than "kill" as quickly as
possible. The absolute way to ensure this is using SIGKILL.

No way to do a `r` command after `k` yet, but might be possible.
2024-12-09 15:13:52 -08:00
Ryan Houdek 2533ed4a63 Context: Constify GPRs passed to ReconstructCompactedEFLAGS
This only reads the GPRs passed in, doesn't modify it.
2024-12-09 15:08:59 -08:00
Ryan Houdek 5a4691fdfc Merge pull request #4201 from Sonicadvance1/remove_lock
FEXCore: Don't `WaitForEmptyJobQueue` if CodeObjectCacheService isn't used
2024-12-09 10:35:20 -08:00
Billy Laws 6c035a0d61 Dispatcher: Split out some common code into lambdas 2024-12-09 14:15:28 +00:00
Billy Laws a234aa300d Dispatcher: Skip extra L1 lookup after CompileBlock 2024-12-09 12:30:00 +00:00
Billy Laws f6abbedbd1 ARM64EC: Fix typo so TF is unset handling exceptions 2024-12-09 12:30:00 +00:00
Billy Laws b6fe4cd6dd Windows: Skip state reconstruction on exceptions in dispatcher
The dispatcher always spills register state before issuing faulting instructions
2024-12-09 12:30:00 +00:00
LC bdae4f6915 Merge pull request #4200 from Sonicadvance1/fix_exit
LinuxSyscalls: Fixes exit syscall
2024-12-08 19:31:16 -05:00
LC f8b6edfb2b Merge pull request #4199 from Sonicadvance1/remove_arch
docs: Remove Arch from the release process.
2024-12-08 19:29:42 -05:00
Ryan Houdek 0a1ecdf6ae FEXCore: Don't WaitForEmptyJobQueue if CodeObjectCacheService isn't used
Seems the unused mutex locking is able to cause some hangs according to #4198
Hard to tell why, but might as well as get rid of that potential
pitfall.
2024-12-08 08:01:28 -08:00
Ryan Houdek beec203f56 LinuxSyscalls: Fixes exit syscall
if an application is using `exit` then it is usually a faulting
condition rather than cleanly exiting. When cleanly exiting
applications will typically use `exit_group` instead.

`exit` is useful to quickly cause a single thread to exit in a
multi-threaded environment as well, where `exit_group` will take down
the entire process group.

FEX had implemented this in a way that would do a double Stop signal,
cascading to a crash. When tied in to a crash handler, this could get
caught in a weird way.

This /should/ fix #4198, but I can't confirm locally. It looks like in
that issue that the steam install is slightly buggered (as evident by
missing srt-logger and steam-runtime-identify-library-abi).

This is a bug regardless so fix it and create a unittest. If it doesn't
fix the user's bug, then we have another workaround that will definitely
solve it.
2024-12-08 05:14:19 -08:00
Ryan Houdek d323032ec9 docs: Remove Arch from the release process.
On December 6th 2024, the fex-emu packages got a deletion request:

> MarsSeed [1] filed a deletion request for fex-emu [2]:
>
> ARM-only package.
> This should be submitted to ArchLinuxARM.org [a], not to AUR - see
> quote from ArchWiki [b]:
>
>     "Packages that do not support the x86_64 architecture
>     are not allowed in the AUR."
>
> [a]:
> https://archlinuxarm.org/forum/viewforum.php?f=4
> [b]:
> https://wiki.archlinux.org/title/AUR_submission_guidelines#Rules_of_submission
>
> [1] https://aur.archlinux.org/account/MarsSeed/
> [2] https://aur.archlinux.org/pkgbase/fex-emu/

This is due to a rule clarification that occured in Arch's forum on November 25th: https://lists.archlinux.org/archives/list/aur-general@lists.archlinux.org/thread/IRZ2LWYX3ECPJQZJXMLAP6JIKL6HLHPZ/#GMYC74CRSFH7GGNENEUOODZUPWHOMX7A

On December 3rd the package submission guidelines on their wiki was
updated to mandate x86-64 support:
https://wiki.archlinux.org/index.php?title=AUR_submission_guidelines&diff=prev&oldid=822050

As of today, December 7th, 2024 the packages have been removed from AUR
due to only supporting aarch64.

> Muflone [1] deleted fex-emu [2].
>
> You will no longer receive notifications about this package.
>
> [1] https://aur.archlinux.org/account/Muflone/
> [2] https://aur.archlinux.org/pkgbase/fex-emu/

ArchLinux is no longer a supported distro for FEX, remove it from the release processes documentation.
2024-12-07 16:15:22 -08:00
Ryan Houdek 7472b21f33 Merge pull request #4197 from Sonicadvance1/revert_4118
Revert #4118
2024-12-07 11:39:39 -08:00
Ryan Houdek 1058575d3a InstcountCI: Update pause instruction 2024-12-06 17:04:28 -08:00
Ryan Houdek e9867ca35a Revert "FEXCore: Change yield implementation to use wfe"
This reverts commit e53f3969e9.
2024-12-06 17:02:27 -08:00
Ryan Houdek 84277319fa Merge pull request #4166 from pmatos/HostFeaturesInPass
Generate SVE for 80bit load/stores when possible
2024-12-06 02:01:07 -08:00
Paulo Matos 8f8aa55c7f instcountci: Cache predicate register generation from pattern 2024-12-06 10:15:38 +01:00
Paulo Matos 72a4063651 Cache predicate register generation from pattern 2024-12-06 10:15:38 +01:00
Paulo Matos 0b1229da55 instcountci: Generate SVE for 80bit load/stores when possible 2024-12-06 10:15:38 +01:00
Paulo Matos 1d3ce30e50 Generate SVE for 80bit load/stores when possible
Fixes #4166.
2024-12-06 10:15:29 +01:00
LC 71187d3ad7 Merge pull request #4195 from Sonicadvance1/fix_clone3
LinuxEmulation: Don't use clone3 for fork
2024-12-06 00:10:07 -05:00
Ryan Houdek dd8a3a9aea LinuxEmulation: Don't use clone3 for fork
clone3 was added in Linux 5.3 but our minimum spec is 5.0. Additionally
the Raspberry Pi 5 kernel seems to complain about clone3 for some
reason?

Just use clone instead of clone3
2024-12-05 15:14:37 -08:00
Tony Wasserka 7b2fc37651 Merge pull request #4193 from WhatAmISupposedToPutHere/main
Thunks/gen: Add support for compiling against clang 19
2024-12-05 15:35:06 -05:00
Sasha Finkelstein 426569d74d Thunks/gen: Add support for compiling against clang 19 2024-12-05 21:16:41 +01:00
Ryan Houdek e877d5b82c unittests: Disable failing x87 tests on simulator 2024-12-05 00:03:33 -08:00
Ryan Houdek 572e0d04d5 unittests/X87: Adds precision and rounding mode tests
Tests all the instructions that are affected by FCW PC (or not!)
Only missing tests are fsincos (More easily tested with just fsin and
fcos), and fpatan
2024-12-04 23:54:36 -08:00
Ryan Houdek e3d7161ac5 FEXCore: Override x87 precision control when necessary
According to the documentation for x87 FCW precision control, this only
affects fadd*, fsub*, fmul*, fdiv*, and fsqrt. FEX was incorrectly
reducing precision for all x87 operations.

Precision is ignored for the following x87 ALU operations:
- fabs
- fscale
- fprem{1,}
- fcos
- fsin
- ftan
- fyl2x
- fyl2xp1
- fpatan
- fsincos
- Plus any operations just doing data movement and conversions

Next commit adds unittests to ensure this is correct for each
instruction.
2024-12-04 23:47:19 -08:00
Ryan Houdek 20caf69951 Docs: Update for release FEX-2412 2024-12-03 11:02:44 -08:00
Paulo Matos fcbf0de05a Enable RA of SVE Predicate Registers 2024-12-02 18:35:31 +01:00
Ryan Houdek 731e4d6271 Merge pull request #4168 from pmatos/SVEStoreInstCountCI
instcountci: testing multiple 80bit ldst using SVE
2024-12-02 07:56:12 -08:00
Paulo Matos 48beb18f29 instcountci: testing multiple 80bit ldst using SVE
In preparation for #4166 which should improve on these results.
2024-12-02 11:37:11 +01:00
LC 41c8731443 Merge pull request #4188 from Sonicadvance1/fexcore_remove_unnecessary
FEXCore: Removes ExitHandler and RunUntilExit
2024-12-01 15:06:57 -05:00
Ryan Houdek efb276f489 FEXCore: Removes ExitHandler and RunUntilExit
Now that all the threading behaviour has been correctly separated/moved
to the frontend, these functions serve no purpose.

- Instead of using RunUntilExit, all threads can use `ExecuteThread`
  directly, since there's nothing special about the primary thread now.
  - This also removes the public function definition of `ExecutionThread` since that was only used for threading logic.
- Instead of using an exit handler, just do the same cleanup after
  `ExecuteThread` has returned.
  - Just make gdbserver is cleaned up early if it exists since it may
    want to send some things to the connected gdb instance before
    threads are exited.
2024-12-01 10:45:38 -08:00
LC 9febddefa3 Merge pull request #4190 from Sonicadvance1/remove_old_gprsize
FEXCore: Removes GetGPRSize and convert all uses to GetGPROpSize
2024-12-01 13:13:21 -05:00
Ryan Houdek ce9a860335 OpcodeDispatcher: Also remove unused CacheIndexToSize 2024-12-01 05:38:11 -08:00
Ryan Houdek 0123946ed1 FEXCore: Removes GetGPRSize and convert all uses to GetGPROpSize
Only a few remaining uses left, easy enough to convert. This finally
switches the final few uses over.

NFC
2024-12-01 05:38:09 -08:00
LC c7098d0da1 Merge pull request #4189 from Sonicadvance1/fexcore_remove_definition
FEXCore: Removes stale function definition
2024-12-01 08:29:11 -05:00
LC 65a162bdf9 Merge pull request #4187 from Sonicadvance1/fexcore_remove_coreshuttingdown
FEXCore: Removes CoreShuttingDown from ContextImpl
2024-12-01 08:28:32 -05:00
Ryan Houdek 2de485d02a FEXCore: Removes stale function definition
`CopyMemoryMapping` was removed a long time ago, the definition happened
to remain. Remove the definition.
2024-11-30 23:20:07 -08:00
Ryan Houdek bf64facaf6 FEXCore: Removes CoreShuttingDown from ContextImpl
This is unused now.

NFC
2024-11-30 22:10:37 -08:00
LC 2e7fc60dbf Merge pull request #4169 from Sonicadvance1/x87_loadstore_tests
unittests/ASM: Fixes x87 80-bit loads on the edge of page boundaries.
2024-11-30 10:43:10 -05:00
LC baddfe00b1 Merge pull request #4186 from Sonicadvance1/remove_remaining_runningevents
FEXCore: Removes remaining RunningEvents from InternalThreadState
2024-11-29 20:00:00 -05:00
Ryan Houdek e7e59204d3 FEXCore: Removes remaining RunningEvents from InternalThreadState
These are all frontend constructs with mostly deprecated constraints.
WaitingToStart isn't used anymore, Running is effectively always true
(and behaviour has changed that if a thread is alive, it's running).

The only one that remains is `ThreadSleeping` which is only handled in
the frontend, and there was some conflation between ThreadSleeping and
Running which was hard to gauge. So delete `Running` and
`WaitingToStart`, but move `ThreadSleeping` to the frontend.
2024-11-29 14:08:55 -08:00
Ryan Houdek 95d5b14f99 Merge pull request #4183 from Sonicadvance1/move_fexcore_executionthread
FEXCore: Moves InternalThreadState ExecutionThread to the frontend
2024-11-29 14:06:42 -08:00
Ryan Houdek 802eaee9c8 FEXCore: Moves InternalThreadState ExecutionThread to the frontend
Once again this is another frontend construct, so move it to
ThreadStateObject
2024-11-29 13:33:56 -08:00
LC e89f48f237 Merge pull request #4182 from Sonicadvance1/move_start_paused
FEXCore: Move InternalThreadState StartRunning to frontend
2024-11-29 16:31:37 -05:00
Ryan Houdek e771e25632 LinuxSyscalls/Thread: Build child thread arguments on parent stack
Now that most of the thread tracking is in the frontend, change this
over to building the thread execution handler on the parent thread.

Removes a memory allocation/free pair, and removes the copy of each
variable in the child thread.
2024-11-29 09:38:36 -08:00
Ryan Houdek 25c202575e FEXCore: Move InternalThreadState StartRunning to frontend
We were using this variable for two things, letting the frontend signal
to the backend that it wants to start executing once the thread is
created, and also for handling thread pausing. These two features are
conflated with one another and actually makes things more confusing.

- Move StartRunning/StartPaused to the frontend, because its a construct
  that only needs to exist in the frontend
- Adds a FEX::HLE::ThreadStateObject CV for handling pausing, which only
  needs to exist for gdbserver
2024-11-29 09:38:24 -08:00
Ryan Houdek f59fc0f747 Merge pull request #4181 from Sonicadvance1/remove_exitreason
FEXCore: Removes ExitReason from InternalThreadState
2024-11-29 09:36:33 -08:00
Ryan Houdek f7a076e00c FEXCore: Removes ExitReason from InternalThreadState
FEXCore hasn't been returning anything other than EXIT_SHUTDOWN for a
long time, so this ended up just moving data around for no reason.

This isn't going to be used for further GdbServer work anyway, so just
completely remove it.
2024-11-29 09:25:44 -08:00
Ryan Houdek 56fadecdaf Merge pull request #4179 from Sonicadvance1/move_thread_waiting_start
FEXCore: Moves ThreadWaiting to the frontend
2024-11-29 09:23:26 -08:00
Ryan Houdek 9f681f9e41 FEXCore: Moves ThreadWaiting to the frontend
Only in one location does the frontend actually care about this, the
backend doesn't care at all.
2024-11-29 09:10:36 -08:00
Ryan Houdek 1b11f2f184 Merge pull request #4185 from asahilina/fix-autoshutdown-regression
FEXServer: Fix auto-shutdown regression
2024-11-29 09:03:53 -08:00
LC b2e61c37be Merge pull request #4170 from Sonicadvance1/gdbserver_work
GdbServer: Minor work
2024-11-29 08:24:43 -05:00
Asahi Lina 7c0cf51f09 FEXServer: Fix auto-shutdown regression
Fixes: #4184
2024-11-29 20:56:07 +09:00
Ryan Houdek 649a49488b Merge pull request #4180 from Sonicadvance1/fexcore_const_ptr_ctx
FEXCore: Constify CTX ptr in InternalThreadState
2024-11-28 17:02:16 -08:00
Ryan Houdek aa2180d494 Merge pull request #4178 from Sonicadvance1/move_statuscode_frontend
FEXCore: Moves StatusCode to the frontend
2024-11-28 16:51:33 -08:00
Ryan Houdek 969cae581c FEXCore: Constify CTX ptr in InternalThreadState
The CTX pointer in the InternalThreadState object will not and must not
change, since it is associated with that CTX object.

Contify it to codify it.
2024-11-28 15:57:36 -08:00
Ryan Houdek 1bf7e2544a FEXCore: Moves StatusCode to the frontend
This is a Linux construct, move it to the frontend.

This is going to need some changes in the future since exit_group and
exit syscalls are supposed to behave differently than how FEX implements
it. For now just move it to the frontend.
2024-11-28 15:55:46 -08:00
Ryan Houdek fad22144a2 Merge pull request #4177 from Sonicadvance1/move_deferred_signal_state
FEXCore: Moves DeferredSignalFrames to the frontend
2024-11-28 15:55:02 -08:00
Ryan Houdek b440e176fb Merge pull request #4176 from Sonicadvance1/move_signalreason
FEXCore: Moves SignalThread/SignalEvent to Frontend
2024-11-28 15:54:21 -08:00
Ryan Houdek 56c6b0d2cb Merge pull request #4175 from Sonicadvance1/gdbserver_remove_earlyexit
FEXCore: Removes EarlyExit running event
2024-11-28 15:53:17 -08:00
Ryan Houdek 0596a963e1 Merge pull request #4174 from Sonicadvance1/gdbserver_move_alloc_tls
FEXCore: Moves TLS initialization for Alloc::OSAllocator
2024-11-28 15:49:19 -08:00
Ryan Houdek 357cc04940 GdbServer: Splits Multi-letter v command handler
Just breaks out the two commands we support and leaves TODOs for
implementing the remaining commands.

NFC
2024-11-28 15:32:50 -08:00
Ryan Houdek 7c6e836865 GdbServer: Split out GDB context definition generation to its own function
GDB has two ways to read the registers. One way is reading the full
GDBContextDefinition, which matches the layout in `BuildTargetXML`.

The other way is to read the individual elements out of
GDBContextDefinition.

These two code paths were independently implemented. Instead generate in
one location and use in either location.

NFC
2024-11-28 15:32:50 -08:00
Ryan Houdek 54a7317312 GdbServer: Split out function searching for thread by TID
This currently happens in two locations, so split it out.

There's some behaviour here where if the TID isn't found, then it
returns the ParentThread of the process. This is working around a bug in
either FEX's gdbserver or binaryninja. Leave it currently before we
figure out what's wrong.

NFC
2024-11-28 15:32:50 -08:00
Ryan Houdek 1fb20710e6 GdbServer: Switch to thread specific stopping break logic
Previous `S AA` logic is legacy for non-multithreaded applications. This
newer command gives more information about what occured and in what
thread id.
2024-11-28 15:32:50 -08:00
Ryan Houdek 811ea093b5 GdbServer: Split out qXfer handlers
NFC, just making this easier to track for me.
2024-11-28 15:32:50 -08:00
Ryan Houdek 71fe9aee21 GdbServer: Fixes thread name setting
When parsing `comm`, by default it will have a newline which breaks gdb
in some cases. Strip out the whitespace to fix that issue.
2024-11-28 15:32:50 -08:00
Ryan Houdek 38c834e731 FEXCore: Removes global StartPaused check for gdb
This doesn't behave properly anymore now that thread management was
moved to the frontend.
2024-11-28 15:32:50 -08:00
Ryan Houdek 740ff60a71 GdbServer: Reorganize packet command handlers
Makes these consistent in the handling and documents the commands in a
way that is easier to parse while working on this.

NFC
2024-11-28 15:32:50 -08:00
Ryan Houdek ee69b9f650 GdbServer: Reconstruct XMM/YMM registers using FEXCore helpers
Previously this would have corrupted data in the upper 128-bits of the
YMM register.
2024-11-28 15:32:50 -08:00
Ryan Houdek a4565ce783 GdbServer: Pass through FCW
We have supported this for a while, just wasn't passed through gdbserver
since it usually doesn't matter.
2024-11-28 15:32:50 -08:00
Ryan Houdek 3131ee4de1 GdbServer: Moves information fetching to independent files
NFC
2024-11-28 15:32:50 -08:00
Ryan Houdek 3ecc66fbcf FEXCore: Moves TLS initialization for Alloc::OSAllocator
Alloc::OSAllocator uses a TLS variable of the thread object so it can
use a forkable mutex plus a deferring signal section. This was setup
when the FEXCore "ExecutionThread" function is called, which is a bit
awkward and is an artifact from when the thread creation was mixed
between the frontend and the backend.

Instead let the frontend inform the backend when to install the TLS
variable.

This is one step required to make GdbServer work correctly again since
the thread initialization and pausing is awkward today.
2024-11-28 02:46:06 -08:00
Ryan Houdek e322e84785 FEXCore: Moves DeferredSignalFrames to the frontend
Deferred signal frames are a frontend construct. Move it there.
2024-11-28 01:14:55 -08:00
Ryan Houdek f0fa7a5b6a FEXCore: Moves SignalThread/SignalEvent to Frontend
This is purely a Linux frontend construct now, move it.
2024-11-28 00:57:50 -08:00
Ryan Houdek 718221be71 FEXCore: Removes EarlyExit running event
This was working around an edge case in the GdbServer where a thread was
getting created while the process was shutting down. This edge case is
getting removed so get rid of it.
2024-11-28 00:46:42 -08:00
Ryan Houdek ee592ba03c Merge pull request #4173 from neobrain/fix_ctest_list
CMake: Generate test list even when testing is disabled
2024-11-27 13:52:32 -08:00
Tony Wasserka 56947f3a94 CMake: Generate test list even when testing is disabled
Previously, running ctest with BUILD_TESTS=OFF would discover and execute
leftover tests from a previous build. This change ensures CTestTestfile.cmake
gets regenerated so that ctest will see an empty test list in that case.
2024-11-27 11:02:54 +01:00
Ryan Houdek f41b9bc514 Merge pull request #4172 from alyssarosenzweig/jit/cf
OpcodeDispatcher: drop PossiblySetNZCV
2024-11-26 15:20:44 -08:00
Ryan Houdek 0463512c6c Merge pull request #4171 from Sonicadvance1/fix_ltrim
Utils/StringUtil: Fixes ltrim and adds a unittest
2024-11-26 13:09:22 -08:00
Ryan Houdek 47369d058e Utils/StringUtil: Fixes ltrim and adds a unittest
ltrim had the issue that it would always consume the left-most character
even if it wasn't whitespace. So `FEXLoader` would turn in to
`EXLoader`, even without any whitespace in the string.

Adds a test to ensure this doesn't occur again.
2024-11-26 12:57:48 -08:00
Billy Laws b6f34fa209 unittests: Add test for carry inversion bug 2024-11-26 09:02:23 -05:00
Alyssa Rosenzweig 03e0ca9833 InstCountCI: Update
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-11-26 09:02:01 -05:00
Alyssa Rosenzweig d19473160d OpcodeDispatcher: drop InvalidateDeferredFlags
it is now useless.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-11-26 09:02:01 -05:00
Alyssa Rosenzweig aeb2c98cbf OpcodeDispatcher: drop PossiblySetNZCV
this is a pain to track and, it turns out, buys us virtually nothing on flagm
systems. rip it out.

this fixes a bug with failing to set in all the right places.

on non-flagm systems there's a slight instcountci impact, but that is mostly
mitigated by the earlier patches in the series. so overall a wash there but
worth it for making the codebase easier
to reason about.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-11-26 08:59:19 -05:00
Alyssa Rosenzweig 2350ae5a07 OpcodeDispatcher: optimize BTC on !flagm
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-11-26 08:59:19 -05:00
Alyssa Rosenzweig 6076d1747e OpcodeDispatcher: use NZV invalidate CF set for BT
this is similar perf on flagm and better on not flagm.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-11-26 08:59:19 -05:00
Alyssa Rosenzweig f4ce6fb621 OpcodeDispatcher: optimize AAS/AAD flag
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-11-26 08:59:19 -05:00
Ryan Houdek f0d25d413b InstcounCI: Update 2024-11-24 00:38:54 -08:00
Ryan Houdek c84503c271 unittests/ASM: Adds x87 loadstore tests for edge of pages
These tests ensure that FEX's x87 80-bit loadstores don't read/write
past the end of the page.
2024-11-24 00:36:50 -08:00
Ryan Houdek 1465df874b OpcodeDispatcher: Fixes 80-bit loads
Ensures reads don't go past the end of the page boundary.
SVE masked loads can make this more effective but `VLoadVectorMasked`
isn't setup to be efficient for this case yet.
2024-11-24 00:36:50 -08:00
Ryan Houdek 60c52e3826 Merge pull request #4167 from pmatos/CheckLDPathNonEmpty
Check that LDPath is not empty
2024-11-22 15:47:42 -08:00
Paulo Matos 13b806130b Check that LDPath is not empty
This ensures we don't underflow at --RootFSLength.
2024-11-22 10:24:14 +01:00
Ryan Houdek 22058c06a1 Merge pull request #4164 from bylaws/swwow
WOW64: Set the software CPU area flag
2024-11-19 09:39:33 -08:00
Ryan Houdek d5c96555f1 Merge pull request #4159 from asahilina/fexserver-two-sockets
FEXServer: Listen on both abstract & named sockets
2024-11-19 09:28:48 -08:00
Asahi Lina c3e8cd8d30 docs: Document that std::filesystem::temp_directory_path() is unsafe 2024-11-20 02:05:32 +09:00
Asahi Lina d761fc44f4 FEXServer: Listen on both abstract & named sockets
Abstract sockets have one limitation: they are bound to a network
namespace. Chromium/CEF sandboxes using a new netns, which breaks
connecting to the FEXServer.

To work around this, use and try *both* abstract and named sockets. As
long as either the filesystem or the network is unsandboxed, things will
work. If both are sandboxed, there isn't much we can do... but at that
point we shouldn't be reinitializing the FEXServer connection anyway
since the FS should be available on FEXInterpreter startup.
2024-11-20 01:58:17 +09:00
Asahi Lina a1aa2547ce FEXServerClient: Do not use strerror() in ConnectToServer()
This triggers glibc allocation.

Signed-off-by: Asahi Lina <lina@asahilina.net>
2024-11-20 01:58:17 +09:00
Asahi Lina 44213c3968 FEXServerClient: Switch GetTempFolder to not use temp_directory_path()
Apparently this causes allocations which are banned in some paths?
2024-11-19 01:46:33 +09:00
Billy Laws 830bd347c5 WOW64: Set the software CPU area flag
This is required for wow64.dll to setup the cross-process queue
that is used to pass through e.g. memory unmap events.
2024-11-18 16:18:31 +00:00
Ryan Houdek bcfdf39d63 Merge pull request #4157 from asahilina/fix-chromium-sandbox
Support CLONE_FS and CLONE_FILES with fork() semantics
2024-11-18 06:42:15 -08:00
Asahi Lina bfed21870f Support CLONE_FS and CLONE_FILES with fork() semantics
Needed by Discord, part of the Chromium sandbox code. The warning still
triggers because Chromium asks for CLONE_VM on x86_64, but that can be
safely ignored (CLONE_FS is the one that matters).
2024-11-18 22:09:33 +09:00
Ryan Houdek 4278c48791 Merge pull request #4158 from asahilina/hide-rootfs-fd-take2
FileManagement: Hide the FEX RootFS fd from /proc/self/fd take 2
2024-11-17 20:27:59 -08:00
Ryan Houdek c1db7a78b1 Merge pull request #4153 from pmatos/VixlSkip
Add some more tests as unsupported by vixl
2024-11-17 19:56:56 -08:00
Ryan Houdek 1bf06f8946 Merge pull request #4162 from pmatos/WarningAvoid
Avoid warning on assertionless builds
2024-11-17 19:03:31 -08:00
Ryan Houdek 09bfe58827 Merge pull request #4160 from asahilina/align-stack
FEXLoader: Align stack base
2024-11-17 18:27:25 -08:00
Paulo Matos 474c780399 Avoid warning on assertionless builds
This was causing unused variable warning due
to the variable only being used in an assertion.
2024-11-13 15:17:36 +01:00
Asahi Lina 4a67893f1d FEXLoader: Align stack base
This ensures that __libc_stack_end is aligned, the same way it is on
native.
2024-11-13 03:47:43 +09:00
Asahi Lina 73ffaa1e18 FileManagement: Hide the FEX RootFS fd from /proc/self/fd take 2
Apparently Chromium/CEF can chroot or otherwise sandbox the filesystem
away before forking and checking for directory FDs, making /proc
inaccessible, which means we can't stat it for our inode check, breaking
the hiding.

So, double down on things and do what Chromium does: open an fd to /proc
ahead of time, so that continues to work. Then we use it to update the
inode of our RootFS fd instead, and finally, also do the /proc fd itself
to hide that one too.

We also don't need to check the st_dev of /proc more than once, since
that's not expected to change anyway.

Fixes cefsimple.
2024-11-13 01:26:42 +09:00
Ryan Houdek e675f4241a Merge pull request #4154 from Liamolucko/check-home
Check if a candidate home directory exists before using it
2024-11-05 20:07:30 -08:00
Liam Murphy 7a61d9d2b4 Check if a candidate home directory exists before using it
This allows FEX to be used in situations where `HOME` is set to
something invalid, e.g. inside Nix builds.
2024-11-06 11:16:53 +11:00
Paulo Matos 06b950a9cd Rounding test doesn't need to be skipped 2024-11-04 19:02:12 +01:00
Paulo Matos de70651406 Add some more tests as unsupported by vixl
It seems a form of `mrs` is unsupported as well as the hint `wfe` used for `pause`.
Remove skipping Rounding(Neg|Pos).asm as they are passing.
2024-11-04 18:52:48 +01:00
LC 5ad7fdb2f3 Merge pull request #4149 from Sonicadvance1/iropsize_convert_class
IR: Convert OpSize over to enum class
2024-10-30 23:55:55 -04:00
Ryan Houdek 9b6cc8f7e0 IR: Convert OpSize over to enum class
NFC

Do the final mopping up to convert the OpSize enum to an enum class!
2024-10-29 16:52:16 -07:00
LC 5c6de4ed14 Merge pull request #4147 from Sonicadvance1/iropsize_convert_irops
IR: Converts base IR operations to store OpSize sizes
2024-10-29 12:02:11 -04:00
Ryan Houdek 82f936cb6d IR: Converts base IR operations to store OpSize sizes
NFC

Finally converts the IR operations themselves to store the OpSize for
the IR operation size and element sizes.

This also finally, FINALLY, converts that remaining `_Constant` helper
to stop using a size field that is specified in bits rather than bytes
like all the other IR op handlers. That thing was so confusing and now
it's gone.
2024-10-28 21:26:59 -07:00
LC 493b952e3f Merge pull request #4146 from Sonicadvance1/iropsize_implicit_jit
JIT: Remove implicit OpSize conversions
2024-10-28 23:47:36 -04:00
Ryan Houdek 460a21625e JIT: Remove implicit OpSize conversions
NFC
2024-10-28 19:48:40 -07:00
LC 00ab3f8440 Merge pull request #4145 from Sonicadvance1/iropsize_various_implicit
OpcodeDispatcher: Various missed OpSize implicit cast fixes
2024-10-28 22:38:41 -04:00
Ryan Houdek 034b62292b Passes/x86StackOptimization: Fixes implicit conversion of OpSize 2024-10-28 19:26:02 -07:00
Ryan Houdek 7b615a07d0 OpcodeDispatcher: Various missed OpSize implicit cast fixes
NFC
Probably more of these around, just tracking the few I found.
2024-10-28 19:18:36 -07:00
LC 704841f004 Merge pull request #4144 from Sonicadvance1/iropsize_addrsize
OpcodeDispatcher: Convert address size helpers to use OpSize
2024-10-28 22:12:40 -04:00
LC c122f3faf9 Merge pull request #4143 from Sonicadvance1/iropsize_flags
OpcodeDispatcher: Convert flags helpers over to OpSize
2024-10-28 22:10:19 -04:00
Ryan Houdek f74f276d64 OpcodeDispatcher: Convert address size helpers to use OpSize
NFC
2024-10-28 19:02:34 -07:00
Ryan Houdek 4b10cbdafd OpcodeDispatcher: Convert flags helpers over to OpSize
NFC

Plus the tertiary bits that require changing to support it.
2024-10-28 18:56:42 -07:00
LC 04c701e912 Merge pull request #4142 from Sonicadvance1/irsize_loadstoregpr
OpcodeDispatcher: Convert {Load,Store}GPRRegister to OpSize
2024-10-28 21:04:01 -04:00
LC 0c29f8faad Merge pull request #4141 from Sonicadvance1/missing_ir_sizes
IR: Fix some missing OpSize conversions
2024-10-28 21:02:46 -04:00
Ryan Houdek 5ed82fa0f6 OpcodeDispatcher: Convert {Load,Store}GPRRegister to OpSize
Trivial but quite a few places pass in a raw integer

NFC
2024-10-28 16:35:23 -07:00
Ryan Houdek 6cca007817 IR: Fix some missing OpSize conversions
Missed these in the previous PR.
2024-10-28 16:25:36 -07:00
LC c0a9463700 Merge pull request #4140 from Sonicadvance1/enforce_irsize
Convert all of the IR operations to use OpSize
2024-10-28 18:20:02 -04:00
Ryan Houdek 55b3d67eb4 Merge pull request #4138 from asahilina/hide-rootfs-fd
FileManagement: Hide the FEX RootFS fd from /proc/self/fd
2024-10-28 14:24:55 -07:00
Ryan Houdek 65ddae1b71 IR: Change F80VBSLStack to use IR::OpSize 2024-10-28 02:25:17 -07:00
Ryan Houdek 063f524084 IR: Change F80CVTToInt to use IR::OpSize 2024-10-28 02:24:25 -07:00
Ryan Houdek 2dd0a82059 IR: Change F80CVTTo to use IR::OpSize 2024-10-28 02:23:47 -07:00
Ryan Houdek b810070e9f IR: Change F80CVTInt to use IR::OpSize 2024-10-28 02:07:20 -07:00
Ryan Houdek 4c7ac17f7d IR: Change F80CVT to use IR::OpSize 2024-10-28 02:06:48 -07:00
Ryan Houdek 84767c8b20 IR: Change PushStack to use IR::OpSize 2024-10-28 02:05:35 -07:00
Ryan Houdek eccfb53bd5 IR: Change StoreStackMemory to use IR::OpSize 2024-10-28 02:01:47 -07:00
Ryan Houdek f4e930262f IR: Change PCLMUL to use IR::OpSize 2024-10-28 01:50:24 -07:00
Ryan Houdek d26d9e7e03 IR: Change CRC32 to use IR::OpSize 2024-10-28 01:50:24 -07:00
Ryan Houdek 51c1998d70 IR: Change VAESDecLast to use IR::OpSize 2024-10-28 01:50:24 -07:00
Ryan Houdek 87f818249d IR: Change VAESDec to use IR::OpSize 2024-10-28 01:50:23 -07:00
Ryan Houdek d81f92f5e2 IR: Change VAESEncLast to use IR::OpSize 2024-10-28 01:50:23 -07:00
Ryan Houdek 29ffe02afe IR: Change VAESEnc to use IR::OpSize 2024-10-28 01:50:23 -07:00
Ryan Houdek dfe4076fe4 IR: Change Vector_F64ToI32 to use IR::OpSize 2024-10-28 01:50:23 -07:00
Ryan Houdek 44f9df062e IR: Change Vector_FToI to use IR::OpSize 2024-10-28 01:50:23 -07:00
Ryan Houdek 6f98ef8cbb IR: Change VFCVTN2 to use IR::OpSize 2024-10-28 01:50:23 -07:00
Ryan Houdek d54888a4c6 IR: Change VFCVTL2 to use IR::OpSize 2024-10-28 01:50:23 -07:00
Ryan Houdek 38c58706da IR: Change Vector_FToF to use IR::OpSize 2024-10-28 01:50:23 -07:00
Ryan Houdek 8ad9286bd4 IR: Change Vector_FToZS to use IR::OpSize 2024-10-28 01:29:32 -07:00
Ryan Houdek 03bc962564 IR: Change Vector_FToS to use IR::OpSize 2024-10-28 01:28:56 -07:00
Ryan Houdek 947b7ae6fe IR: Change Vector_SToF to use IR::OpSize 2024-10-28 01:23:20 -07:00
Ryan Houdek 30317ac979 IR: Change Float_FToF to use IR::OpSize 2024-10-28 01:22:45 -07:00
Ryan Houdek 4544e7c1af IR: Change Float_FromGPR_S to use IR::OpSize 2024-10-28 01:21:48 -07:00
Ryan Houdek 34050431ff IR: Change VDupFromGPR to use IR::OpSize 2024-10-28 01:20:06 -07:00
Ryan Houdek 65439956bf IR: Change VCastFromGPR to use IR::OpSize 2024-10-28 01:18:21 -07:00
Ryan Houdek a6cbce4fd7 IR: Change VFNMLS to use IR::OpSize 2024-10-28 01:16:11 -07:00
Ryan Houdek 17b851d4f3 IR: Change VFNMLA to use IR::OpSize 2024-10-28 01:15:47 -07:00
Ryan Houdek 362b5728be IR: Change VFMLS to use IR::OpSize 2024-10-28 01:15:25 -07:00
Ryan Houdek bd5159c7d5 IR: Change VFMLA to use IR::OpSize 2024-10-28 01:15:01 -07:00
Ryan Houdek 660dfcd1f9 IR: Change VFCADD to use IR::OpSize 2024-10-28 01:14:21 -07:00
Ryan Houdek 4e21177988 IR: Change VBSL to use IR::OpSize 2024-10-28 01:13:53 -07:00
Ryan Houdek 5a83a65905 IR: Change VTBX1 to use IR::OpSize 2024-10-28 01:13:15 -07:00
Ryan Houdek 061fc44923 IR: Change VTBL2 to use IR::OpSize 2024-10-28 01:13:00 -07:00
Ryan Houdek 0c7afa0672 IR: Change VTBL1 to use IR::OpSize 2024-10-28 01:12:43 -07:00
Ryan Houdek 30bf0d5767 IR: Change VFCMPUNO to use IR::OpSize 2024-10-28 01:12:22 -07:00
Ryan Houdek ee8e3127d2 IR: Change VFCMPORD to use IR::OpSize 2024-10-28 01:11:54 -07:00
Ryan Houdek e8f64f2976 IR: Change VFCMPLE to use IR::OpSize 2024-10-28 01:11:31 -07:00
Ryan Houdek 0a4b21da87 IR: Change VFCMPGT to use IR::OpSize 2024-10-28 01:11:11 -07:00
Ryan Houdek 074777bc75 IR: Change VFCMPLT to use IR::OpSize 2024-10-28 01:10:50 -07:00
Ryan Houdek 47c403998a IR: Change VFCMPNEQ to use IR::OpSize 2024-10-28 01:10:28 -07:00
Ryan Houdek 0fa095cec5 IR: Change VFCMPEQ to use IR::OpSize 2024-10-28 01:10:07 -07:00
Ryan Houdek cfa4e7f165 IR: Change VCMPGT to use IR::OpSize 2024-10-28 01:09:23 -07:00
Ryan Houdek 75b8226e7d IR: Change VCMPEQ to use IR::OpSize 2024-10-28 01:09:04 -07:00
Ryan Houdek 9e3c50ca2c IR: Change VExtr to use IR::OpSize 2024-10-28 01:08:35 -07:00
Ryan Houdek 365d8b9508 IR: Change VInsGPR to use IR::OpSize 2024-10-28 01:07:16 -07:00
Ryan Houdek 02ebe06496 IR: Change VInsElement to use IR::OpSize 2024-10-28 01:05:34 -07:00
Ryan Houdek 34d5e70e6b IR: Change VUShlSWide to use IR::OpSize 2024-10-28 00:59:33 -07:00
Ryan Houdek 2d8bd7b59d IR: Change VSShrSWide to use IR::OpSize 2024-10-28 00:57:02 -07:00
Ryan Houdek 3a6f5e638b IR: Change VUShrSWide to use IR::OpSize 2024-10-28 00:54:45 -07:00
Ryan Houdek c06274066f IR: Change VSShrS to use IR::OpSize 2024-10-28 00:51:19 -07:00
Ryan Houdek 820b0be9f2 IR: Change VUShrS to use IR::OpSize 2024-10-28 00:50:57 -07:00
Ryan Houdek 3e74817dd9 IR: Change VUShlS to use IR::OpSize 2024-10-28 00:50:36 -07:00
Ryan Houdek 3b9a2d2141 IR: Change VSShr to use IR::OpSize 2024-10-28 00:50:15 -07:00
Ryan Houdek 9ba431a51d IR: Change VUShr to use IR::OpSize 2024-10-28 00:49:52 -07:00
Ryan Houdek d0d2229db6 IR: Change VUShl to use IR::OpSize 2024-10-28 00:49:26 -07:00
Ryan Houdek 286258a2f2 IR: Change VUABDL2 to use IR::OpSize 2024-10-28 00:48:59 -07:00
Ryan Houdek ac14a88647 IR: Change VUABDL to use IR::OpSize 2024-10-28 00:48:32 -07:00
Ryan Houdek bf19578673 IR: Change VSMulH to use IR::OpSize 2024-10-28 00:47:44 -07:00
Ryan Houdek c5f396d889 IR: Change VUMulH to use IR::OpSize 2024-10-28 00:47:08 -07:00
Ryan Houdek 22325500d9 IR: Change VSMull2 to use IR::OpSize 2024-10-28 00:46:12 -07:00
Ryan Houdek 51eede080c IR: Change VUMull2 to use IR::OpSize 2024-10-28 00:45:50 -07:00
Ryan Houdek 503c86d47d IR: Change VSMull to use IR::OpSize 2024-10-28 00:45:23 -07:00
Ryan Houdek 0e0887181d IR: Change VUMull to use IR::OpSize 2024-10-28 00:44:20 -07:00
Ryan Houdek 974cc591bd IR: Change VMul to use IR::OpSize 2024-10-28 00:43:57 -07:00
Ryan Houdek 844afdb653 IR: Change VFMax to use IR::OpSize 2024-10-28 00:43:36 -07:00
Ryan Houdek c3c643d9b7 IR: Change VFMin to use IR::OpSize 2024-10-28 00:43:14 -07:00
Ryan Houdek b51b0c5f62 IR: Change VFDiv to use IR::OpSize 2024-10-28 00:42:54 -07:00
Ryan Houdek 96bdefddc9 IR: Change VFMul to use IR::OpSize 2024-10-28 00:42:32 -07:00
Ryan Houdek ba74a6a252 IR: Change VFSub to use IR::OpSize 2024-10-28 00:42:09 -07:00
Ryan Houdek 79427097e1 IR: Change VFAddV to use IR::OpSize 2024-10-28 00:41:46 -07:00
Ryan Houdek 7ec21b7121 IR: Change VFAddP to use IR::OpSize 2024-10-28 00:41:18 -07:00
Ryan Houdek 29115b3185 IR: Change VFAdd to use IR::OpSize 2024-10-28 00:38:55 -07:00
Ryan Houdek 33b3814642 IR: Change VTrn2 to use IR::OpSize 2024-10-28 00:37:44 -07:00
Ryan Houdek 85431a8132 IR: Change VTrn to use IR::OpSize 2024-10-28 00:35:12 -07:00
Ryan Houdek 2d9ef56a8b IR: Change VUnZip2 to use IR::OpSize 2024-10-28 00:31:48 -07:00
Ryan Houdek b2ae829731 IR: Change VUnZip to use IR::OpSize 2024-10-28 00:31:23 -07:00
Ryan Houdek a3c544a9a1 IR: Change VZip2 to use IR::OpSize 2024-10-28 00:27:28 -07:00
Ryan Houdek 9c2292289b IR: Change VZip to use IR::OpSize 2024-10-28 00:26:34 -07:00
Ryan Houdek b514548ca2 IR: Change VSMax to use IR::OpSize 2024-10-28 00:22:02 -07:00
Ryan Houdek 692a4a8fcd IR: Change VSMin to use IR::OpSize 2024-10-28 00:21:39 -07:00
Ryan Houdek 3a07cf7d70 IR: Change VUMax to use IR::OpSize 2024-10-28 00:21:22 -07:00
Ryan Houdek 7f5421fc26 IR: Change VUMin to use IR::OpSize 2024-10-28 00:21:00 -07:00
Ryan Houdek 04d62cd269 IR: Change VURAvg to use IR::OpSize 2024-10-28 00:20:35 -07:00
Ryan Houdek ac55e468a7 IR: Change VAddP to use IR::OpSize 2024-10-28 00:20:07 -07:00
Ryan Houdek e20db7dc88 IR: Change VSQSub to use IR::OpSize 2024-10-28 00:18:03 -07:00
Ryan Houdek 235bee9191 IR: Change VSQAdd to use IR::OpSize 2024-10-28 00:17:20 -07:00
Ryan Houdek 7a429b01c7 IR: Change VUQSub to use IR::OpSize 2024-10-28 00:16:31 -07:00
Ryan Houdek fbf5b14933 IR: Change VUQAdd to use IR::OpSize 2024-10-28 00:16:01 -07:00
Ryan Houdek 0dfd5dd96f IR: Change VXor to use IR::OpSize 2024-10-28 00:15:33 -07:00
Ryan Houdek 6d5acec958 IR: Change VOr to use IR::OpSize 2024-10-28 00:13:06 -07:00
Ryan Houdek beed43e577 IR: Change VAndn to use IR::OpSize 2024-10-28 00:12:37 -07:00
Ryan Houdek fc04b9113e IR: Change VAnd to use IR::OpSize 2024-10-28 00:11:44 -07:00
Ryan Houdek 68c038085a IR: Change VSub to use IR::OpSize 2024-10-28 00:10:21 -07:00
Ryan Houdek 176f5a2860 IR: Change VAdd to use IR::OpSize 2024-10-28 00:07:30 -07:00
Ryan Houdek e7c9623aa9 IR: Change VRev64 to use IR::OpSize 2024-10-27 23:36:03 -07:00
Ryan Houdek 53ca2ac378 IR: Change VRev32 to use IR::OpSize 2024-10-27 23:34:51 -07:00
Ryan Houdek d140cb4450 IR: Change VSQSHL to use IR::OpSize 2024-10-27 23:34:22 -07:00
Ryan Houdek 0ba501636e IR: Change VSRSHR to use IR::OpSize 2024-10-27 23:33:49 -07:00
Ryan Houdek ceca9fff17 IR: Change VSQXTUNPair to use IR::OpSize 2024-10-27 23:30:02 -07:00
Ryan Houdek 3e31abb645 IR: Change VSQXTUN2 to use IR::OpSize 2024-10-27 23:27:53 -07:00
Ryan Houdek 6c07cd319b IR: Change VSQXTUN to use IR::OpSize 2024-10-27 23:27:27 -07:00
Ryan Houdek 5247b7124f IR: Change VSQXTNPair to use IR::OpSize 2024-10-27 23:26:55 -07:00
Ryan Houdek 6cec557855 IR: Change VSQXTN2 to use IR::OpSize 2024-10-27 23:24:50 -07:00
Ryan Houdek 1868bd6777 IR: Change VSQXTN to use IR::OpSize 2024-10-27 23:24:25 -07:00
Ryan Houdek 7c7efeda82 IR: Change VUXTL2 to use IR::OpSize 2024-10-27 23:23:58 -07:00
Ryan Houdek 414486f1dd IR: Change VUXTL to use IR::OpSize 2024-10-27 23:23:12 -07:00
Ryan Houdek b8bc9659d4 IR: Change VSSHLL2 to use IR::OpSize 2024-10-27 23:22:47 -07:00
Ryan Houdek d52a6e6fc4 IR: Change VSSHLL to use IR::OpSize 2024-10-27 23:22:19 -07:00
Ryan Houdek 5ab41056ab IR: Change VSXTL2 to use IR::OpSize 2024-10-27 23:21:49 -07:00
Ryan Houdek 0692b34192 IR: Change VSXTL to use IR::OpSize 2024-10-27 23:21:17 -07:00
Ryan Houdek 3636c332ff IR: Change VUShrNI2 to use IR::OpSize 2024-10-27 23:05:41 -07:00
Ryan Houdek cb18963ded IR: Change VUShrNI to use IR::OpSize 2024-10-27 23:05:08 -07:00
Ryan Houdek 8cf92d3303 IR: Change VSShrI to use IR::OpSize 2024-10-27 23:03:21 -07:00
Ryan Houdek cdc5c15b4b IR: Change VUShraI to use IR::OpSize 2024-10-27 22:58:35 -07:00
Ryan Houdek ed313edd07 IR: Change VUShrI to use IR::OpSize 2024-10-27 22:57:57 -07:00
Ryan Houdek 9b981a4f61 IR: Change VShlI to use IR::OpSize 2024-10-27 22:55:08 -07:00
Ryan Houdek c791893b4a IR: Change VDupElement to use IR::OpSize 2024-10-27 22:50:47 -07:00
Ryan Houdek 2605c7e0b3 IR: Change VCMPLTZ to use IR::OpSize 2024-10-27 22:44:45 -07:00
Ryan Houdek f315948028 IR: Change VCMPGTZ to use IR::OpSize 2024-10-27 22:44:14 -07:00
Ryan Houdek 869367f7e2 IR: Change VCMPEQZ to use IR::OpSize 2024-10-27 22:43:42 -07:00
Ryan Houdek 3a2c7e8edd IR: Change VFRSqrt to use IR::OpSize 2024-10-27 22:43:12 -07:00
Ryan Houdek 0a34a43976 IR: Change VFSqrt to use IR::OpSize 2024-10-27 22:42:37 -07:00
Ryan Houdek 78cd21d78f IR: Change VFRecp to use IR::OpSize 2024-10-27 22:33:53 -07:00
Ryan Houdek 9f18de0196 IR: Change VFNeg to use IR::OpSize 2024-10-27 22:33:25 -07:00
Ryan Houdek 0bffdc4e27 IR: Change VFAbs to use IR::OpSize 2024-10-27 22:32:37 -07:00
Ryan Houdek 0f5ff53386 IR: Change VUMaxV to use IR::OpSize 2024-10-27 22:31:31 -07:00
Ryan Houdek 21611fc1ad IR: Change VUMinV to use IR::OpSize 2024-10-27 22:31:04 -07:00
Ryan Houdek 99e1eb5452 IR: Change VAddv to use IR::OpSize 2024-10-27 22:30:30 -07:00
Ryan Houdek 8c3ca44c57 IR: Change VPopcount to use IR::OpSize 2024-10-27 22:28:51 -07:00
Ryan Houdek 7a85e17d14 IR: Change VAbs to use IR::OpSize 2024-10-27 22:28:24 -07:00
Ryan Houdek b533dcd86d IR: Change VNot to use IR::OpSize 2024-10-27 22:27:28 -07:00
Ryan Houdek a379ce6fed IR: Change VNeg to use IR::OpSize 2024-10-27 22:09:02 -07:00
Ryan Houdek efd5c51110 IR: Change LoadNamedVectorIndexedConstant to use IR::OpSize 2024-10-27 22:08:27 -07:00
Ryan Houdek 886db4ffca IR: Change LoadNamedVectorConstant to use IR::OpSize 2024-10-27 22:05:42 -07:00
Ryan Houdek e6f6ee2bcd IR: Change VectorImm to use IR::OpSize 2024-10-27 21:58:34 -07:00
Ryan Houdek 4a6b5d4ec7 IR: Change VMov to use IR::OpSize 2024-10-27 21:52:11 -07:00
Ryan Houdek 027e7624cb IR: Change VFNMLSScalarInsert to use IR::OpSize 2024-10-27 18:37:30 -07:00
Ryan Houdek 0e31077735 IR: Change VFNMLAScalarInsert to use IR::OpSize 2024-10-27 18:36:53 -07:00
Ryan Houdek c8a9dd0d0a IR: Change VFMLSScalarInsert to use IR::OpSize 2024-10-27 18:36:14 -07:00
Ryan Houdek 2bd7ddaa31 IR: Change VFMLAScalarInsert to use IR::OpSize 2024-10-27 18:35:40 -07:00
Ryan Houdek 5566b4455b IR: Change VFCMPScalarInsert to use IR::OpSize 2024-10-27 18:35:03 -07:00
Ryan Houdek fed2c13521 IR: Change VFToIScalarInsert to use IR::OpSize 2024-10-27 18:32:11 -07:00
Ryan Houdek 37d092aab8 IR: Change VSToFGPRInsert to use IR::OpSize 2024-10-27 18:29:34 -07:00
Ryan Houdek 5626f4e50a IR: Change VSToFVectorInsert to use IR::OpSize 2024-10-27 18:27:39 -07:00
Ryan Houdek efbc42dac3 IR: Change VFToFScalarInsert to use IR::OpSize 2024-10-27 18:25:13 -07:00
Ryan Houdek 160934884d IR: Change VFRecpScalarInsert to use IR::OpSize 2024-10-27 18:21:05 -07:00
Ryan Houdek af1cfcb9bd IR: Change VFRSqrtScalarInsert to use IR::OpSize 2024-10-27 18:20:35 -07:00
Ryan Houdek d6f726fc23 IR: Change VFSqrtScalarInsert to use IR::OpSize 2024-10-27 18:20:06 -07:00
Ryan Houdek 92ee071eb2 IR: Change VFMaxScalarInsert to use IR::OpSize 2024-10-27 18:17:43 -07:00
Ryan Houdek bdfa8ad4f3 IR: Change VFMinScalarInsert to use IR::OpSize 2024-10-27 18:17:17 -07:00
Ryan Houdek 37540f4927 IR: Change VFDivScalarInsert to use IR::OpSize 2024-10-27 18:16:30 -07:00
Ryan Houdek 000ab5ff19 IR: Change VFMulScalarInsert to use IR::OpSize 2024-10-27 18:15:58 -07:00
Ryan Houdek f054274948 IR: Change VFSubScalarInsert to use IR::OpSize 2024-10-27 18:15:22 -07:00
Ryan Houdek 081907e168 IR: Change VFAddScalarInsert to use IR::OpSize 2024-10-27 18:14:37 -07:00
Ryan Houdek 1a115a8ce6 IR: Change FCmp to use IR::OpSize 2024-10-27 18:06:34 -07:00
Ryan Houdek fd9158c75f IR: Change Float_ToGPR_ZS to use IR::OpSize 2024-10-27 18:03:18 -07:00
Ryan Houdek 1bde30a196 IR: Change Float_ToGPR_S to use IR::OpSize 2024-10-27 18:02:45 -07:00
Ryan Houdek 764aacaa8f IR: Change VExtractToGPR to use IR::OpSize 2024-10-27 17:58:53 -07:00
Ryan Houdek a848211926 IR: Change NZCVSelectV to use IR::OpSize 2024-10-27 17:53:15 -07:00
Ryan Houdek f1a42869d5 IR: Change CondJump to use IR::OpSize 2024-10-27 17:50:51 -07:00
Ryan Houdek 97a6ba9931 IR: Change VLoadNonTemporal to use IR::OpSize 2024-10-27 17:45:29 -07:00
Ryan Houdek f4744f1e79 IR: Change VStoreNonTemporalPair to use IR::OpSize 2024-10-27 17:44:57 -07:00
Ryan Houdek 321f686108 IR: Change VStoreNonTemporal to use IR::OpSize 2024-10-27 17:44:23 -07:00
Ryan Houdek 90340350fa IR: Change MemCpy to use IR::OpSize 2024-10-27 17:43:10 -07:00
Ryan Houdek 6f4fd4467b IR: Change MemSet to use IR::OpSize 2024-10-27 17:42:39 -07:00
Ryan Houdek b31ce13f68 IR: Change Pop to use IR::OpSize 2024-10-27 17:41:30 -07:00
Ryan Houdek 260d3b0b4e IR: Change Push to use IR::OpSize 2024-10-27 17:39:21 -07:00
Ryan Houdek c8c7ffbf05 IR: Change VBroadcastFromMem to use IR::OpSize 2024-10-27 17:35:56 -07:00
Ryan Houdek 4b03185b77 IR: Change VStoreVectorElement to use IR::OpSize 2024-10-27 17:32:25 -07:00
Ryan Houdek 52ec572db3 IR: Change VLoadVectorElement to use IR::OpSize 2024-10-27 17:27:42 -07:00
Ryan Houdek dc31cf83c6 IR: Change VLoadVectorGatherMaskedQPS to use IR::OpSize 2024-10-27 17:22:07 -07:00
Ryan Houdek 8a4f51257d IR: Change VLoadVectorGatherMasked to use IR::OpSize 2024-10-27 17:21:33 -07:00
Ryan Houdek 051469fa16 IR: Change VStoreVectorMasked to use IR::OpSize 2024-10-27 17:20:56 -07:00
Ryan Houdek 3f6cdc2e03 IR: Change VLoadVectorMasked to use IR::OpSize 2024-10-27 17:20:18 -07:00
Ryan Houdek f3449f2b00 IR: Change StoreMemTSO to use IR::OpSize 2024-10-27 17:17:26 -07:00
Ryan Houdek d7691d9a25 IR: Change LoadMemTSO to use IR::OpSize 2024-10-27 17:16:23 -07:00
Ryan Houdek f414d4934c IR: Change StoreMemPair to use IR::OpSize 2024-10-27 17:15:00 -07:00
Ryan Houdek 014917301a IR: Change StoreMem to use IR::OpSize 2024-10-27 17:14:18 -07:00
Ryan Houdek cc483acbde IR: Change LoadMemPair to use IR::OpSize 2024-10-27 16:33:52 -07:00
Ryan Houdek 07f8a4eadd IR: Change LoadMem to use IR::OpSize 2024-10-27 16:33:14 -07:00
Ryan Houdek 5fd127b53a IR: Change StoreContextIndexed to use IR::OpSize 2024-10-27 15:51:30 -07:00
Ryan Houdek ece89ddeab IR: Change LoadContextIndexed to use IR::OpSize 2024-10-27 15:50:03 -07:00
Ryan Houdek a1565a7d99 IR: Change StoreContextPair to use IR::OpSize 2024-10-27 15:47:08 -07:00
Ryan Houdek 2f9b0de742 IR: Change StoreContext to use IR::OpSize 2024-10-27 15:46:32 -07:00
Ryan Houdek 7e5f1b5859 IR: Change LoadContextPair to use IR::OpSize 2024-10-27 15:42:39 -07:00
Ryan Houdek 40fd4bbb66 IR: Change LoadContext to use IR::OpSize 2024-10-27 15:42:07 -07:00
Ryan Houdek e4143352c9 IR: Change Store{PF,AF} to use IR::OpSize 2024-10-27 15:37:41 -07:00
Ryan Houdek c045e14837 IR: Change StoreRegister to use IR::OpSize 2024-10-27 15:37:05 -07:00
Ryan Houdek f0f3c215ce IR: Change Load{PF,AF} to use IR::OpSize 2024-10-27 15:35:26 -07:00
Ryan Houdek 8f4113d859 IR: Change LoadRegister to use IR::OpSize 2024-10-27 15:34:46 -07:00
Ryan Houdek 4cfc2ac1a4 IR: Change AllocateFPR to use IR::OpSize 2024-10-27 15:30:11 -07:00
LC d2aa5217dc Merge pull request #4132 from Sonicadvance1/fix_irsize
Fix IR operation usage to use OpSize when possible
2024-10-27 17:26:11 -04:00
Ryan Houdek e8baf4a28c OpcodeDispatcher: Ensure IR ops use OpSize
NFC
2024-10-27 14:11:25 -07:00
Ryan Houdek e438d32879 OpcodeDispatcher/Vector: Ensure IR ops use OpSize
NFC
2024-10-27 14:11:07 -07:00
Ryan Houdek 32ef10b273 OpcodeDispatcher/AVX128: Ensure IR ops use OpSize
NFC
2024-10-27 14:11:07 -07:00
Ryan Houdek ad296051b7 OpcodeDispatcher/Crypto: Ensure IR ops use OpSize
NFC
2024-10-27 14:11:07 -07:00
Ryan Houdek e603136918 OpcodeDispatcher/Flags: Ensure IR ops use OpSize
NFC
2024-10-27 14:11:07 -07:00
Ryan Houdek f8a61f7d7e OpcodeDispatcher/X87: Ensure IR ops use OpSize
NFC
2024-10-27 14:11:07 -07:00
Ryan Houdek cb5ba8baae OpcodeDispatcher/X87F64: Ensure IR ops use OpSize
NFC
2024-10-27 14:11:07 -07:00
LC 079e70fc4e Merge pull request #4134 from Sonicadvance1/move_jit
JIT: Moves Arm64 JIT up one folder
2024-10-27 17:10:26 -04:00
Asahi Lina 3d701f5fcf FileManagement: Hide the FEX RootFS fd from /proc/self/fd
Chromium/CEF has code that iterates through all open FDs and bails if
any are directories (apparently a sandboxing sanity check). To avoid
this check, we need to hide the RootFS FD. This requires hooking all the
getdents variants to skip that entry.

To keep the runtime cost low, we keep track of the inode of
/proc/self/fd/<rootfs fd> (note: not the RootFS inode, the inode of the
magic symlink in /proc), and first do a quick check on that. If it
matches, then we stat the dirfd we are reading and check against the
procfs device, to complete the inode equality check.

As an extra benefit, this also fixes code that tries to iterate and
close all/extra FDs and ends up closing the RootFS fd.
2024-10-27 07:05:00 +09:00
Ryan Houdek b31e4a3c27 JIT: Moves Arm64 JIT up one folder
We have only one JIT, there is no reason to subfolder this. Move it up
one folder.

NFC
2024-10-25 18:44:20 -07:00
LC 992d6e8477 Merge pull request #4136 from Sonicadvance1/support_tpidrro
FEXCore: Adds support for CPU Index through TPIDRRO
2024-10-25 21:43:33 -04:00
LC f6cdb165a3 Merge pull request #4137 from Sonicadvance1/minor_pushf_opt
OpcodeDispatcher: Minor optimization to small pushf
2024-10-25 21:42:45 -04:00
LC f60388d160 Merge pull request #4135 from Sonicadvance1/remove_xop
X86Tables: Removes XOP tables
2024-10-25 19:41:31 -04:00
Ryan Houdek 96fa2ad8eb InstcountCI: Update 2024-10-25 15:43:10 -07:00
Ryan Houdek f143462ebe OpcodeDispatcher: Minor optimization to small pushf
The push operation already truncates the result, there's no need to bfe
it. Noticed this while cleaning up in #4134. Removes one instruction for
16-bit and 32-bit pushf instructions.
2024-10-25 15:41:22 -07:00
Ryan Houdek 51fa61a1cd InstcountCI: Adds missing pushf implementations
pushf was aliasing to pushfq, needed an o16 prefix.
We also weren't testing the 32-bit path, which only exists on 32-bit, so
add that as well.
2024-10-25 15:40:44 -07:00
Ryan Houdek 048e967546 FEXCore: Adds support for CPU Index through TPIDRRO 2024-10-25 15:07:57 -07:00
Ryan Houdek 608fd49ac3 CodeEmitter/unittests: Add support for TPIDRRO_EL0 2024-10-25 15:07:29 -07:00
Ryan Houdek bb630797b5 CodeEmitter: Add support for TPIDRRO_EL0 2024-10-25 15:07:15 -07:00
Ryan Houdek 01a6e914f2 X86Tables: Removes XOP tables
These weren't even wired up to the frontend. We aren't going to support
XOP, so just remove the tables.
2024-10-25 14:32:13 -07:00
Ryan Houdek 0190e1a00b Merge pull request #4130 from pmatos/X87F64Simp
X87 Code Simplification
2024-10-24 10:57:12 -07:00
Paulo Matos 11a87c22f9 instcountci: X87 code simplification 2024-10-24 18:17:48 +02:00
Paulo Matos 5f6c0d2245 X87 code simplification
Merges some of the code from reduced precision into the main path
since they are practically the same.
2024-10-24 18:15:49 +02:00
Ryan Houdek caaacb6c15 Merge pull request #4127 from alyssarosenzweig/opt/masking
Optimize bsf, bsr, register cmpxchg, pcmpistri
2024-10-23 07:28:29 -07:00
Ryan Houdek 767c61c08b Merge pull request #4129 from pmatos/RPRESInstcountci
Disable RPRES in instcounci files
2024-10-23 07:27:57 -07:00
Paulo Matos dc93e30451 Disable RPRES in instcounci files
This was giving false changes on RPRES enabled HW.
2024-10-23 15:09:17 +02:00
Ryan Houdek 368162df87 Merge pull request #4128 from ahoneybun/update-ubuntu-support
add Ubuntu 24.10 and remove unsupported releases
2024-10-22 14:43:22 -07:00
Aaron Honeycutt 9eb2106ed2 update supported list 2024-10-22 15:35:41 -06:00
Aaron Honeycutt cfc05b78fe add Ubuntu 24.10 2024-10-22 15:27:21 -06:00
Alyssa Rosenzweig d2a42c0038 InstCountCI: Update
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-10-22 13:39:21 -04:00
Alyssa Rosenzweig 58a3d174ec OpcodeDispatcher: explain why we provide defined bsf behaviour
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-10-22 13:36:24 -04:00
Alyssa Rosenzweig 9c605e7333 OpcodeDispatcher: optimize bsf/bsr
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-10-22 13:31:49 -04:00
Alyssa Rosenzweig 1fd7e88ffd OpcodeDispatcher: optimize cmp in cmpxchg
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-10-22 13:29:47 -04:00
Alyssa Rosenzweig c5e7da0631 OpcodeDispatcher: optimize more cmpxchg mask
none of it matters.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-10-22 13:29:47 -04:00
Alyssa Rosenzweig 68f58e415f OpcodeDispatcher: optimize cmpxchg masking
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-10-22 13:29:47 -04:00
Alyssa Rosenzweig 698abec25c JIT: drop FindMSB zero handling
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-10-22 13:29:47 -04:00
Alyssa Rosenzweig 1578f5ed47 JIT: drop FindLSB masking
consequence of the UB

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-10-22 13:29:47 -04:00
Alyssa Rosenzweig 80d7b5a5c9 JIT: drop FindLSB zero handling
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-10-22 13:29:47 -04:00
Alyssa Rosenzweig eb023ceb51 IR: make FindLSB/FindMSB undefined for zero
these are used in places:

* bsf/bsr
* pcmpblabla
* x87 fild

In all cases we explicitly check for zero and change the behaviour accordingly.
So weaken the IR op to let us optimize. No sense checking twice.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-10-22 13:29:47 -04:00
LC 5d1fda7d7f Merge pull request #4125 from Sonicadvance1/unify_pshuflhw
OpcodeDispatcher: Unify PSHUF{L,H}W implementations
2024-10-21 14:37:39 -04:00
Ryan Houdek fafc04a59e OpcodeDispatcher/AVX128: Fixes glibc allocation 2024-10-21 09:34:59 -07:00
Ryan Houdek ddcca58f64 InstcountCI: Update for PSHUF{L,H}W changes 2024-10-21 09:25:58 -07:00
Ryan Houdek fe8f5c745d OpcodeDispatcher/AVX128: Use unified PSHUF{L,H}W implementation
Allows the AVX128 implementation to use the same implementation as the
128-bit SSE implementation, because it works per 128-bit lane just like
SSE. Is a minor optimization.
2024-10-21 09:24:38 -07:00
Ryan Houdek d876224358 OpcodeDispatcher: Unify MMX and SSE PSHUF{L,H}W implementations
No functional change.
2024-10-21 09:11:05 -07:00
LC c4306f2f0a Merge pull request #4122 from Sonicadvance1/avx_fixes
AVX128: Fixes some AVX bugs
2024-10-18 19:34:28 -04:00
Ryan Houdek 8b7a227820 InstcountCI: Update 2024-10-18 15:49:25 -07:00
Ryan Houdek d7afcee622 AVX128: Fixes asome AVX bugs
vblendvps, vblendvpd, vpblendvb all broke because of failing to zext the
128-bit register correctly.

vinsertps broke because it was accidentally using the wrong
implementation.

Extends each of their unittests to handle these cases.
2024-10-18 15:49:25 -07:00
Ryan Houdek a421ff1105 Merge pull request #4093 from pmatos/FXtractFix
Fix FXTRACT for 0.0 and -0.0
2024-10-17 05:31:34 -07:00
Paulo Matos 5997030c97 instcountci: Fix FXTRACT for 0.0 and -0.0 2024-10-17 09:13:40 +02:00
Paulo Matos 3c8086373b FXTRACT fix ASM tests 2024-10-17 09:05:17 +02:00
Paulo Matos 10ec6b63b6 Fix FXTRACT for 0.0 and -0.0
Fixes fxtract by returning the correct values for 0.0 and -0.0. We moved the split of fxtract into _sig and _exp, to the opcode dispatcher, to ease some comparisons.

Also removed the IR node F80XTRACTStack which is not needed anymore.
2024-10-17 09:05:10 +02:00
Paulo Matos 49087007be Implement NZCVSelectV for selection on FPRs
Behaves like NZCVSelect for FPRs.
2024-10-17 08:42:50 +02:00
Ryan Houdek 0897cd8777 Merge pull request #4116 from Sonicadvance1/personality_handling
LinuxEmulation: Personality handling
2024-10-16 15:52:35 -07:00
Ryan Houdek 4b945a9041 Merge pull request #4105 from pmatos/X87MMXState
Implement explicit state switch between X87 and MMX
2024-10-16 15:52:27 -07:00
Ryan Houdek d66ed71bc6 Merge pull request #4082 from Sonicadvance1/fix_wine
FEXLoader: Fixes newer wine versions and Fedora
2024-10-16 15:52:19 -07:00
LC b5b34df155 Merge pull request #4119 from Sonicadvance1/unaligned_lock
unittests/ASM: Adds missing unaligned atomic tests
2024-10-15 12:16:37 -04:00
LC ff51435747 Merge pull request #4118 from Sonicadvance1/wfe_for_pause
FEXCore: Change yield implementation to use wfe
2024-10-15 12:15:11 -04:00
Paulo Matos def561986b instcountci: Implements explicit state switch between X87 and MMX 2024-10-15 17:58:59 +02:00
Paulo Matos 5c258d4a2a ASM Test: Implements explicit state switch between X87 and MMX
Tags is set to all valid in FEX, but in host it's set to all valid _and_
reinterpreted. Adding this to known failures in the host runner.
2024-10-15 17:58:59 +02:00
Paulo Matos 0d53f2b45c Implements explicit state switch between X87 and MMX
Fixes #3850
2024-10-15 17:58:53 +02:00
Ryan Houdek 4f03044fe7 unittests/ASM: Adds missing unaligned atomic tests
Fixes #2670

Walked through all the unaligned atomic tests to find which ones were
missing. Turns out it was only ADC, NEG, NOT, and SBB.
2024-10-15 06:59:38 -07:00
Ryan Houdek b967538435 InstcountCI: Add pause instruction 2024-10-15 05:52:41 -07:00
Ryan Houdek e53f3969e9 FEXCore: Change yield implementation to use wfe
According to
https://github.com/rust-lang/rust/commit/c064b6560b7ce0adeb9bbf5d7dcf12b1acb0c807
turns out that the arm yield instruction is effectively a nop on all
reasonably new CPUs.
Instead switch over to wfe because it matches x86 `PAUSE` semantics more
closely.
2024-10-15 05:50:26 -07:00
Ryan Houdek 5026bf8247 Merge pull request #4117 from pmatos/NoTest
Remove file since FXAM_Simple is not a Linux test
2024-10-11 18:17:09 -07:00
Paulo Matos 09cb4f5fc5 Remove file since FXAM_Simple is not a Linux test
Already properly skipped in the right place.
File added accidentally.
2024-10-11 16:55:07 +02:00
Paulo Matos ddd7a550e4 Remove check on top 16bits
Makes it uniform among all 3DNow tests instead of
some checking and some don't.
2024-10-11 16:18:43 +02:00
Ryan Houdek 3398f22c16 FEXLinuxTests: Adds personality test
These would have failed before the prior changes.
2024-10-11 05:03:28 -07:00
Ryan Houdek 7f17519fbf LinuxEmulation/personality: Support PER_LINUX32 2024-10-11 04:52:23 -07:00
Ryan Houdek a65884f9ae LinuxEmulation/personality: Support UNAME26 2024-10-11 04:52:21 -07:00
Ryan Houdek d70766f4c8 LinuxEmulation: Support personality tracking
Doesn't handle the emulation of it, but handle passing it to the host
kernel, tracking the value, and inheriting it through new threads.
2024-10-11 04:52:21 -07:00
Ryan Houdek 1365aa8881 FEXLinuxTests: Update to c++20 2024-10-11 04:52:21 -07:00
Ryan Houdek fe5bc02682 FEXLoader: Fixes newer wine versions and Fedora
This was brought up by #3831 but I finally got the courage to look at
the hard problem.

Although I'm only tackling half of the problem with this PR, which is
that FEXLoader needs to strip the rootfs path from the executed path if
it begins with the rootfs, plus some changes to the surrounding code.

The primary concern here is that when an application has been executed
under FEX, specifically through binfmt_misc, then FEX needs to prepend
the full rootfs path otherwise Linux can't find the program.
Additionally execveat with an FD will resolve a full path to the rootfs.

So past FEX's initial setup, we need to strip off the rootfs path to
provide an "absolute" path that is visible to the guest application
later. Which is kind of funny since we have a `RootFSRedirect` function
which did the exact opposite. This was due to legacy problems in the
original ELFLoader that couldn't handle symlinks correctly, which has
since been resolved, so that no longer needs to exist.

There was also some weirdness in `GetApplicationNames` where the passed
in argument list was modifying Args[0] and then saving the Program as
well. Which I just got rid of. Also stopped passing in the arguments by
value because....why did I write it like that?

In InterpreterHandler we now need to check if we can open the path
inside the rootfs or fallback without it. Plus I had to change the
shebang handling so it stopped prefixing the rootfs AGAIN. Took the time
to change the shebang handling there so it stops creating string copies
and instead just generates views.

Overall this fixes a fairly major flaw with how we were representing
`/proc/self` to the application, which was breaking wine since it would
prefix the rootfs multiple times, which was weird.

It doesn't address the remaining problem in #3831, which is that
applications can still see some of the leaky abstractions with symlinks
through the rootfs, but I want to get at least this step in.
2024-10-11 01:40:36 -07:00
Ryan Houdek 6f096e7c4b FHU: Add StringArgumentParser function
Split this out so we can unittest it.

Adds a unittest to handle specific edge cases.
2024-10-11 01:40:36 -07:00
LC 389ad737e6 Merge pull request #4103 from Sonicadvance1/shared_vdso_mmap
VDSOEmulation: Support loading VDSO thunk as shared
2024-10-10 01:25:12 -04:00
LC c00f7813a2 Merge pull request #4108 from Sonicadvance1/ensure_x87_size_save_restore
unittests/ASM: Ensures FNSAVE and FRSTOR only store as much data as required
2024-10-10 01:00:54 -04:00
LC e5ceaa182d Merge pull request #4110 from Sonicadvance1/remove_unused_memory_regions
unitests/ASM: Removes unused MemoryRegion configs
2024-10-10 00:59:31 -04:00
LC eeb8eb1824 Merge pull request #4109 from Sonicadvance1/fix_fsgs_testharness
TestHarnessRunner: Fixes FS/GS usage in tests
2024-10-10 00:56:20 -04:00
Ryan Houdek 7c6444c37c unitests/ASM: Removes unused MemoryRegion configs
FEX's ASM unitests had the problem that they were copy and pasted
templates and MemoryRegion was copied in to almost all tests.

Very few tests actually use the MemoryRegion they were asking for and
instead used none, or the hardcoded memory regions that the
TestHarnessRunner provides.

This is entirely a sed replacement and minor fixups plus reverts for the
few tests that actually use the region asked for.
2024-10-08 16:07:25 -07:00
Ryan Houdek 4a179c8f87 TestHarnessRunner: Fixes FS/GS usage in tests
When writing `FEX_bugs/tls_vector_element.asm` I had to switch to using
GS segment instead of FS segment because the unittests didn't correctly
restore FS after running. This is because GS is unused on Linux
applications, but FS would become broken and break glibc cleanup on
shutdown.

Now that xbyak has been updated to v7.09, it now supports
{rd,wr}{fs,gs}base which allows us to save and restore the segments
correctly. This lets us drop in TLS tests in to unittests more easily
now.

Ensured this works by modifying the test temporarily to use fs instead
of gs again, seeing it crash before HostRunner changes, and work after
HostRunner changes.

Fixes #4104
2024-10-08 15:42:44 -07:00
Ryan Houdek 2b3895a514 Update xbyak to v7.09 2024-10-08 15:38:00 -07:00
Ryan Houdek 0bb0f9cec7 unittests/ASM: Ensures FNSAVE and FRSTOR only store as much data as required
The instruction definition only allows these instructions to load/store
94 or 108 bytes, not affecting any bytes afterwards. This is a bit
awkward because 80-bit x87 registers are stored at the end.

FEX has an optimization today where it uses overlapping loads and stores
for the first seven x87 registers, and a split loadstore for the final
register. This ensures that we get the correct data while reducing the
number of loadstores.

We didn't have a unittest in place to ensure we only ever write the
correct amount of data, so changes like in #4107 which look correct from
an initial glance, would have resulted in broken behaviour.

This unittest ensures both that the instructions don't try to access
beyond the end of the page, and also ensures that they don't overwrite
subsequent data. Making sure that potentially broken behaviour doesn't
make its way in.
2024-10-08 15:31:36 -07:00
Ryan Houdek 6a07ea73a8 Merge pull request #4106 from slp/compat-input-prctl
FEXCore: adds support for compat input prctl
2024-10-08 13:22:19 -07:00
Sergio Lopez 5c51c54ccc FEXCore: adds support for compat input prctl
The size of the input_event struct differs between 32 bits applications
and 64 bits applications. To deal with this, the kernel implements a
compat variant for the input syscalls, but it's only enabled for 32 bit
processes.

In libkrunfw we're introducing a prctl that enables a 64 bit process to
request the kernel to enable the compat variant for the input syscalls.
This commit makes use of that interface for enabling/disabling the
compat input variant as required.

The visible effect is that input devices such as gamepads work properly
on emulated 32 bit applications.

Signed-off-by: Sergio Lopez <slp@redhat.com>
2024-10-08 11:38:42 +02:00
Ryan Houdek cac3767c20 Docs: Update for release FEX-2410 2024-10-07 11:14:53 -07:00
Ryan Houdek e82d2c70c3 Merge pull request #4095 from alyssarosenzweig/opt/jit-time
speed up the JIT
2024-10-04 09:00:03 -07:00
LC 9716fc73d3 Merge pull request #4099 from Sonicadvance1/v6.11
LinuxSyscalls: Update max reported kernel version to 6.11
2024-10-04 11:58:20 -04:00
LC aaa8eef9f1 Merge pull request #4098 from Sonicadvance1/linux_v6.11_syscalls
LinuxEmulation: Update syscalls for v6.11
2024-10-04 11:57:52 -04:00
LC c036868938 Merge pull request #4097 from Sonicadvance1/linux_v6.11_drm
LinuxEmulation: Update DRM for v6.11
2024-10-04 11:57:21 -04:00
LC 2726f35a10 Merge pull request #4102 from Sonicadvance1/fix_vector_element_loadstore
OpcodeDispatcher: Fixes segment prefixing on vector element loadstores
2024-10-04 11:56:15 -04:00
Ryan Houdek c740801ea5 VDSOEmulation: Support loading VDSO thunk as shared
Just requires a thread pointer check to be fixed in FEXCore.
This doesn't need unique pages to exist for the file mapping and can be
shared since it's readonly mapped.
2024-10-03 21:06:35 -07:00
Ryan Houdek 407dc5d4dd unittests: Adds TLS vector element loadstore test
Hits all the instructions that would have previously crashed.
2024-10-03 20:53:28 -07:00
Ryan Houdek 7dda75646d OpcodeDispatcher: Fixes segment prefixing on vector element loadstores
These had completely forgotten to load segment prefixes on the element
loadstore operations. This had caused a crash on the Linux version of
"Halls of Tormet" (steamid 2218750).

This game has TLS vector data which it was loading with a movhps and hit
this path.
2024-10-03 20:51:37 -07:00
Ryan Houdek b89f5b8a03 Merge pull request #4101 from Sonicadvance1/fprem_fixes
Fixes fprem
2024-10-03 15:49:22 -07:00
Ryan Houdek 16869d8954 unittests: fprem tests work since 688af49485 2024-10-03 14:03:51 -07:00
Ryan Houdek e353ae8408 unittests: Adds x87 fprem test
Exposes a bug in FEX's x87 FPREM implementation.
2024-10-03 14:03:51 -07:00
Alyssa Rosenzweig aa3c963df1 SoftFloat: fix FPREM
fixes incorrect signs on FPREM. in turn should fix end-to-end failures logging
into Steam.

Thank you to Sergio Lopez for tracking down the JavaScript fail, and Ryan for
finding the bug.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-10-03 13:56:14 -07:00
Ryan Houdek 0c8cfa7bb2 IoctlEmulation: Update AssignDeviceTypeToFD
Adds known good drivers to remove a log.
2024-10-02 19:29:45 -07:00
Ryan Houdek dd837fa693 LinuxSyscalls: Update max reported kernel version to 6.11
Investigated the uapi differences and nothing terrifying leapt out that
we haven't already implemented.
2024-10-02 15:36:47 -07:00
Ryan Houdek eeff198ff1 LinuxEmulation: Update syscalls for v6.11
Only thing added was uretprobe on x86-64. We can't support this so just
return ENOSYS.
2024-10-02 14:40:11 -07:00
Ryan Houdek 0cd6371c1b IoctlEmulation: Add panthor_drm 2024-10-02 14:27:24 -07:00
Ryan Houdek 5aff16f72b IoctlEmulation: Update xe_drm 2024-10-02 14:26:38 -07:00
Ryan Houdek 2c7896bce4 IoctlEmulation: Update v3d_drm 2024-10-02 14:26:30 -07:00
Ryan Houdek 6dd72a09e5 Scripts: Fixes newer clang-python
It provided this in the newer clang-python
2024-10-02 14:25:59 -07:00
Ryan Houdek 4e6a4d9b69 Linux: Update drm-headers to v6.11 2024-10-02 13:43:43 -07:00
Alyssa Rosenzweig 44484a7b05 ConstProp: drop non-loadbearing opts
every pattern costs us JIT time, but not every pattern is doing anything.
especially with the flag rework of 2023-2024, a lot of patterns just don't make
sense anymore. constant folding in particular isn't too useful now.

only instcountci change is AAD which i'm contractually forbidden from caring
about.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-10-02 14:01:32 -04:00
Alyssa Rosenzweig 6ac7ba388f ConstProp: rm leftover
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-10-02 13:38:48 -04:00
Alyssa Rosenzweig 36d0a67070 ConstProp: don't pool with a hash table
hash tables have high overhead! but individual blocks are small, so the
quadratic search is _much_ faster in practice. knocks 8% off node.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-10-02 13:38:48 -04:00
Alyssa Rosenzweig 0650dd1992 ConstProp: defer initialization
for blocks that don't need pooling. hopefully that's most of them.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-10-02 13:38:48 -04:00
Alyssa Rosenzweig e2d58809ed OpcodeDispatcher: rm dead
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-10-02 13:38:48 -04:00
Alyssa Rosenzweig 967a74cda9 OpcodeDispatcher: manually inline constants
less work for constprop.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-10-02 13:38:48 -04:00
Alyssa Rosenzweig 6cc6181261 OpcodeDispatcher/Flags: dont emit zero
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-10-02 13:38:48 -04:00
Alyssa Rosenzweig f175b525f4 OpcodeDispatcher/Flags: inline constants
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-10-02 13:38:48 -04:00
Alyssa Rosenzweig 319f1e66cb RegisterAllocationPass: eliminate a silly sxtb
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-10-02 13:38:48 -04:00
Alyssa Rosenzweig d16969bdce RegisterAllocationPass: simplify
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-10-02 13:38:48 -04:00
Alyssa Rosenzweig c17858bfeb RegisterAllocationPass: don't lookup sources
lot of silly chasing. not needed.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-10-02 13:38:48 -04:00
Alyssa Rosenzweig ec24fc3d5d RegisterAllocationPass: infer AnyRemapped
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-10-02 13:38:48 -04:00
Alyssa Rosenzweig 8819fa88d5 RegisterAllocationPass: avoid allocating remaps
shaves 6ms off.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-10-02 13:38:48 -04:00
Alyssa Rosenzweig 5e9c2110db Merge pull request #4096 from pmatos/x87Cleanup
x87 small cleanups; NFC
2024-10-02 12:43:13 -04:00
Alyssa Rosenzweig 5aaa18e7a2 RegisterAllocationPass: remove Class->Allocated
not needed anymore.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-10-02 12:41:37 -04:00
Alyssa Rosenzweig 8a551b9e64 RegisterAllocationPass: rm leftover comment
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-10-02 12:41:37 -04:00
Alyssa Rosenzweig aa548bd19c RegisterAllocationPass: track if we need a remap
should be faster on average.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-10-02 12:41:37 -04:00
Alyssa Rosenzweig 9565f16d84 RegisterAllocationPass: simplify
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-10-02 12:41:37 -04:00
Paulo Matos eb76dbdf4d x87 small cleanups; NFC 2024-10-02 18:27:51 +02:00
Ryan Houdek 967c04e252 Merge pull request #4088 from bylaws/winesys
ARM64EC: Dynamically determine syscall numbers under wine.
2024-10-01 09:07:35 -07:00
Ryan Houdek 49de1fac59 Merge pull request #4028 from neobrain/refactor_jemallocless_tools
Build host tools without jemalloc
2024-09-30 11:59:45 -07:00
Ryan Houdek 3c205eb35e Merge pull request #4094 from Sonicadvance1/strict_unittest_args
unittests: Be more strict with -- separator
2024-09-30 11:38:30 -07:00
Tony Wasserka 682b8ef705 Build host tools without jemalloc
Jemalloc blocks use on platforms with 16K pages.
2024-09-30 20:38:24 +02:00
Tony Wasserka 8c47a40625 Update jemalloc 2024-09-30 20:35:46 +02:00
Ryan Houdek ee5c6af868 unittests: Be more strict with -- separator 2024-09-30 11:30:56 -07:00
Ryan Houdek 1670c89b5e Merge pull request #4090 from Sonicadvance1/update_vulkan_headers
Thunks/Vulkan: Updates thunks to v1.3.296
2024-09-30 09:54:15 -07:00
Ryan Houdek e9d34c7ded Merge pull request #4089 from bylaws/wineavx
ARM64EC: AVX register save/restore support under wine
2024-09-30 09:46:33 -07:00
Ryan Houdek c2f2c58367 Merge pull request #4060 from bylaws/ucrt
Windows: Switch supported MinGW CRT to UCRT
2024-09-30 09:31:27 -07:00
Billy Laws 38acd9e18c ARM64EC: Dynamically determine syscall numbers under wine
Takes advantage of the consistent alphabetical syscall ordering that
wine follows.
2024-09-29 19:42:49 +00:00
Billy Laws 21604708ca Windows: Commonise the linker loadcfg and support CFGuard toolchains 2024-09-29 19:42:13 +00:00
Billy Laws 5fec6faeb6 Windows: Switch supported MinGW CRT to UCRT
MSVCRT is deprecated and the reasons FEX used it no longer apply, so
support UCRT and UCRT only.
2024-09-29 19:42:13 +00:00
Billy Laws b659701cef ARM64EC: AVX register save/restore support under wine
Requires corresponding a wine patch to function:
https://github.com/bylaws/wine/commit/7d411bd74bd03ba90cf202623560733cb229cc7d

Since AVX still works in almost all cases without this, I chose not to
e.g. detect the patch and disable AVX support when it is missing.
2024-09-29 18:59:22 +00:00
LC e902ad5278 Merge pull request #3946 from Sonicadvance1/vdso_getrandom
VDSO: Implements v6.11 vdso getrandom
2024-09-28 15:26:35 -04:00
Ryan Houdek 0f54369f9b VDSO: Implements v6.11 vdso getrandom
This is an interesting vdso implementation because it only exists in
x86-64 with kernel v6.11. For Arm64 the implementation is likely to land
in a couple kernel versions from now.

Some differences with vdso_getrandom versus the regular getrandom
syscall
- Has two additional arguments, opaque_state and opaque_len
- Expects the userspace to mmap/munmap this opaque state structure
- Lets userspace query information about this structure upfront
- If the opaque data structure isn't provided then it falls back to
  regular SYS_getrandom

With the "glibc" implementation, we can tell the interface to allocate a
single page that gets unused (otherwise glibc ends up not using the
interface), and fallback to the regular SYS_getrandom.

In the case of the vdso interface, currently the arguments just get
passed through.

Keeping this as a WIP until the ARM64 kernel patches land and I can
actually test the things. Running the "glibc" path works today with the
selftest, but the vdso path is currently completely untested.
2024-09-28 01:06:56 -07:00
LC f640dcc7a4 Merge pull request #4075 from Sonicadvance1/move_thunkhandler
Thunks: Move to the frontend
2024-09-28 03:26:10 -04:00
Ryan Houdek b59da0e049 Thunks/Vulkan: Updates thunks to v1.3.296
Between this version and our previous version v1.3.278 this adds a few
extensions.
- VK_EXT_device_generated_commands
- VK_EXT_depth_clamp_control
- VK_AMD_anti_lag
- VK_KHR_pipeline_binary

These extensions are almost immediately getting picked up by mesa and
the D3D layers, so we want to support them as soon as possible.

Once again updated as described in #2076. This is starting to take more
time as more features and extensions get added to Vulkan. We probably
want to update the `DefinitionExtract.py` script at some point to
contain all the tertiary data in some json so everything gets generated
automatically. Not doing that today but something to think about.

Fixes #4083
2024-09-27 17:59:31 -07:00
Ryan Houdek 4ae3aef502 Vulkan-Headers: Update to v1.3.296 2024-09-27 17:17:50 -07:00
Ryan Houdek 926fa3c24c Thunks: Just move code around for a minor cleanup
NFC
2024-09-27 15:50:23 -07:00
Ryan Houdek 8c1740f592 Thunks: Remove the temporary TLS variables
From prep commit 511103ee5dcc9474b0b7468c05f61ce10fea4393.
Now that the frontend is setup, we can remove the temporary TLS
variables and use the ThreadObject directly.
2024-09-27 15:50:23 -07:00
Ryan Houdek 611aa0a5b9 Thunks: Wire up TLS handling in the frontend
Because it was moved from the backend to the frontend, re-wire up the
TLS handling which requires going through our syscall handler.
2024-09-27 15:50:21 -07:00
Ryan Houdek 0b8b5108b9 Thunks: Moves AppendThunkDefinitions function to the frontend
Backend doesn't need access to this at all
2024-09-27 15:48:31 -07:00
Ryan Houdek e409a0afec Thunks: Shift namespace name
Moves from FEXCore to FEX::HLE. Also moves the ThunkFunctions that get
exposed to a namespace to make it more obvious that these are
thunkhandlers rather than just static functions.
2024-09-27 15:48:31 -07:00
Ryan Houdek 24211f8523 FEXCore: Move ThunksHandler class to FEXLoader
With as little changes as possible, because this is fairly tricky.
2024-09-27 15:48:31 -07:00
Ryan Houdek 94462e4dd0 Thunks: Temporarily add more TLS variables while moving
Prep commit to give Thunks the data it needs before moving.
2024-09-27 15:48:31 -07:00
Ryan Houdek aa44668a41 FEXCore/Thunks: Moves the implementation of LinkAddressToGuestFunction to ContextImpl
No functional change, just moving code here.
2024-09-27 15:48:31 -07:00
Ryan Houdek 33a675624c FEXCore/Thunks: Remove lock passing in AddCustomIREntrypoint
This was a legacy feature that is no longer necessary every since we
removed the frontend IRLoader.
2024-09-27 15:48:31 -07:00
LC 5745b419d9 Merge pull request #4045 from Sonicadvance1/stop_using_dlopen_for_vdso
VDSOEmulation: Stop using dlopen for VDSO
2024-09-27 18:20:55 -04:00
LC 9100235041 Merge pull request #4077 from Sonicadvance1/frontend_informed_supportsavx
SignalDelegator: Let the frontend inform AVX support
2024-09-27 18:15:13 -04:00
LC 4d62e75d5a Merge pull request #4085 from pmatos/FScaleZero
Fix FScale'ing zero that should return zero
2024-09-27 18:13:07 -04:00
LC b6d3df0e54 Merge pull request #4086 from Sonicadvance1/stop_breaking_x86
Stop installing binfmt_misc on x86
2024-09-27 03:34:41 -04:00
Ryan Houdek d5d1230422 Stop installing binfmt_misc on x86
With the new systemd support, we had forgotten to wrap this check in an
arm64 check. I had done an install and broke my x86 machine.

Add the check back so we stop breaking x86 machines...again. Took me
only about three hours to find the issue this time.
2024-09-26 15:49:48 -07:00
Paulo Matos f35a212f06 Fix FScale'ing zero that should return zero
Added tests will fail without the current patch.
This will properly check if we return the correct sign of zero.
2024-09-26 17:57:39 +02:00
Ryan Houdek f49d82deb3 Merge pull request #4074 from Sonicadvance1/global_rootfs
FEXConfig: Add the ability to watch and configure global rootfs
2024-09-24 03:30:47 -07:00
Ryan Houdek 77b801b8ae FEXConfig: Add the ability to watch and configure global rootfs
Global rootfs doesn't support a named rootfs (only local supports that).
So when it is a global rootfs option just give the full path to the
config path.

Just needs some special handling to ensure the full path is retained for
global paths.

A reimplementation of #4067 that doesn't hardcode paths and routes the
global data path through FEXCore.
2024-09-24 01:17:45 -07:00
Ryan Houdek 163a3fc899 FEXConfig: Convert select over to ppoll usage
Using select initially was a mistake, it has issues with FD limits and
is less efficient due to fd_set. It's recommended to switch select using
over to polling, so do it.
2024-09-24 01:17:45 -07:00
Ryan Houdek 9b627e8743 FEXCore: Add global data directory support
Currently unused as all data still comes from local installs.
2024-09-24 01:17:43 -07:00
LC 9d6865f62d Merge pull request #4073 from Sonicadvance1/constexpr_all_tables
OpcodeDispatcher: Constexpr-ify all the tables possible
2024-09-23 21:40:37 -04:00
LC d547f2b8f2 Merge pull request #4080 from Sonicadvance1/improve_createthread_stack
Linux: Optimize CreateNewThread and HandleNewClone stack usage
2024-09-23 21:28:56 -04:00
Ryan Houdek b5c9eb3463 Merge pull request #4081 from Sonicadvance1/smarter_meta_reload
Config: Be a bit smarter in ReloadMetaLayer
2024-09-23 04:52:25 -07:00
Ryan Houdek 00a3793814 Config: Be a bit smarter in ReloadMetaLayer
Reduces stack usage from ~1024 bytes to ~704 bytes and stops making a
bunch of string copies when they can just be pointers instead.
2024-09-23 04:21:10 -07:00
LC 4f8d7cb89c Merge pull request #4078 from Sonicadvance1/fix_envloader_stackusage
Config: Fixes stack usage in EnvLoader
2024-09-22 19:19:46 -04:00
LC 9310ac59f4 Merge pull request #4076 from Sonicadvance1/remove_block_sampling
FEXCore: Remove BlockSamplingData
2024-09-22 19:18:37 -04:00
Ryan Houdek 9e4de3bfe8 Merge pull request #4079 from alyssarosenzweig/bug/maxss-nan
Fix (v)maxss on non-AFP platforms
2024-09-22 11:45:30 -07:00
Alyssa Rosenzweig 598b99fe58 InstCountCI: Update
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-09-22 14:06:27 -04:00
Alyssa Rosenzweig 9dda76ebe6 unittests: add min/max tests
we had a bug with max of nans. since x86 behaviour is weird, add a unit test to
try all the weird things. this fails on main without AFP

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-09-22 14:06:27 -04:00
Ryan Houdek f5def7ae1c Linux: Also optimize HandleNewClone stack usage
Drops from ~1392 bytes of stack usage to ~80 bytes
2024-09-22 10:51:25 -07:00
Ryan Houdek 58bcf691d6 Linux: Optimize CreateNewThread stack usage
We were creating a copy of the FEXCore::Core::CPUState object when we
didn't need to. We can pass the host thread's CPUState frame through to
the creation handlers since it's read-only (so modify it to be const).

We then just move the RAX and RSP setting to /after/ the CreateThread
handling instead of before.
This reduces stack usage from ~1392 bytes to ~80 bytes.
2024-09-22 10:45:21 -07:00
Alyssa Rosenzweig 33186b803d JIT: fix scalar fmax on non-AFP platforms
NaN and signed zero handling was busted.  Pretty nasty.

Fixes the following Vulkan CTS test run under emulation without thunking with a clang-built Mesa:

  dEQP-VK.spirv_assembly.instruction.graphics.float_controls.fp32.generated_args.denorm_nmax_nan_flush_to_zero_vert

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-09-22 12:14:21 -04:00
Alyssa Rosenzweig bb1d7d0750 JIT: rm broken dead code
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-09-22 12:14:21 -04:00
Ryan Houdek 4897c1e80c Config: Fixes stack usage in EnvLoader
EnvLoader::Load was generating strings on the stack and then copying
them to the FEXCore::Config state. FEXCore::Config::Set supports
string_views directly which will emplace in to the map directly.

This improves this functions stack usage from ~4464 bytes to ~288 bytes.
2024-09-22 08:45:02 -07:00
Ryan Houdek e7605c94dd SignalDelegator: Let the frontend inform AVX support
Since the frontend has changed to informing the backend if AVX is
supported, there is no reason to feed that configuration back in to
SignalDelegator from the backend.

Instead inform the SignalDelegator directly in the frontend instead of
this now weird round-about path.
2024-09-22 08:23:06 -07:00
Ryan Houdek ab5a10374b FEXCore: Remove BlockSamplingData
This was a legacy interface that has long lost its implementation and
has long lost its use. This interface was similar to an older
Dolphin-Emu interface that was useful in that project, but isn't really
useful in a world with hundreds of threads and processes running at the
same time.
2024-09-22 08:04:39 -07:00
Ryan Houdek dc2a26582f FEXCore/OpcodeDispatcher: Fixes missing static on some tables
This was causing egregious stack usage in these two functions
2024-09-16 18:52:55 -07:00
Ryan Houdek e6512fbe4d X86Tables: Remove function wraps from VEX tables 2024-09-16 18:52:55 -07:00
Ryan Houdek dbca441573 X86Tables: Remove function wraps from DDD table 2024-09-16 18:52:55 -07:00
Ryan Houdek 31adc953b7 X86Tables: Remove function wraps from H0F38/H0F3A table 2024-09-16 18:52:55 -07:00
Ryan Houdek 0dd7f5e2f7 X86Tables: Remove function wraps from SecondaryModRM table 2024-09-16 18:52:54 -07:00
Ryan Houdek 36556c4705 X86Tables: Remove function wraps from Group tables 2024-09-16 18:52:54 -07:00
Ryan Houdek e29fac3f25 X86Tables: Remove function wraps from Secondary tables 2024-09-16 18:52:54 -07:00
Ryan Houdek c984bdb42a OpcodeDispatcher: Remove previous template instantiantions and use Bind 2024-09-16 18:52:54 -07:00
Ryan Houdek 2645b374a7 OpcodeDispatcher: Deduplicate InstallToTable helper 2024-09-16 18:52:54 -07:00
Ryan Houdek 10192eebe5 X86Tables: Remove EVEX Table
Still track the telemetry if EVEX is attempted, but remove the tables
which we won't be supporting.
2024-09-16 17:40:20 -07:00
Ryan Houdek 0e57cbf5b9 OpcodeDispatcher: Remove EVEX table install
This hasn't been necessary for quite a while as we handle
invalid/unsupported opcodes in the frontend now.
2024-09-16 17:40:20 -07:00
Ryan Houdek c7413d96ad OpcodeDispatcher: Constexpr-ify VEX & VEXGroup tables 2024-09-16 17:40:20 -07:00
Ryan Houdek f3ba8cb33c OpcodeDispatcher: Constexpr-ify DDD tables 2024-09-16 17:40:20 -07:00
Ryan Houdek e714933b10 OpcodeDispatcher: Constexpr-ify H0F3A tables 2024-09-16 17:40:20 -07:00
Ryan Houdek 9a0f83cbf6 OpcodeDispatcher: Constexpr-ify H0F38 tables 2024-09-16 17:40:20 -07:00
Ryan Houdek c4d30e8437 OpcodeDispatcher: Constexpr-ify SecondaryModRM tables 2024-09-16 17:40:20 -07:00
Ryan Houdek 7b30df8a56 OpcodeDispatcher: Constexpr-ify SecondaryGroup tables 2024-09-16 17:40:20 -07:00
Ryan Houdek 86aa459dd1 OpcodeDispatcher: Constexpr-ify PrimaryGroup tables 2024-09-16 17:40:20 -07:00
Ryan Houdek 6c9a47fc71 OpcodeDispatcher: Constexpr-ify Secondary OpSizeMod tables 2024-09-16 17:40:20 -07:00
Ryan Houdek 7a72cf631b OpcodeDispatcher: Constexpr-ify Secondary RepNEMod tables 2024-09-16 17:40:20 -07:00
Ryan Houdek 3d98eef7b8 OpcodeDispatcher: Constexpr-ify Secondary RepMod tables 2024-09-16 17:40:20 -07:00
Ryan Houdek f2011b0b79 OpcodeDispatcher: Constexpr-ify Secondary base tables 2024-09-16 17:40:19 -07:00
Ryan Houdek c0b8a0d9c3 Merge pull request #4066 from Sonicadvance1/remove_imgui
Tools: Delete imgui FEXConfig
2024-09-16 17:25:46 -07:00
Ryan Houdek 5bd0afa307 Merge pull request #4068 from Sonicadvance1/update_cpuid
CPUID: Update to something a little more modern
2024-09-16 06:33:04 -07:00
Ryan Houdek 20fb0da7d9 Merge pull request #4048 from Sonicadvance1/constexpr_more_tables
FEXCore: Convert Base tables over to constexpr
2024-09-16 06:32:23 -07:00
Ryan Houdek 35cb1f710c FEXQConfig->FEXConfig 2024-09-16 02:11:27 -07:00
Ryan Houdek 7c7b99f9c5 FEXConfig: Remove 2024-09-16 02:09:25 -07:00
Ryan Houdek 09879bd962 CPUID: Update to something a little more modern
Previously reported as some old CPU without AVX and SSE4 and other
things.
Start advertising as something more modern that actually shipped with
AVX2 and other features. Should help some modern libraries that do bad
family and model checks rather that CPUID features checks.

Also removes the silly `(ES)` tag from CPU-Z.

Also moves generation in to a constexpr function that can actually range
check these 4-bit and 8-bit values.
2024-09-16 01:42:35 -07:00
Ryan Houdek c8f3fe3788 Merge pull request #4071 from Sonicadvance1/unittests_incorrect_arg
unittests: Fixes incorrect argument
2024-09-15 15:20:54 -07:00
Ryan Houdek 88e3164b55 unittests: Fixes incorrect argument
Turns out our unittests have been passing an incorrect argument for
years.
2024-09-14 23:15:52 -07:00
Ryan Houdek bebd7400c0 Merge pull request #4063 from Sonicadvance1/fallback_if_no_zenity
FEXRootFSFetcher: Fallback to TTY if zenity isn't installed
2024-09-14 16:14:32 -07:00
Ryan Houdek e190d029dc Merge pull request #4064 from Sonicadvance1/remove_binfmt_flag
FEXLoader: Drop the binfmt_misc `I` flag
2024-09-13 12:51:24 -07:00
Ryan Houdek 37a70e2ec6 FEXCore: Convert Base tables over to constexpr
Only doing the single table for review purposes. Once reviewed I will
hammer out the remaining tables.

Similar to #3320, most of the OpcodeDispatcher tables can be consteval
and made to be a compile time constant. This just requires shuffling the
code slightly. The idea is to get almost all of the table setup out of
the `InstallOpcodeHandlers` function and instead only install the
handlers that change based on 32-bit or 64-bit, just like the x86 tables
we also did.
2024-09-13 11:39:09 -07:00
Ryan Houdek 6ff073ac11 FEXRootFSFetcher: convert vector usage over to array and span
Most uses of vector could actually be converted to array and a lot of
argument passing could actually be converted over to spans instead.
2024-09-13 11:29:44 -07:00
Ryan Houdek b6e4c47abc FEXLoader: Drop the binfmt_misc I flag
Upstream has rejected this flag which would have let FEX close the gap
in functionality between binfmt_misc interpreters and PT_INTERP
interpreters for how `/proc/exe` is handled. Since we are unable to
change their opinions, just remove the code from FEX since it's never
going to happen.

Removes a little bit of confusing behaviour in execveat and
filemanagement handling. Leaving us with only the regular confusing
behaviour of trying to track accesses to `/proc/exe` instead.
2024-09-12 19:58:35 -07:00
Ryan Houdek daf8b409f3 FEXRootFSFetcher: Fallback to TTY if zenity isn't installed
If packaged improperly or if someone has just did a source install then
they may not have zenity. Fallback to TTY path if zenity isn't
installed.
2024-09-11 17:08:20 -07:00
Ryan Houdek 0a35029d9a Merge pull request #4062 from Sonicadvance1/fix_fexserver_writing_to_pipe
FEXServer/PipeScanner: Stop writing null to incoming pipes
2024-09-11 13:59:24 -07:00
Ryan Houdek e8b51ac0a4 FEXServer/PipeScanner: Stop writing null to incoming pipes
This was causing problems in a typical use case of piping stdout/stderr
to another application. So if FEXInterpreter was started, with stdout
piped to another application, and FEXServer wasn't running so a new
instance needed to be started. FEXServer would write a null to every
pipe then close it. This includes the stdout that is being piped to the
new application!

This writing was legacy behaviour before FEX was properly listening to
POLLHUP and is no longer necessary. Just remove the writing completely
to fix this issue.

Easiest test case was just `./Bin/FEXInterpreter `which ls` / | xxd | head -n 3`

Returned:
```
$ ./Bin/FEXInterpreter `which ls` / | xxd | head -n 3
00000000: 0000 0000 6269 6e0a 6269 6e2e 7573 722d  ....bin.bin.usr-
00000010: 6973 2d6d 6572 6765 640a 626f 6f74 0a64  is-merged.boot.d
00000020: 6576 0a65 7463 0a68 6f6d 650a 6c69 620a  ev.etc.home.lib.
```

That first null shouldn't be there.
2024-09-11 12:52:57 -07:00
Ryan Houdek f20e626fda Merge pull request #4059 from bylaws/log
Windows: Don't assume the log file was successfully opened
2024-09-11 10:16:14 -07:00
Ryan Houdek 2672e06aa5 Merge pull request #4058 from bylaws/cef
ARM64EC: Patch the call checker before calling any syscall exports
2024-09-11 10:16:02 -07:00
Ryan Houdek 4a2b4be59d Merge pull request #4057 from bylaws/overcommit
Support for emulated overcommit on Windows
2024-09-11 10:15:43 -07:00
Ryan Houdek 05be9445cd Merge pull request #4056 from Sonicadvance1/context_remove_unused_features
FEXCore/Context: Removes unused features
2024-09-11 10:09:24 -07:00
Ryan Houdek 4cb4c0e277 Merge pull request #4055 from Sonicadvance1/remove_header
FEXCore: Remove Context header include when unneeded
2024-09-11 10:09:14 -07:00
Billy Laws 0135e2d78c LookupCache: Use emulated overcommit if necessary
Allocates uncommited memory and calls the frontend callbacks so it can
commit pages on faults as necessary.
2024-09-10 16:08:45 +00:00
Billy Laws 1266a5a5ce Windows: Support emulated memory overcommit 2024-09-10 16:08:45 +00:00
Billy Laws 7255850e9d ARM64EC: Patch the call checker before calling any syscall exports
CEF on Windows patches these before FEX is even loaded, so this needs to
be setup as early as possible using some direct syscalls which skip any
such patches.
2024-09-10 16:05:37 +00:00
Billy Laws 002a8bebd8 Windows: Cleanup exception handling logic 2024-09-10 16:05:02 +00:00
Billy Laws 452c9fd19b AllocatorHooks: Make commiting allocated memory optional
Ignored for linux as it supports overcommit.
2024-09-10 16:05:02 +00:00
Billy Laws 0b0e15c9b4 Windows: Don't assume the log file was successfully opened 2024-09-10 15:58:03 +00:00
Ryan Houdek ae9db336e7 FEXCore/Context: Removes unused features
No functional change here.
- CoreRunningMode enum and variable wasn't used anymore.
   - Code was moved to the frontend
- CustomCPUFactory wasn't used anymore
   - All special signal handling and various features were moved to
     TestHarnessRunner
   - We also don't want to support actual custom CPU cores.
   - TestHarnessRunner just runs as a host runner if compiled on an
     x86-64 device if vixl sim isn't enabled now.
   - Removes the Core config option entirely.
- Moves VDSOPointers struct to the frontend
   - Every use of this lives in the Linux frontend instead now
2024-09-09 18:38:33 -07:00
Ryan Houdek 360ea538b7 Merge pull request #4054 from Sonicadvance1/fix_xxhash
FEXRootFSFetcher/XXHash: Fix double fd close
2024-09-09 18:03:55 -07:00
Ryan Houdek f99900bdc5 Merge pull request #4053 from Sonicadvance1/disable_vixl_compiling
CMake: Disable vixl compiling if not enabled
2024-09-09 18:03:46 -07:00
Ryan Houdek 34f2bce203 FEXCore: Remove Context header include when unneeded
This is one of the most expensive headers in FEX, averaging 1.2 seconds
of compile time per include.
We include this header 31 times around the project. Remove
the five instances where it is unused to help this issue. Next step
would be to make the header lighter if possible.
2024-09-09 17:43:35 -07:00
Ryan Houdek a05d6ae082 FEXRootFSFetcher/XXHash: Check for errors in two locations that were missed 2024-09-09 14:07:23 -07:00
Ryan Houdek 6a4eb434f7 FEXRootFSFetcher/XXHash: Fix double fd close
Since `HadError` is a lambda that handles FD closing this was a double
FD close. Just move the xxhash state freeing in there as well and remove
the FD and hash freeing. Fixes the double free.
2024-09-09 14:00:44 -07:00
Ryan Houdek 0c3eb41f57 CMake: Disable vixl compiling if not enabled
Like how FEXCore disabled vixl linking, make sure we aren't accidentally
compiling and including it when vixl stuff is disabled. Noticed this in
the build logs.
2024-09-09 13:54:35 -07:00
Ryan Houdek e4bfbda008 VDSOEmulation: Stop using dlopen for VDSO
We have been relying on the
`YesIKnowImNotSupposedToUseTheGlibcAllocator` fault avoidance for a long
while now. The plan was to rewrite the symbol fetching in the future to
avoid this since VDSO is kind of special anyway.

That time is now, I have had this VDSO symbol parsing code living in a
different project for a few months now and it is working great.

The basic things here are that the Linux kernel provides the VDSO
mapping base pointer through the auxv value `AT_SYSINFO_EHDR`. We then
need a minimal ELF parser that /only/ parses the dynamic symbol header
(and accompanying string header).

Then it's a simple case of walking the symbol table and recording the
pointers. Confirmed this fetches all the correct symbols (As I've been
using it for a while already I already knew it worked.)

This lets us stop allocating memory through the glibc allocator due to
dlopen.
2024-09-09 13:26:06 -07:00
Ryan Houdek b4c797c199 Merge pull request #4052 from Sonicadvance1/various_warnings
Bunch of little things reported by coverity
2024-09-09 12:54:22 -07:00
Ryan Houdek ebae1bf71d Merge pull request #4051 from Sonicadvance1/fix_warning
Arm64: Fix warning
2024-09-09 08:24:48 -07:00
Ryan Houdek fa293e5e8f GDBServer: Fix buffer overrun in x87 register copying 2024-09-08 20:24:17 -07:00
Ryan Houdek 720c89e648 GDBServer: Make it clear we're copying all 512-bytes to clear a warning 2024-09-08 20:23:03 -07:00
Ryan Houdek abc5f5abd4 GDBServer: Use std::move for move assignment 2024-09-08 20:22:39 -07:00
Ryan Houdek 22ab16b217 Linux/x32/ipc: Fixes version check for msgrcv
Also adds an EINVAL return if version 1 is attempted with SHMAT,
mirroring what the kernel returns.
2024-09-08 20:18:11 -07:00
Ryan Houdek b80f05dd86 Seccomp/BPFEmitter: Reorder check to avoid integer overflow warning 2024-09-08 20:13:14 -07:00
Ryan Houdek 1642a0f4d9 Seccomp/BPFEmitter: Initialize Func,FuncSize just to make coverity happy 2024-09-08 20:10:44 -07:00
Ryan Houdek 96e990dade SignalDelegator: Use const auto ref to avoid copy 2024-09-08 20:07:45 -07:00
Ryan Houdek 3423a12ab6 SignalDelegator: Remove unused function copying for delegators 2024-09-08 20:06:38 -07:00
Ryan Houdek c4a0fb6609 Ioctl/drm: Fix missing member copy 2024-09-08 20:02:51 -07:00
Ryan Houdek 8cf23eba9c FEXLoader/ELFCodeLoader: Use const ref auto to avoid copy 2024-09-08 20:02:51 -07:00
Ryan Houdek fbe817f1c4 Config: Use std::move on strings to avoid copy 2024-09-08 20:02:51 -07:00
Ryan Houdek 3b61394548 FEXRootFSFetcher: Fixes some extract logic
`Extract` was never set to false even if the user said not to overwrite.
Ensure that if the user decided not to overwrite an existing folder,
that it returns false so it correctly doesn't overwrite the folder.
2024-09-08 20:02:51 -07:00
Ryan Houdek 8ed6b36181 FileManagement: Use const ref for AppConfigName 2024-09-08 20:02:51 -07:00
Ryan Houdek 9141322666 Ioctl/drm: Adds missing flags member copying 2024-09-08 20:02:51 -07:00
Ryan Houdek 2d91c5441e Ioctl/Radeon: Adds missing func member copying 2024-09-08 20:02:51 -07:00
Ryan Houdek 2ff1589c35 FileManagement: Use const ref for Filename 2024-09-08 20:02:51 -07:00
Ryan Houdek a308b9edbe Linux/GenerateMap: Fixes FD leak
This FD was unused
2024-09-08 19:34:38 -07:00
Ryan Houdek f6a8e595df Linux/Syscalls: Add todo for futimesat 2024-09-08 19:34:38 -07:00
Ryan Houdek eaaf62e4b7 X87: Make it a bit clearer that 80-bit floats can't hit the memory path
Makes the static analysis happy.
2024-09-08 18:03:20 -07:00
Ryan Houdek dc5fc57e19 FEXServer: Check if prctl errors 2024-09-08 18:03:20 -07:00
Ryan Houdek c3c0bbb060 F80Fallbacks: Fix Uninitialized variable warning that can't occur 2024-09-08 18:03:20 -07:00
Ryan Houdek 98a91d242d FEXRootFSFetcher: Fixes resource leak on XXHash error 2024-09-08 18:03:20 -07:00
Ryan Houdek 8da6b1e729 FEXCore/Allocator: Fixes uninitialized scalar variable warning that can't occur 2024-09-08 18:03:20 -07:00
Ryan Houdek d260bce31e FEXRootFSFetcher: Fixes uninitialized variable 2024-09-08 18:03:20 -07:00
Ryan Houdek a94aaca0ec OpcodeDispatcher: Add default FEX_UNREACHABLE
This can't happen but if it changes then make sure we capture it.
2024-09-08 18:03:20 -07:00
Ryan Houdek 2ac3ca3ead pidof: Fix uninitialized variable warning 2024-09-08 18:03:20 -07:00
Ryan Houdek c559445b62 Linux/IoctlEmulation: Fixes incorrect direction copy for a few ioctls
Plus convert remaining handlers over to using CPYF/CPYT handlers so
ensure this doesn't happen again.
2024-09-08 18:03:20 -07:00
Ryan Houdek 8ea4e4f663 Linux: Adds some missing brace initializers on conversion operators 2024-09-08 18:03:20 -07:00
Ryan Houdek 2da700fe82 CodeEmitter: Removes some ternaries that are unnecessary
NFC
2024-09-08 18:03:20 -07:00
Ryan Houdek 0a32ba9b29 instcountCI: Fixes for shifts 2024-09-08 18:03:20 -07:00
Ryan Houdek f51832ca6b FEXCore: Fixes a bug with VPSRLDQ/VPSLLDQ with >= 16-byte shifts
When the shift amount is >= 16-bytes then we need to zero the register.
We had a bug where we were assigning `Result.High` to itself, which
effectively made the top 128-bits of the ymm register not modify itself.

Adds a unit test to ensure that doesn't happen again.
2024-09-08 15:39:22 -07:00
Ryan Houdek 5f9a30af7a FileManagement: Use const ref for RootFSPath
No need to make a copy
2024-09-08 15:20:04 -07:00
Ryan Houdek 453ef0b94c Linux/ExecveHandler: Move a path rather than copy
We can do move assignment instead of copy assignment here.
2024-09-08 15:17:04 -07:00
Ryan Houdek 4779e64cf5 Linux/EmulatedFiles: Check return for seal
This shouldn't be possible to occur, but have a log message just in
case.
2024-09-08 15:15:38 -07:00
Ryan Houdek e6025cc087 Arm64: Fix warning
This variable is only used in a log message no, so move it there.
2024-09-08 11:15:46 -07:00
Ryan Houdek cd4c224dc0 Merge pull request #4050 from alyssarosenzweig/sra/move-size
JIT: always use 64-bit moves for SRA
2024-09-08 10:30:17 -07:00
Ryan Houdek 07f117b957 Merge pull request #4047 from bylaws/arm64ec-sysv2
ARM64EC: Switch to exception-based native syscall dispatch
2024-09-08 10:30:05 -07:00
Ryan Houdek 36e4f9a070 Merge pull request #4049 from alyssarosenzweig/opt/constprop-merges
ConstProp: speed it up
2024-09-08 10:13:23 -07:00
Alyssa Rosenzweig 67c751d3e5 JIT: always use 64-bit moves for SRA
I see no reason not to. Simpler code and it's slightly faster on Firestorm.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-09-08 13:02:57 -04:00
Ryan Houdek 1c59bfeb9f Merge pull request #4027 from alyssarosenzweig/opt/global-flag
Global flag optimizations
2024-09-08 09:42:18 -07:00
Alyssa Rosenzweig 2b0041a291 InstCountCI: Update
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-09-08 10:27:20 -04:00
Alyssa Rosenzweig 39cc1e5116 ConstProp: merge folding+inlining
2% off nodejs

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-09-08 10:26:58 -04:00
Alyssa Rosenzweig f46856c090 ConstProp: template constant inlining
DRY

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-09-08 10:26:58 -04:00
Alyssa Rosenzweig b7859052da ConstProp: don't mask shifts
done in the emitter now

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-09-08 10:26:58 -04:00
Alyssa Rosenzweig 3b30d8103f ConstProp: combine like cases
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-09-08 10:26:58 -04:00
Alyssa Rosenzweig d19fc36f8f ConstProp: delay constant pooling
more efficient to pool the output of constant folding and such. but we need to
avoid that becoming accidentally quadratic, so rework that too

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-09-08 10:26:58 -04:00
Alyssa Rosenzweig 5fbf76ee9e CodeEmitter: mask immediate shifts
more convenient for constprop and not really any more expensive.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-09-08 10:26:58 -04:00
Ryan Houdek 5f79761fe4 Merge pull request #4046 from Sonicadvance1/cpubackend_remove_unused
CPUBackend: Remove unused functions
2024-09-07 13:11:32 -07:00
Billy Laws 621af9e7e8 ARM64EC: Switch to exception-based native syscall dispatch
The previous approach assumed that ntdll wouldn't have been patched
before FEX was loaded, but that isn't necessarily the case thanks to
CEF. This takes the same approach used by xtajit which wine recently gained
support for wine. The overhead to this isn't ideal but most syscalls
aren't directly done through x86 code anyway and in the future FEX could
forward unpatched thunks to their ARM variants at JIT time.
2024-09-07 15:30:00 +00:00
Billy Laws f4d107c8f0 ARM64EC: Split out FEX -> packed EC context conversion from reconstruction 2024-09-07 15:29:48 +00:00
Ryan Houdek 59643db331 CPUBackend: Remove unused functions
Some of these were bad ideas, some ideas were something we effectively
grew out of.
2024-09-07 08:07:00 -07:00
Alyssa Rosenzweig 46f36dc89d InstCountCI: Update
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-09-07 10:59:41 -04:00
Alyssa Rosenzweig 384882744c DeadStoreElimination: drop now that it's redundant
it's only really load bearing for pf/af, which is handled as a global flag opt
now. this mitigates some of the compile time hit from globalizing flag opts.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-09-07 10:59:05 -04:00
Alyssa Rosenzweig b36d1f7e7b RedundantFlagCalculationElimination: optimize parity
another global CFG-based optimization -- if we know that the raw PF is already
1-bit we can skip parity evaluation, saving work with floating point compares.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-09-07 10:59:05 -04:00
Alyssa Rosenzweig dd7c66db2e RedundantFlagCalculationElimination: use LUT for flags
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-09-07 10:59:05 -04:00
Alyssa Rosenzweig 47ac9edd7a RedundantFlagCalculationElimination: select testz
this saves uops.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-09-07 10:59:05 -04:00
Alyssa Rosenzweig 6e6d640fec RedundantFlagCalculationElimination: fold compares/axflag into branches
now that we know whether flags are killed on the edge, we can improve branch
isel

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-09-07 10:59:05 -04:00
Alyssa Rosenzweig eb88366614 RedundantFlagCalculationElimination: globalize
Gather a control flow graph and use it to propagate flags throughout the
program.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-09-07 10:59:05 -04:00
Alyssa Rosenzweig 50d6ddd591 RedundantFlagCalculationElimination: extract per-block logic
nfc

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-09-07 10:59:05 -04:00
Alyssa Rosenzweig 249de7c758 RedundantFlagCalculationElimination: drop unneeded bools
nonzero <==> not dummy

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-09-07 10:59:05 -04:00
Alyssa Rosenzweig c350888d70 RedundantFlagCalculationElimination: add missing adczero case
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-09-07 10:59:05 -04:00
Alyssa Rosenzweig f23bef1653 RedundantFlagCalculationElimination: add missing testnz case
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-09-07 10:59:05 -04:00
Alyssa Rosenzweig d17f33a922 OpcodeDispatcher: don't emit fake 0 for condjump
not needed and getting in the way

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-09-07 10:59:05 -04:00
Alyssa Rosenzweig 50a3ca0d6d IR: introduce dedicated PF/AF instructions
this makes reasoning about them a little easier, e.g. for flags.  about 1% win
in nodejs.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-09-07 10:59:05 -04:00
Alyssa Rosenzweig 8745455a5b IR: track whether parity is read
so we can gate optimizations efficiently

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-09-07 08:26:04 -04:00
Alyssa Rosenzweig e9ab514962 IR: push parity evaluation down
so we can optimize it globally

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-09-07 08:26:04 -04:00
Alyssa Rosenzweig 4eb0948451 IR: push down AXFLAG lowering
so we can get the new axflag optimizations on billy's x13s.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-09-07 08:26:04 -04:00
Alyssa Rosenzweig 8d9f19bd73 IR: add TestZ op
more optimized than TestNZ if we don't care about the sign bit.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-09-07 08:26:04 -04:00
Alyssa Rosenzweig 9fe7bc5818 InstCountCI: add ucomiss+pf opt case
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-09-07 08:26:04 -04:00
Alyssa Rosenzweig 5a590a9b11 InstCountCI: add 8-bit test cases
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-09-07 08:26:04 -04:00
Ryan Houdek a4acd64246 Merge pull request #4042 from neobrain/refactor_system_libs
CMake: Compile with system libraries for xxhash, Catch2, and fmt, if available
2024-09-06 11:48:12 -07:00
Ryan Houdek 304b5de1af Merge pull request #4038 from Sonicadvance1/move_aotir_debugdata
AOTIR: Move debugdata structure to internal header
2024-09-06 10:50:42 -07:00
Ryan Houdek ca2fe0b301 AOTIR: Move debugdata structure to internal header
This definition doesn't need to be exposed in the public API.
2024-09-06 10:19:45 -07:00
Ryan Houdek bbef4d762c Merge pull request #4044 from bylaws/int2e
OpcodeDispatcher: Do not forbid INT 2E syscalls on 64-bit Windows
2024-09-06 10:18:18 -07:00
Ryan Houdek f77841d784 Merge pull request #4041 from Sonicadvance1/fix_poll
LinuxSyscalls: With poll syscall, ensure fds is writable only if nfds is not zero
2024-09-06 10:17:55 -07:00
Ryan Houdek 219a4777c6 Merge pull request #4040 from Sonicadvance1/delete_threadsstate
FEXCore: Delete ThreadsState struct
2024-09-06 10:16:28 -07:00
Ryan Houdek 29f0e9b4d6 Merge pull request #4039 from Sonicadvance1/move_customir_entry
FEXCore: Move `CustomIRResult` to internal header
2024-09-06 10:16:18 -07:00
Ryan Houdek b75efc42bf Merge pull request #4037 from Sonicadvance1/move_symbol
FEXCore: Move `JITSymbolBuffer` to internal header
2024-09-06 10:15:54 -07:00
Tony Wasserka 90dbd4766d Merge pull request #4043 from alyssarosenzweig/gitignore/build
gitignore: ignore build symlink
2024-09-06 18:08:47 +02:00
Billy Laws f7b911ca43 OpcodeDispatcher: Do not forbid INT 2E syscalls on 64-bit Windows
This works fine on real Windows and is relied on by wine as SystemCall
is set to 1 in KUSER_SHARED_DATA, which causes the ntdll thunks to use
it over `syscall`
2024-09-06 15:58:17 +00:00
Alyssa Rosenzweig 6de0333708 gitignore: ignore build symlink
I symlink build -> Build for YouCompleteMe.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
2024-09-06 11:22:31 -04:00
Tony Wasserka ab5d3ab22b CMake: Don't use pkgconfig when cross-compiling 2024-09-06 10:30:48 +02:00
Tony Wasserka 355c3428c0 CMake: Don't use system libraries when compiling with mingw 2024-09-06 10:30:24 +02:00
Tony Wasserka c76b7bfe8d CMake: Compile with system libraries for xxhash, Catch2, and fmt, if available 2024-09-06 09:47:49 +02:00
Ryan Houdek 64c5362580 LinuxSyscalls: With poll syscall, ensure fds is writable only if nfds is not zero
The pointer is allowed to be null or garbage if the number of nfds
passed in is zero.
Unify the x86 and x86-64 implementations to ensure consistency.

Fixes Warhammer 40k: Relics of War
2024-09-05 14:33:52 -07:00
Ryan Houdek 77469de247 FEXCore: Delete ThreadsState struct
All uses of this have been removed.
2024-09-05 13:39:02 -07:00
Ryan Houdek 48fd827004 FEXCore: Move CustomIRResult to internal header
This definition doesn't need to be exposed in the public API.

Stopped needing to be public once we removed our IR CI thing.
2024-09-05 13:36:15 -07:00
Ryan Houdek f09d511ac8 FEXCore: Move JITSymbolBuffer to internal header
This definition doesn't need to be exposed in the public API.
2024-09-05 13:28:45 -07:00
1381 changed files with 72010 additions and 37546 deletions

No files matched your search

+4 -1
View File
@@ -3,10 +3,13 @@
# Ignore all files in the External directory
External/*
# SoftFloat-3e code doesn't belong to us
# SoftFloat-3e code doesn't belong to us
FEXCore/Source/Common/SoftFloat-3e/*
Source/Common/cpp-optparse/*
# Files with human-indented tables for readability - don't mess with these
FEXCore/Source/Interface/Core/X86Tables/*
# Inline headers with list-like content that can't be processed individually
Source/Tools/LinuxEmulation/LinuxSyscalls/x*/SyscallsNames.inl
Source/Tools/LinuxEmulation/LinuxSyscalls/x*/Ioctl/*.inl
+3
View File
@@ -13,3 +13,6 @@
# Second reformat to find fixed point PR#3577
905aa935f5ce344a48ef4d5edab3c31efa8d793e
# Reformat of CodeEmitter inl files
8760c593ece92d7e9fa94c40da0368fd367c9cad
+1 -1
View File
@@ -250,7 +250,7 @@ jobs:
- name: Upload results
if: ${{ always() }}
uses: 'actions/upload-artifact@v3'
uses: 'actions/upload-artifact@v4'
timeout-minutes: 1
with:
name: Results-${{ env.runner_name }}
+1 -1
View File
@@ -184,7 +184,7 @@ jobs:
- name: Upload results
if: ${{ always() }}
uses: 'actions/upload-artifact@v3'
uses: 'actions/upload-artifact@v4'
timeout-minutes: 1
with:
name: Results-${{ env.runner_name }}
+1 -1
View File
@@ -97,7 +97,7 @@ jobs:
- name: Upload results
if: ${{ always() }}
uses: 'actions/upload-artifact@v3'
uses: 'actions/upload-artifact@v4'
timeout-minutes: 1
with:
name: Results-${{ env.runner_name }}
+2 -2
View File
@@ -128,7 +128,7 @@ jobs:
- name: Upload results
if: ${{ always() }}
uses: 'actions/upload-artifact@v3'
uses: 'actions/upload-artifact@v4'
timeout-minutes: 1
with:
name: Results-${{ env.runner_name }}
@@ -137,7 +137,7 @@ jobs:
- name: Upload results InstCountCI
if: ${{ always() }}
uses: 'actions/upload-artifact@v3'
uses: 'actions/upload-artifact@v4'
timeout-minutes: 1
with:
name: Results-${{ env.runner_name }}-instcountci
+1 -1
View File
@@ -92,7 +92,7 @@ jobs:
- name: Upload results
if: ${{ always() }}
uses: 'actions/upload-artifact@v3'
uses: 'actions/upload-artifact@v4'
timeout-minutes: 1
with:
name: Results-${{ env.runner_name }}
+1 -1
View File
@@ -126,7 +126,7 @@ jobs:
- name: Upload results
if: ${{ always() }}
uses: 'actions/upload-artifact@v3'
uses: 'actions/upload-artifact@v4'
timeout-minutes: 1
with:
name: Results-${{ env.runner_name }}
+1 -1
View File
@@ -4,7 +4,7 @@ compile_commands.json
vim_rc
Config.json
[Bb]uild*/
[Bb]uild*
[Bb]in/
out/
.vscode/
-3
View File
@@ -5,9 +5,6 @@
[submodule "External/cpp-optparse"]
path = Source/Common/cpp-optparse
url = https://github.com/Sonicadvance1/cpp-optparse
[submodule "External/imgui"]
path = External/imgui
url = https://github.com/Sonicadvance1/imgui.git
[submodule "External/xbyak"]
shallow = true
path = External/xbyak
+36 -29
View File
@@ -7,8 +7,8 @@ CHECK_INCLUDE_FILES ("gdb/jit-reader.h" HAVE_GDB_JIT_READER_H)
option(BUILD_TESTS "Build unit tests to ensure sanity" TRUE)
option(BUILD_FEX_LINUX_TESTS "Build FEXLinuxTests, requires x86 compiler" FALSE)
option(BUILD_THUNKS "Build thunks" FALSE)
set(USE_FEXCONFIG_TOOLKIT "imgui" CACHE STRING "If set, build FEXConfig (qt or imgui)")
option(ENABLE_CLANG_THUNKS "Build thunks with clang" FALSE)
option(BUILD_FEXCONFIG "Build FEXConfig" TRUE)
option(ENABLE_CLANG_THUNKS "Build thunks with clang" TRUE)
option(ENABLE_IWYU "Enables include what you use program" FALSE)
option(ENABLE_LTO "Enable LTO with compilation" TRUE)
option(ENABLE_XRAY "Enable building with LLVM X-Ray" FALSE)
@@ -26,10 +26,9 @@ option(ENABLE_OFFLINE_TELEMETRY "Enables FEX offline telemetry" TRUE)
option(ENABLE_COMPILE_TIME_TRACE "Enables time trace compile option" FALSE)
option(ENABLE_LIBCXX "Enables LLVM libc++" FALSE)
option(ENABLE_CCACHE "Enables ccache for compile caching" TRUE)
option(ENABLE_VIXL_SIMULATOR "Forces the FEX JIT to use the VIXL simulator" FALSE)
option(ENABLE_VIXL_SIMULATOR "Enable use of VIXL simulator for emulation (only useful for CI testing)" FALSE)
option(ENABLE_VIXL_DISASSEMBLER "Enables debug disassembler output with VIXL" FALSE)
option(USE_LEGACY_BINFMTMISC "Uses legacy method of setting up binfmt_misc" FALSE)
option(COMPILE_VIXL_DISASSEMBLER "Compiles the vixl disassembler in to vixl" FALSE)
option(ENABLE_FEXCORE_PROFILER "Enables use of the FEXCore timeline profiling capabilities" FALSE)
set (FEXCORE_PROFILER_BACKEND "gpuvis" CACHE STRING "Set which backend you want to use for the FEXCore profiler")
option(ENABLE_GLIBC_ALLOCATOR_HOOK_FAULT "Enables glibc memory allocation hooking with fault for CI testing")
@@ -37,6 +36,7 @@ option(USE_PDB_DEBUGINFO "Builds debug info in PDB format" FALSE)
set (X86_32_TOOLCHAIN_FILE "${CMAKE_CURRENT_SOURCE_DIR}/toolchain_x86_32.cmake" CACHE FILEPATH "Toolchain file for the (cross-)compiler targeting i686")
set (X86_64_TOOLCHAIN_FILE "${CMAKE_CURRENT_SOURCE_DIR}/toolchain_x86_64.cmake" CACHE FILEPATH "Toolchain file for the (cross-)compiler targeting x86_64")
set (X86_DEV_ROOTFS "/" CACHE FILEPATH "Path to the sysroot used for cross-compiling for i686 and x86_64")
set (DATA_DIRECTORY "${CMAKE_INSTALL_PREFIX}/share/fex-emu" CACHE PATH "global data directory")
string(FIND ${CMAKE_BASE_NAME} mingw CONTAINS_MINGW)
@@ -232,7 +232,6 @@ if (ENABLE_JEMALLOC_GLIBC_ALLOC)
# The glibc jemalloc subproject which hooks the glibc allocator.
# Required for thunks to work.
# All host native libraries will use this allocator, while *most* other FEX internal allocations will use the other jemalloc allocator.
add_definitions(-DENABLE_JEMALLOC_GLIBC=1)
add_subdirectory(External/jemalloc_glibc/)
elseif (NOT MINGW_BUILD)
message (STATUS
@@ -244,9 +243,7 @@ endif()
if (ENABLE_JEMALLOC)
# The jemalloc subproject that all FEXCore fextl objects allocate through.
add_definitions(-DENABLE_JEMALLOC=1)
add_subdirectory(External/jemalloc/)
include_directories(External/jemalloc/pregen/include/)
elseif (NOT MINGW_BUILD)
message (STATUS
" jemalloc disabled!\n"
@@ -268,14 +265,11 @@ set (CMAKE_LINKER_FLAGS_RELEASE "${CMAKE_LINKER_FLAGS_RELEASE} -fomit-frame-poin
include_directories(External/robin-map/include/)
if (BUILD_TESTS)
# Enable vixl disassembler if tests are enabled.
set(COMPILE_VIXL_DISASSEMBLER TRUE)
if (BUILD_TESTS OR ENABLE_VIXL_DISASSEMBLER OR ENABLE_VIXL_SIMULATOR)
add_subdirectory(External/vixl/)
include_directories(SYSTEM External/vixl/src/)
endif()
add_subdirectory(External/vixl/)
include_directories(SYSTEM External/vixl/src/)
if (CMAKE_CXX_COMPILER_ID STREQUAL "GNU")
# This means we were attempted to get compiled with GCC
message(FATAL_ERROR "FEX doesn't support getting compiled with GCC!")
@@ -284,29 +278,37 @@ endif()
find_package(PkgConfig REQUIRED)
find_package(Python 3.0 REQUIRED COMPONENTS Interpreter)
set(XXHASH_BUNDLED_MODE TRUE)
set(XXHASH_BUILD_XXHSUM FALSE)
set(BUILD_SHARED_LIBS OFF)
add_subdirectory(External/xxhash/cmake_unofficial/)
pkg_search_module(xxhash IMPORTED_TARGET xxhash libxxhash)
if (TARGET PkgConfig::xxhash AND NOT CMAKE_CROSSCOMPILING)
add_library(xxHash::xxhash ALIAS PkgConfig::xxhash)
else()
set(XXHASH_BUNDLED_MODE TRUE)
set(XXHASH_BUILD_XXHSUM FALSE)
add_subdirectory(External/xxhash/cmake_unofficial/)
endif()
add_definitions(-Wno-trigraphs)
add_definitions(-DGLOBAL_DATA_DIRECTORY="${DATA_DIRECTORY}/")
if (BUILD_TESTS)
add_subdirectory(External/Catch2/)
find_package(Catch2 3 QUIET)
if (NOT Catch2_FOUND)
add_subdirectory(External/Catch2/)
# Pull in catch_discover_tests definition
list(APPEND CMAKE_MODULE_PATH "${CMAKE_CURRENT_SOURCE_DIR}/External/Catch2/contrib/")
endif()
# Pull in catch_discover_tests definition
list(APPEND CMAKE_MODULE_PATH "${CMAKE_CURRENT_SOURCE_DIR}/External/Catch2/contrib/")
include(Catch)
endif()
# Disable fmt install
set(FMT_INSTALL OFF)
add_subdirectory(External/fmt/)
if (USE_FEXCONFIG_TOOLKIT STREQUAL "imgui")
add_subdirectory(External/imgui/)
include_directories(External/imgui/)
find_package(fmt QUIET)
if (NOT fmt_FOUND)
# Disable fmt install
set(FMT_INSTALL OFF)
add_subdirectory(External/fmt/)
endif()
add_subdirectory(External/tiny-json/)
@@ -405,10 +407,13 @@ configure_file(
${CMAKE_CURRENT_SOURCE_DIR}/include/Config.h.in
${CMAKE_BINARY_DIR}/generated/ConfigDefines.h)
include(CTest)
if (BUILD_TESTS)
include(CTest)
enable_testing()
message(STATUS "Unit tests are enabled")
if (NOT BUILD_TESTING)
# CMake checks this variable before generating CTestTestfile.cmake
message(SEND_ERROR "Unit tests require BUILD_TESTING to be enabled")
endif()
set (TEST_JOB_COUNT "" CACHE STRING "Override number of parallel jobs to use while running tests")
if (TEST_JOB_COUNT)
@@ -423,7 +428,7 @@ add_subdirectory(FEXHeaderUtils/)
add_subdirectory(CodeEmitter/)
add_subdirectory(FEXCore/)
if (NOT MINGW_BUILD)
if (_M_ARM_64 AND NOT MINGW_BUILD)
# Binfmt_misc files must be installed prior to Source/ installs
add_subdirectory(Data/binfmts/)
endif()
@@ -469,6 +474,7 @@ if (BUILD_THUNKS)
"-DCMAKE_INSTALL_PREFIX=${CMAKE_INSTALL_PREFIX}"
"-DFEX_PROJECT_SOURCE_DIR=${FEX_PROJECT_SOURCE_DIR}"
"-DGENERATOR_EXE=$<TARGET_FILE:thunkgen>"
"-DX86_DEV_ROOTFS=${X86_DEV_ROOTFS}"
INSTALL_COMMAND ""
BUILD_ALWAYS ON
DEPENDS thunkgen
@@ -487,6 +493,7 @@ if (BUILD_THUNKS)
"-DCMAKE_INSTALL_PREFIX=${CMAKE_INSTALL_PREFIX}"
"-DFEX_PROJECT_SOURCE_DIR=${FEX_PROJECT_SOURCE_DIR}"
"-DGENERATOR_EXE=$<TARGET_FILE:thunkgen>"
"-DX86_DEV_ROOTFS=${X86_DEV_ROOTFS}"
INSTALL_COMMAND ""
BUILD_ALWAYS ON
DEPENDS thunkgen
+167 -191
View File
@@ -11,6 +11,14 @@
* FEX-Emu ALU operations usually have a 32-bit or 64-bit operating size encoded in the IR operation,
* This allows FEX to use a single helper function which decodes to both handlers.
*/
#pragma once
#ifndef INCLUDED_BY_EMITTER
#include <CodeEmitter/Emitter.h>
namespace ARMEmitter {
struct EmitterOps : Emitter {
#endif
private:
static bool IsADRRange(int64_t Imm) {
return Imm >= -1048576 && Imm <= 1048575;
@@ -28,26 +36,23 @@ public:
DataProcessing_PCRel_Imm(Op, rd, Imm);
}
void adr(ARMEmitter::Register rd, BackwardLabel const* Label) {
void adr(ARMEmitter::Register rd, const BackwardLabel* Label) {
int32_t Imm = static_cast<int32_t>(Label->Location - GetCursorAddress<uint8_t*>());
LOGMAN_THROW_A_FMT(IsADRRange(Imm), "Unscaled offset too large");
constexpr uint32_t Op = 0b0001'0000 << 24;
DataProcessing_PCRel_Imm(Op, rd, Imm);
}
template<typename LabelType>
requires (std::is_same_v<LabelType, ForwardLabel> || std::is_same_v<LabelType, SingleUseForwardLabel>)
void adr(ARMEmitter::Register rd, LabelType *Label) {
AddLocationToLabel(Label, SingleUseForwardLabel{ .Location = GetCursorAddress<uint8_t*>(), .Type = SingleUseForwardLabel::InstType::ADR });
void adr(ARMEmitter::Register rd, ForwardLabel* Label) {
AddLocationToLabel(Label, ForwardLabel::Reference {.Location = GetCursorAddress<uint8_t*>(), .Type = ForwardLabel::InstType::ADR});
constexpr uint32_t Op = 0b0001'0000 << 24;
DataProcessing_PCRel_Imm(Op, rd, 0);
}
void adr(ARMEmitter::Register rd, BiDirectionalLabel *Label) {
void adr(ARMEmitter::Register rd, BiDirectionalLabel* Label) {
if (Label->Backward.Location) {
adr(rd, &Label->Backward);
}
else {
} else {
adr(rd, &Label->Forward);
}
}
@@ -57,39 +62,34 @@ public:
DataProcessing_PCRel_Imm(Op, rd, Imm);
}
void adrp(ARMEmitter::Register rd, BackwardLabel const* Label) {
void adrp(ARMEmitter::Register rd, const BackwardLabel* Label) {
int64_t Imm = reinterpret_cast<int64_t>(Label->Location) - (GetCursorAddress<int64_t>() & ~0xFFFLL);
LOGMAN_THROW_A_FMT(IsADRPRange(Imm) && IsADRPAligned(Imm), "Unscaled offset too large");
constexpr uint32_t Op = 0b1001'0000 << 24;
DataProcessing_PCRel_Imm(Op, rd, Imm);
}
template<typename LabelType>
requires (std::is_same_v<LabelType, ForwardLabel> || std::is_same_v<LabelType, SingleUseForwardLabel>)
void adrp(ARMEmitter::Register rd, LabelType *Label) {
AddLocationToLabel(Label, SingleUseForwardLabel{ .Location = GetCursorAddress<uint8_t*>(), .Type = SingleUseForwardLabel::InstType::ADRP });
void adrp(ARMEmitter::Register rd, ForwardLabel* Label) {
AddLocationToLabel(Label, ForwardLabel::Reference {.Location = GetCursorAddress<uint8_t*>(), .Type = ForwardLabel::InstType::ADRP});
constexpr uint32_t Op = 0b1001'0000 << 24;
DataProcessing_PCRel_Imm(Op, rd, 0);
}
void adrp(ARMEmitter::Register rd, BiDirectionalLabel *Label) {
void adrp(ARMEmitter::Register rd, BiDirectionalLabel* Label) {
if (Label->Backward.Location) {
adrp(rd, &Label->Backward);
}
else {
} else {
adrp(rd, &Label->Forward);
}
}
void LongAddressGen(ARMEmitter::Register rd, BackwardLabel const* Label) {
void LongAddressGen(ARMEmitter::Register rd, const BackwardLabel* Label) {
int64_t Imm = reinterpret_cast<int64_t>(Label->Location) - (GetCursorAddress<int64_t>());
if (IsADRRange(Imm)) {
// If the range is in ADR range then we can just use ADR.
adr(rd, Label);
}
else if (IsADRPRange(Imm)) {
int64_t ADRPImm = (reinterpret_cast<int64_t>(Label->Location) & ~0xFFFLL)
- (GetCursorAddress<int64_t>() & ~0xFFFLL);
} else if (IsADRPRange(Imm)) {
int64_t ADRPImm = (reinterpret_cast<int64_t>(Label->Location) & ~0xFFFLL) - (GetCursorAddress<int64_t>() & ~0xFFFLL);
// If the range is in the ADRP range then we can use ADRP.
bool NeedsOffset = !IsADRPAligned(reinterpret_cast<uint64_t>(Label->Location));
@@ -102,24 +102,22 @@ public:
// Now even an add
add(ARMEmitter::Size::i64Bit, rd, rd, AlignedOffset);
}
}
else {
} else {
LOGMAN_MSG_A_FMT("Unscaled offset too large");
FEX_UNREACHABLE;
}
}
void LongAddressGen(ARMEmitter::Register rd, ForwardLabel* Label) {
Label->Insts.emplace_back(SingleUseForwardLabel{ .Location = GetCursorAddress<uint8_t*>(), .Type = SingleUseForwardLabel::InstType::LONG_ADDRESS_GEN });
AddLocationToLabel(Label, ForwardLabel::Reference {.Location = GetCursorAddress<uint8_t*>(), .Type = ForwardLabel::InstType::LONG_ADDRESS_GEN});
// Emit a register index and a nop. These will be backpatched.
dc32(rd.Idx());
nop();
}
void LongAddressGen(ARMEmitter::Register rd, BiDirectionalLabel *Label) {
void LongAddressGen(ARMEmitter::Register rd, BiDirectionalLabel* Label) {
if (Label->Backward.Location) {
LongAddressGen(rd, &Label->Backward);
}
else {
} else {
LongAddressGen(rd, &Label->Forward);
}
}
@@ -176,11 +174,7 @@ public:
// Logical immediate
void and_(ARMEmitter::Size s, ARMEmitter::Register rd, ARMEmitter::Register rn, uint64_t Imm) {
uint32_t n, immr, imms;
[[maybe_unused]] const auto IsImm = IsImmLogical(Imm,
RegSizeInBits(s),
&n,
&imms,
&immr);
[[maybe_unused]] const auto IsImm = IsImmLogical(Imm, RegSizeInBits(s), &n, &imms, &immr);
LOGMAN_THROW_A_FMT(IsImm, "Couldn't encode immediate to logical op");
and_(s, rd, rn, n, immr, imms);
}
@@ -191,11 +185,7 @@ public:
void ands(ARMEmitter::Size s, ARMEmitter::Register rd, ARMEmitter::Register rn, uint64_t Imm) {
uint32_t n, immr, imms;
[[maybe_unused]] const auto IsImm = IsImmLogical(Imm,
RegSizeInBits(s),
&n,
&imms,
&immr);
[[maybe_unused]] const auto IsImm = IsImmLogical(Imm, RegSizeInBits(s), &n, &imms, &immr);
LOGMAN_THROW_A_FMT(IsImm, "Couldn't encode immediate to logical op");
ands(s, rd, rn, n, immr, imms);
}
@@ -206,22 +196,14 @@ public:
void orr(ARMEmitter::Size s, ARMEmitter::Register rd, ARMEmitter::Register rn, uint64_t Imm) {
uint32_t n, immr, imms;
[[maybe_unused]] const auto IsImm = IsImmLogical(Imm,
RegSizeInBits(s),
&n,
&imms,
&immr);
[[maybe_unused]] const auto IsImm = IsImmLogical(Imm, RegSizeInBits(s), &n, &imms, &immr);
LOGMAN_THROW_A_FMT(IsImm, "Couldn't encode immediate to logical op");
orr(s, rd, rn, n, immr, imms);
}
void eor(ARMEmitter::Size s, ARMEmitter::Register rd, ARMEmitter::Register rn, uint64_t Imm) {
uint32_t n, immr, imms;
[[maybe_unused]] const auto IsImm = IsImmLogical(Imm,
RegSizeInBits(s),
&n,
&imms,
&immr);
[[maybe_unused]] const auto IsImm = IsImmLogical(Imm, RegSizeInBits(s), &n, &imms, &immr);
LOGMAN_THROW_A_FMT(IsImm, "Couldn't encode immediate to logical op");
eor(s, rd, rn, n, immr, imms);
}
@@ -301,8 +283,9 @@ public:
xbfiz_helper(true, s, rd, rn, lsb, width);
}
void asr(ARMEmitter::Size s, ARMEmitter::Register rd, ARMEmitter::Register rn, uint32_t shift) {
LOGMAN_THROW_A_FMT(shift <= RegSizeInBits(s), "Tried to asr a region larger than the register");
sbfm(s, rd, rn, shift, RegSizeInBits(s) - 1);
const auto RegSize_m1 = RegSizeInBits(s) - 1;
shift &= RegSize_m1;
sbfm(s, rd, rn, shift, RegSize_m1);
}
void uxtb(ARMEmitter::Size s, ARMEmitter::Register rd, ARMEmitter::Register rn) {
@@ -325,14 +308,14 @@ public:
}
void lsl(ARMEmitter::Size s, ARMEmitter::Register rd, ARMEmitter::Register rn, uint32_t shift) {
const auto RegSize = RegSizeInBits(s);
LOGMAN_THROW_A_FMT(shift < RegSize, "Tried to lsl a region larger than the register");
ubfm(s, rd, rn, (RegSize - shift) % RegSize, RegSize - shift - 1);
const auto RegSize_m1 = RegSizeInBits(s) - 1;
shift &= RegSize_m1;
ubfm(s, rd, rn, (RegSizeInBits(s) - shift) & RegSize_m1, RegSize_m1 - shift);
}
void lsr(ARMEmitter::Size s, ARMEmitter::Register rd, ARMEmitter::Register rn, uint32_t shift) {
const auto RegSize = RegSizeInBits(s);
LOGMAN_THROW_A_FMT(shift < RegSize, "Tried to lsr a region larger than the register");
ubfm(s, rd, rn, shift, RegSize - 1);
const auto RegSize_m1 = RegSizeInBits(s) - 1;
shift &= RegSize_m1;
ubfm(s, rd, rn, shift, RegSize_m1);
}
void ubfx(ARMEmitter::Size s, ARMEmitter::Register rd, ARMEmitter::Register rn, uint32_t lsb, uint32_t width) {
LOGMAN_THROW_A_FMT(width > 0, "ubfx needs width > 0");
@@ -354,8 +337,8 @@ public:
const auto lsb_p_width = lsb + width;
LOGMAN_THROW_A_FMT(width >= 1, "bfxil needs width >= 1");
LOGMAN_THROW_A_FMT(lsb_p_width <= reg_size_bits, "bfxil lsb + width ({}) must be <= {}. lsb={}, width={}",
lsb_p_width, reg_size_bits, lsb, width);
LOGMAN_THROW_A_FMT(lsb_p_width <= reg_size_bits, "bfxil lsb + width ({}) must be <= {}. lsb={}, width={}", lsb_p_width, reg_size_bits,
lsb, width);
bfm(s, rd, rn, lsb, lsb_p_width - 1);
}
@@ -368,193 +351,148 @@ public:
}
void ror(ARMEmitter::Size s, ARMEmitter::Register rd, ARMEmitter::Register rn, uint32_t Imm) {
Imm &= RegSizeInBits(s) - 1;
extr(s, rd, rn, rn, Imm);
}
// Data processing - 2 source
void udiv(ARMEmitter::Size s, ARMEmitter::Register rd, ARMEmitter::Register rn, ARMEmitter::Register rm) {
constexpr uint32_t Op = (0b001'1010'110U << 21) |
(0b0000'10U << 10);
constexpr uint32_t Op = (0b001'1010'110U << 21) | (0b0000'10U << 10);
DataProcessing_2Source(Op, s, rd, rn, rm);
}
void sdiv(ARMEmitter::Size s, ARMEmitter::Register rd, ARMEmitter::Register rn, ARMEmitter::Register rm) {
constexpr uint32_t Op = (0b001'1010'110U << 21) |
(0b0000'11U << 10);
constexpr uint32_t Op = (0b001'1010'110U << 21) | (0b0000'11U << 10);
DataProcessing_2Source(Op, s, rd, rn, rm);
}
void lslv(ARMEmitter::Size s, ARMEmitter::Register rd, ARMEmitter::Register rn, ARMEmitter::Register rm) {
constexpr uint32_t Op = (0b001'1010'110U << 21) |
(0b0010'00U << 10);
constexpr uint32_t Op = (0b001'1010'110U << 21) | (0b0010'00U << 10);
DataProcessing_2Source(Op, s, rd, rn, rm);
}
void lsrv(ARMEmitter::Size s, ARMEmitter::Register rd, ARMEmitter::Register rn, ARMEmitter::Register rm) {
constexpr uint32_t Op = (0b001'1010'110U << 21) |
(0b0010'01U << 10);
constexpr uint32_t Op = (0b001'1010'110U << 21) | (0b0010'01U << 10);
DataProcessing_2Source(Op, s, rd, rn, rm);
}
void asrv(ARMEmitter::Size s, ARMEmitter::Register rd, ARMEmitter::Register rn, ARMEmitter::Register rm) {
constexpr uint32_t Op = (0b001'1010'110U << 21) |
(0b0010'10U << 10);
constexpr uint32_t Op = (0b001'1010'110U << 21) | (0b0010'10U << 10);
DataProcessing_2Source(Op, s, rd, rn, rm);
}
void rorv(ARMEmitter::Size s, ARMEmitter::Register rd, ARMEmitter::Register rn, ARMEmitter::Register rm) {
constexpr uint32_t Op = (0b001'1010'110U << 21) |
(0b0010'11U << 10);
constexpr uint32_t Op = (0b001'1010'110U << 21) | (0b0010'11U << 10);
DataProcessing_2Source(Op, s, rd, rn, rm);
}
void crc32b(ARMEmitter::WRegister rd, ARMEmitter::WRegister rn, ARMEmitter::WRegister rm) {
constexpr uint32_t Op = (0b001'1010'110U << 21) |
(0b0100'00U << 10);
constexpr uint32_t Op = (0b001'1010'110U << 21) | (0b0100'00U << 10);
DataProcessing_2Source(Op, ARMEmitter::Size::i32Bit, rd, rn, rm);
}
void crc32h(ARMEmitter::WRegister rd, ARMEmitter::WRegister rn, ARMEmitter::WRegister rm) {
constexpr uint32_t Op = (0b001'1010'110U << 21) |
(0b0100'01U << 10);
constexpr uint32_t Op = (0b001'1010'110U << 21) | (0b0100'01U << 10);
DataProcessing_2Source(Op, ARMEmitter::Size::i32Bit, rd, rn, rm);
}
void crc32w(ARMEmitter::WRegister rd, ARMEmitter::WRegister rn, ARMEmitter::WRegister rm) {
constexpr uint32_t Op = (0b001'1010'110U << 21) |
(0b0100'10U << 10);
constexpr uint32_t Op = (0b001'1010'110U << 21) | (0b0100'10U << 10);
DataProcessing_2Source(Op, ARMEmitter::Size::i32Bit, rd, rn, rm);
}
void crc32cb(ARMEmitter::WRegister rd, ARMEmitter::WRegister rn, ARMEmitter::WRegister rm) {
constexpr uint32_t Op = (0b001'1010'110U << 21) |
(0b0101'00U << 10);
constexpr uint32_t Op = (0b001'1010'110U << 21) | (0b0101'00U << 10);
DataProcessing_2Source(Op, ARMEmitter::Size::i32Bit, rd, rn, rm);
}
void crc32ch(ARMEmitter::WRegister rd, ARMEmitter::WRegister rn, ARMEmitter::WRegister rm) {
constexpr uint32_t Op = (0b001'1010'110U << 21) |
(0b0101'01U << 10);
constexpr uint32_t Op = (0b001'1010'110U << 21) | (0b0101'01U << 10);
DataProcessing_2Source(Op, ARMEmitter::Size::i32Bit, rd, rn, rm);
}
void crc32cw(ARMEmitter::WRegister rd, ARMEmitter::WRegister rn, ARMEmitter::WRegister rm) {
constexpr uint32_t Op = (0b001'1010'110U << 21) |
(0b0101'10U << 10);
constexpr uint32_t Op = (0b001'1010'110U << 21) | (0b0101'10U << 10);
DataProcessing_2Source(Op, ARMEmitter::Size::i32Bit, rd, rn, rm);
}
void smax(ARMEmitter::Size s, ARMEmitter::Register rd, ARMEmitter::Register rn, ARMEmitter::Register rm) {
constexpr uint32_t Op = (0b001'1010'110U << 21) |
(0b0110'00U << 10);
constexpr uint32_t Op = (0b001'1010'110U << 21) | (0b0110'00U << 10);
DataProcessing_2Source(Op, s, rd, rn, rm);
}
void umax(ARMEmitter::Size s, ARMEmitter::Register rd, ARMEmitter::Register rn, ARMEmitter::Register rm) {
constexpr uint32_t Op = (0b001'1010'110U << 21) |
(0b0110'01U << 10);
constexpr uint32_t Op = (0b001'1010'110U << 21) | (0b0110'01U << 10);
DataProcessing_2Source(Op, s, rd, rn, rm);
}
void smin(ARMEmitter::Size s, ARMEmitter::Register rd, ARMEmitter::Register rn, ARMEmitter::Register rm) {
constexpr uint32_t Op = (0b001'1010'110U << 21) |
(0b0110'10U << 10);
constexpr uint32_t Op = (0b001'1010'110U << 21) | (0b0110'10U << 10);
DataProcessing_2Source(Op, s, rd, rn, rm);
}
void umin(ARMEmitter::Size s, ARMEmitter::Register rd, ARMEmitter::Register rn, ARMEmitter::Register rm) {
constexpr uint32_t Op = (0b001'1010'110U << 21) |
(0b0110'11U << 10);
constexpr uint32_t Op = (0b001'1010'110U << 21) | (0b0110'11U << 10);
DataProcessing_2Source(Op, s, rd, rn, rm);
}
void subp(ARMEmitter::XRegister rd, ARMEmitter::XRegister rn, ARMEmitter::XRegister rm) {
constexpr uint32_t Op = (0b001'1010'110U << 21) |
(0b0000'00U << 10);
constexpr uint32_t Op = (0b001'1010'110U << 21) | (0b0000'00U << 10);
DataProcessing_2Source(Op, ARMEmitter::Size::i64Bit, rd, rn, rm);
}
void irg(ARMEmitter::XRegister rd, ARMEmitter::XRegister rn, ARMEmitter::XRegister rm) {
constexpr uint32_t Op = (0b001'1010'110U << 21) |
(0b0001'00U << 10);
constexpr uint32_t Op = (0b001'1010'110U << 21) | (0b0001'00U << 10);
DataProcessing_2Source(Op, ARMEmitter::Size::i64Bit, rd, rn, rm);
}
void gmi(ARMEmitter::XRegister rd, ARMEmitter::XRegister rn, ARMEmitter::XRegister rm) {
constexpr uint32_t Op = (0b001'1010'110U << 21) |
(0b0001'01U << 10);
constexpr uint32_t Op = (0b001'1010'110U << 21) | (0b0001'01U << 10);
DataProcessing_2Source(Op, ARMEmitter::Size::i64Bit, rd, rn, rm);
}
void pacga(ARMEmitter::XRegister rd, ARMEmitter::XRegister rn, ARMEmitter::XRegister rm) {
constexpr uint32_t Op = (0b001'1010'110U << 21) |
(0b0011'00U << 10);
constexpr uint32_t Op = (0b001'1010'110U << 21) | (0b0011'00U << 10);
DataProcessing_2Source(Op, ARMEmitter::Size::i64Bit, rd, rn, rm);
}
void crc32x(ARMEmitter::XRegister rd, ARMEmitter::XRegister rn, ARMEmitter::XRegister rm) {
constexpr uint32_t Op = (0b001'1010'110U << 21) |
(0b0100'11U << 10);
constexpr uint32_t Op = (0b001'1010'110U << 21) | (0b0100'11U << 10);
DataProcessing_2Source(Op, ARMEmitter::Size::i64Bit, rd, rn, rm);
}
void crc32cx(ARMEmitter::XRegister rd, ARMEmitter::XRegister rn, ARMEmitter::XRegister rm) {
constexpr uint32_t Op = (0b001'1010'110U << 21) |
(0b0101'11U << 10);
constexpr uint32_t Op = (0b001'1010'110U << 21) | (0b0101'11U << 10);
DataProcessing_2Source(Op, ARMEmitter::Size::i64Bit, rd, rn, rm);
}
void subps(ARMEmitter::XRegister rd, ARMEmitter::XRegister rn, ARMEmitter::XRegister rm) {
constexpr uint32_t Op = (0b011'1010'110U << 21) |
(0b0000'00U << 10);
constexpr uint32_t Op = (0b011'1010'110U << 21) | (0b0000'00U << 10);
DataProcessing_2Source(Op, ARMEmitter::Size::i64Bit, rd, rn, rm);
}
// Data processing - 1 source
void rbit(ARMEmitter::Size s, ARMEmitter::Register rd, ARMEmitter::Register rn) {
constexpr uint32_t Op = (0b101'1010'110U << 21) |
(0b0'0000U << 16) |
(0b0000'00U << 10);
constexpr uint32_t Op = (0b101'1010'110U << 21) | (0b0'0000U << 16) | (0b0000'00U << 10);
DataProcessing_1Source(Op, s, rd, rn);
}
void rev16(ARMEmitter::Size s, ARMEmitter::Register rd, ARMEmitter::Register rn) {
constexpr uint32_t Op = (0b101'1010'110U << 21) |
(0b0'0000U << 16) |
(0b0000'01U << 10);
constexpr uint32_t Op = (0b101'1010'110U << 21) | (0b0'0000U << 16) | (0b0000'01U << 10);
DataProcessing_1Source(Op, s, rd, rn);
}
void rev(ARMEmitter::WRegister rd, ARMEmitter::WRegister rn) {
constexpr uint32_t Op = (0b101'1010'110U << 21) |
(0b0'0000U << 16) |
(0b0000'10U << 10);
constexpr uint32_t Op = (0b101'1010'110U << 21) | (0b0'0000U << 16) | (0b0000'10U << 10);
DataProcessing_1Source(Op, ARMEmitter::Size::i32Bit, rd, rn);
}
void rev32(ARMEmitter::XRegister rd, ARMEmitter::XRegister rn) {
constexpr uint32_t Op = (0b101'1010'110U << 21) |
(0b0'0000U << 16) |
(0b0000'10U << 10);
constexpr uint32_t Op = (0b101'1010'110U << 21) | (0b0'0000U << 16) | (0b0000'10U << 10);
DataProcessing_1Source(Op, ARMEmitter::Size::i64Bit, rd, rn);
}
void clz(ARMEmitter::Size s, ARMEmitter::Register rd, ARMEmitter::Register rn) {
constexpr uint32_t Op = (0b101'1010'110U << 21) |
(0b0'0000U << 16) |
(0b0001'00U << 10);
constexpr uint32_t Op = (0b101'1010'110U << 21) | (0b0'0000U << 16) | (0b0001'00U << 10);
DataProcessing_1Source(Op, s, rd, rn);
}
void cls(ARMEmitter::Size s, ARMEmitter::Register rd, ARMEmitter::Register rn) {
constexpr uint32_t Op = (0b101'1010'110U << 21) |
(0b0'0000U << 16) |
(0b0001'01U << 10);
constexpr uint32_t Op = (0b101'1010'110U << 21) | (0b0'0000U << 16) | (0b0001'01U << 10);
DataProcessing_1Source(Op, s, rd, rn);
}
void rev(ARMEmitter::XRegister rd, ARMEmitter::XRegister rn) {
constexpr uint32_t Op = (0b101'1010'110U << 21) |
(0b0'0000U << 16) |
(0b0000'11U << 10);
constexpr uint32_t Op = (0b101'1010'110U << 21) | (0b0'0000U << 16) | (0b0000'11U << 10);
DataProcessing_1Source(Op, ARMEmitter::Size::i64Bit, rd, rn);
}
void rev(ARMEmitter::Size s, ARMEmitter::Register rd, ARMEmitter::Register rn) {
uint32_t Op = (0b101'1010'110U << 21) |
(0b0'0000U << 16) |
(0b0000'10U << 10) |
(s == ARMEmitter::Size::i64Bit ? (1U << 10) : 0);
uint32_t Op = (0b101'1010'110U << 21) | (0b0'0000U << 16) | (0b0000'10U << 10) | (s == ARMEmitter::Size::i64Bit ? (1U << 10) : 0);
DataProcessing_1Source(Op, s, rd, rn);
}
void ctz(ARMEmitter::Size s, ARMEmitter::Register rd, ARMEmitter::Register rn) {
constexpr uint32_t Op = (0b101'1010'110U << 21) |
(0b0'0000U << 16) |
(0b0001'10U << 10);
constexpr uint32_t Op = (0b101'1010'110U << 21) | (0b0'0000U << 16) | (0b0001'10U << 10);
DataProcessing_1Source(Op, s, rd, rn);
}
void cnt(ARMEmitter::Size s, ARMEmitter::Register rd, ARMEmitter::Register rn) {
constexpr uint32_t Op = (0b101'1010'110U << 21) |
(0b0'0000U << 16) |
(0b0001'11U << 10);
constexpr uint32_t Op = (0b101'1010'110U << 21) | (0b0'0000U << 16) | (0b0001'11U << 10);
DataProcessing_1Source(Op, s, rd, rn);
}
void abs(ARMEmitter::Size s, ARMEmitter::Register rd, ARMEmitter::Register rn) {
constexpr uint32_t Op = (0b101'1010'110U << 21) |
(0b0'0000U << 16) |
(0b0010'00U << 10);
constexpr uint32_t Op = (0b101'1010'110U << 21) | (0b0'0000U << 16) | (0b0010'00U << 10);
DataProcessing_1Source(Op, s, rd, rn);
}
@@ -571,27 +509,33 @@ public:
orr(ARMEmitter::Size::i32Bit, rd.R(), ARMEmitter::Reg::zr, rn.R(), ARMEmitter::ShiftType::LSL, 0);
}
void mvn(ARMEmitter::Size s, ARMEmitter::Register rd, ARMEmitter::Register rn, ARMEmitter::ShiftType Shift = ARMEmitter::ShiftType::LSL, uint32_t amt = 0) {
void mvn(ARMEmitter::Size s, ARMEmitter::Register rd, ARMEmitter::Register rn, ARMEmitter::ShiftType Shift = ARMEmitter::ShiftType::LSL,
uint32_t amt = 0) {
orn(s, rd, ARMEmitter::Reg::zr, rn, Shift, amt);
}
void and_(ARMEmitter::Size s, ARMEmitter::Register rd, ARMEmitter::Register rn, ARMEmitter::Register rm, ARMEmitter::ShiftType Shift = ARMEmitter::ShiftType::LSL, uint32_t amt = 0) {
void and_(ARMEmitter::Size s, ARMEmitter::Register rd, ARMEmitter::Register rn, ARMEmitter::Register rm,
ARMEmitter::ShiftType Shift = ARMEmitter::ShiftType::LSL, uint32_t amt = 0) {
constexpr uint32_t Op = 0b000'1010'000U << 21;
DataProcessing_Shifted_Reg(Op, s, rd, rn, rm, Shift, amt);
}
void ands(ARMEmitter::Size s, ARMEmitter::Register rd, ARMEmitter::Register rn, ARMEmitter::Register rm, ARMEmitter::ShiftType Shift = ARMEmitter::ShiftType::LSL, uint32_t amt = 0) {
void ands(ARMEmitter::Size s, ARMEmitter::Register rd, ARMEmitter::Register rn, ARMEmitter::Register rm,
ARMEmitter::ShiftType Shift = ARMEmitter::ShiftType::LSL, uint32_t amt = 0) {
constexpr uint32_t Op = 0b110'1010'000U << 21;
DataProcessing_Shifted_Reg(Op, s, rd, rn, rm, Shift, amt);
}
void bic(ARMEmitter::Size s, ARMEmitter::Register rd, ARMEmitter::Register rn, ARMEmitter::Register rm, ARMEmitter::ShiftType Shift = ARMEmitter::ShiftType::LSL, uint32_t amt = 0) {
void bic(ARMEmitter::Size s, ARMEmitter::Register rd, ARMEmitter::Register rn, ARMEmitter::Register rm,
ARMEmitter::ShiftType Shift = ARMEmitter::ShiftType::LSL, uint32_t amt = 0) {
constexpr uint32_t Op = 0b000'1010'001U << 21;
DataProcessing_Shifted_Reg(Op, s, rd, rn, rm, Shift, amt);
}
void bics(ARMEmitter::Size s, ARMEmitter::Register rd, ARMEmitter::Register rn, ARMEmitter::Register rm, ARMEmitter::ShiftType Shift = ARMEmitter::ShiftType::LSL, uint32_t amt = 0) {
void bics(ARMEmitter::Size s, ARMEmitter::Register rd, ARMEmitter::Register rn, ARMEmitter::Register rm,
ARMEmitter::ShiftType Shift = ARMEmitter::ShiftType::LSL, uint32_t amt = 0) {
constexpr uint32_t Op = 0b110'1010'001U << 21;
DataProcessing_Shifted_Reg(Op, s, rd, rn, rm, Shift, amt);
}
void orr(ARMEmitter::Size s, ARMEmitter::Register rd, ARMEmitter::Register rn, ARMEmitter::Register rm, ARMEmitter::ShiftType Shift = ARMEmitter::ShiftType::LSL, uint32_t amt = 0) {
void orr(ARMEmitter::Size s, ARMEmitter::Register rd, ARMEmitter::Register rn, ARMEmitter::Register rm,
ARMEmitter::ShiftType Shift = ARMEmitter::ShiftType::LSL, uint32_t amt = 0) {
constexpr uint32_t Op = 0b010'1010'000U << 21;
DataProcessing_Shifted_Reg(Op, s, rd, rn, rm, Shift, amt);
}
@@ -599,30 +543,36 @@ public:
ands(s, Reg::zr, rn, rm, shift, amt);
}
void orn(ARMEmitter::Size s, ARMEmitter::Register rd, ARMEmitter::Register rn, ARMEmitter::Register rm, ARMEmitter::ShiftType Shift = ARMEmitter::ShiftType::LSL, uint32_t amt = 0) {
void orn(ARMEmitter::Size s, ARMEmitter::Register rd, ARMEmitter::Register rn, ARMEmitter::Register rm,
ARMEmitter::ShiftType Shift = ARMEmitter::ShiftType::LSL, uint32_t amt = 0) {
constexpr uint32_t Op = 0b010'1010'001U << 21;
DataProcessing_Shifted_Reg(Op, s, rd, rn, rm, Shift, amt);
}
void eor(ARMEmitter::Size s, ARMEmitter::Register rd, ARMEmitter::Register rn, ARMEmitter::Register rm, ARMEmitter::ShiftType Shift = ARMEmitter::ShiftType::LSL, uint32_t amt = 0) {
void eor(ARMEmitter::Size s, ARMEmitter::Register rd, ARMEmitter::Register rn, ARMEmitter::Register rm,
ARMEmitter::ShiftType Shift = ARMEmitter::ShiftType::LSL, uint32_t amt = 0) {
constexpr uint32_t Op = 0b100'1010'000U << 21;
DataProcessing_Shifted_Reg(Op, s, rd, rn, rm, Shift, amt);
}
void eon(ARMEmitter::Size s, ARMEmitter::Register rd, ARMEmitter::Register rn, ARMEmitter::Register rm, ARMEmitter::ShiftType Shift = ARMEmitter::ShiftType::LSL, uint32_t amt = 0) {
void eon(ARMEmitter::Size s, ARMEmitter::Register rd, ARMEmitter::Register rn, ARMEmitter::Register rm,
ARMEmitter::ShiftType Shift = ARMEmitter::ShiftType::LSL, uint32_t amt = 0) {
constexpr uint32_t Op = 0b100'1010'001U << 21;
DataProcessing_Shifted_Reg(Op, s, rd, rn, rm, Shift, amt);
}
// AddSub - shifted register
void add(ARMEmitter::XRegister rd, ARMEmitter::XRegister rn, ARMEmitter::XRegister rm, ARMEmitter::ShiftType Shift = ARMEmitter::ShiftType::LSL, uint32_t amt = 0) {
void add(ARMEmitter::XRegister rd, ARMEmitter::XRegister rn, ARMEmitter::XRegister rm,
ARMEmitter::ShiftType Shift = ARMEmitter::ShiftType::LSL, uint32_t amt = 0) {
add(ARMEmitter::Size::i64Bit, rd.R(), rn.R(), rm.R(), Shift, amt);
}
void adds(ARMEmitter::XRegister rd, ARMEmitter::XRegister rn, ARMEmitter::XRegister rm, ARMEmitter::ShiftType Shift = ARMEmitter::ShiftType::LSL, uint32_t amt = 0) {
void adds(ARMEmitter::XRegister rd, ARMEmitter::XRegister rn, ARMEmitter::XRegister rm,
ARMEmitter::ShiftType Shift = ARMEmitter::ShiftType::LSL, uint32_t amt = 0) {
adds(ARMEmitter::Size::i64Bit, rd.R(), rn.R(), rm.R(), Shift, amt);
}
void cmn(ARMEmitter::XRegister rn, ARMEmitter::XRegister rm, ARMEmitter::ShiftType Shift = ARMEmitter::ShiftType::LSL, uint32_t amt = 0) {
adds(ARMEmitter::Size::i64Bit, ARMEmitter::XReg::zr, rn.R(), rm.R(), Shift, amt);
}
void sub(ARMEmitter::XRegister rd, ARMEmitter::XRegister rn, ARMEmitter::XRegister rm, ARMEmitter::ShiftType Shift = ARMEmitter::ShiftType::LSL, uint32_t amt = 0) {
void sub(ARMEmitter::XRegister rd, ARMEmitter::XRegister rn, ARMEmitter::XRegister rm,
ARMEmitter::ShiftType Shift = ARMEmitter::ShiftType::LSL, uint32_t amt = 0) {
sub(ARMEmitter::Size::i64Bit, rd.R(), rn.R(), rm.R(), Shift, amt);
}
void neg(ARMEmitter::XRegister rd, ARMEmitter::XRegister rm, ARMEmitter::ShiftType Shift = ARMEmitter::ShiftType::LSL, uint32_t amt = 0) {
@@ -631,23 +581,27 @@ public:
void cmp(ARMEmitter::XRegister rn, ARMEmitter::XRegister rm, ARMEmitter::ShiftType Shift = ARMEmitter::ShiftType::LSL, uint32_t amt = 0) {
subs(ARMEmitter::Size::i64Bit, ARMEmitter::Reg::rsp, rn.R(), rm.R(), Shift, amt);
}
void subs(ARMEmitter::XRegister rd, ARMEmitter::XRegister rn, ARMEmitter::XRegister rm, ARMEmitter::ShiftType Shift = ARMEmitter::ShiftType::LSL, uint32_t amt = 0) {
void subs(ARMEmitter::XRegister rd, ARMEmitter::XRegister rn, ARMEmitter::XRegister rm,
ARMEmitter::ShiftType Shift = ARMEmitter::ShiftType::LSL, uint32_t amt = 0) {
subs(ARMEmitter::Size::i64Bit, rd.R(), rn.R(), rm.R(), Shift, amt);
}
void negs(ARMEmitter::XRegister rd, ARMEmitter::XRegister rm, ARMEmitter::ShiftType Shift = ARMEmitter::ShiftType::LSL, uint32_t amt = 0) {
subs(rd, ARMEmitter::XReg::zr, rm, Shift, amt);
}
void add(ARMEmitter::WRegister rd, ARMEmitter::WRegister rn, ARMEmitter::WRegister rm, ARMEmitter::ShiftType Shift = ARMEmitter::ShiftType::LSL, uint32_t amt = 0) {
void add(ARMEmitter::WRegister rd, ARMEmitter::WRegister rn, ARMEmitter::WRegister rm,
ARMEmitter::ShiftType Shift = ARMEmitter::ShiftType::LSL, uint32_t amt = 0) {
add(ARMEmitter::Size::i32Bit, rd.R(), rn.R(), rm.R(), Shift, amt);
}
void adds(ARMEmitter::WRegister rd, ARMEmitter::WRegister rn, ARMEmitter::WRegister rm, ARMEmitter::ShiftType Shift = ARMEmitter::ShiftType::LSL, uint32_t amt = 0) {
void adds(ARMEmitter::WRegister rd, ARMEmitter::WRegister rn, ARMEmitter::WRegister rm,
ARMEmitter::ShiftType Shift = ARMEmitter::ShiftType::LSL, uint32_t amt = 0) {
adds(ARMEmitter::Size::i32Bit, rd.R(), rn.R(), rm.R(), Shift, amt);
}
void cmn(ARMEmitter::WRegister rn, ARMEmitter::WRegister rm, ARMEmitter::ShiftType Shift = ARMEmitter::ShiftType::LSL, uint32_t amt = 0) {
adds(ARMEmitter::Size::i32Bit, ARMEmitter::WReg::zr, rn.R(), rm.R(), Shift, amt);
}
void sub(ARMEmitter::WRegister rd, ARMEmitter::WRegister rn, ARMEmitter::WRegister rm, ARMEmitter::ShiftType Shift = ARMEmitter::ShiftType::LSL, uint32_t amt = 0) {
void sub(ARMEmitter::WRegister rd, ARMEmitter::WRegister rn, ARMEmitter::WRegister rm,
ARMEmitter::ShiftType Shift = ARMEmitter::ShiftType::LSL, uint32_t amt = 0) {
sub(ARMEmitter::Size::i32Bit, rd.R(), rn.R(), rm.R(), Shift, amt);
}
void neg(ARMEmitter::WRegister rd, ARMEmitter::WRegister rm, ARMEmitter::ShiftType Shift = ARMEmitter::ShiftType::LSL, uint32_t amt = 0) {
@@ -656,65 +610,78 @@ public:
void cmp(ARMEmitter::WRegister rn, ARMEmitter::WRegister rm, ARMEmitter::ShiftType Shift = ARMEmitter::ShiftType::LSL, uint32_t amt = 0) {
subs(ARMEmitter::Size::i32Bit, ARMEmitter::Reg::rsp, rn.R(), rm.R(), Shift, amt);
}
void subs(ARMEmitter::WRegister rd, ARMEmitter::WRegister rn, ARMEmitter::WRegister rm, ARMEmitter::ShiftType Shift = ARMEmitter::ShiftType::LSL, uint32_t amt = 0) {
void subs(ARMEmitter::WRegister rd, ARMEmitter::WRegister rn, ARMEmitter::WRegister rm,
ARMEmitter::ShiftType Shift = ARMEmitter::ShiftType::LSL, uint32_t amt = 0) {
subs(ARMEmitter::Size::i32Bit, rd.R(), rn.R(), rm.R(), Shift, amt);
}
void negs(ARMEmitter::WRegister rd, ARMEmitter::WRegister rm, ARMEmitter::ShiftType Shift = ARMEmitter::ShiftType::LSL, uint32_t amt = 0) {
subs(rd, ARMEmitter::WReg::zr, rm, Shift, amt);
}
void add(ARMEmitter::Size s, ARMEmitter::Register rd, ARMEmitter::Register rn, ARMEmitter::Register rm, ARMEmitter::ShiftType Shift = ARMEmitter::ShiftType::LSL, uint32_t amt = 0) {
LOGMAN_THROW_AA_FMT(Shift != ARMEmitter::ShiftType::ROR, "Doesn't support ROR");
void add(ARMEmitter::Size s, ARMEmitter::Register rd, ARMEmitter::Register rn, ARMEmitter::Register rm,
ARMEmitter::ShiftType Shift = ARMEmitter::ShiftType::LSL, uint32_t amt = 0) {
LOGMAN_THROW_A_FMT(Shift != ARMEmitter::ShiftType::ROR, "Doesn't support ROR");
constexpr uint32_t Op = 0b000'1011'000U << 21;
DataProcessing_Shifted_Reg(Op, s, rd, rn, rm, Shift, amt);
}
void adds(ARMEmitter::Size s, ARMEmitter::Register rd, ARMEmitter::Register rn, ARMEmitter::Register rm, ARMEmitter::ShiftType Shift = ARMEmitter::ShiftType::LSL, uint32_t amt = 0) {
LOGMAN_THROW_AA_FMT(Shift != ARMEmitter::ShiftType::ROR, "Doesn't support ROR");
void adds(ARMEmitter::Size s, ARMEmitter::Register rd, ARMEmitter::Register rn, ARMEmitter::Register rm,
ARMEmitter::ShiftType Shift = ARMEmitter::ShiftType::LSL, uint32_t amt = 0) {
LOGMAN_THROW_A_FMT(Shift != ARMEmitter::ShiftType::ROR, "Doesn't support ROR");
constexpr uint32_t Op = 0b010'1011'000U << 21;
DataProcessing_Shifted_Reg(Op, s, rd, rn, rm, Shift, amt);
}
void cmn(ARMEmitter::Size s, ARMEmitter::Register rn, ARMEmitter::Register rm, ARMEmitter::ShiftType Shift = ARMEmitter::ShiftType::LSL, uint32_t amt = 0) {
void cmn(ARMEmitter::Size s, ARMEmitter::Register rn, ARMEmitter::Register rm, ARMEmitter::ShiftType Shift = ARMEmitter::ShiftType::LSL,
uint32_t amt = 0) {
adds(s, ARMEmitter::Reg::zr, rn, rm, Shift, amt);
}
void sub(ARMEmitter::Size s, ARMEmitter::Register rd, ARMEmitter::Register rn, ARMEmitter::Register rm, ARMEmitter::ShiftType Shift = ARMEmitter::ShiftType::LSL, uint32_t amt = 0) {
LOGMAN_THROW_AA_FMT(Shift != ARMEmitter::ShiftType::ROR, "Doesn't support ROR");
void sub(ARMEmitter::Size s, ARMEmitter::Register rd, ARMEmitter::Register rn, ARMEmitter::Register rm,
ARMEmitter::ShiftType Shift = ARMEmitter::ShiftType::LSL, uint32_t amt = 0) {
LOGMAN_THROW_A_FMT(Shift != ARMEmitter::ShiftType::ROR, "Doesn't support ROR");
constexpr uint32_t Op = 0b100'1011'000U << 21;
DataProcessing_Shifted_Reg(Op, s, rd, rn, rm, Shift, amt);
}
void neg(ARMEmitter::Size s, ARMEmitter::Register rd, ARMEmitter::Register rm, ARMEmitter::ShiftType Shift = ARMEmitter::ShiftType::LSL, uint32_t amt = 0) {
void neg(ARMEmitter::Size s, ARMEmitter::Register rd, ARMEmitter::Register rm, ARMEmitter::ShiftType Shift = ARMEmitter::ShiftType::LSL,
uint32_t amt = 0) {
sub(s, rd, ARMEmitter::Reg::zr, rm, Shift, amt);
}
void cmp(ARMEmitter::Size s, ARMEmitter::Register rn, ARMEmitter::Register rm, ARMEmitter::ShiftType Shift = ARMEmitter::ShiftType::LSL, uint32_t amt = 0) {
void cmp(ARMEmitter::Size s, ARMEmitter::Register rn, ARMEmitter::Register rm, ARMEmitter::ShiftType Shift = ARMEmitter::ShiftType::LSL,
uint32_t amt = 0) {
subs(s, ARMEmitter::Reg::zr, rn, rm, Shift, amt);
}
void subs(ARMEmitter::Size s, ARMEmitter::Register rd, ARMEmitter::Register rn, ARMEmitter::Register rm, ARMEmitter::ShiftType Shift = ARMEmitter::ShiftType::LSL, uint32_t amt = 0) {
LOGMAN_THROW_AA_FMT(Shift != ARMEmitter::ShiftType::ROR, "Doesn't support ROR");
void subs(ARMEmitter::Size s, ARMEmitter::Register rd, ARMEmitter::Register rn, ARMEmitter::Register rm,
ARMEmitter::ShiftType Shift = ARMEmitter::ShiftType::LSL, uint32_t amt = 0) {
LOGMAN_THROW_A_FMT(Shift != ARMEmitter::ShiftType::ROR, "Doesn't support ROR");
constexpr uint32_t Op = 0b110'1011'000U << 21;
DataProcessing_Shifted_Reg(Op, s, rd, rn, rm, Shift, amt);
}
void negs(ARMEmitter::Size s, ARMEmitter::Register rd, ARMEmitter::Register rm, ARMEmitter::ShiftType Shift = ARMEmitter::ShiftType::LSL, uint32_t amt = 0) {
void negs(ARMEmitter::Size s, ARMEmitter::Register rd, ARMEmitter::Register rm, ARMEmitter::ShiftType Shift = ARMEmitter::ShiftType::LSL,
uint32_t amt = 0) {
subs(s, rd, ARMEmitter::Reg::zr, rm, Shift, amt);
}
// AddSub - extended register
void add(ARMEmitter::Size s, ARMEmitter::Register rd, ARMEmitter::Register rn, ARMEmitter::Register rm, ARMEmitter::ExtendedType Option, uint32_t Shift = 0) {
LOGMAN_THROW_AA_FMT(Shift <= 4, "Shift amount is too large");
void add(ARMEmitter::Size s, ARMEmitter::Register rd, ARMEmitter::Register rn, ARMEmitter::Register rm, ARMEmitter::ExtendedType Option,
uint32_t Shift = 0) {
LOGMAN_THROW_A_FMT(Shift <= 4, "Shift amount is too large");
constexpr uint32_t Op = 0b000'1011'001U << 21;
DataProcessing_Extended_Reg(Op, s, rd, rn, rm, Option, Shift);
}
void adds(ARMEmitter::Size s, ARMEmitter::Register rd, ARMEmitter::Register rn, ARMEmitter::Register rm, ARMEmitter::ExtendedType Option, uint32_t Shift = 0) {
void adds(ARMEmitter::Size s, ARMEmitter::Register rd, ARMEmitter::Register rn, ARMEmitter::Register rm, ARMEmitter::ExtendedType Option,
uint32_t Shift = 0) {
constexpr uint32_t Op = 0b010'1011'001U << 21;
DataProcessing_Extended_Reg(Op, s, rd, rn, rm, Option, Shift);
}
void cmn(ARMEmitter::Size s, ARMEmitter::Register rn, ARMEmitter::Register rm, ARMEmitter::ExtendedType Option, uint32_t Shift = 0) {
adds(s, ARMEmitter::Reg::zr, rn, rm, Option, Shift);
}
void sub(ARMEmitter::Size s, ARMEmitter::Register rd, ARMEmitter::Register rn, ARMEmitter::Register rm, ARMEmitter::ExtendedType Option, uint32_t Shift = 0) {
void sub(ARMEmitter::Size s, ARMEmitter::Register rd, ARMEmitter::Register rn, ARMEmitter::Register rm, ARMEmitter::ExtendedType Option,
uint32_t Shift = 0) {
constexpr uint32_t Op = 0b100'1011'001U << 21;
DataProcessing_Extended_Reg(Op, s, rd, rn, rm, Option, Shift);
}
void subs(ARMEmitter::Size s, ARMEmitter::Register rd, ARMEmitter::Register rn, ARMEmitter::Register rm, ARMEmitter::ExtendedType Option, uint32_t Shift = 0) {
void subs(ARMEmitter::Size s, ARMEmitter::Register rd, ARMEmitter::Register rn, ARMEmitter::Register rm, ARMEmitter::ExtendedType Option,
uint32_t Shift = 0) {
constexpr uint32_t Op = 0b110'1011'001U << 21;
DataProcessing_Extended_Reg(Op, s, rd, rn, rm, Option, Shift);
}
@@ -749,8 +716,8 @@ public:
// Rotate right into flags
void rmif(XRegister rn, uint32_t shift, uint32_t mask) {
LOGMAN_THROW_AA_FMT(shift <= 63, "Shift must be within 0-63. Shift: {}", shift);
LOGMAN_THROW_AA_FMT(mask <= 15, "Mask must be within 0-15. Mask: {}", mask);
LOGMAN_THROW_A_FMT(shift <= 63, "Shift must be within 0-63. Shift: {}", shift);
LOGMAN_THROW_A_FMT(mask <= 15, "Mask must be within 0-15. Mask: {}", mask);
uint32_t Op = 0b1011'1010'0000'0000'0000'0100'0000'0000;
Op |= rn.Idx() << 5;
@@ -814,7 +781,8 @@ public:
}
void cset(ARMEmitter::Size s, ARMEmitter::Register rd, ARMEmitter::Condition Cond) {
constexpr uint32_t Op = 0b0001'1010'100 << 21;
ConditionalCompare(Op, 0, 0b01, s, rd, ARMEmitter::Reg::zr, ARMEmitter::Reg::zr, static_cast<ARMEmitter::Condition>(FEXCore::ToUnderlying(Cond) ^ FEXCore::ToUnderlying(ARMEmitter::Condition::CC_NE)));
ConditionalCompare(Op, 0, 0b01, s, rd, ARMEmitter::Reg::zr, ARMEmitter::Reg::zr,
static_cast<ARMEmitter::Condition>(FEXCore::ToUnderlying(Cond) ^ FEXCore::ToUnderlying(ARMEmitter::Condition::CC_NE)));
}
void csinc(ARMEmitter::Size s, ARMEmitter::Register rd, ARMEmitter::Register rn, ARMEmitter::Register rm, ARMEmitter::Condition Cond) {
constexpr uint32_t Op = 0b0001'1010'100 << 21;
@@ -896,8 +864,7 @@ public:
private:
static constexpr Condition InvertCondition(Condition cond) {
// These behave as always, so it makes no sense to allow inverting these.
LOGMAN_THROW_AA_FMT(cond != Condition::CC_AL && cond != Condition::CC_NV,
"Cannot invert CC_AL or CC_NV");
LOGMAN_THROW_A_FMT(cond != Condition::CC_AL && cond != Condition::CC_NV, "Cannot invert CC_AL or CC_NV");
return static_cast<Condition>(FEXCore::ToUnderlying(cond) ^ 1);
}
@@ -948,7 +915,7 @@ private:
LSL12 = true;
Imm >>= 12;
}
LOGMAN_THROW_AA_FMT(TooLarge == false, "Imm amount too large: 0x{:x}", Imm);
LOGMAN_THROW_A_FMT(TooLarge == false, "Imm amount too large: 0x{:x}", Imm);
const uint32_t SF = s == ARMEmitter::Size::i64Bit ? (1U << 31) : 0;
@@ -993,7 +960,8 @@ private:
}
// Logical immediate
void DataProcessing_Logical_Imm(uint32_t Op, ARMEmitter::Size s, ARMEmitter::Register rd, ARMEmitter::Register rn, uint32_t n, uint32_t immr, uint32_t imms) {
void DataProcessing_Logical_Imm(uint32_t Op, ARMEmitter::Size s, ARMEmitter::Register rd, ARMEmitter::Register rn, uint32_t n,
uint32_t immr, uint32_t imms) {
const uint32_t SF = s == ARMEmitter::Size::i64Bit ? (1U << 31) : 0;
uint32_t Instr = Op;
@@ -1012,9 +980,8 @@ private:
[[maybe_unused]] const auto lsb_p_width = lsb + width;
const auto reg_size_bits = RegSizeInBits(s);
LOGMAN_THROW_AA_FMT(lsb_p_width <= reg_size_bits, "lsb + width ({}) must be <= {}. lsb={}, width={}",
lsb_p_width, reg_size_bits, lsb, width);
LOGMAN_THROW_AA_FMT(width >= 1, "xbfiz width must be >= 1");
LOGMAN_THROW_A_FMT(lsb_p_width <= reg_size_bits, "lsb + width ({}) must be <= {}. lsb={}, width={}", lsb_p_width, reg_size_bits, lsb, width);
LOGMAN_THROW_A_FMT(width >= 1, "xbfiz width must be >= 1");
const auto immr = (reg_size_bits - lsb) & (reg_size_bits - 1);
const auto imms = width - 1;
@@ -1026,12 +993,13 @@ private:
}
}
void DataProcessing_Extract(uint32_t Op, ARMEmitter::Size s, ARMEmitter::Register rd, ARMEmitter::Register rn, ARMEmitter::Register rm, uint32_t Imm) {
void DataProcessing_Extract(uint32_t Op, ARMEmitter::Size s, ARMEmitter::Register rd, ARMEmitter::Register rn, ARMEmitter::Register rm,
uint32_t Imm) {
const uint32_t SF = s == ARMEmitter::Size::i64Bit ? (1U << 31) : 0;
// Current ARMv8 spec hardcodes SF == N for this class of instructions.
// Anythign else is undefined behaviour.
const uint32_t N = s == ARMEmitter::Size::i64Bit ? (1U << 22) : 0;
const uint32_t N = s == ARMEmitter::Size::i64Bit ? (1U << 22) : 0;
uint32_t Instr = Op;
@@ -1074,10 +1042,11 @@ private:
}
// AddSub - shifted register
void DataProcessing_Shifted_Reg(uint32_t Op, ARMEmitter::Size s, ARMEmitter::Register rd, ARMEmitter::Register rn, ARMEmitter::Register rm, ARMEmitter::ShiftType Shift, uint32_t amt) {
LOGMAN_THROW_AA_FMT((amt & ~0b11'1111U) == 0, "Shift amount too large");
void DataProcessing_Shifted_Reg(uint32_t Op, ARMEmitter::Size s, ARMEmitter::Register rd, ARMEmitter::Register rn,
ARMEmitter::Register rm, ARMEmitter::ShiftType Shift, uint32_t amt) {
LOGMAN_THROW_A_FMT((amt & ~0b11'1111U) == 0, "Shift amount too large");
if (s == ARMEmitter::Size::i32Bit) {
LOGMAN_THROW_AA_FMT(amt < 32, "Shift amount for 32-bit must be below 32");
LOGMAN_THROW_A_FMT(amt < 32, "Shift amount for 32-bit must be below 32");
}
const uint32_t SF = s == ARMEmitter::Size::i64Bit ? (1U << 31) : 0;
@@ -1095,7 +1064,8 @@ private:
}
// AddSub - extended register
void DataProcessing_Extended_Reg(uint32_t Op, ARMEmitter::Size s, ARMEmitter::Register rd, ARMEmitter::Register rn, ARMEmitter::Register rm, ARMEmitter::ExtendedType Option, uint32_t Shift) {
void DataProcessing_Extended_Reg(uint32_t Op, ARMEmitter::Size s, ARMEmitter::Register rd, ARMEmitter::Register rn,
ARMEmitter::Register rm, ARMEmitter::ExtendedType Option, uint32_t Shift) {
const uint32_t SF = s == ARMEmitter::Size::i64Bit ? (1U << 31) : 0;
uint32_t Instr = Op;
@@ -1111,7 +1081,8 @@ private:
}
// Conditional compare - register
template<typename T>
void ConditionalCompare(uint32_t Op, uint32_t o1, uint32_t o2, uint32_t o3, ARMEmitter::Size s, ARMEmitter::Register rn, T rm, ARMEmitter::StatusFlags flags, ARMEmitter::Condition Cond) {
void ConditionalCompare(uint32_t Op, uint32_t o1, uint32_t o2, uint32_t o3, ARMEmitter::Size s, ARMEmitter::Register rn, T rm,
ARMEmitter::StatusFlags flags, ARMEmitter::Condition Cond) {
const uint32_t SF = s == ARMEmitter::Size::i64Bit ? (1U << 31) : 0;
uint32_t Instr = Op;
@@ -1129,7 +1100,8 @@ private:
}
template<typename T>
void ConditionalCompare(uint32_t Op, uint32_t o1, uint32_t o2, ARMEmitter::Size s, ARMEmitter::Register rd, ARMEmitter::Register rn, T rm, ARMEmitter::Condition Cond) {
void ConditionalCompare(uint32_t Op, uint32_t o1, uint32_t o2, ARMEmitter::Size s, ARMEmitter::Register rd, ARMEmitter::Register rn, T rm,
ARMEmitter::Condition Cond) {
const uint32_t SF = s == ARMEmitter::Size::i64Bit ? (1U << 31) : 0;
uint32_t Instr = Op;
@@ -1146,7 +1118,8 @@ private:
}
// Data-processing - 3 source
void DataProcessing_3Source(uint32_t Op, uint32_t Op0, ARMEmitter::Size s, ARMEmitter::Register rd, ARMEmitter::Register rn, ARMEmitter::Register rm, ARMEmitter::Register ra) {
void DataProcessing_3Source(uint32_t Op, uint32_t Op0, ARMEmitter::Size s, ARMEmitter::Register rd, ARMEmitter::Register rn,
ARMEmitter::Register rm, ARMEmitter::Register ra) {
const uint32_t SF = s == ARMEmitter::Size::i64Bit ? (1U << 31) : 0;
uint32_t Instr = Op;
@@ -1168,4 +1141,7 @@ private:
dc32(Instr);
}
#ifndef INCLUDED_BY_EMITTER
}; // struct LoadstoreEmitterOps
} // namespace ARMEmitter
#endif
File diff suppressed because it is too large. Load diff
+291 -305
View File
@@ -3,339 +3,325 @@
*
* Most of these instructions will use `BackwardLabel`, `ForwardLabel`, or `BiDirectionLabel` to determine where a branch targets.
*/
#pragma once
#ifndef INCLUDED_BY_EMITTER
#include <CodeEmitter/Emitter.h>
namespace ARMEmitter {
struct EmitterOps : Emitter {
#endif
public:
// Branches, Exception Generating and System instructions
public:
// Conditional branch immediate
///< Branch conditional
void b(ARMEmitter::Condition Cond, uint32_t Imm) {
constexpr uint32_t Op = 0b0101'010 << 25;
Branch_Conditional(Op, 0, 0, Cond, Imm);
public:
// Conditional branch immediate
///< Branch conditional
void b(ARMEmitter::Condition Cond, uint32_t Imm) {
constexpr uint32_t Op = 0b0101'010 << 25;
Branch_Conditional(Op, 0, 0, Cond, Imm);
}
void b(ARMEmitter::Condition Cond, const BackwardLabel* Label) {
int32_t Imm = static_cast<int32_t>(Label->Location - GetCursorAddress<uint8_t*>());
LOGMAN_THROW_A_FMT(Imm >= -1048576 && Imm <= 1048575 && ((Imm & 0b11) == 0), "Unscaled offset too large");
constexpr uint32_t Op = 0b0101'010 << 25;
Branch_Conditional(Op, 0, 0, Cond, Imm >> 2);
}
void b(ARMEmitter::Condition Cond, ForwardLabel* Label) {
AddLocationToLabel(Label, ForwardLabel::Reference {.Location = GetCursorAddress<uint8_t*>(), .Type = ForwardLabel::InstType::BC});
constexpr uint32_t Op = 0b0101'010 << 25;
Branch_Conditional(Op, 0, 0, Cond, 0);
}
void b(ARMEmitter::Condition Cond, BiDirectionalLabel* Label) {
if (Label->Backward.Location) {
b(Cond, &Label->Backward);
} else {
b(Cond, &Label->Forward);
}
void b(ARMEmitter::Condition Cond, BackwardLabel const* Label) {
int32_t Imm = static_cast<int32_t>(Label->Location - GetCursorAddress<uint8_t*>());
LOGMAN_THROW_A_FMT(Imm >= -1048576 && Imm <= 1048575 && ((Imm & 0b11) == 0), "Unscaled offset too large");
constexpr uint32_t Op = 0b0101'010 << 25;
Branch_Conditional(Op, 0, 0, Cond, Imm >> 2);
}
///< Branch consistent conditional
void bc(ARMEmitter::Condition Cond, uint32_t Imm) {
constexpr uint32_t Op = 0b0101'010 << 25;
Branch_Conditional(Op, 0, 1, Cond, Imm);
}
void bc(ARMEmitter::Condition Cond, const BackwardLabel* Label) {
int32_t Imm = static_cast<int32_t>(Label->Location - GetCursorAddress<uint8_t*>());
LOGMAN_THROW_A_FMT(Imm >= -1048576 && Imm <= 1048575 && ((Imm & 0b11) == 0), "Unscaled offset too large");
constexpr uint32_t Op = 0b0101'010 << 25;
Branch_Conditional(Op, 0, 1, Cond, Imm >> 2);
}
void bc(ARMEmitter::Condition Cond, ForwardLabel* Label) {
AddLocationToLabel(Label, ForwardLabel::Reference {.Location = GetCursorAddress<uint8_t*>(), .Type = ForwardLabel::InstType::BC});
constexpr uint32_t Op = 0b0101'010 << 25;
Branch_Conditional(Op, 0, 1, Cond, 0);
}
void bc(ARMEmitter::Condition Cond, BiDirectionalLabel* Label) {
if (Label->Backward.Location) {
bc(Cond, &Label->Backward);
} else {
bc(Cond, &Label->Forward);
}
template<typename LabelType>
requires (std::is_same_v<LabelType, ForwardLabel> || std::is_same_v<LabelType, SingleUseForwardLabel>)
void b(ARMEmitter::Condition Cond, LabelType *Label) {
AddLocationToLabel(Label, SingleUseForwardLabel{ .Location = GetCursorAddress<uint8_t*>(), .Type = SingleUseForwardLabel::InstType::BC });
constexpr uint32_t Op = 0b0101'010 << 25;
Branch_Conditional(Op, 0, 0, Cond, 0);
}
// Unconditional branch register
void br(ARMEmitter::Register rn) {
constexpr uint32_t Op = 0b1101011 << 25 | 0b0'000 << 21 | // opc
0b1'1111 << 16 | // op2
0b0000'00 << 10 | // op3
0b0'0000; // op4
UnconditionalBranch(Op, rn);
}
void blr(ARMEmitter::Register rn) {
constexpr uint32_t Op = 0b1101011 << 25 | 0b0'001 << 21 | // opc
0b1'1111 << 16 | // op2
0b0000'00 << 10 | // op3
0b0'0000; // op4
UnconditionalBranch(Op, rn);
}
void ret(ARMEmitter::Register rn = ARMEmitter::Reg::r30) {
constexpr uint32_t Op = 0b1101011 << 25 | 0b0'010 << 21 | // opc
0b1'1111 << 16 | // op2
0b0000'00 << 10 | // op3
0b0'0000; // op4
UnconditionalBranch(Op, rn);
}
// Unconditional branch immediate
void b(uint32_t Imm) {
constexpr uint32_t Op = 0b0001'01 << 26;
UnconditionalBranch(Op, Imm);
}
void b(const BackwardLabel* Label) {
int32_t Imm = static_cast<int32_t>(Label->Location - GetCursorAddress<uint8_t*>());
LOGMAN_THROW_A_FMT(Imm >= -134217728 && Imm <= 134217724 && ((Imm & 0b11) == 0), "Unscaled offset too large");
constexpr uint32_t Op = 0b0001'01 << 26;
UnconditionalBranch(Op, Imm >> 2);
}
void b(ForwardLabel* Label) {
AddLocationToLabel(Label, ForwardLabel::Reference {.Location = GetCursorAddress<uint8_t*>(), .Type = ForwardLabel::InstType::B});
constexpr uint32_t Op = 0b0001'01 << 26;
UnconditionalBranch(Op, 0);
}
void b(BiDirectionalLabel* Label) {
if (Label->Backward.Location) {
b(&Label->Backward);
} else {
b(&Label->Forward);
}
}
void b(ARMEmitter::Condition Cond, BiDirectionalLabel *Label) {
if (Label->Backward.Location) {
b(Cond, &Label->Backward);
}
else {
b(Cond, &Label->Forward);
}
void bl(uint32_t Imm) {
constexpr uint32_t Op = 0b1001'01 << 26;
UnconditionalBranch(Op, Imm);
}
void bl(const BackwardLabel* Label) {
int32_t Imm = static_cast<int32_t>(Label->Location - GetCursorAddress<uint8_t*>());
LOGMAN_THROW_A_FMT(Imm >= -134217728 && Imm <= 134217724 && ((Imm & 0b11) == 0), "Unscaled offset too large");
constexpr uint32_t Op = 0b1001'01 << 26;
UnconditionalBranch(Op, Imm >> 2);
}
void bl(ForwardLabel* Label) {
AddLocationToLabel(Label, ForwardLabel::Reference {.Location = GetCursorAddress<uint8_t*>(), .Type = ForwardLabel::InstType::B});
constexpr uint32_t Op = 0b1001'01 << 26;
UnconditionalBranch(Op, 0);
}
void bl(BiDirectionalLabel* Label) {
if (Label->Backward.Location) {
bl(&Label->Backward);
} else {
bl(&Label->Forward);
}
}
///< Branch consistent conditional
void bc(ARMEmitter::Condition Cond, uint32_t Imm) {
constexpr uint32_t Op = 0b0101'010 << 25;
Branch_Conditional(Op, 0, 1, Cond, Imm);
// Compare and branch
void cbz(ARMEmitter::Size s, ARMEmitter::Register rt, uint32_t Imm) {
constexpr uint32_t Op = 0b0011'0100 << 24;
CompareAndBranch(Op, s, rt, Imm);
}
void cbz(ARMEmitter::Size s, ARMEmitter::Register rt, const BackwardLabel* Label) {
int32_t Imm = static_cast<int32_t>(Label->Location - GetCursorAddress<uint8_t*>());
LOGMAN_THROW_A_FMT(Imm >= -1048576 && Imm <= 1048575 && ((Imm & 0b11) == 0), "Unscaled offset too large");
constexpr uint32_t Op = 0b0011'0100 << 24;
CompareAndBranch(Op, s, rt, Imm >> 2);
}
void cbz(ARMEmitter::Size s, ARMEmitter::Register rt, ForwardLabel* Label) {
AddLocationToLabel(Label, ForwardLabel::Reference {.Location = GetCursorAddress<uint8_t*>(), .Type = ForwardLabel::InstType::BC});
constexpr uint32_t Op = 0b0011'0100 << 24;
CompareAndBranch(Op, s, rt, 0);
}
void cbz(ARMEmitter::Size s, ARMEmitter::Register rt, BiDirectionalLabel* Label) {
if (Label->Backward.Location) {
cbz(s, rt, &Label->Backward);
} else {
cbz(s, rt, &Label->Forward);
}
void bc(ARMEmitter::Condition Cond, BackwardLabel const* Label) {
int32_t Imm = static_cast<int32_t>(Label->Location - GetCursorAddress<uint8_t*>());
LOGMAN_THROW_A_FMT(Imm >= -1048576 && Imm <= 1048575 && ((Imm & 0b11) == 0), "Unscaled offset too large");
constexpr uint32_t Op = 0b0101'010 << 25;
Branch_Conditional(Op, 0, 1, Cond, Imm >> 2);
}
void cbnz(ARMEmitter::Size s, ARMEmitter::Register rt, uint32_t Imm) {
constexpr uint32_t Op = 0b0011'0101 << 24;
CompareAndBranch(Op, s, rt, Imm);
}
void cbnz(ARMEmitter::Size s, ARMEmitter::Register rt, const BackwardLabel* Label) {
int32_t Imm = static_cast<int32_t>(Label->Location - GetCursorAddress<uint8_t*>());
LOGMAN_THROW_A_FMT(Imm >= -1048576 && Imm <= 1048575 && ((Imm & 0b11) == 0), "Unscaled offset too large");
constexpr uint32_t Op = 0b0011'0101 << 24;
CompareAndBranch(Op, s, rt, Imm >> 2);
}
void cbnz(ARMEmitter::Size s, ARMEmitter::Register rt, ForwardLabel* Label) {
AddLocationToLabel(Label, ForwardLabel::Reference {.Location = GetCursorAddress<uint8_t*>(), .Type = ForwardLabel::InstType::BC});
constexpr uint32_t Op = 0b0011'0101 << 24;
CompareAndBranch(Op, s, rt, 0);
}
void cbnz(ARMEmitter::Size s, ARMEmitter::Register rt, BiDirectionalLabel* Label) {
if (Label->Backward.Location) {
cbnz(s, rt, &Label->Backward);
} else {
cbnz(s, rt, &Label->Forward);
}
}
template<typename LabelType>
requires (std::is_same_v<LabelType, ForwardLabel> || std::is_same_v<LabelType, SingleUseForwardLabel>)
void bc(ARMEmitter::Condition Cond, LabelType *Label) {
AddLocationToLabel(Label, SingleUseForwardLabel{ .Location = GetCursorAddress<uint8_t*>(), .Type = SingleUseForwardLabel::InstType::BC });
constexpr uint32_t Op = 0b0101'010 << 25;
Branch_Conditional(Op, 0, 1, Cond, 0);
// Test and branch immediate
void tbz(ARMEmitter::Register rt, uint32_t Bit, uint32_t Imm) {
constexpr uint32_t Op = 0b0011'0110 << 24;
TestAndBranch(Op, rt, Bit, Imm);
}
void tbz(ARMEmitter::Register rt, uint32_t Bit, const BackwardLabel* Label) {
int32_t Imm = static_cast<int32_t>(Label->Location - GetCursorAddress<uint8_t*>());
LOGMAN_THROW_A_FMT(Imm >= -32768 && Imm <= 32764 && ((Imm & 0b11) == 0), "Unscaled offset too large");
constexpr uint32_t Op = 0b0011'0110 << 24;
TestAndBranch(Op, rt, Bit, Imm >> 2);
}
void tbz(ARMEmitter::Register rt, uint32_t Bit, ForwardLabel* Label) {
AddLocationToLabel(Label, ForwardLabel::Reference {.Location = GetCursorAddress<uint8_t*>(), .Type = ForwardLabel::InstType::TEST_BRANCH});
constexpr uint32_t Op = 0b0011'0110 << 24;
TestAndBranch(Op, rt, Bit, 0);
}
void tbz(ARMEmitter::Register rt, uint32_t Bit, BiDirectionalLabel* Label) {
if (Label->Backward.Location) {
tbz(rt, Bit, &Label->Backward);
} else {
tbz(rt, Bit, &Label->Forward);
}
}
void bc(ARMEmitter::Condition Cond, BiDirectionalLabel *Label) {
if (Label->Backward.Location) {
bc(Cond, &Label->Backward);
}
else {
bc(Cond, &Label->Forward);
}
}
void tbnz(ARMEmitter::Register rt, uint32_t Bit, uint32_t Imm) {
constexpr uint32_t Op = 0b0011'0111 << 24;
// Unconditional branch register
void br(ARMEmitter::Register rn) {
constexpr uint32_t Op = 0b1101011 << 25 |
0b0'000 << 21 | // opc
0b1'1111 << 16 | // op2
0b0000'00 << 10 | // op3
0b0'0000; // op4
TestAndBranch(Op, rt, Bit, Imm);
}
void tbnz(ARMEmitter::Register rt, uint32_t Bit, const BackwardLabel* Label) {
int32_t Imm = static_cast<int32_t>(Label->Location - GetCursorAddress<uint8_t*>());
LOGMAN_THROW_A_FMT(Imm >= -32768 && Imm <= 32764 && ((Imm & 0b11) == 0), "Unscaled offset too large");
UnconditionalBranch(Op, rn);
}
void blr(ARMEmitter::Register rn) {
constexpr uint32_t Op = 0b1101011 << 25 |
0b0'001 << 21 | // opc
0b1'1111 << 16 | // op2
0b0000'00 << 10 | // op3
0b0'0000; // op4
constexpr uint32_t Op = 0b0011'0111 << 24;
UnconditionalBranch(Op, rn);
}
void ret(ARMEmitter::Register rn = ARMEmitter::Reg::r30) {
constexpr uint32_t Op = 0b1101011 << 25 |
0b0'010 << 21 | // opc
0b1'1111 << 16 | // op2
0b0000'00 << 10 | // op3
0b0'0000; // op4
TestAndBranch(Op, rt, Bit, Imm >> 2);
}
UnconditionalBranch(Op, rn);
}
void tbnz(ARMEmitter::Register rt, uint32_t Bit, ForwardLabel* Label) {
AddLocationToLabel(Label, ForwardLabel::Reference {.Location = GetCursorAddress<uint8_t*>(), .Type = ForwardLabel::InstType::TEST_BRANCH});
constexpr uint32_t Op = 0b0011'0111 << 24;
// Unconditional branch immediate
void b(uint32_t Imm) {
constexpr uint32_t Op = 0b0001'01 << 26;
TestAndBranch(Op, rt, Bit, 0);
}
UnconditionalBranch(Op, Imm);
}
void b(BackwardLabel const* Label) {
int32_t Imm = static_cast<int32_t>(Label->Location - GetCursorAddress<uint8_t*>());
LOGMAN_THROW_A_FMT(Imm >= -134217728 && Imm <= 134217724 && ((Imm & 0b11) == 0), "Unscaled offset too large");
constexpr uint32_t Op = 0b0001'01 << 26;
UnconditionalBranch(Op, Imm >> 2);
}
template<typename LabelType>
requires (std::is_same_v<LabelType, ForwardLabel> || std::is_same_v<LabelType, SingleUseForwardLabel>)
void b(LabelType *Label) {
AddLocationToLabel(Label, SingleUseForwardLabel{ .Location = GetCursorAddress<uint8_t*>(), .Type = SingleUseForwardLabel::InstType::B });
constexpr uint32_t Op = 0b0001'01 << 26;
UnconditionalBranch(Op, 0);
}
void b(BiDirectionalLabel *Label) {
if (Label->Backward.Location) {
b(&Label->Backward);
}
else {
b(&Label->Forward);
}
}
void bl(uint32_t Imm) {
constexpr uint32_t Op = 0b1001'01 << 26;
UnconditionalBranch(Op, Imm);
}
void bl(BackwardLabel const* Label) {
int32_t Imm = static_cast<int32_t>(Label->Location - GetCursorAddress<uint8_t*>());
LOGMAN_THROW_A_FMT(Imm >= -134217728 && Imm <= 134217724 && ((Imm & 0b11) == 0), "Unscaled offset too large");
constexpr uint32_t Op = 0b1001'01 << 26;
UnconditionalBranch(Op, Imm >> 2);
}
template<typename LabelType>
requires (std::is_same_v<LabelType, ForwardLabel> || std::is_same_v<LabelType, SingleUseForwardLabel>)
void bl(LabelType *Label) {
AddLocationToLabel(Label, SingleUseForwardLabel{ .Location = GetCursorAddress<uint8_t*>(), .Type = SingleUseForwardLabel::InstType::B });
constexpr uint32_t Op = 0b1001'01 << 26;
UnconditionalBranch(Op, 0);
}
void bl(BiDirectionalLabel *Label) {
if (Label->Backward.Location) {
bl(&Label->Backward);
}
else {
bl(&Label->Forward);
}
}
// Compare and branch
void cbz(ARMEmitter::Size s, ARMEmitter::Register rt, uint32_t Imm) {
constexpr uint32_t Op = 0b0011'0100 << 24;
CompareAndBranch(Op, s, rt, Imm);
}
void cbz(ARMEmitter::Size s, ARMEmitter::Register rt, BackwardLabel const* Label) {
int32_t Imm = static_cast<int32_t>(Label->Location - GetCursorAddress<uint8_t*>());
LOGMAN_THROW_A_FMT(Imm >= -1048576 && Imm <= 1048575 && ((Imm & 0b11) == 0), "Unscaled offset too large");
constexpr uint32_t Op = 0b0011'0100 << 24;
CompareAndBranch(Op, s, rt, Imm >> 2);
}
template<typename LabelType>
requires (std::is_same_v<LabelType, ForwardLabel> || std::is_same_v<LabelType, SingleUseForwardLabel>)
void cbz(ARMEmitter::Size s, ARMEmitter::Register rt, LabelType *Label) {
AddLocationToLabel(Label, SingleUseForwardLabel{ .Location = GetCursorAddress<uint8_t*>(), .Type = SingleUseForwardLabel::InstType::BC });
constexpr uint32_t Op = 0b0011'0100 << 24;
CompareAndBranch(Op, s, rt, 0);
}
void cbz(ARMEmitter::Size s, ARMEmitter::Register rt, BiDirectionalLabel *Label) {
if (Label->Backward.Location) {
cbz(s, rt, &Label->Backward);
}
else {
cbz(s, rt, &Label->Forward);
}
}
void cbnz(ARMEmitter::Size s, ARMEmitter::Register rt, uint32_t Imm) {
constexpr uint32_t Op = 0b0011'0101 << 24;
CompareAndBranch(Op, s, rt, Imm);
}
void cbnz(ARMEmitter::Size s, ARMEmitter::Register rt, BackwardLabel const* Label) {
int32_t Imm = static_cast<int32_t>(Label->Location - GetCursorAddress<uint8_t*>());
LOGMAN_THROW_A_FMT(Imm >= -1048576 && Imm <= 1048575 && ((Imm & 0b11) == 0), "Unscaled offset too large");
constexpr uint32_t Op = 0b0011'0101 << 24;
CompareAndBranch(Op, s, rt, Imm >> 2);
}
template<typename LabelType>
requires (std::is_same_v<LabelType, ForwardLabel> || std::is_same_v<LabelType, SingleUseForwardLabel>)
void cbnz(ARMEmitter::Size s, ARMEmitter::Register rt, LabelType *Label) {
AddLocationToLabel(Label, SingleUseForwardLabel{ .Location = GetCursorAddress<uint8_t*>(), .Type = SingleUseForwardLabel::InstType::BC });
constexpr uint32_t Op = 0b0011'0101 << 24;
CompareAndBranch(Op, s, rt, 0);
}
void cbnz(ARMEmitter::Size s, ARMEmitter::Register rt, BiDirectionalLabel *Label) {
if (Label->Backward.Location) {
cbnz(s, rt, &Label->Backward);
}
else {
cbnz(s, rt, &Label->Forward);
}
}
// Test and branch immediate
void tbz(ARMEmitter::Register rt, uint32_t Bit, uint32_t Imm) {
constexpr uint32_t Op = 0b0011'0110 << 24;
TestAndBranch(Op, rt, Bit, Imm);
}
void tbz(ARMEmitter::Register rt, uint32_t Bit, BackwardLabel const* Label) {
int32_t Imm = static_cast<int32_t>(Label->Location - GetCursorAddress<uint8_t*>());
LOGMAN_THROW_A_FMT(Imm >= -32768 && Imm <= 32764 && ((Imm & 0b11) == 0), "Unscaled offset too large");
constexpr uint32_t Op = 0b0011'0110 << 24;
TestAndBranch(Op, rt, Bit, Imm >> 2);
}
template<typename LabelType>
requires (std::is_same_v<LabelType, ForwardLabel> || std::is_same_v<LabelType, SingleUseForwardLabel>)
void tbz(ARMEmitter::Register rt, uint32_t Bit, LabelType *Label) {
AddLocationToLabel(Label, SingleUseForwardLabel{ .Location = GetCursorAddress<uint8_t*>(), .Type = SingleUseForwardLabel::InstType::TEST_BRANCH });
constexpr uint32_t Op = 0b0011'0110 << 24;
TestAndBranch(Op, rt, Bit, 0);
}
void tbz(ARMEmitter::Register rt, uint32_t Bit, BiDirectionalLabel *Label) {
if (Label->Backward.Location) {
tbz(rt, Bit, &Label->Backward);
}
else {
tbz(rt, Bit, &Label->Forward);
}
}
void tbnz(ARMEmitter::Register rt, uint32_t Bit, uint32_t Imm) {
constexpr uint32_t Op = 0b0011'0111 << 24;
TestAndBranch(Op, rt, Bit, Imm);
}
void tbnz(ARMEmitter::Register rt, uint32_t Bit, BackwardLabel const* Label) {
int32_t Imm = static_cast<int32_t>(Label->Location - GetCursorAddress<uint8_t*>());
LOGMAN_THROW_A_FMT(Imm >= -32768 && Imm <= 32764 && ((Imm & 0b11) == 0), "Unscaled offset too large");
constexpr uint32_t Op = 0b0011'0111 << 24;
TestAndBranch(Op, rt, Bit, Imm >> 2);
}
template<typename LabelType>
requires (std::is_same_v<LabelType, ForwardLabel> || std::is_same_v<LabelType, SingleUseForwardLabel>)
void tbnz(ARMEmitter::Register rt, uint32_t Bit, LabelType *Label) {
AddLocationToLabel(Label, SingleUseForwardLabel{ .Location = GetCursorAddress<uint8_t*>(), .Type = SingleUseForwardLabel::InstType::TEST_BRANCH });
constexpr uint32_t Op = 0b0011'0111 << 24;
TestAndBranch(Op, rt, Bit, 0);
}
void tbnz(ARMEmitter::Register rt, uint32_t Bit, BiDirectionalLabel *Label) {
if (Label->Backward.Location) {
tbnz(rt, Bit, &Label->Backward);
}
else {
tbnz(rt, Bit, &Label->Forward);
}
void tbnz(ARMEmitter::Register rt, uint32_t Bit, BiDirectionalLabel* Label) {
if (Label->Backward.Location) {
tbnz(rt, Bit, &Label->Backward);
} else {
tbnz(rt, Bit, &Label->Forward);
}
}
private:
// Conditional branch immediate
void Branch_Conditional(uint32_t Op, uint32_t Op1, uint32_t Op0, ARMEmitter::Condition Cond, uint32_t Imm) {
uint32_t Instr = Op;
// Conditional branch immediate
void Branch_Conditional(uint32_t Op, uint32_t Op1, uint32_t Op0, ARMEmitter::Condition Cond, uint32_t Imm) {
uint32_t Instr = Op;
Instr |= Op1 << 24;
Instr |= (Imm & 0x7'FFFF) << 5;
Instr |= Op0 << 4;
Instr |= FEXCore::ToUnderlying(Cond);
Instr |= Op1 << 24;
Instr |= (Imm & 0x7'FFFF) << 5;
Instr |= Op0 << 4;
Instr |= FEXCore::ToUnderlying(Cond);
dc32(Instr);
}
dc32(Instr);
}
// Unconditional branch register
void UnconditionalBranch(uint32_t Op, ARMEmitter::Register rn) {
uint32_t Instr = Op;
Instr |= Encode_rn(rn);
dc32(Instr);
}
// Unconditional branch register
void UnconditionalBranch(uint32_t Op, ARMEmitter::Register rn) {
uint32_t Instr = Op;
Instr |= Encode_rn(rn);
dc32(Instr);
}
// Unconditional branch - immediate
void UnconditionalBranch(uint32_t Op, uint32_t Imm) {
uint32_t Instr = Op;
Instr |= Imm & 0x3FF'FFFF;
dc32(Instr);
}
// Unconditional branch - immediate
void UnconditionalBranch(uint32_t Op, uint32_t Imm) {
uint32_t Instr = Op;
Instr |= Imm & 0x3FF'FFFF;
dc32(Instr);
}
// Compare and branch
void CompareAndBranch(uint32_t Op, ARMEmitter::Size s, ARMEmitter::Register rt, uint32_t Imm) {
const uint32_t SF = s == ARMEmitter::Size::i64Bit ? (1U << 31) : 0;
// Compare and branch
void CompareAndBranch(uint32_t Op, ARMEmitter::Size s, ARMEmitter::Register rt, uint32_t Imm) {
const uint32_t SF = s == ARMEmitter::Size::i64Bit ? (1U << 31) : 0;
uint32_t Instr = Op;
uint32_t Instr = Op;
Instr |= SF;
Instr |= (Imm & 0x7'FFFF) << 5;
Instr |= Encode_rt(rt);
dc32(Instr);
}
Instr |= SF;
Instr |= (Imm & 0x7'FFFF) << 5;
Instr |= Encode_rt(rt);
dc32(Instr);
}
// Test and branch - immediate
void TestAndBranch(uint32_t Op, ARMEmitter::Register rt, uint32_t Bit, uint32_t Imm) {
uint32_t Instr = Op;
// Test and branch - immediate
void TestAndBranch(uint32_t Op, ARMEmitter::Register rt, uint32_t Bit, uint32_t Imm) {
uint32_t Instr = Op;
Instr |= (Bit >> 5) << 31;
Instr |= (Bit & 0b1'1111) << 19;
Instr |= (Imm & 0x3FFF) << 5;
Instr |= Encode_rt(rt);
dc32(Instr);
}
Instr |= (Bit >> 5) << 31;
Instr |= (Bit & 0b1'1111) << 19;
Instr |= (Imm & 0x3FFF) << 5;
Instr |= Encode_rt(rt);
dc32(Instr);
}
#ifndef INCLUDED_BY_EMITTER
}; // struct LoadstoreEmitterOps
} // namespace ARMEmitter
#endif
+94 -94
View File
@@ -341,93 +341,88 @@ public:
};
template<uint32_t op0, uint32_t op1, uint32_t CRn, uint32_t CRm, uint32_t op2>
constexpr uint32_t GenSystemReg() {
return op0 << 19 | op1 << 16 | CRn << 12 | CRm << 8 | op2 << 5;
};
inline constexpr uint32_t GenSystemReg = op0 << 19 | op1 << 16 | CRn << 12 | CRm << 8 | op2 << 5;
// This `SystemRegister` enum is used for the mrs/msr instructions.
enum class SystemRegister : uint32_t {
CTR_EL0 = GenSystemReg<0b11, 0b011, 0b0000, 0b0000, 0b001>(),
DCZID_EL0 = GenSystemReg<0b11, 0b011, 0b0000, 0b0000, 0b111>(),
TPIDR_EL0 = GenSystemReg<0b11, 0b011, 0b1101, 0b0000, 0b010>(),
RNDR = GenSystemReg<0b11, 0b011, 0b0010, 0b0100, 0b000>(),
RNDRRS = GenSystemReg<0b11, 0b011, 0b0010, 0b0100, 0b001>(),
NZCV = GenSystemReg<0b11, 0b011, 0b0100, 0b0010, 0b000>(),
FPCR = GenSystemReg<0b11, 0b011, 0b0100, 0b0100, 0b000>(),
CNTFRQ_EL0 = GenSystemReg<0b11, 0b011, 0b1110, 0b0000, 0b000>(),
CNTVCT_EL0 = GenSystemReg<0b11, 0b011, 0b1110, 0b0000, 0b010>(),
CTR_EL0 = GenSystemReg<0b11, 0b011, 0b0000, 0b0000, 0b001>,
DCZID_EL0 = GenSystemReg<0b11, 0b011, 0b0000, 0b0000, 0b111>,
TPIDR_EL0 = GenSystemReg<0b11, 0b011, 0b1101, 0b0000, 0b010>,
RNDR = GenSystemReg<0b11, 0b011, 0b0010, 0b0100, 0b000>,
RNDRRS = GenSystemReg<0b11, 0b011, 0b0010, 0b0100, 0b001>,
NZCV = GenSystemReg<0b11, 0b011, 0b0100, 0b0010, 0b000>,
FPCR = GenSystemReg<0b11, 0b011, 0b0100, 0b0100, 0b000>,
TPIDRRO_EL0 = GenSystemReg<0b11, 0b011, 0b1101, 0b0000, 0b011>,
CNTFRQ_EL0 = GenSystemReg<0b11, 0b011, 0b1110, 0b0000, 0b000>,
CNTVCT_EL0 = GenSystemReg<0b11, 0b011, 0b1110, 0b0000, 0b010>,
};
template<uint32_t op1, uint32_t CRm, uint32_t op2>
constexpr uint32_t GenDCReg() {
return op1 << 16 | CRm << 8 | op2 << 5;
};
inline constexpr uint32_t GenDCReg = op1 << 16 | CRm << 8 | op2 << 5;
// This `DataCacheOperation` enum is used for the dc instruction.
enum class DataCacheOperation : uint32_t {
IVAC = GenDCReg<0b000, 0b0110, 0b001>(),
ISW = GenDCReg<0b000, 0b0110, 0b010>(),
CSW = GenDCReg<0b000, 0b1010, 0b010>(),
CISW = GenDCReg<0b000, 0b1110, 0b010>(),
ZVA = GenDCReg<0b011, 0b0100, 0b001>(),
CVAC = GenDCReg<0b011, 0b1010, 0b001>(),
CVAU = GenDCReg<0b011, 0b1011, 0b001>(),
CIVAC = GenDCReg<0b011, 0b1110, 0b001>(),
IVAC = GenDCReg<0b000, 0b0110, 0b001>,
ISW = GenDCReg<0b000, 0b0110, 0b010>,
CSW = GenDCReg<0b000, 0b1010, 0b010>,
CISW = GenDCReg<0b000, 0b1110, 0b010>,
ZVA = GenDCReg<0b011, 0b0100, 0b001>,
CVAC = GenDCReg<0b011, 0b1010, 0b001>,
CVAU = GenDCReg<0b011, 0b1011, 0b001>,
CIVAC = GenDCReg<0b011, 0b1110, 0b001>,
// MTE2
IGVAC = GenDCReg<0b000, 0b0110, 0b011>(),
IGSW = GenDCReg<0b000, 0b0110, 0b100>(),
IGDVAC = GenDCReg<0b000, 0b0110, 0b101>(),
IGDSW = GenDCReg<0b000, 0b0110, 0b110>(),
CGSW = GenDCReg<0b000, 0b1010, 0b100>(),
CGDSW = GenDCReg<0b000, 0b1010, 0b110>(),
CIGSW = GenDCReg<0b000, 0b1110, 0b100>(),
CIGDSW = GenDCReg<0b000, 0b1110, 0b110>(),
IGVAC = GenDCReg<0b000, 0b0110, 0b011>,
IGSW = GenDCReg<0b000, 0b0110, 0b100>,
IGDVAC = GenDCReg<0b000, 0b0110, 0b101>,
IGDSW = GenDCReg<0b000, 0b0110, 0b110>,
CGSW = GenDCReg<0b000, 0b1010, 0b100>,
CGDSW = GenDCReg<0b000, 0b1010, 0b110>,
CIGSW = GenDCReg<0b000, 0b1110, 0b100>,
CIGDSW = GenDCReg<0b000, 0b1110, 0b110>,
// MTE
GVA = GenDCReg<0b011, 0b0100, 0b011>(),
GZVA = GenDCReg<0b011, 0b0100, 0b100>(),
CGVAC = GenDCReg<0b011, 0b1010, 0b011>(),
CGDVAC = GenDCReg<0b011, 0b1010, 0b101>(),
CGVAP = GenDCReg<0b011, 0b1100, 0b011>(),
CGDVAP = GenDCReg<0b011, 0b1100, 0b101>(),
CGVADP = GenDCReg<0b011, 0b1101, 0b011>(),
CGDVADP = GenDCReg<0b011, 0b1101, 0b101>(),
CIGVAC = GenDCReg<0b011, 0b1110, 0b011>(),
CIGDVAC = GenDCReg<0b011, 0b1110, 0b101>(),
GVA = GenDCReg<0b011, 0b0100, 0b011>,
GZVA = GenDCReg<0b011, 0b0100, 0b100>,
CGVAC = GenDCReg<0b011, 0b1010, 0b011>,
CGDVAC = GenDCReg<0b011, 0b1010, 0b101>,
CGVAP = GenDCReg<0b011, 0b1100, 0b011>,
CGDVAP = GenDCReg<0b011, 0b1100, 0b101>,
CGVADP = GenDCReg<0b011, 0b1101, 0b011>,
CGDVADP = GenDCReg<0b011, 0b1101, 0b101>,
CIGVAC = GenDCReg<0b011, 0b1110, 0b011>,
CIGDVAC = GenDCReg<0b011, 0b1110, 0b101>,
// DPB
CVAP = GenDCReg<0b011, 0b1100, 0b001>(),
CVAP = GenDCReg<0b011, 0b1100, 0b001>,
// DPB2
CVADP = GenDCReg<0b011, 0b1101, 0b001>(),
CVADP = GenDCReg<0b011, 0b1101, 0b001>,
};
template<uint32_t CRm, uint32_t op2>
constexpr uint32_t GenHintBarrierReg() {
return CRm << 8 | op2 << 5;
}
inline constexpr uint32_t GenHintBarrierReg = CRm << 8 | op2 << 5;
// This `HintRegister` enum is used for the hint instruction.
enum class HintRegister : uint32_t {
NOP = GenHintBarrierReg<0b0000, 0b000>(),
YIELD = GenHintBarrierReg<0b0000, 0b001>(),
WFE = GenHintBarrierReg<0b0000, 0b010>(),
WFI = GenHintBarrierReg<0b0000, 0b011>(),
SEV = GenHintBarrierReg<0b0000, 0b100>(),
SEVL = GenHintBarrierReg<0b0000, 0b101>(),
DGH = GenHintBarrierReg<0b0000, 0b110>(),
CSDB = GenHintBarrierReg<0b0010, 0b100>(),
NOP = GenHintBarrierReg<0b0000, 0b000>,
YIELD = GenHintBarrierReg<0b0000, 0b001>,
WFE = GenHintBarrierReg<0b0000, 0b010>,
WFI = GenHintBarrierReg<0b0000, 0b011>,
SEV = GenHintBarrierReg<0b0000, 0b100>,
SEVL = GenHintBarrierReg<0b0000, 0b101>,
DGH = GenHintBarrierReg<0b0000, 0b110>,
CSDB = GenHintBarrierReg<0b0010, 0b100>,
};
// This `BarrierRegister` enum is used for the various barrier instructions.
enum class BarrierRegister : uint32_t {
CLREX = GenHintBarrierReg<0b0000, 0b010>(),
TCOMMIT = GenHintBarrierReg<0b0000, 0b011>(),
DSB = GenHintBarrierReg<0b0000, 0b100>(),
DMB = GenHintBarrierReg<0b0000, 0b101>(),
ISB = GenHintBarrierReg<0b0000, 0b110>(),
SB = GenHintBarrierReg<0b0000, 0b111>(),
CLREX = GenHintBarrierReg<0b0000, 0b010>,
TCOMMIT = GenHintBarrierReg<0b0000, 0b011>,
DSB = GenHintBarrierReg<0b0000, 0b100>,
DMB = GenHintBarrierReg<0b0000, 0b101>,
ISB = GenHintBarrierReg<0b0000, 0b110>,
SB = GenHintBarrierReg<0b0000, 0b111>,
};
// This `BarrierScope` enum is used for the dsb/dmb instructions.
@@ -512,7 +507,7 @@ enum class SVEFMaxMinImm : uint32_t {
_1_0,
};
/* This `BackwardLabel` struct used for retaining a location for PC-Relative instructions.
/* This `BackwardLabel` struct is used for retaining a location for PC-Relative instructions.
* This is specifically a label for a target that is logically `below` an instruction that uses it.
* Which means that a branch would jump backwards.
*/
@@ -520,13 +515,11 @@ struct BackwardLabel {
uint8_t* Location {};
};
/* This `SingleUseForwardLabel` struct used for retaining a location for PC-Relative instructions.
/* This `ForwardLabel` struct is used for retaining a location for PC-Relative instructions.
* This is specifically a label for a target that is logically `above` an instruction that uses it.
* Which means that a branch would jump forwards.
*
* The `ForwardLabel` struct can be bound to multiple instructions, so it needs a vector for each bind instruction type.
*/
struct SingleUseForwardLabel {
struct ForwardLabel {
enum class InstType {
UNKNOWN,
ADR,
@@ -537,12 +530,16 @@ struct SingleUseForwardLabel {
RELATIVE_LOAD,
LONG_ADDRESS_GEN,
};
uint8_t* Location {};
InstType Type = InstType::UNKNOWN;
};
struct ForwardLabel {
fextl::vector<SingleUseForwardLabel> Insts {};
struct Reference {
uint8_t* Location {};
InstType Type = InstType::UNKNOWN;
};
// The first element is stored separately to avoid allocations for simple cases
Reference FirstInst;
fextl::vector<Reference> Insts;
};
/* This `BiDirectionalLabel` struct used for retaining a location for PC-Relative instructions.
@@ -554,14 +551,12 @@ struct BiDirectionalLabel {
ForwardLabel Forward;
};
static inline void AddLocationToLabel(SingleUseForwardLabel* Label, SingleUseForwardLabel&& Location) {
LOGMAN_THROW_A_FMT(Label->Type == SingleUseForwardLabel::InstType::UNKNOWN, "Trying to bind a SingleUseForwardLabel to multiple "
"locations. Use ForwardLabel instead.");
*Label = std::move(Location);
}
static inline void AddLocationToLabel(ForwardLabel* Label, SingleUseForwardLabel&& Location) {
Label->Insts.emplace_back(std::move(Location));
static inline void AddLocationToLabel(ForwardLabel* Label, ForwardLabel::Reference&& Location) {
if (Label->FirstInst.Location == nullptr) {
Label->FirstInst = Location;
} else {
Label->Insts.push_back(Location);
}
}
// Some FCMA ASIMD instructions support a rotation argument.
@@ -630,15 +625,15 @@ public:
// Bind a backward label to an address.
// Address that is bound is the current emitter location.
void Bind(BackwardLabel* Label) {
LOGMAN_THROW_AA_FMT(Label->Location == nullptr, "Trying to bind a label twice");
LOGMAN_THROW_A_FMT(Label->Location == nullptr, "Trying to bind a label twice");
Label->Location = GetCursorAddress<uint8_t*>();
}
void Bind(const SingleUseForwardLabel* Label) {
void Bind(const ForwardLabel::Reference* Label) {
uint8_t* CurrentAddress = GetCursorAddress<uint8_t*>();
// Patch up the instructions
switch (Label->Type) {
case SingleUseForwardLabel::InstType::ADR: {
case ForwardLabel::InstType::ADR: {
uint32_t* Instruction = reinterpret_cast<uint32_t*>(Label->Location);
int64_t Imm = reinterpret_cast<int64_t>(CurrentAddress) - reinterpret_cast<int64_t>(Instruction);
LOGMAN_THROW_A_FMT(IsADRRange(Imm), "Unscaled offset too large");
@@ -650,7 +645,7 @@ public:
*Instruction = Inst;
break;
}
case SingleUseForwardLabel::InstType::ADRP: {
case ForwardLabel::InstType::ADRP: {
uint32_t* Instruction = reinterpret_cast<uint32_t*>(Label->Location);
int64_t Imm = reinterpret_cast<int64_t>(CurrentAddress) - reinterpret_cast<int64_t>(Instruction);
LOGMAN_THROW_A_FMT(IsADRPRange(Imm) && IsADRPAligned(Imm), "Unscaled offset too large");
@@ -664,7 +659,7 @@ public:
break;
}
case SingleUseForwardLabel::InstType::B: {
case ForwardLabel::InstType::B: {
uint32_t* Instruction = reinterpret_cast<uint32_t*>(Label->Location);
int64_t Imm = reinterpret_cast<int64_t>(CurrentAddress) - reinterpret_cast<int64_t>(Instruction);
LOGMAN_THROW_A_FMT(Imm >= -134217728 && Imm <= 134217724 && ((Imm & 0b11) == 0), "Unscaled offset too large");
@@ -678,7 +673,7 @@ public:
break;
}
case SingleUseForwardLabel::InstType::TEST_BRANCH: {
case ForwardLabel::InstType::TEST_BRANCH: {
uint32_t* Instruction = reinterpret_cast<uint32_t*>(Label->Location);
int64_t Imm = reinterpret_cast<int64_t>(CurrentAddress) - reinterpret_cast<int64_t>(Instruction);
LOGMAN_THROW_A_FMT(Imm >= -32768 && Imm <= 32764 && ((Imm & 0b11) == 0), "Unscaled offset too large");
@@ -691,8 +686,8 @@ public:
break;
}
case SingleUseForwardLabel::InstType::BC:
case SingleUseForwardLabel::InstType::RELATIVE_LOAD: {
case ForwardLabel::InstType::BC:
case ForwardLabel::InstType::RELATIVE_LOAD: {
uint32_t* Instruction = reinterpret_cast<uint32_t*>(Label->Location);
int64_t Imm = reinterpret_cast<int64_t>(CurrentAddress) - reinterpret_cast<int64_t>(Instruction);
LOGMAN_THROW_A_FMT(Imm >= -1048576 && Imm <= 1048575 && ((Imm & 0b11) == 0), "Unscaled offset too large");
@@ -704,7 +699,7 @@ public:
*Instruction = Inst;
break;
}
case SingleUseForwardLabel::InstType::LONG_ADDRESS_GEN: {
case ForwardLabel::InstType::LONG_ADDRESS_GEN: {
uint32_t* Instructions = reinterpret_cast<uint32_t*>(Label->Location);
int64_t ImmInstOne = reinterpret_cast<int64_t>(CurrentAddress) - reinterpret_cast<int64_t>(&Instructions[0]);
int64_t ImmInstTwo = reinterpret_cast<int64_t>(CurrentAddress) - reinterpret_cast<int64_t>(&Instructions[1]);
@@ -749,10 +744,9 @@ public:
// Bind a forward label to a location.
// This walks all the instructions in the label's vector.
// Then backpatching all instructions that have used the label.
template<bool WarnAboutEmpty = false>
void Bind(ForwardLabel* Label) {
if constexpr (WarnAboutEmpty) {
LOGMAN_THROW_A_FMT(Label->Insts.empty() == false, "Binding forward label that didn't have any instructions using it");
if (Label->FirstInst.Location) {
Bind(&Label->FirstInst);
}
for (auto& Inst : Label->Insts) {
Bind(&Inst);
@@ -765,12 +759,18 @@ public:
if (!Label->Backward.Location) {
Bind(&Label->Backward);
}
Bind<false>(&Label->Forward);
Bind(&Label->Forward);
}
#include <CodeEmitter/VixlUtils.inl>
public:
// This symbol is used to allow external tooling (IDEs, clang-format, ...) to process the included files individually:
// If defined, the files will inject member functions into this class.
// If not, the files will wrap the member functions in a class so that tooling will process them properly.
#define INCLUDED_BY_EMITTER
// TODO: Implement SME when it matters.
#include <CodeEmitter/ALUOps.inl>
#include <CodeEmitter/BranchOps.inl>
@@ -780,7 +780,9 @@ public:
#include <CodeEmitter/ASIMDOps.inl>
#include <CodeEmitter/SVEOps.inl>
private:
#undef INCLUDED_BY_EMITTER
protected:
template<typename T>
uint32_t Encode_ra(T Reg) const {
return Reg.Idx() << 10;
@@ -792,7 +794,6 @@ private:
uint32_t Encode_rt2(T Reg) const {
return Reg.Idx() << 10;
}
template<>
uint32_t Encode_rt2(uint32_t Reg) const {
return Reg << 10;
}
@@ -828,7 +829,6 @@ private:
uint32_t Encode_rt(T Reg) const {
return Reg.Idx();
}
template<>
uint32_t Encode_rt(Prefetch Reg) const {
return FEXCore::ToUnderlying(Reg);
}
File diff suppressed because it is too large. Load diff
File diff suppressed because it is too large. Load diff
+160 -210
View File
@@ -16,17 +16,25 @@
* Exceptions to this rule will have asserts in the emitter implementation when misused.
*
*/
#pragma once
#ifndef INCLUDED_BY_EMITTER
#include <CodeEmitter/Emitter.h>
namespace ARMEmitter {
struct EmitterOps : Emitter {
#endif
public:
// Advanced SIMD scalar copy
// Advanced SIMD scalar copy
void dup(ScalarRegSize size, VRegister rd, VRegister rn, uint32_t Index) {
constexpr uint32_t Op = 0b0101'1110'0000'0000'0000'01 << 10;
const uint32_t SizeImm = FEXCore::ToUnderlying(size);
const uint32_t IndexShift = SizeImm + 1;
const uint32_t ElementSize = 1U << SizeImm;
const uint32_t MaxIndex = 128U / (ElementSize * 8);
[[maybe_unused]] const uint32_t MaxIndex = 128U / (ElementSize * 8);
LOGMAN_THROW_AA_FMT(Index < MaxIndex, "Index too large. Index={}, Max Index: {}", Index, MaxIndex);
LOGMAN_THROW_A_FMT(Index < MaxIndex, "Index too large. Index={}, Max Index: {}", Index, MaxIndex);
const uint32_t imm5 = (Index << IndexShift) | ElementSize;
@@ -37,7 +45,7 @@ public:
dup(size, rd, rn, Index);
}
// Advanced SIMD scalar three same FP16
// Advanced SIMD scalar three same FP16
void fmulx(HRegister rd, HRegister rn, HRegister rm) {
ASIMDScalarThreeSameFP16(0, 0, 0b011, rm, rn, rd);
}
@@ -66,7 +74,7 @@ public:
ASIMDScalarThreeSameFP16(1, 1, 0b101, rm, rn, rd);
}
// Advanced SIMD scalar two-register miscellaneous FP16
// Advanced SIMD scalar two-register miscellaneous FP16
void fcvtns(HRegister rd, HRegister rn) {
ASIMDScalarTwoRegMiscFP16(0, 0, 0b11010, rn, rd);
}
@@ -128,9 +136,9 @@ public:
ASIMDScalarTwoRegMiscFP16(1, 1, 0b11101, rn, rd);
}
// Advanced SIMD scalar three same extra
// XXX:
// Advanced SIMD scalar two-register miscellaneous
// Advanced SIMD scalar three same extra
// XXX:
// Advanced SIMD scalar two-register miscellaneous
void suqadd(ScalarRegSize size, VRegister rd, VRegister rn) {
ASIMDScalar2RegMisc(0, 0, size, 0b00011, rd, rn);
}
@@ -140,67 +148,55 @@ public:
///< Comparison against 0.0
void cmgt(ScalarRegSize size, VRegister rd, VRegister rn) {
LOGMAN_THROW_AA_FMT(size == ScalarRegSize::i64Bit, "Only supports 64-bit");
LOGMAN_THROW_A_FMT(size == ScalarRegSize::i64Bit, "Only supports 64-bit");
ASIMDScalar2RegMisc(0, 0, size, 0b01000, rd, rn);
}
///< Comparison against 0.0
void cmeq(ScalarRegSize size, VRegister rd, VRegister rn) {
LOGMAN_THROW_AA_FMT(size == ScalarRegSize::i64Bit, "Only supports 64-bit");
LOGMAN_THROW_A_FMT(size == ScalarRegSize::i64Bit, "Only supports 64-bit");
ASIMDScalar2RegMisc(0, 0, size, 0b01001, rd, rn);
}
///< Comparison against 0.0
void cmlt(ScalarRegSize size, VRegister rd, VRegister rn) {
LOGMAN_THROW_AA_FMT(size == ScalarRegSize::i64Bit, "Only supports 64-bit");
LOGMAN_THROW_A_FMT(size == ScalarRegSize::i64Bit, "Only supports 64-bit");
ASIMDScalar2RegMisc(0, 0, size, 0b01010, rd, rn);
}
void abs(ScalarRegSize size, VRegister rd, VRegister rn) {
LOGMAN_THROW_AA_FMT(size == ScalarRegSize::i64Bit, "Only supports 64-bit");
LOGMAN_THROW_A_FMT(size == ScalarRegSize::i64Bit, "Only supports 64-bit");
ASIMDScalar2RegMisc(0, 0, size, 0b01011, rd, rn);
}
///< size is destination size.
void sqxtn(ScalarRegSize size, VRegister rd, VRegister rn) {
LOGMAN_THROW_AA_FMT(size != ScalarRegSize::i64Bit, "64-bit destination not supported");
LOGMAN_THROW_A_FMT(size != ScalarRegSize::i64Bit, "64-bit destination not supported");
ASIMDScalar2RegMisc(0, 0, size, 0b10100, rd, rn);
}
void fcvtns(ScalarRegSize size, VRegister rd, VRegister rn) {
LOGMAN_THROW_A_FMT(size == ScalarRegSize::i64Bit || size == ScalarRegSize::i32Bit, "Invalid size selected for float convert");
const ScalarRegSize ConvertedSize =
size == ScalarRegSize::i64Bit ?
ScalarRegSize::i16Bit :
ScalarRegSize::i8Bit;
const ScalarRegSize ConvertedSize = size == ScalarRegSize::i64Bit ? ScalarRegSize::i16Bit : ScalarRegSize::i8Bit;
ASIMDScalar2RegMisc(0, 0, ConvertedSize, 0b11010, rd, rn);
}
void fcvtms(ScalarRegSize size, VRegister rd, VRegister rn) {
LOGMAN_THROW_A_FMT(size == ScalarRegSize::i64Bit || size == ScalarRegSize::i32Bit, "Invalid size selected for float convert");
const ScalarRegSize ConvertedSize =
size == ScalarRegSize::i64Bit ?
ScalarRegSize::i16Bit :
ScalarRegSize::i8Bit;
const ScalarRegSize ConvertedSize = size == ScalarRegSize::i64Bit ? ScalarRegSize::i16Bit : ScalarRegSize::i8Bit;
ASIMDScalar2RegMisc(0, 0, ConvertedSize, 0b11011, rd, rn);
}
void fcvtas(ScalarRegSize size, VRegister rd, VRegister rn) {
LOGMAN_THROW_A_FMT(size == ScalarRegSize::i64Bit || size == ScalarRegSize::i32Bit, "Invalid size selected for float convert");
const ScalarRegSize ConvertedSize =
size == ScalarRegSize::i64Bit ?
ScalarRegSize::i16Bit :
ScalarRegSize::i8Bit;
const ScalarRegSize ConvertedSize = size == ScalarRegSize::i64Bit ? ScalarRegSize::i16Bit : ScalarRegSize::i8Bit;
ASIMDScalar2RegMisc(0, 0, ConvertedSize, 0b11100, rd, rn);
}
void scvtf(ScalarRegSize size, VRegister rd, VRegister rn) {
LOGMAN_THROW_A_FMT(size == ScalarRegSize::i64Bit || size == ScalarRegSize::i32Bit, "Invalid size selected for float convert");
const ScalarRegSize ConvertedSize =
size == ScalarRegSize::i64Bit ?
ScalarRegSize::i16Bit :
ScalarRegSize::i8Bit;
const ScalarRegSize ConvertedSize = size == ScalarRegSize::i64Bit ? ScalarRegSize::i16Bit : ScalarRegSize::i8Bit;
ASIMDScalar2RegMisc(0, 0, ConvertedSize, 0b11101, rd, rn);
}
@@ -249,70 +245,58 @@ public:
}
///< Comparison against 0.0
void cmge(ScalarRegSize size, VRegister rd, VRegister rn) {
LOGMAN_THROW_AA_FMT(size == ScalarRegSize::i64Bit, "Only supports 64-bit");
LOGMAN_THROW_A_FMT(size == ScalarRegSize::i64Bit, "Only supports 64-bit");
ASIMDScalar2RegMisc(0, 1, size, 0b01000, rd, rn);
}
///< Comparison against 0.0
void cmle(ScalarRegSize size, VRegister rd, VRegister rn) {
LOGMAN_THROW_AA_FMT(size == ScalarRegSize::i64Bit, "Only supports 64-bit");
LOGMAN_THROW_A_FMT(size == ScalarRegSize::i64Bit, "Only supports 64-bit");
ASIMDScalar2RegMisc(0, 1, size, 0b01001, rd, rn);
}
void neg(ScalarRegSize size, VRegister rd, VRegister rn) {
LOGMAN_THROW_AA_FMT(size == ScalarRegSize::i64Bit, "Only supports 64-bit");
LOGMAN_THROW_A_FMT(size == ScalarRegSize::i64Bit, "Only supports 64-bit");
ASIMDScalar2RegMisc(0, 1, size, 0b01011, rd, rn);
}
///< size is destination.
void sqxtun(ScalarRegSize size, VRegister rd, VRegister rn) {
LOGMAN_THROW_AA_FMT(size != ScalarRegSize::i64Bit, "64-bit destination not supported");
LOGMAN_THROW_A_FMT(size != ScalarRegSize::i64Bit, "64-bit destination not supported");
ASIMDScalar2RegMisc(0, 1, size, 0b10010, rd, rn);
}
///< size is destination.
void uqxtn(ScalarRegSize size, VRegister rd, VRegister rn) {
LOGMAN_THROW_AA_FMT(size != ScalarRegSize::i64Bit, "64-bit destination not supported");
LOGMAN_THROW_A_FMT(size != ScalarRegSize::i64Bit, "64-bit destination not supported");
ASIMDScalar2RegMisc(0, 1, size, 0b10100, rd, rn);
}
///< size is destination.
void fcvtxn(ScalarRegSize size, VRegister rd, VRegister rn) {
LOGMAN_THROW_AA_FMT(size == ScalarRegSize::i32Bit, "Invalid size selected for float convert");
LOGMAN_THROW_A_FMT(size == ScalarRegSize::i32Bit, "Invalid size selected for float convert");
ASIMDScalar2RegMisc(0, 1, ScalarRegSize::i16Bit, 0b10110, rd, rn);
}
void fcvtnu(ScalarRegSize size, VRegister rd, VRegister rn) {
LOGMAN_THROW_A_FMT(size == ScalarRegSize::i64Bit || size == ScalarRegSize::i32Bit, "Invalid size selected for float convert");
const ScalarRegSize ConvertedSize =
size == ScalarRegSize::i64Bit ?
ScalarRegSize::i16Bit :
ScalarRegSize::i8Bit;
const ScalarRegSize ConvertedSize = size == ScalarRegSize::i64Bit ? ScalarRegSize::i16Bit : ScalarRegSize::i8Bit;
ASIMDScalar2RegMisc(0, 1, ConvertedSize, 0b11010, rd, rn);
}
void fcvtmu(ScalarRegSize size, VRegister rd, VRegister rn) {
LOGMAN_THROW_A_FMT(size == ScalarRegSize::i64Bit || size == ScalarRegSize::i32Bit, "Invalid size selected for float convert");
const ScalarRegSize ConvertedSize =
size == ScalarRegSize::i64Bit ?
ScalarRegSize::i16Bit :
ScalarRegSize::i8Bit;
const ScalarRegSize ConvertedSize = size == ScalarRegSize::i64Bit ? ScalarRegSize::i16Bit : ScalarRegSize::i8Bit;
ASIMDScalar2RegMisc(0, 1, ConvertedSize, 0b11011, rd, rn);
}
void fcvtau(ScalarRegSize size, VRegister rd, VRegister rn) {
LOGMAN_THROW_A_FMT(size == ScalarRegSize::i64Bit || size == ScalarRegSize::i32Bit, "Invalid size selected for float convert");
const ScalarRegSize ConvertedSize =
size == ScalarRegSize::i64Bit ?
ScalarRegSize::i16Bit :
ScalarRegSize::i8Bit;
const ScalarRegSize ConvertedSize = size == ScalarRegSize::i64Bit ? ScalarRegSize::i16Bit : ScalarRegSize::i8Bit;
ASIMDScalar2RegMisc(0, 1, ConvertedSize, 0b11100, rd, rn);
}
void ucvtf(ScalarRegSize size, VRegister rd, VRegister rn) {
LOGMAN_THROW_A_FMT(size == ScalarRegSize::i64Bit || size == ScalarRegSize::i32Bit, "Invalid size selected for float convert");
const ScalarRegSize ConvertedSize =
size == ScalarRegSize::i64Bit ?
ScalarRegSize::i16Bit :
ScalarRegSize::i8Bit;
const ScalarRegSize ConvertedSize = size == ScalarRegSize::i64Bit ? ScalarRegSize::i16Bit : ScalarRegSize::i8Bit;
ASIMDScalar2RegMisc(0, 1, ConvertedSize, 0b11101, rd, rn);
}
@@ -366,73 +350,55 @@ public:
}
void fmaxnmp(ScalarRegSize size, VRegister rd, VRegister rn) {
LOGMAN_THROW_AA_FMT(size == ScalarRegSize::i64Bit || size == ScalarRegSize::i32Bit, "Invalid size selected for float convert");
LOGMAN_THROW_A_FMT(size == ScalarRegSize::i64Bit || size == ScalarRegSize::i32Bit, "Invalid size selected for float convert");
const ScalarRegSize ConvertedSize =
size == ScalarRegSize::i64Bit ?
ScalarRegSize::i16Bit :
ScalarRegSize::i8Bit;
const ScalarRegSize ConvertedSize = size == ScalarRegSize::i64Bit ? ScalarRegSize::i16Bit : ScalarRegSize::i8Bit;
ASIMDScalar2RegMisc(1, 1, ConvertedSize, 0b01100, rd, rn);
}
void faddp(ScalarRegSize size, VRegister rd, VRegister rn) {
LOGMAN_THROW_AA_FMT(size == ScalarRegSize::i64Bit || size == ScalarRegSize::i32Bit, "Invalid size selected for float convert");
LOGMAN_THROW_A_FMT(size == ScalarRegSize::i64Bit || size == ScalarRegSize::i32Bit, "Invalid size selected for float convert");
const ScalarRegSize ConvertedSize =
size == ScalarRegSize::i64Bit ?
ScalarRegSize::i16Bit :
ScalarRegSize::i8Bit;
const ScalarRegSize ConvertedSize = size == ScalarRegSize::i64Bit ? ScalarRegSize::i16Bit : ScalarRegSize::i8Bit;
ASIMDScalar2RegMisc(1, 1, ConvertedSize, 0b01101, rd, rn);
}
void fmaxp(ScalarRegSize size, VRegister rd, VRegister rn) {
LOGMAN_THROW_AA_FMT(size == ScalarRegSize::i64Bit || size == ScalarRegSize::i32Bit, "Invalid size selected for float convert");
LOGMAN_THROW_A_FMT(size == ScalarRegSize::i64Bit || size == ScalarRegSize::i32Bit, "Invalid size selected for float convert");
const ScalarRegSize ConvertedSize =
size == ScalarRegSize::i64Bit ?
ScalarRegSize::i16Bit :
ScalarRegSize::i8Bit;
const ScalarRegSize ConvertedSize = size == ScalarRegSize::i64Bit ? ScalarRegSize::i16Bit : ScalarRegSize::i8Bit;
ASIMDScalar2RegMisc(1, 1, ConvertedSize, 0b01111, rd, rn);
}
void fminnmp(ScalarRegSize size, VRegister rd, VRegister rn) {
LOGMAN_THROW_AA_FMT(size == ScalarRegSize::i64Bit || size == ScalarRegSize::i32Bit, "Invalid size selected for float convert");
LOGMAN_THROW_A_FMT(size == ScalarRegSize::i64Bit || size == ScalarRegSize::i32Bit, "Invalid size selected for float convert");
ASIMDScalar2RegMisc(1, 1, size, 0b01100, rd, rn);
}
void fminp(ScalarRegSize size, VRegister rd, VRegister rn) {
LOGMAN_THROW_AA_FMT(size == ScalarRegSize::i64Bit || size == ScalarRegSize::i32Bit, "Invalid size selected for float convert");
LOGMAN_THROW_A_FMT(size == ScalarRegSize::i64Bit || size == ScalarRegSize::i32Bit, "Invalid size selected for float convert");
ASIMDScalar2RegMisc(1, 1, size, 0b01111, rd, rn);
}
// Advanced SIMD scalar three different
// Advanced SIMD scalar three different
///< size is destination.
void sqdmlal(ScalarRegSize size, VRegister rd, VRegister rn, VRegister rm) {
LOGMAN_THROW_AA_FMT(size == ScalarRegSize::i64Bit || size == ScalarRegSize::i32Bit, "Invalid size selected for float convert");
const ScalarRegSize ConvertedSize =
size == ScalarRegSize::i64Bit ?
ScalarRegSize::i32Bit :
ScalarRegSize::i16Bit;
LOGMAN_THROW_A_FMT(size == ScalarRegSize::i64Bit || size == ScalarRegSize::i32Bit, "Invalid size selected for float convert");
const ScalarRegSize ConvertedSize = size == ScalarRegSize::i64Bit ? ScalarRegSize::i32Bit : ScalarRegSize::i16Bit;
ASIMD3RegDifferent(0, ConvertedSize, 0b1001, rd, rn, rm);
}
///< size is destination.
void sqdmlsl(ScalarRegSize size, VRegister rd, VRegister rn, VRegister rm) {
LOGMAN_THROW_AA_FMT(size == ScalarRegSize::i64Bit || size == ScalarRegSize::i32Bit, "Invalid size selected for float convert");
const ScalarRegSize ConvertedSize =
size == ScalarRegSize::i64Bit ?
ScalarRegSize::i32Bit :
ScalarRegSize::i16Bit;
LOGMAN_THROW_A_FMT(size == ScalarRegSize::i64Bit || size == ScalarRegSize::i32Bit, "Invalid size selected for float convert");
const ScalarRegSize ConvertedSize = size == ScalarRegSize::i64Bit ? ScalarRegSize::i32Bit : ScalarRegSize::i16Bit;
ASIMD3RegDifferent(0, ConvertedSize, 0b1011, rd, rn, rm);
}
///< size is destination.
void sqdmull(ScalarRegSize size, VRegister rd, VRegister rn, VRegister rm) {
LOGMAN_THROW_AA_FMT(size == ScalarRegSize::i64Bit || size == ScalarRegSize::i32Bit, "Invalid size selected for float convert");
const ScalarRegSize ConvertedSize =
size == ScalarRegSize::i64Bit ?
ScalarRegSize::i32Bit :
ScalarRegSize::i16Bit;
LOGMAN_THROW_A_FMT(size == ScalarRegSize::i64Bit || size == ScalarRegSize::i32Bit, "Invalid size selected for float convert");
const ScalarRegSize ConvertedSize = size == ScalarRegSize::i64Bit ? ScalarRegSize::i32Bit : ScalarRegSize::i16Bit;
ASIMD3RegDifferent(0, ConvertedSize, 0b1101, rd, rn, rm);
}
// Advanced SIMD scalar three same
// Advanced SIMD scalar three same
void sqadd(ScalarRegSize size, VRegister rd, VRegister rn, VRegister rm) {
ASIMD3RegSame(0, size, 0b00001, rd, rn, rm);
}
@@ -440,71 +406,62 @@ public:
ASIMD3RegSame(0, size, 0b00101, rd, rn, rm);
}
void cmgt(ScalarRegSize size, VRegister rd, VRegister rn, VRegister rm) {
LOGMAN_THROW_AA_FMT(size == ScalarRegSize::i64Bit, "Only supports 64-bit");
LOGMAN_THROW_A_FMT(size == ScalarRegSize::i64Bit, "Only supports 64-bit");
ASIMD3RegSame(0, size, 0b00110, rd, rn, rm);
}
void cmge(ScalarRegSize size, VRegister rd, VRegister rn, VRegister rm) {
LOGMAN_THROW_AA_FMT(size == ScalarRegSize::i64Bit, "Only supports 64-bit");
LOGMAN_THROW_A_FMT(size == ScalarRegSize::i64Bit, "Only supports 64-bit");
ASIMD3RegSame(0, size, 0b00111, rd, rn, rm);
}
void sshl(ScalarRegSize size, VRegister rd, VRegister rn, VRegister rm) {
LOGMAN_THROW_AA_FMT(size == ScalarRegSize::i64Bit, "Only supports 64-bit");
LOGMAN_THROW_A_FMT(size == ScalarRegSize::i64Bit, "Only supports 64-bit");
ASIMD3RegSame(0, size, 0b01000, rd, rn, rm);
}
void sqshl(ScalarRegSize size, VRegister rd, VRegister rn, VRegister rm) {
ASIMD3RegSame(0, size, 0b01001, rd, rn, rm);
}
void srshl(ScalarRegSize size, VRegister rd, VRegister rn, VRegister rm) {
LOGMAN_THROW_AA_FMT(size == ScalarRegSize::i64Bit, "Only supports 64-bit");
LOGMAN_THROW_A_FMT(size == ScalarRegSize::i64Bit, "Only supports 64-bit");
ASIMD3RegSame(0, size, 0b01010, rd, rn, rm);
}
void sqrshl(ScalarRegSize size, VRegister rd, VRegister rn, VRegister rm) {
ASIMD3RegSame(0, size, 0b01011, rd, rn, rm);
}
void add(ScalarRegSize size, VRegister rd, VRegister rn, VRegister rm) {
LOGMAN_THROW_AA_FMT(size == ScalarRegSize::i64Bit, "Only supports 64-bit");
LOGMAN_THROW_A_FMT(size == ScalarRegSize::i64Bit, "Only supports 64-bit");
ASIMD3RegSame(0, size, 0b10000, rd, rn, rm);
}
void cmtst(ScalarRegSize size, VRegister rd, VRegister rn, VRegister rm) {
LOGMAN_THROW_AA_FMT(size == ScalarRegSize::i64Bit, "Only supports 64-bit");
LOGMAN_THROW_A_FMT(size == ScalarRegSize::i64Bit, "Only supports 64-bit");
ASIMD3RegSame(0, size, 0b10001, rd, rn, rm);
}
void sqdmulh(ScalarRegSize size, VRegister rd, VRegister rn, VRegister rm) {
LOGMAN_THROW_AA_FMT(size == ScalarRegSize::i32Bit || size == ScalarRegSize::i16Bit, "Invalid size");
LOGMAN_THROW_A_FMT(size == ScalarRegSize::i32Bit || size == ScalarRegSize::i16Bit, "Invalid size");
ASIMD3RegSame(0, size, 0b10110, rd, rn, rm);
}
void fmulx(ScalarRegSize size, VRegister rd, VRegister rn, VRegister rm) {
LOGMAN_THROW_AA_FMT(size == ScalarRegSize::i64Bit || size == ScalarRegSize::i32Bit, "Invalid size selected for float convert");
LOGMAN_THROW_A_FMT(size == ScalarRegSize::i64Bit || size == ScalarRegSize::i32Bit, "Invalid size selected for float convert");
const ScalarRegSize ConvertedSize =
size == ScalarRegSize::i64Bit ?
ScalarRegSize::i16Bit :
ScalarRegSize::i8Bit;
const ScalarRegSize ConvertedSize = size == ScalarRegSize::i64Bit ? ScalarRegSize::i16Bit : ScalarRegSize::i8Bit;
ASIMD3RegSame(0, ConvertedSize, 0b11011, rd, rn, rm);
}
void fcmeq(ScalarRegSize size, VRegister rd, VRegister rn, VRegister rm) {
LOGMAN_THROW_AA_FMT(size == ScalarRegSize::i64Bit || size == ScalarRegSize::i32Bit, "Invalid size selected for float convert");
LOGMAN_THROW_A_FMT(size == ScalarRegSize::i64Bit || size == ScalarRegSize::i32Bit, "Invalid size selected for float convert");
const ScalarRegSize ConvertedSize =
size == ScalarRegSize::i64Bit ?
ScalarRegSize::i16Bit :
ScalarRegSize::i8Bit;
const ScalarRegSize ConvertedSize = size == ScalarRegSize::i64Bit ? ScalarRegSize::i16Bit : ScalarRegSize::i8Bit;
ASIMD3RegSame(0, ConvertedSize, 0b11100, rd, rn, rm);
}
void frecps(ScalarRegSize size, VRegister rd, VRegister rn, VRegister rm) {
LOGMAN_THROW_AA_FMT(size == ScalarRegSize::i64Bit || size == ScalarRegSize::i32Bit, "Invalid size selected for float convert");
LOGMAN_THROW_A_FMT(size == ScalarRegSize::i64Bit || size == ScalarRegSize::i32Bit, "Invalid size selected for float convert");
const ScalarRegSize ConvertedSize =
size == ScalarRegSize::i64Bit ?
ScalarRegSize::i16Bit :
ScalarRegSize::i8Bit;
const ScalarRegSize ConvertedSize = size == ScalarRegSize::i64Bit ? ScalarRegSize::i16Bit : ScalarRegSize::i8Bit;
ASIMD3RegSame(0, ConvertedSize, 0b11111, rd, rn, rm);
}
void frsqrts(ScalarRegSize size, VRegister rd, VRegister rn, VRegister rm) {
LOGMAN_THROW_AA_FMT(size == ScalarRegSize::i64Bit || size == ScalarRegSize::i32Bit, "Invalid size selected for float convert");
LOGMAN_THROW_A_FMT(size == ScalarRegSize::i64Bit || size == ScalarRegSize::i32Bit, "Invalid size selected for float convert");
ASIMD3RegSame(0, size, 0b11111, rd, rn, rm);
}
void uqadd(ScalarRegSize size, VRegister rd, VRegister rn, VRegister rm) {
@@ -514,75 +471,69 @@ public:
ASIMD3RegSame(1, size, 0b00101, rd, rn, rm);
}
void cmhi(ScalarRegSize size, VRegister rd, VRegister rn, VRegister rm) {
LOGMAN_THROW_AA_FMT(size == ScalarRegSize::i64Bit, "Only supports 64-bit");
LOGMAN_THROW_A_FMT(size == ScalarRegSize::i64Bit, "Only supports 64-bit");
ASIMD3RegSame(1, size, 0b00110, rd, rn, rm);
}
void cmhs(ScalarRegSize size, VRegister rd, VRegister rn, VRegister rm) {
LOGMAN_THROW_AA_FMT(size == ScalarRegSize::i64Bit, "Only supports 64-bit");
LOGMAN_THROW_A_FMT(size == ScalarRegSize::i64Bit, "Only supports 64-bit");
ASIMD3RegSame(1, size, 0b00111, rd, rn, rm);
}
void ushl(ScalarRegSize size, VRegister rd, VRegister rn, VRegister rm) {
LOGMAN_THROW_AA_FMT(size == ScalarRegSize::i64Bit, "Only supports 64-bit");
LOGMAN_THROW_A_FMT(size == ScalarRegSize::i64Bit, "Only supports 64-bit");
ASIMD3RegSame(1, size, 0b01000, rd, rn, rm);
}
void uqshl(ScalarRegSize size, VRegister rd, VRegister rn, VRegister rm) {
ASIMD3RegSame(1, size, 0b01001, rd, rn, rm);
}
void urshl(ScalarRegSize size, VRegister rd, VRegister rn, VRegister rm) {
LOGMAN_THROW_AA_FMT(size == ScalarRegSize::i64Bit, "Only supports 64-bit");
LOGMAN_THROW_A_FMT(size == ScalarRegSize::i64Bit, "Only supports 64-bit");
ASIMD3RegSame(1, size, 0b01010, rd, rn, rm);
}
void uqrshl(ScalarRegSize size, VRegister rd, VRegister rn, VRegister rm) {
ASIMD3RegSame(1, size, 0b01011, rd, rn, rm);
}
void sub(ScalarRegSize size, VRegister rd, VRegister rn, VRegister rm) {
LOGMAN_THROW_AA_FMT(size == ScalarRegSize::i64Bit, "Only supports 64-bit");
LOGMAN_THROW_A_FMT(size == ScalarRegSize::i64Bit, "Only supports 64-bit");
ASIMD3RegSame(1, size, 0b10000, rd, rn, rm);
}
void cmeq(ScalarRegSize size, VRegister rd, VRegister rn, VRegister rm) {
LOGMAN_THROW_AA_FMT(size == ScalarRegSize::i64Bit, "Only supports 64-bit");
LOGMAN_THROW_A_FMT(size == ScalarRegSize::i64Bit, "Only supports 64-bit");
ASIMD3RegSame(1, size, 0b10001, rd, rn, rm);
}
void sqrdmulh(ScalarRegSize size, VRegister rd, VRegister rn, VRegister rm) {
LOGMAN_THROW_AA_FMT(size == ScalarRegSize::i32Bit || size == ScalarRegSize::i16Bit, "Invalid size");
LOGMAN_THROW_A_FMT(size == ScalarRegSize::i32Bit || size == ScalarRegSize::i16Bit, "Invalid size");
ASIMD3RegSame(1, size, 0b10110, rd, rn, rm);
}
void fcmge(ScalarRegSize size, VRegister rd, VRegister rn, VRegister rm) {
LOGMAN_THROW_AA_FMT(size == ScalarRegSize::i64Bit || size == ScalarRegSize::i32Bit, "Invalid size selected for float convert");
LOGMAN_THROW_A_FMT(size == ScalarRegSize::i64Bit || size == ScalarRegSize::i32Bit, "Invalid size selected for float convert");
const ScalarRegSize ConvertedSize =
size == ScalarRegSize::i64Bit ?
ScalarRegSize::i16Bit :
ScalarRegSize::i8Bit;
const ScalarRegSize ConvertedSize = size == ScalarRegSize::i64Bit ? ScalarRegSize::i16Bit : ScalarRegSize::i8Bit;
ASIMD3RegSame(1, ConvertedSize, 0b11100, rd, rn, rm);
}
void facge(ScalarRegSize size, VRegister rd, VRegister rn, VRegister rm) {
LOGMAN_THROW_AA_FMT(size == ScalarRegSize::i64Bit || size == ScalarRegSize::i32Bit, "Invalid size selected for float convert");
LOGMAN_THROW_A_FMT(size == ScalarRegSize::i64Bit || size == ScalarRegSize::i32Bit, "Invalid size selected for float convert");
const ScalarRegSize ConvertedSize =
size == ScalarRegSize::i64Bit ?
ScalarRegSize::i16Bit :
ScalarRegSize::i8Bit;
const ScalarRegSize ConvertedSize = size == ScalarRegSize::i64Bit ? ScalarRegSize::i16Bit : ScalarRegSize::i8Bit;
ASIMD3RegSame(1, ConvertedSize, 0b11101, rd, rn, rm);
}
void fabd(ScalarRegSize size, VRegister rd, VRegister rn, VRegister rm) {
LOGMAN_THROW_AA_FMT(size == ScalarRegSize::i64Bit || size == ScalarRegSize::i32Bit, "Invalid size selected for float convert");
LOGMAN_THROW_A_FMT(size == ScalarRegSize::i64Bit || size == ScalarRegSize::i32Bit, "Invalid size selected for float convert");
ASIMD3RegSame(1, size, 0b11010, rd, rn, rm);
}
void fcmgt(ScalarRegSize size, VRegister rd, VRegister rn, VRegister rm) {
LOGMAN_THROW_AA_FMT(size == ScalarRegSize::i64Bit || size == ScalarRegSize::i32Bit, "Invalid size selected for float convert");
LOGMAN_THROW_A_FMT(size == ScalarRegSize::i64Bit || size == ScalarRegSize::i32Bit, "Invalid size selected for float convert");
ASIMD3RegSame(1, size, 0b11100, rd, rn, rm);
}
void facgt(ScalarRegSize size, VRegister rd, VRegister rn, VRegister rm) {
LOGMAN_THROW_AA_FMT(size == ScalarRegSize::i64Bit || size == ScalarRegSize::i32Bit, "Invalid size selected for float convert");
LOGMAN_THROW_A_FMT(size == ScalarRegSize::i64Bit || size == ScalarRegSize::i32Bit, "Invalid size selected for float convert");
ASIMD3RegSame(1, size, 0b11101, rd, rn, rm);
}
// Advanced SIMD scalar shift by immediate
// Advanced SIMD scalar shift by immediate
void sshr(ScalarRegSize size, VRegister rd, VRegister rn, uint32_t Shift) {
LOGMAN_THROW_AA_FMT(Shift > 0 && Shift < 64, "Invalid shift for sshr");
LOGMAN_THROW_AA_FMT(size == ARMEmitter::ScalarRegSize::i64Bit, "Invalid size selected for sshr");
LOGMAN_THROW_A_FMT(Shift > 0 && Shift < 64, "Invalid shift for sshr");
LOGMAN_THROW_A_FMT(size == ARMEmitter::ScalarRegSize::i64Bit, "Invalid size selected for sshr");
const size_t SubregSizeInBits = ScalarRegSizeInBits(size);
// Shift encoded in immh:immb, but inverted with 128-bit source
// shift = (esize * 2) - immh:immb
@@ -592,8 +543,8 @@ public:
ASIMDScalarShiftByImm(0, immh, immb, 0b00000, rd, rn);
}
void ssra(ScalarRegSize size, VRegister rd, VRegister rn, uint32_t Shift) {
LOGMAN_THROW_AA_FMT(Shift > 0 && Shift < 64, "Invalid shift for sshr");
LOGMAN_THROW_AA_FMT(size == ARMEmitter::ScalarRegSize::i64Bit, "Invalid size selected for sshr");
LOGMAN_THROW_A_FMT(Shift > 0 && Shift < 64, "Invalid shift for sshr");
LOGMAN_THROW_A_FMT(size == ARMEmitter::ScalarRegSize::i64Bit, "Invalid size selected for sshr");
const size_t SubregSizeInBits = ScalarRegSizeInBits(size);
// Shift encoded in immh:immb, but inverted with 128-bit source
// shift = (esize * 2) - immh:immb
@@ -603,8 +554,8 @@ public:
ASIMDScalarShiftByImm(0, immh, immb, 0b00010, rd, rn);
}
void srshr(ScalarRegSize size, VRegister rd, VRegister rn, uint32_t Shift) {
LOGMAN_THROW_AA_FMT(Shift > 0 && Shift < 64, "Invalid shift for sshr");
LOGMAN_THROW_AA_FMT(size == ARMEmitter::ScalarRegSize::i64Bit, "Invalid size selected for sshr");
LOGMAN_THROW_A_FMT(Shift > 0 && Shift < 64, "Invalid shift for sshr");
LOGMAN_THROW_A_FMT(size == ARMEmitter::ScalarRegSize::i64Bit, "Invalid size selected for sshr");
const size_t SubregSizeInBits = ScalarRegSizeInBits(size);
// Shift encoded in immh:immb, but inverted with 128-bit source
// shift = (esize * 2) - immh:immb
@@ -614,8 +565,8 @@ public:
ASIMDScalarShiftByImm(0, immh, immb, 0b00100, rd, rn);
}
void srsra(ScalarRegSize size, VRegister rd, VRegister rn, uint32_t Shift) {
LOGMAN_THROW_AA_FMT(Shift > 0 && Shift < 64, "Invalid shift for sshr");
LOGMAN_THROW_AA_FMT(size == ARMEmitter::ScalarRegSize::i64Bit, "Invalid size selected for sshr");
LOGMAN_THROW_A_FMT(Shift > 0 && Shift < 64, "Invalid shift for sshr");
LOGMAN_THROW_A_FMT(size == ARMEmitter::ScalarRegSize::i64Bit, "Invalid size selected for sshr");
const size_t SubregSizeInBits = ScalarRegSizeInBits(size);
// Shift encoded in immh:immb, but inverted with 128-bit source
// shift = (esize * 2) - immh:immb
@@ -625,8 +576,8 @@ public:
ASIMDScalarShiftByImm(0, immh, immb, 0b00110, rd, rn);
}
void shl(ScalarRegSize size, VRegister rd, VRegister rn, uint32_t Shift) {
LOGMAN_THROW_AA_FMT(Shift > 0 && Shift < 64, "Invalid shift for sshr");
LOGMAN_THROW_AA_FMT(size == ARMEmitter::ScalarRegSize::i64Bit, "Invalid size selected for sshr");
LOGMAN_THROW_A_FMT(Shift > 0 && Shift < 64, "Invalid shift for sshr");
LOGMAN_THROW_A_FMT(size == ARMEmitter::ScalarRegSize::i64Bit, "Invalid size selected for sshr");
// Shift encoded a bit weirdly.
// shift = immh:immb - elementsize but immh is /also/ used for element size.
const uint32_t immh = 1 << FEXCore::ToUnderlying(size) | (Shift >> 3);
@@ -644,7 +595,7 @@ public:
///< size is destination
void sqshrn(ScalarRegSize size, VRegister rd, VRegister rn, uint32_t Shift) {
LOGMAN_THROW_A_FMT(Shift > 0 && Shift < ScalarRegSizeInBits(size), "Invalid shift for sshr");
LOGMAN_THROW_AA_FMT(size != ARMEmitter::ScalarRegSize::i64Bit, "Invalid size selected for sqshrn");
LOGMAN_THROW_A_FMT(size != ARMEmitter::ScalarRegSize::i64Bit, "Invalid size selected for sqshrn");
const size_t SubregSizeInBits = ScalarRegSizeInBits(size);
// Shift encoded in immh:immb, but inverted with 128-bit source
// shift = (esize * 2) - immh:immb
@@ -655,7 +606,7 @@ public:
}
void sqrshrn(ScalarRegSize size, VRegister rd, VRegister rn, uint32_t Shift) {
LOGMAN_THROW_A_FMT(Shift > 0 && Shift < ScalarRegSizeInBits(size), "Invalid shift for sshr");
LOGMAN_THROW_AA_FMT(size != ARMEmitter::ScalarRegSize::i64Bit, "Invalid size selected for sqshrn");
LOGMAN_THROW_A_FMT(size != ARMEmitter::ScalarRegSize::i64Bit, "Invalid size selected for sqshrn");
const size_t SubregSizeInBits = ScalarRegSizeInBits(size);
// Shift encoded in immh:immb, but inverted with 128-bit source
// shift = (esize * 2) - immh:immb
@@ -666,8 +617,8 @@ public:
}
// TODO: SCVTF, FCVTZS
void ushr(ScalarRegSize size, VRegister rd, VRegister rn, uint32_t Shift) {
LOGMAN_THROW_AA_FMT(Shift > 0 && Shift < 64, "Invalid shift for sshr");
LOGMAN_THROW_AA_FMT(size == ARMEmitter::ScalarRegSize::i64Bit, "Invalid size selected for sshr");
LOGMAN_THROW_A_FMT(Shift > 0 && Shift < 64, "Invalid shift for sshr");
LOGMAN_THROW_A_FMT(size == ARMEmitter::ScalarRegSize::i64Bit, "Invalid size selected for sshr");
const size_t SubregSizeInBits = ScalarRegSizeInBits(size);
// Shift encoded in immh:immb, but inverted with 128-bit source
// shift = (esize * 2) - immh:immb
@@ -677,8 +628,8 @@ public:
ASIMDScalarShiftByImm(1, immh, immb, 0b00000, rd, rn);
}
void usra(ScalarRegSize size, VRegister rd, VRegister rn, uint32_t Shift) {
LOGMAN_THROW_AA_FMT(Shift > 0 && Shift < 64, "Invalid shift for sshr");
LOGMAN_THROW_AA_FMT(size == ARMEmitter::ScalarRegSize::i64Bit, "Invalid size selected for sshr");
LOGMAN_THROW_A_FMT(Shift > 0 && Shift < 64, "Invalid shift for sshr");
LOGMAN_THROW_A_FMT(size == ARMEmitter::ScalarRegSize::i64Bit, "Invalid size selected for sshr");
const size_t SubregSizeInBits = ScalarRegSizeInBits(size);
// Shift encoded in immh:immb, but inverted with 128-bit source
// shift = (esize * 2) - immh:immb
@@ -688,8 +639,8 @@ public:
ASIMDScalarShiftByImm(1, immh, immb, 0b00010, rd, rn);
}
void urshr(ScalarRegSize size, VRegister rd, VRegister rn, uint32_t Shift) {
LOGMAN_THROW_AA_FMT(Shift > 0 && Shift < 64, "Invalid shift for sshr");
LOGMAN_THROW_AA_FMT(size == ARMEmitter::ScalarRegSize::i64Bit, "Invalid size selected for sshr");
LOGMAN_THROW_A_FMT(Shift > 0 && Shift < 64, "Invalid shift for sshr");
LOGMAN_THROW_A_FMT(size == ARMEmitter::ScalarRegSize::i64Bit, "Invalid size selected for sshr");
const size_t SubregSizeInBits = ScalarRegSizeInBits(size);
// Shift encoded in immh:immb, but inverted with 128-bit source
// shift = (esize * 2) - immh:immb
@@ -699,8 +650,8 @@ public:
ASIMDScalarShiftByImm(1, immh, immb, 0b00100, rd, rn);
}
void ursra(ScalarRegSize size, VRegister rd, VRegister rn, uint32_t Shift) {
LOGMAN_THROW_AA_FMT(Shift > 0 && Shift < 64, "Invalid shift for sshr");
LOGMAN_THROW_AA_FMT(size == ARMEmitter::ScalarRegSize::i64Bit, "Invalid size selected for sshr");
LOGMAN_THROW_A_FMT(Shift > 0 && Shift < 64, "Invalid shift for sshr");
LOGMAN_THROW_A_FMT(size == ARMEmitter::ScalarRegSize::i64Bit, "Invalid size selected for sshr");
const size_t SubregSizeInBits = ScalarRegSizeInBits(size);
// Shift encoded in immh:immb, but inverted with 128-bit source
// shift = (esize * 2) - immh:immb
@@ -710,8 +661,8 @@ public:
ASIMDScalarShiftByImm(1, immh, immb, 0b00110, rd, rn);
}
void sri(ScalarRegSize size, VRegister rd, VRegister rn, uint32_t Shift) {
LOGMAN_THROW_AA_FMT(Shift > 0 && Shift < 64, "Invalid shift for sshr");
LOGMAN_THROW_AA_FMT(size == ARMEmitter::ScalarRegSize::i64Bit, "Invalid size selected for sshr");
LOGMAN_THROW_A_FMT(Shift > 0 && Shift < 64, "Invalid shift for sshr");
LOGMAN_THROW_A_FMT(size == ARMEmitter::ScalarRegSize::i64Bit, "Invalid size selected for sshr");
const size_t SubregSizeInBits = ScalarRegSizeInBits(size);
// Shift encoded in immh:immb, but inverted with 128-bit source
// shift = (esize * 2) - immh:immb
@@ -721,8 +672,8 @@ public:
ASIMDScalarShiftByImm(1, immh, immb, 0b01000, rd, rn);
}
void sli(ScalarRegSize size, VRegister rd, VRegister rn, uint32_t Shift) {
LOGMAN_THROW_AA_FMT(Shift > 0 && Shift < 64, "Invalid shift for sshr");
LOGMAN_THROW_AA_FMT(size == ARMEmitter::ScalarRegSize::i64Bit, "Invalid size selected for sshr");
LOGMAN_THROW_A_FMT(Shift > 0 && Shift < 64, "Invalid shift for sshr");
LOGMAN_THROW_A_FMT(size == ARMEmitter::ScalarRegSize::i64Bit, "Invalid size selected for sshr");
// Shift encoded a bit weirdly.
// shift = immh:immb - elementsize but immh is /also/ used for element size.
const uint32_t immh = 1 << FEXCore::ToUnderlying(size) | (Shift >> 3);
@@ -748,7 +699,7 @@ public:
///< size is destination.
void sqshrun(ScalarRegSize size, VRegister rd, VRegister rn, uint32_t Shift) {
LOGMAN_THROW_A_FMT(Shift > 0 && Shift < ScalarRegSizeInBits(size), "Invalid shift for sshr");
LOGMAN_THROW_AA_FMT(size != ARMEmitter::ScalarRegSize::i64Bit, "Invalid size selected for sqshrun");
LOGMAN_THROW_A_FMT(size != ARMEmitter::ScalarRegSize::i64Bit, "Invalid size selected for sqshrun");
const size_t SubregSizeInBits = ScalarRegSizeInBits(size);
// Shift encoded in immh:immb, but inverted with 128-bit source
// shift = (esize * 2) - immh:immb
@@ -760,7 +711,7 @@ public:
///< size is destination.
void sqrshrun(ScalarRegSize size, VRegister rd, VRegister rn, uint32_t Shift) {
LOGMAN_THROW_A_FMT(Shift > 0 && Shift < ScalarRegSizeInBits(size), "Invalid shift for sshr");
LOGMAN_THROW_AA_FMT(size != ARMEmitter::ScalarRegSize::i64Bit, "Invalid size selected for sqrshrun");
LOGMAN_THROW_A_FMT(size != ARMEmitter::ScalarRegSize::i64Bit, "Invalid size selected for sqrshrun");
const size_t SubregSizeInBits = ScalarRegSizeInBits(size);
// Shift encoded in immh:immb, but inverted with 128-bit source
// shift = (esize * 2) - immh:immb
@@ -772,7 +723,7 @@ public:
///< size is destination.
void uqshrn(ScalarRegSize size, VRegister rd, VRegister rn, uint32_t Shift) {
LOGMAN_THROW_A_FMT(Shift > 0 && Shift < ScalarRegSizeInBits(size), "Invalid shift for sshr");
LOGMAN_THROW_AA_FMT(size != ARMEmitter::ScalarRegSize::i64Bit, "Invalid size selected for sqrshrun");
LOGMAN_THROW_A_FMT(size != ARMEmitter::ScalarRegSize::i64Bit, "Invalid size selected for sqrshrun");
const size_t SubregSizeInBits = ScalarRegSizeInBits(size);
// Shift encoded in immh:immb, but inverted with 128-bit source
// shift = (esize * 2) - immh:immb
@@ -784,7 +735,7 @@ public:
///< size is destination.
void uqrshrn(ScalarRegSize size, VRegister rd, VRegister rn, uint32_t Shift) {
LOGMAN_THROW_A_FMT(Shift > 0 && Shift < ScalarRegSizeInBits(size), "Invalid shift for sshr");
LOGMAN_THROW_AA_FMT(size != ARMEmitter::ScalarRegSize::i64Bit, "Invalid size selected for sqrshrun");
LOGMAN_THROW_A_FMT(size != ARMEmitter::ScalarRegSize::i64Bit, "Invalid size selected for sqrshrun");
const size_t SubregSizeInBits = ScalarRegSizeInBits(size);
// Shift encoded in immh:immb, but inverted with 128-bit source
// shift = (esize * 2) - immh:immb
@@ -794,10 +745,10 @@ public:
ASIMDScalarShiftByImm(1, immh, immb, 0b10011, rd, rn);
}
// TODO: UCVTF, FCVTZU
// Advanced SIMD scalar x indexed element
// XXX:
//
// Floating-point data-processing (1 source)
// Advanced SIMD scalar x indexed element
// XXX:
//
// Floating-point data-processing (1 source)
void fmov(ScalarRegSize size, VRegister rd, VRegister rn) {
Float1Source(size, 0, 0, 0b000000, rd, rn);
}
@@ -991,14 +942,14 @@ public:
Float1Source(0, 0, 0b11, 0b001111, rd.V(), rn.V());
}
// Floating-point compare
// Floating-point compare
void fcmp(ScalarRegSize Size, VRegister rn, VRegister rm) {
LOGMAN_THROW_AA_FMT(Size != ScalarRegSize::i8Bit, "8-bit destination not supported");
LOGMAN_THROW_A_FMT(Size != ScalarRegSize::i8Bit, "8-bit destination not supported");
const auto ConvertedSize =
Size == ARMEmitter::ScalarRegSize::i64Bit ? 0b01 :
Size == ARMEmitter::ScalarRegSize::i32Bit ? 0b00 :
Size == ARMEmitter::ScalarRegSize::i16Bit ? 0b11 : 0;
const auto ConvertedSize = Size == ARMEmitter::ScalarRegSize::i64Bit ? 0b01 :
Size == ARMEmitter::ScalarRegSize::i32Bit ? 0b00 :
Size == ARMEmitter::ScalarRegSize::i16Bit ? 0b11 :
0;
FloatCompare(0, 0, ConvertedSize, 0b00, 0b00000, rn, rm);
}
@@ -1051,7 +1002,7 @@ public:
FloatCompare(0, 0, 0b11, 0b00, 0b11000, rn.V(), VReg::v0);
}
// Floating-point immediate
// Floating-point immediate
void fmov(ARMEmitter::ScalarRegSize size, ARMEmitter::VRegister rd, float Value) {
uint32_t M = 0;
uint32_t S = 0;
@@ -1061,16 +1012,13 @@ public:
if (size == ARMEmitter::ScalarRegSize::i16Bit) {
LOGMAN_MSG_A_FMT("Unsupported");
FEX_UNREACHABLE;
}
else if (size == ARMEmitter::ScalarRegSize::i32Bit) {
} else if (size == ARMEmitter::ScalarRegSize::i32Bit) {
ptype = 0b00;
imm8 = FP32ToImm8(Value);
}
else if (size == ARMEmitter::ScalarRegSize::i64Bit) {
} else if (size == ARMEmitter::ScalarRegSize::i64Bit) {
ptype = 0b01;
imm8 = FP64ToImm8(Value);
}
else {
} else {
FEX_UNREACHABLE;
}
@@ -1090,7 +1038,7 @@ public:
dc32(Instr);
}
// Floating-point conditional compare
// Floating-point conditional compare
void fccmp(SRegister rn, SRegister rm, StatusFlags flags, Condition Cond) {
FloatConditionalCompare(0, 0, 0b00, 0b0, rn.V(), rm.V(), flags, Cond);
}
@@ -1110,7 +1058,7 @@ public:
FloatConditionalCompare(0, 0, 0b11, 0b1, rn.V(), rm.V(), flags, Cond);
}
// Floating-point data-processing (2 source)
// Floating-point data-processing (2 source)
void fmul(ScalarRegSize size, VRegister rd, VRegister rn, VRegister rm) {
Float2Source(size, 0, 0, 0b0000, rd, rn, rm);
}
@@ -1225,11 +1173,10 @@ public:
// Floating-point conditional select
void fcsel(ScalarRegSize size, VRegister rd, VRegister rn, VRegister rm, Condition Cond) {
LOGMAN_THROW_AA_FMT(size == ScalarRegSize::i16Bit || size == ScalarRegSize::i64Bit || size == ScalarRegSize::i32Bit, "Invalid size selected for {}", __func__);
LOGMAN_THROW_A_FMT(size == ScalarRegSize::i16Bit || size == ScalarRegSize::i64Bit || size == ScalarRegSize::i32Bit,
"Invalid size selected for {}", __func__);
const uint32_t ConvertedSize =
size == ScalarRegSize::i64Bit ? 0b01 :
size == ScalarRegSize::i32Bit ? 0b00 : 0b11;
const uint32_t ConvertedSize = size == ScalarRegSize::i64Bit ? 0b01 : size == ScalarRegSize::i32Bit ? 0b00 : 0b11;
FloatConditionalSelect(0, 0, ConvertedSize, rd, rn, rm, Cond);
}
@@ -1244,7 +1191,7 @@ public:
FloatConditionalSelect(0, 0, 0b11, rd.V(), rn.V(), rm.V(), Cond);
}
// Floating-point data-processing (3 source)
// Floating-point data-processing (3 source)
void fmadd(SRegister rd, SRegister rn, SRegister rm, SRegister ra) {
Float3Source(0, 0, 0b00, 0, 0, rd.V(), rn.V(), rm.V(), ra.V());
}
@@ -1285,7 +1232,7 @@ public:
}
private:
// Advanced SIMD scalar copy
// Advanced SIMD scalar copy
void ASIMDScalarCopy(uint32_t Op, uint32_t Q, uint32_t imm5, uint32_t imm4, ARMEmitter::VRegister rd, ARMEmitter::VRegister rn) {
uint32_t Instr = Op;
@@ -1297,7 +1244,7 @@ private:
dc32(Instr);
}
// Advanced SIMD scalar three same FP16
// Advanced SIMD scalar three same FP16
void ASIMDScalarThreeSameFP16(uint32_t U, uint32_t a, uint32_t opcode, HRegister rm, HRegister rn, HRegister rd) {
uint32_t Instr = 0b0101'1110'0100'0000'0000'0100'0000'0000;
@@ -1309,7 +1256,7 @@ private:
Instr |= rd.Idx();
dc32(Instr);
}
// Advanced SIMD scalar two-register miscellaneous FP16
// Advanced SIMD scalar two-register miscellaneous FP16
void ASIMDScalarTwoRegMiscFP16(uint32_t U, uint32_t a, uint32_t opcode, HRegister rn, HRegister rd) {
uint32_t Instr = 0b0101'1110'0111'1000'0000'1000'0000'0000;
@@ -1321,9 +1268,9 @@ private:
dc32(Instr);
}
// Advanced SIMD scalar three same extra
// XXX:
// Advanced SIMD scalar two-register miscellaneous
// Advanced SIMD scalar three same extra
// XXX:
// Advanced SIMD scalar two-register miscellaneous
void ASIMDScalar2RegMisc(uint32_t b20, uint32_t U, ScalarRegSize size, uint32_t opcode, VRegister rd, VRegister rn) {
uint32_t Instr = 0b0101'1110'0010'0000'0000'1000'0000'0000;
@@ -1336,9 +1283,9 @@ private:
dc32(Instr);
}
// Advanced SIMD scalar pairwise
// XXX:
// Advanced SIMD scalar three different
// Advanced SIMD scalar pairwise
// XXX:
// Advanced SIMD scalar three different
void ASIMD3RegDifferent(uint32_t U, ScalarRegSize size, uint32_t opcode, VRegister rd, VRegister rn, VRegister rm) {
uint32_t Instr = 0b0101'1110'0010'0000'0000'0000'0000'0000;
@@ -1350,7 +1297,7 @@ private:
Instr |= Encode_rd(rd);
dc32(Instr);
}
// Advanced SIMD scalar three same
// Advanced SIMD scalar three same
void ASIMD3RegSame(uint32_t U, ScalarRegSize size, uint32_t opcode, VRegister rd, VRegister rn, VRegister rm) {
uint32_t Instr = 0b0101'1110'0010'0000'0000'0100'0000'0000;
@@ -1362,7 +1309,7 @@ private:
Instr |= Encode_rd(rd);
dc32(Instr);
}
// Advanced SIMD scalar shift by immediate
// Advanced SIMD scalar shift by immediate
void ASIMDScalarShiftByImm(uint32_t U, uint32_t immh, uint32_t immb, uint32_t opcode, VRegister rd, VRegister rn) {
uint32_t Instr = 0b0101'1111'0000'0000'0000'0100'0000'0000;
@@ -1374,9 +1321,9 @@ private:
Instr |= Encode_rd(rd);
dc32(Instr);
}
// Advanced SIMD scalar x indexed element
// XXX:
// Floating-point data-processing (1 source)
// Advanced SIMD scalar x indexed element
// XXX:
// Floating-point data-processing (1 source)
void Float1Source(uint32_t M, uint32_t S, uint32_t ptype, uint32_t opcode, VRegister rd, VRegister rn) {
uint32_t Instr = 0b0001'1110'0010'0000'0100'0000'0000'0000;
@@ -1390,16 +1337,15 @@ private:
dc32(Instr);
}
void Float1Source(ScalarRegSize size, uint32_t M, uint32_t S, uint32_t opcode, VRegister rd, VRegister rn) {
LOGMAN_THROW_AA_FMT(size == ScalarRegSize::i16Bit || size == ScalarRegSize::i64Bit || size == ScalarRegSize::i32Bit, "Invalid size selected for {}", __func__);
LOGMAN_THROW_A_FMT(size == ScalarRegSize::i16Bit || size == ScalarRegSize::i64Bit || size == ScalarRegSize::i32Bit,
"Invalid size selected for {}", __func__);
const uint32_t ConvertedSize =
size == ScalarRegSize::i64Bit ? 0b01 :
size == ScalarRegSize::i32Bit ? 0b00 : 0b11;
const uint32_t ConvertedSize = size == ScalarRegSize::i64Bit ? 0b01 : size == ScalarRegSize::i32Bit ? 0b00 : 0b11;
Float1Source(M, S, ConvertedSize, opcode, rd, rn);
}
// Floating-point compare
// Floating-point compare
void FloatCompare(uint32_t M, uint32_t S, uint32_t ftype, uint32_t op, uint32_t opcode2, VRegister rn, VRegister rm) {
uint32_t Instr = 0b0001'1110'0010'0000'0010'0000'0000'0000;
@@ -1413,9 +1359,9 @@ private:
dc32(Instr);
}
// Floating-point immediate
// XXX:
// Floating-point conditional compare
// Floating-point immediate
// XXX:
// Floating-point conditional compare
void FloatConditionalCompare(uint32_t M, uint32_t S, uint32_t ptype, uint32_t op, VRegister rn, VRegister rm, StatusFlags flags, Condition Cond) {
uint32_t Instr = 0b0001'1110'0010'0000'0000'0100'0000'0000;
@@ -1430,7 +1376,7 @@ private:
dc32(Instr);
}
// Floating-point data-processing (2 source)
// Floating-point data-processing (2 source)
void Float2Source(uint32_t M, uint32_t S, uint32_t ptype, uint32_t opcode, VRegister rd, VRegister rn, VRegister rm) {
uint32_t Instr = 0b0001'1110'0010'0000'0000'1000'0000'0000;
@@ -1447,16 +1393,15 @@ private:
}
void Float2Source(ScalarRegSize size, uint32_t M, uint32_t S, uint32_t opcode, VRegister rd, VRegister rn, VRegister rm) {
LOGMAN_THROW_AA_FMT(size == ScalarRegSize::i16Bit || size == ScalarRegSize::i64Bit || size == ScalarRegSize::i32Bit, "Invalid size selected for {}", __func__);
LOGMAN_THROW_A_FMT(size == ScalarRegSize::i16Bit || size == ScalarRegSize::i64Bit || size == ScalarRegSize::i32Bit,
"Invalid size selected for {}", __func__);
const uint32_t ConvertedSize =
size == ScalarRegSize::i64Bit ? 0b01 :
size == ScalarRegSize::i32Bit ? 0b00 : 0b11;
const uint32_t ConvertedSize = size == ScalarRegSize::i64Bit ? 0b01 : size == ScalarRegSize::i32Bit ? 0b00 : 0b11;
Float2Source(M, S, ConvertedSize, opcode, rd, rn, rm);
}
// Floating-point conditional select
// Floating-point conditional select
void FloatConditionalSelect(uint32_t M, uint32_t S, uint32_t ptype, VRegister rd, VRegister rn, VRegister rm, Condition Cond) {
uint32_t Instr = 0b0001'1110'0010'0000'0000'1100'0000'0000;
@@ -1470,7 +1415,7 @@ private:
dc32(Instr);
}
// Floating-point data-processing (3 source)
// Floating-point data-processing (3 source)
void Float3Source(uint32_t M, uint32_t S, uint32_t ptype, uint32_t o1, uint32_t o0, VRegister rd, VRegister rn, VRegister rm, VRegister ra) {
uint32_t Instr = 0b0001'1111'0000'0000'0000'0000'0000'0000;
@@ -1485,3 +1430,8 @@ private:
Instr |= Encode_rd(rd);
dc32(Instr);
}
#ifndef INCLUDED_BY_EMITTER
}; // struct LoadstoreEmitterOps
} // namespace ARMEmitter
#endif
+160 -148
View File
@@ -4,173 +4,185 @@
* This is mostly a mashup of various instruction types.
* Nothing follows an explicit pattern since they are mostly different.
*/
#pragma once
#ifndef INCLUDED_BY_EMITTER
#include <CodeEmitter/Emitter.h>
namespace ARMEmitter {
struct EmitterOps : Emitter {
#endif
public:
// System with result
// TODO: SYSL
// System Instruction
// TODO: AT
// TODO: CFP
// TODO: CPP
void dc(ARMEmitter::DataCacheOperation DCOp, ARMEmitter::Register rt) {
constexpr uint32_t Op = 0b1101'0101'0000'1000'0111 << 12;
SystemInstruction(Op, 0, FEXCore::ToUnderlying(DCOp), rt);
}
// TODO: DVP
// TODO: IC
// TODO: TLBI
// System with result
// TODO: SYSL
// System Instruction
// TODO: AT
// TODO: CFP
// TODO: CPP
void dc(ARMEmitter::DataCacheOperation DCOp, ARMEmitter::Register rt) {
constexpr uint32_t Op = 0b1101'0101'0000'1000'0111 << 12;
SystemInstruction(Op, 0, FEXCore::ToUnderlying(DCOp), rt);
}
// TODO: DVP
// TODO: IC
// TODO: TLBI
// Exception generation
void svc(uint32_t Imm) {
ExceptionGeneration(0b000, 0b000, 0b01, Imm);
}
void hvc(uint32_t Imm) {
ExceptionGeneration(0b000, 0b000, 0b10, Imm);
}
void smc(uint32_t Imm) {
ExceptionGeneration(0b000, 0b000, 0b11, Imm);
}
void brk(uint32_t Imm) {
ExceptionGeneration(0b001, 0b000, 0b00, Imm);
}
void hlt(uint32_t Imm) {
ExceptionGeneration(0b010, 0b000, 0b00, Imm);
}
void tcancel(uint32_t Imm) {
ExceptionGeneration(0b011, 0b000, 0b00, Imm);
}
void dcps1(uint32_t Imm) {
ExceptionGeneration(0b101, 0b000, 0b01, Imm);
}
void dcps2(uint32_t Imm) {
ExceptionGeneration(0b101, 0b000, 0b10, Imm);
}
void dcps3(uint32_t Imm) {
ExceptionGeneration(0b101, 0b000, 0b11, Imm);
}
// System instructions with register argument
void wfet(ARMEmitter::Register rt) {
SystemInstructionWithReg(0b0000, 0b000, rt);
}
void wfit(ARMEmitter::Register rt) {
SystemInstructionWithReg(0b0000, 0b001, rt);
}
// Exception generation
void svc(uint32_t Imm) {
ExceptionGeneration(0b000, 0b000, 0b01, Imm);
}
void hvc(uint32_t Imm) {
ExceptionGeneration(0b000, 0b000, 0b10, Imm);
}
void smc(uint32_t Imm) {
ExceptionGeneration(0b000, 0b000, 0b11, Imm);
}
void brk(uint32_t Imm) {
ExceptionGeneration(0b001, 0b000, 0b00, Imm);
}
void hlt(uint32_t Imm) {
ExceptionGeneration(0b010, 0b000, 0b00, Imm);
}
void tcancel(uint32_t Imm) {
ExceptionGeneration(0b011, 0b000, 0b00, Imm);
}
void dcps1(uint32_t Imm) {
ExceptionGeneration(0b101, 0b000, 0b01, Imm);
}
void dcps2(uint32_t Imm) {
ExceptionGeneration(0b101, 0b000, 0b10, Imm);
}
void dcps3(uint32_t Imm) {
ExceptionGeneration(0b101, 0b000, 0b11, Imm);
}
// System instructions with register argument
void wfet(ARMEmitter::Register rt) {
SystemInstructionWithReg(0b0000, 0b000, rt);
}
void wfit(ARMEmitter::Register rt) {
SystemInstructionWithReg(0b0000, 0b001, rt);
}
// Hints
void nop() {
Hint(ARMEmitter::HintRegister::NOP);
}
void yield() {
Hint(ARMEmitter::HintRegister::YIELD);
}
void wfe() {
Hint(ARMEmitter::HintRegister::WFE);
}
void wfi() {
Hint(ARMEmitter::HintRegister::WFI);
}
void sev() {
Hint(ARMEmitter::HintRegister::SEV);
}
void sevl() {
Hint(ARMEmitter::HintRegister::SEVL);
}
void dgh() {
Hint(ARMEmitter::HintRegister::DGH);
}
void csdb() {
Hint(ARMEmitter::HintRegister::CSDB);
}
// Hints
void nop() {
Hint(ARMEmitter::HintRegister::NOP);
}
void yield() {
Hint(ARMEmitter::HintRegister::YIELD);
}
void wfe() {
Hint(ARMEmitter::HintRegister::WFE);
}
void wfi() {
Hint(ARMEmitter::HintRegister::WFI);
}
void sev() {
Hint(ARMEmitter::HintRegister::SEV);
}
void sevl() {
Hint(ARMEmitter::HintRegister::SEVL);
}
void dgh() {
Hint(ARMEmitter::HintRegister::DGH);
}
void csdb() {
Hint(ARMEmitter::HintRegister::CSDB);
}
// Barriers
void clrex(uint32_t imm = 15) {
LOGMAN_THROW_AA_FMT(imm < 16, "Immediate out of range");
Barrier(ARMEmitter::BarrierRegister::CLREX, imm);
}
void dsb(ARMEmitter::BarrierScope Scope) {
Barrier(ARMEmitter::BarrierRegister::DSB, FEXCore::ToUnderlying(Scope));
}
void dmb(ARMEmitter::BarrierScope Scope) {
Barrier(ARMEmitter::BarrierRegister::DMB, FEXCore::ToUnderlying(Scope));
}
void isb() {
Barrier(ARMEmitter::BarrierRegister::ISB, FEXCore::ToUnderlying(ARMEmitter::BarrierScope::SY));
}
void sb() {
Barrier(ARMEmitter::BarrierRegister::SB, 0);
}
void tcommit() {
Barrier(ARMEmitter::BarrierRegister::TCOMMIT, 0);
}
// Barriers
void clrex(uint32_t imm = 15) {
LOGMAN_THROW_A_FMT(imm < 16, "Immediate out of range");
Barrier(ARMEmitter::BarrierRegister::CLREX, imm);
}
void dsb(ARMEmitter::BarrierScope Scope) {
Barrier(ARMEmitter::BarrierRegister::DSB, FEXCore::ToUnderlying(Scope));
}
void dmb(ARMEmitter::BarrierScope Scope) {
Barrier(ARMEmitter::BarrierRegister::DMB, FEXCore::ToUnderlying(Scope));
}
void isb() {
Barrier(ARMEmitter::BarrierRegister::ISB, FEXCore::ToUnderlying(ARMEmitter::BarrierScope::SY));
}
void sb() {
Barrier(ARMEmitter::BarrierRegister::SB, 0);
}
void tcommit() {
Barrier(ARMEmitter::BarrierRegister::TCOMMIT, 0);
}
// System register move
void msr(ARMEmitter::SystemRegister reg, ARMEmitter::Register rt) {
constexpr uint32_t Op = 0b1101'0101'0001 << 20;
SystemRegisterMove(Op, rt, reg);
}
// System register move
void msr(ARMEmitter::SystemRegister reg, ARMEmitter::Register rt) {
constexpr uint32_t Op = 0b1101'0101'0001 << 20;
SystemRegisterMove(Op, rt, reg);
}
void mrs(ARMEmitter::Register rd, ARMEmitter::SystemRegister reg) {
constexpr uint32_t Op = 0b1101'0101'0011 << 20;
SystemRegisterMove(Op, rd, reg);
}
void mrs(ARMEmitter::Register rd, ARMEmitter::SystemRegister reg) {
constexpr uint32_t Op = 0b1101'0101'0011 << 20;
SystemRegisterMove(Op, rd, reg);
}
private:
// Exception Generation
void ExceptionGeneration(uint32_t opc, uint32_t op2, uint32_t LL, uint32_t Imm) {
LOGMAN_THROW_AA_FMT((Imm & 0xFFFF'0000) == 0, "Imm amount too large");
// Exception Generation
void ExceptionGeneration(uint32_t opc, uint32_t op2, uint32_t LL, uint32_t Imm) {
LOGMAN_THROW_A_FMT((Imm & 0xFFFF'0000) == 0, "Imm amount too large");
uint32_t Instr = 0b1101'0100 << 24;
uint32_t Instr = 0b1101'0100 << 24;
Instr |= opc << 21;
Instr |= Imm << 5;
Instr |= op2 << 2;
Instr |= LL;
Instr |= opc << 21;
Instr |= Imm << 5;
Instr |= op2 << 2;
Instr |= LL;
dc32(Instr);
}
dc32(Instr);
}
// System instructions with register argument
void SystemInstructionWithReg(uint32_t CRm, uint32_t op2, ARMEmitter::Register rt) {
uint32_t Instr = 0b1101'0101'0000'0011'0001 << 12;
// System instructions with register argument
void SystemInstructionWithReg(uint32_t CRm, uint32_t op2, ARMEmitter::Register rt) {
uint32_t Instr = 0b1101'0101'0000'0011'0001 << 12;
Instr |= CRm << 8;
Instr |= op2 << 5;
Instr |= Encode_rt(rt);
dc32(Instr);
}
Instr |= CRm << 8;
Instr |= op2 << 5;
Instr |= Encode_rt(rt);
dc32(Instr);
}
// Hints
void Hint(ARMEmitter::HintRegister Reg) {
uint32_t Instr = 0b1101'0101'0000'0011'0010'0000'0001'1111U;
Instr |= FEXCore::ToUnderlying(Reg);
dc32(Instr);
}
// Barriers
void Barrier(ARMEmitter::BarrierRegister Reg, uint32_t CRm) {
uint32_t Instr = 0b1101'0101'0000'0011'0011'0000'0001'1111U;
Instr |= CRm << 8;
Instr |= FEXCore::ToUnderlying(Reg);
dc32(Instr);
}
// Hints
void Hint(ARMEmitter::HintRegister Reg) {
uint32_t Instr = 0b1101'0101'0000'0011'0010'0000'0001'1111U;
Instr |= FEXCore::ToUnderlying(Reg);
dc32(Instr);
}
// Barriers
void Barrier(ARMEmitter::BarrierRegister Reg, uint32_t CRm) {
uint32_t Instr = 0b1101'0101'0000'0011'0011'0000'0001'1111U;
Instr |= CRm << 8;
Instr |= FEXCore::ToUnderlying(Reg);
dc32(Instr);
}
// System Instruction
void SystemInstruction(uint32_t Op, uint32_t L, uint32_t SubOp, ARMEmitter::Register rt) {
uint32_t Instr = Op;
// System Instruction
void SystemInstruction(uint32_t Op, uint32_t L, uint32_t SubOp, ARMEmitter::Register rt) {
uint32_t Instr = Op;
Instr |= L << 21;
Instr |= SubOp;
Instr |= Encode_rt(rt);
Instr |= L << 21;
Instr |= SubOp;
Instr |= Encode_rt(rt);
dc32(Instr);
}
dc32(Instr);
}
// System register move
void SystemRegisterMove(uint32_t Op, ARMEmitter::Register rt, ARMEmitter::SystemRegister reg) {
uint32_t Instr = Op;
// System register move
void SystemRegisterMove(uint32_t Op, ARMEmitter::Register rt, ARMEmitter::SystemRegister reg) {
uint32_t Instr = Op;
Instr |= FEXCore::ToUnderlying(reg);
Instr |= Encode_rt(rt);
Instr |= FEXCore::ToUnderlying(reg);
Instr |= Encode_rt(rt);
dc32(Instr);
}
dc32(Instr);
}
#ifndef INCLUDED_BY_EMITTER
}; // struct LoadstoreEmitterOps
} // namespace ARMEmitter
#endif
+17 -21
View File
@@ -34,11 +34,7 @@
// by the corresponding fields in the logical instruction.
// If it can not be encoded, the function returns false, and the values pointed
// to by n, imm_s and imm_r are undefined.
static bool IsImmLogical(uint64_t value,
unsigned width,
unsigned* n = nullptr,
unsigned* imm_s = nullptr,
unsigned* imm_r = nullptr) {
static bool IsImmLogical(uint64_t value, unsigned width, unsigned* n = nullptr, unsigned* imm_s = nullptr, unsigned* imm_r = nullptr) {
[[maybe_unused]] constexpr auto kBRegSize = 8;
[[maybe_unused]] constexpr auto kHRegSize = 16;
[[maybe_unused]] constexpr auto kSRegSize = 32;
@@ -47,8 +43,7 @@ static bool IsImmLogical(uint64_t value,
constexpr auto kWRegSize = 32;
constexpr auto kXRegSize = 64;
LOGMAN_THROW_A_FMT((width == kBRegSize) || (width == kHRegSize) ||
(width == kSRegSize) || (width == kDRegSize), "Unexpected imm size");
LOGMAN_THROW_A_FMT((width == kBRegSize) || (width == kHRegSize) || (width == kSRegSize) || (width == kDRegSize), "Unexpected imm size");
bool negate = false;
@@ -182,12 +177,7 @@ static bool IsImmLogical(uint64_t value,
// (1 + 2^d + 2^(2d) + ...), i.e. 0x0001000100010001 or similar. These can
// be derived using a table lookup on CLZ(d).
static const uint64_t multipliers[] = {
0x0000000000000001UL,
0x0000000100000001UL,
0x0001000100010001UL,
0x0101010101010101UL,
0x1111111111111111UL,
0x5555555555555555UL,
0x0000000000000001UL, 0x0000000100000001UL, 0x0001000100010001UL, 0x0101010101010101UL, 0x1111111111111111UL, 0x5555555555555555UL,
};
uint64_t multiplier = multipliers[CountLeadingZeros(d, kXRegSize) - 57];
uint64_t candidate = (b - a) * multiplier;
@@ -244,7 +234,9 @@ static bool IsImmLogical(uint64_t value,
}
static inline bool IsIntN(unsigned n, int64_t x) {
if (n == 64) return true;
if (n == 64) {
return true;
}
int64_t limit = INT64_C(1) << (n - 1);
return (-limit <= x) && (x < limit);
}
@@ -271,11 +263,15 @@ V(57) V(58) V(59) V(60) V(61) V(62) V(63)
// clang-format on
#define DECLARE_IS_INT_N(N) \
static inline bool IsInt##N(int64_t x) { return IsIntN(N, x); }
#define DECLARE_IS_INT_N(N) \
static inline bool IsInt##N(int64_t x) { \
return IsIntN(N, x); \
}
#define DECLARE_IS_UINT_N(N) \
static inline bool IsUint##N(int64_t x) { return IsUintN(N, x); }
#define DECLARE_IS_UINT_N(N) \
static inline bool IsUint##N(int64_t x) { \
return IsUintN(N, x); \
}
INT_1_TO_63_LIST(DECLARE_IS_INT_N)
INT_1_TO_63_LIST(DECLARE_IS_UINT_N)
@@ -285,14 +281,14 @@ INT_1_TO_63_LIST(DECLARE_IS_UINT_N)
private:
template <typename V>
template<typename V>
static inline bool IsPowerOf2(V value) {
return (value != 0) && ((value & (value - 1)) == 0);
}
// Some compilers dislike negating unsigned integers,
// so we provide an equivalent.
template <typename T>
template<typename T>
static inline T UnsignedNegate(T value) {
static_assert(std::is_unsigned<T>::value);
return ~value + 1;
@@ -302,7 +298,7 @@ static inline uint64_t LowestSetBit(uint64_t value) {
return value & UnsignedNegate(value);
}
template <typename V>
template<typename V>
static inline int CountLeadingZeros(V value, int width = (sizeof(V) * 8)) {
#if COMPILER_HAS_BUILTIN_CLZ
if (width == 32) {
-1
View File
@@ -6,4 +6,3 @@ mask \xff\xff\xff\xff\xff\xfe\xfe\x00\x00\x00\x00\xff\xff\xff\xff\xff\xfe\xff\xf
credentials yes
fix_binary yes
preserve yes
expose_interpreter optional
-1
View File
@@ -6,4 +6,3 @@ mask \xff\xff\xff\xff\xff\xfe\xfe\x00\x00\x00\x00\xff\xff\xff\xff\xff\xfe\xff\xf
credentials yes
fix_binary yes
preserve yes
expose_interpreter optional
+1 -1
-1
Submodule External/imgui deleted from 4c986ecb8d.
+1 -1
View File
@@ -1,3 +1,3 @@
set(NAME tiny-json)
set(SRCS tiny-json.c)
add_library(${NAME} ${SRCS})
add_library(${NAME} STATIC ${SRCS})
+1 -1
+1 -1
+6 -2
View File
@@ -220,7 +220,11 @@ def print_man_environment_tail():
print_man_env_option(
"FEX_PORTABLE",
[
"Allows FEX to run without installation. Global locations for configuration and binfmt_misc are ignored. These files are instead read from <FEXInterpreterPath>/fex-emu/ by default.",
"Allows FEX to run without installation. Global locations for configuration and binfmt_misc are ignored.",
"For FEXInterpreter on Linux:",
"These files are instead read from <FEXInterpreterPath>/fex-emu/ by default.",
"For Arm64ec/Wow64 WINE builds:",
"These files are instead read from $LOCALAPPDATA/fex-emu/ by default.",
"For further customization, see FEX_APP_CONFIG_LOCATION and FEX_APP_DATA_LOCATION."
],
"''", True)
@@ -401,7 +405,7 @@ def print_parse_argloader_options(options):
conversion_func = "FEXCore::Config::Handler::{0}(".format(op_vals["ArgumentHandler"])
if (value_type == "str"):
NeedsString = True
conversion_func = "("
conversion_func = "std::move("
if (value_type == "bool"):
# boolean values need a decimal specifier. Otherwise fmt prints strings.
conversion_func = "fextl::fmt::format(\"{:d}\", "
+19 -17
View File
@@ -44,7 +44,7 @@ class OpDefinition:
HasDest: bool
DestType: str
DestSize: str
NumElements: str
ElementSize: str
OpClass: str
HasSideEffects: bool
ImplicitFlagClobber: bool
@@ -67,7 +67,7 @@ class OpDefinition:
self.HasDest = False
self.DestType = None
self.DestSize = None
self.NumElements = None
self.ElementSize = None
self.OpClass = None
self.OpSize = 0
self.HasSideEffects = False
@@ -232,8 +232,8 @@ def parse_ops(ops):
if "DestSize" in op_val:
OpDef.DestSize = op_val["DestSize"]
if "NumElements" in op_val:
OpDef.NumElements = op_val["NumElements"]
if "ElementSize" in op_val:
OpDef.ElementSize = op_val["ElementSize"]
if len(op_class):
OpDef.OpClass = op_class
@@ -323,8 +323,8 @@ def print_ir_structs(defines):
output_file.write("struct __attribute__((packed)) IROp_Header {\n")
output_file.write("\tvoid* Data[0];\n")
output_file.write("\tIROps Op;\n\n")
output_file.write("\tuint8_t Size;\n")
output_file.write("\tuint8_t ElementSize;\n")
output_file.write("\tIR::OpSize Size;\n")
output_file.write("\tIR::OpSize ElementSize;\n")
output_file.write("\ttemplate<typename T>\n")
output_file.write("\tT const* C() const { return reinterpret_cast<T const*>(Data); }\n")
@@ -630,20 +630,19 @@ def print_ir_allocator_helpers():
output_file.write("\t\treturn IRPair<T>{Op, CreateNode(&Op->Header)};\n")
output_file.write("\t}\n\n")
output_file.write("\tuint8_t GetOpSize(const OrderedNode *Op) const {\n")
output_file.write("\tIR::OpSize GetOpSize(const OrderedNode *Op) const {\n")
output_file.write("\t\tauto HeaderOp = Op->Header.Value.GetNode(DualListData.DataBegin());\n")
output_file.write("\t\treturn HeaderOp->Size;\n")
output_file.write("\t}\n\n")
output_file.write("\tuint8_t GetOpElementSize(const OrderedNode *Op) const {\n")
output_file.write("\tIR::OpSize GetOpElementSize(const OrderedNode *Op) const {\n")
output_file.write("\t\tauto HeaderOp = Op->Header.Value.GetNode(DualListData.DataBegin());\n")
output_file.write("\t\treturn HeaderOp->ElementSize;\n")
output_file.write("\t}\n\n")
output_file.write("\tuint8_t GetOpElements(const OrderedNode *Op) const {\n")
output_file.write("\t\tauto HeaderOp = Op->Header.Value.GetNode(DualListData.DataBegin());\n")
output_file.write("\t\tLOGMAN_THROW_A_FMT(OpHasDest(Op), \"Op {} has no dest\\n\", GetName(HeaderOp->Op));\n")
output_file.write("\t\treturn HeaderOp->Size / HeaderOp->ElementSize;\n")
output_file.write("\t\tLOGMAN_THROW_A_FMT(OpHasDest(Op), \"Op {} has no dest\\n\", GetOpName(Op));\n")
output_file.write("\t\treturn IR::OpSizeToSize(GetOpSize(Op)) / IR::OpSizeToSize(GetOpElementSize(Op));\n")
output_file.write("\t}\n\n")
output_file.write("\tbool OpHasDest(const OrderedNode *Op) const {\n")
@@ -699,8 +698,12 @@ def print_ir_allocator_helpers():
# We gather the "has x87?" flag as we go. This saves the user from
# having to keep track of whether they emitted any x87.
# Also changes the mmx state to X87.
if op.LoweredX87:
output_file.write("\t\tRecordX87Use();\n")
output_file.write(
"\t\tif(MMXState == MMXState_MMX) ChgStateMMX_X87();\n"
)
output_file.write("\t\tauto _Op = AllocateOp<IROp_{}, IROps::OP_{}>();\n".format(op.Name, op.Name.upper()))
@@ -724,11 +727,11 @@ def print_ir_allocator_helpers():
# We can only infer a size if we have arguments
if op.DestSize == None:
# We need to infer destination size
output_file.write("\t\tuint8_t InferSize = 0;\n")
output_file.write("\t\tIR::OpSize InferSize = OpSize::iUnsized;\n")
if len(op.Arguments) != 0:
for arg in op.Arguments:
if arg.IsSSA:
output_file.write("\t\tuint8_t Size{} = GetOpSize({});\n".format(arg.Name, arg.Name))
output_file.write("\t\tauto Size{} = GetOpSize({});\n".format(arg.Name, arg.Name))
for arg in op.Arguments:
if arg.IsSSA:
output_file.write("\t\tInferSize = std::max(InferSize, Size{});\n".format(arg.Name))
@@ -740,10 +743,10 @@ def print_ir_allocator_helpers():
if op.DestSize != None:
output_file.write("\t\t_Op.first->Header.Size = {};\n".format(op.DestSize))
if op.NumElements == None:
output_file.write("\t\t_Op.first->Header.ElementSize = _Op.first->Header.Size / ({});\n".format(1))
if op.ElementSize == None:
output_file.write("\t\t_Op.first->Header.ElementSize = _Op.first->Header.Size;\n")
else:
output_file.write("\t\t_Op.first->Header.ElementSize = _Op.first->Header.Size / ({});\n".format(op.NumElements))
output_file.write("\t\t_Op.first->Header.ElementSize = {};\n".format(op.ElementSize))
# Insert validation here
if op.EmitValidation != None:
@@ -826,4 +829,3 @@ print_ir_dispatcher_defs()
print_ir_dispatcher_dispatch()
output_dispatch_file.close()
+33 -24
View File
@@ -86,7 +86,6 @@ set (SRCS
Common/SoftFloat-3e/s_f32UIToCommonNaN.c
Interface/Context/Context.cpp
Interface/Core/LookupCache.cpp
Interface/Core/BlockSamplingData.cpp
Interface/Core/Core.cpp
Interface/Core/CPUBackend.cpp
Interface/Core/CPUID.cpp
@@ -106,20 +105,19 @@ set (SRCS
Interface/Core/ArchHelpers/Arm64Emitter.cpp
Interface/Core/Dispatcher/Dispatcher.cpp
Interface/Core/Interpreter/Fallbacks/InterpreterFallbacks.cpp
Interface/Core/JIT/Arm64/JIT.cpp
Interface/Core/JIT/Arm64/ALUOps.cpp
Interface/Core/JIT/Arm64/AtomicOps.cpp
Interface/Core/JIT/Arm64/BranchOps.cpp
Interface/Core/JIT/Arm64/ConversionOps.cpp
Interface/Core/JIT/Arm64/EncryptionOps.cpp
Interface/Core/JIT/Arm64/MemoryOps.cpp
Interface/Core/JIT/Arm64/MiscOps.cpp
Interface/Core/JIT/Arm64/MoveOps.cpp
Interface/Core/JIT/Arm64/VectorOps.cpp
Interface/Core/JIT/Arm64/Arm64Relocations.cpp
Interface/Core/JIT/JIT.cpp
Interface/Core/JIT/ALUOps.cpp
Interface/Core/JIT/AtomicOps.cpp
Interface/Core/JIT/BranchOps.cpp
Interface/Core/JIT/ConversionOps.cpp
Interface/Core/JIT/EncryptionOps.cpp
Interface/Core/JIT/MemoryOps.cpp
Interface/Core/JIT/MiscOps.cpp
Interface/Core/JIT/MoveOps.cpp
Interface/Core/JIT/VectorOps.cpp
Interface/Core/JIT/Arm64Relocations.cpp
Interface/Core/X86Tables/BaseTables.cpp
Interface/Core/X86Tables/DDDTables.cpp
Interface/Core/X86Tables/EVEXTables.cpp
Interface/Core/X86Tables/H0F38Tables.cpp
Interface/Core/X86Tables/H0F3ATables.cpp
Interface/Core/X86Tables/PrimaryGroupTables.cpp
@@ -128,8 +126,6 @@ set (SRCS
Interface/Core/X86Tables/SecondaryTables.cpp
Interface/Core/X86Tables/VEXTables.cpp
Interface/Core/X86Tables/X87Tables.cpp
Interface/Core/X86Tables/XOPTables.cpp
Interface/HLE/Thunks/Thunks.cpp
Interface/GDBJIT/GDBJIT.cpp
Interface/IR/AOTIR.cpp
Interface/IR/IRDumper.cpp
@@ -140,7 +136,6 @@ set (SRCS
Interface/IR/Passes/IRValidation.cpp
Interface/IR/Passes/RAValidation.cpp
Interface/IR/Passes/RedundantFlagCalculationElimination.cpp
Interface/IR/Passes/DeadStoreElimination.cpp
Interface/IR/Passes/RegisterAllocationPass.cpp
Interface/IR/Passes/x87StackOptimizationPass.cpp
Utils/Telemetry.cpp
@@ -197,14 +192,6 @@ else()
endif()
endif()
if (ENABLE_JEMALLOC)
list (APPEND LIBS FEX_jemalloc)
endif()
if (ENABLE_JEMALLOC_GLIBC_ALLOC)
list (APPEND LIBS FEX_jemalloc_glibc)
endif()
# Generate config
configure_file(
${CMAKE_CURRENT_SOURCE_DIR}/Interface/Config/Config.json.in
@@ -379,3 +366,25 @@ if (NOT MINGW_BUILD)
DESTINATION ${CMAKE_INSTALL_LIBDIR}
COMPONENT Libraries)
endif()
# Meta-library to link jemalloc libraries enabled in the build configuration.
# Only needed for targets that run emulation. For others, use JemallocDummy.
add_library(JemallocLibs STATIC Utils/AllocatorHooks.cpp)
if (ENABLE_JEMALLOC)
target_compile_definitions(JemallocLibs PRIVATE ENABLE_JEMALLOC=1 JEMALLOC_NO_RENAME=1)
target_link_libraries(JemallocLibs PUBLIC FEX_jemalloc)
endif()
if (ENABLE_JEMALLOC_GLIBC_ALLOC)
set_source_files_properties(Interface/HLE/Thunks/Thunks.cpp PROPERTIES COMPILE_DEFINITIONS ENABLE_JEMALLOC_GLIBC=1)
target_link_libraries(JemallocLibs INTERFACE FEX_jemalloc_glibc)
endif()
if (NOT MINGW_BUILD)
# Dummy project to use for host tools.
# This overrides use of jemalloc in FEXCore with the normal glibc allocator.
add_library(JemallocDummy STATIC Utils/AllocatorHooks.cpp)
target_include_directories(JemallocDummy PRIVATE "${PROJECT_SOURCE_DIR}/include/")
endif()
# The shared library should always link enabled jemalloc libraries
target_link_libraries(${PROJECT_NAME}_shared JemallocLibs)
+3 -3
View File
@@ -21,12 +21,12 @@ struct BitSet final {
ElementType* Memory;
void Allocate(size_t Elements) {
size_t AllocateSize = ToBytes(Elements);
LOGMAN_THROW_AA_FMT((AllocateSize * MinimumSize) >= Elements, "Fail");
LOGMAN_THROW_A_FMT((AllocateSize * MinimumSize) >= Elements, "Fail");
Memory = static_cast<ElementType*>(FEXCore::Allocator::malloc(AllocateSize));
}
void Realloc(size_t Elements) {
size_t AllocateSize = ToBytes(Elements);
LOGMAN_THROW_AA_FMT((AllocateSize * MinimumSize) >= Elements, "Fail");
LOGMAN_THROW_A_FMT((AllocateSize * MinimumSize) >= Elements, "Fail");
Memory = static_cast<ElementType*>(FEXCore::Allocator::realloc(Memory, AllocateSize));
}
void Free() {
@@ -68,7 +68,7 @@ struct BitSetView final {
ElementType* Memory;
void GetView(BitSet<T>& Set, uint64_t ElementOffset) {
LOGMAN_THROW_AA_FMT((ElementOffset % MinimumSize) == 0, "Bitset view offset needs to be aligned to size of backing element");
LOGMAN_THROW_A_FMT((ElementOffset % MinimumSize) == 0, "Bitset view offset needs to be aligned to size of backing element");
Memory = &Set.Memory[ElementOffset / MinimumSizeBits];
}
+4 -4
View File
@@ -55,7 +55,7 @@ void JITSymbols::RegisterJITSpace(const void* HostAddr, uint32_t CodeSize) {
}
// Buffered JIT symbols.
void JITSymbols::Register(Core::JITSymbolBuffer* Buffer, const void* HostAddr, uint64_t GuestAddr, uint32_t CodeSize) {
void JITSymbols::Register(FEXCore::JITSymbolBuffer* Buffer, const void* HostAddr, uint64_t GuestAddr, uint32_t CodeSize) {
if (fd == -1) {
return;
}
@@ -79,7 +79,7 @@ void JITSymbols::Register(Core::JITSymbolBuffer* Buffer, const void* HostAddr, u
WriteBuffer(Buffer);
}
void JITSymbols::Register(Core::JITSymbolBuffer* Buffer, const void* HostAddr, uint32_t CodeSize, std::string_view Name, uintptr_t Offset) {
void JITSymbols::Register(FEXCore::JITSymbolBuffer* Buffer, const void* HostAddr, uint32_t CodeSize, std::string_view Name, uintptr_t Offset) {
if (fd == -1) {
return;
}
@@ -104,7 +104,7 @@ void JITSymbols::Register(Core::JITSymbolBuffer* Buffer, const void* HostAddr, u
WriteBuffer(Buffer);
}
void JITSymbols::RegisterNamedRegion(Core::JITSymbolBuffer* Buffer, const void* HostAddr, uint32_t CodeSize, std::string_view Name) {
void JITSymbols::RegisterNamedRegion(FEXCore::JITSymbolBuffer* Buffer, const void* HostAddr, uint32_t CodeSize, std::string_view Name) {
if (fd == -1) {
return;
}
@@ -128,7 +128,7 @@ void JITSymbols::RegisterNamedRegion(Core::JITSymbolBuffer* Buffer, const void*
WriteBuffer(Buffer);
}
void JITSymbols::WriteBuffer(Core::JITSymbolBuffer* Buffer, bool ForceWrite) {
void JITSymbols::WriteBuffer(FEXCore::JITSymbolBuffer* Buffer, bool ForceWrite) {
auto Now = std::chrono::steady_clock::now();
if (!ForceWrite) {
if (((Buffer->LastWrite - Now) < Buffer->MAXIMUM_THRESHOLD) && Buffer->Offset < Buffer->NEEDS_WRITE_DISTANCE) {
+26 -6
View File
@@ -11,6 +11,26 @@
#include <string_view>
namespace FEXCore {
// Buffered JIT symbol tracking.
struct JITSymbolBuffer {
// Maximum buffer size to ensure we are a page in size.
constexpr static size_t BUFFER_SIZE = 4096 - (8 * 2);
// Maximum distance until the end of the buffer to do a write.
constexpr static size_t NEEDS_WRITE_DISTANCE = BUFFER_SIZE - 64;
// Maximum time threshhold to wait before a buffer write occurs.
constexpr static std::chrono::milliseconds MAXIMUM_THRESHOLD {100};
JITSymbolBuffer()
: LastWrite {std::chrono::steady_clock::now()} {}
// stead_clock to ensure a monotonic increasing clock.
// In highly stressed situations this can still cause >2% CPU time in vdso_clock_gettime.
// If we need lower CPU time when JIT symbols are enabled then FEX can read the cycle counter directly.
std::chrono::steady_clock::time_point LastWrite {};
size_t Offset {};
char Buffer[BUFFER_SIZE] {};
};
static_assert(sizeof(JITSymbolBuffer) == 4096, "Ensure this is one page in size");
class JITSymbols final {
public:
JITSymbols();
@@ -21,16 +41,16 @@ public:
void RegisterJITSpace(const void* HostAddr, uint32_t CodeSize);
// Allocate JIT buffer.
static fextl::unique_ptr<Core::JITSymbolBuffer> AllocateBuffer() {
return fextl::make_unique<Core::JITSymbolBuffer>();
static fextl::unique_ptr<FEXCore::JITSymbolBuffer> AllocateBuffer() {
return fextl::make_unique<FEXCore::JITSymbolBuffer>();
}
void Register(Core::JITSymbolBuffer* Buffer, const void* HostAddr, uint64_t GuestAddr, uint32_t CodeSize);
void Register(Core::JITSymbolBuffer* Buffer, const void* HostAddr, uint32_t CodeSize, std::string_view Name, uintptr_t Offset);
void RegisterNamedRegion(Core::JITSymbolBuffer* Buffer, const void* HostAddr, uint32_t CodeSize, std::string_view Name);
void Register(FEXCore::JITSymbolBuffer* Buffer, const void* HostAddr, uint64_t GuestAddr, uint32_t CodeSize);
void Register(FEXCore::JITSymbolBuffer* Buffer, const void* HostAddr, uint32_t CodeSize, std::string_view Name, uintptr_t Offset);
void RegisterNamedRegion(FEXCore::JITSymbolBuffer* Buffer, const void* HostAddr, uint32_t CodeSize, std::string_view Name);
private:
int fd {-1};
void WriteBuffer(Core::JITSymbolBuffer* Buffer, bool ForceWrite = false);
void WriteBuffer(FEXCore::JITSymbolBuffer* Buffer, bool ForceWrite = false);
};
} // namespace FEXCore
+39 -1
View File
@@ -160,7 +160,31 @@ struct FEX_PACKED X80SoftFloat {
return Result;
#else
return extF80_rem(state, lhs, rhs);
/*
* FPREM is not an IEEE-754 remainder. From the spec:
*
* Computes the remainder obtained from dividing the value in the ST(0)
* register (the dividend) by the value in the ST(1) register (the divisor
* or modulus), and stores the result in ST(0). The remainder represents the
* following value:
*
* Remainder := ST(0) − (Q * ST(1))
*
* Here, Q is an integer value that is obtained by truncating the
* floating-point number quotient of [ST(0) / ST(1)] toward zero.
*
* We implement this sequence literally. softfloat_round_minMag means
* "truncate towards zero".
*/
extFloat80_t quotient = extF80_div(state, lhs, rhs);
extFloat80_t Q = extF80_roundToInt(state, quotient, softfloat_round_minMag, true);
bool Q_zero = Q.signif == 0 && (Q.signExp & ~(1 << 15)) == 0;
if (Q_zero) {
return lhs;
} else {
return extF80_sub(state, lhs, extF80_mul(state, Q, rhs));
}
#endif
}
@@ -209,6 +233,10 @@ struct FEX_PACKED X80SoftFloat {
return Result;
#else
// Zero is a special case, the significand for +/- 0 is +/- zero.
if (lhs.Exponent == 0x0 && lhs.Significand == 0x0) {
return lhs;
}
X80SoftFloat Tmp = lhs;
Tmp.Exponent = 0x3FFF;
Tmp.Sign = lhs.Sign;
@@ -232,6 +260,12 @@ struct FEX_PACKED X80SoftFloat {
return Result;
#else
// Zero is a special case, the exponent is always -inf
if (lhs.Exponent == 0x0 && lhs.Significand == 0x0) {
X80SoftFloat Result(1, 0x7FFFUL, 0x8000'0000'0000'0000UL);
return Result;
}
int32_t TrueExp = lhs.Exponent - ExponentBias;
return i32_to_extF80(TrueExp);
#endif
@@ -262,6 +296,10 @@ struct FEX_PACKED X80SoftFloat {
return Result;
#else
extFloat80_t Zero {0, 0};
if (extF80_eq(state, lhs, Zero)) {
return lhs;
}
X80SoftFloat Int = FRNDINT(state, rhs, softfloat_round_minMag);
LIBRARY_PRECISION Src2_d = Int.ToFMax(state);
Src2_d = exp2l(Src2_d);
+47 -57
View File
@@ -43,7 +43,8 @@ namespace DefaultValues {
} // namespace DefaultValues
enum Paths {
PATH_DATA_DIR = 0,
PATH_DATA_DIR_LOCAL = 0,
PATH_DATA_DIR_GLOBAL,
PATH_CONFIG_DIR_LOCAL,
PATH_CONFIG_DIR_GLOBAL,
PATH_CONFIG_FILE_LOCAL,
@@ -53,8 +54,8 @@ enum Paths {
};
static std::array<fextl::string, Paths::PATH_LAST> Paths;
void SetDataDirectory(const std::string_view Path) {
Paths[PATH_DATA_DIR] = Path;
void SetDataDirectory(const std::string_view Path, bool Global) {
Paths[PATH_DATA_DIR_LOCAL + Global] = Path;
}
void SetConfigDirectory(const std::string_view Path, bool Global) {
@@ -73,15 +74,15 @@ const fextl::string& GetTelemetryDirectory() {
Path = TelemetryDirectory;
Path += "/";
} else {
Path = Config::GetDataDirectory() + "Telemetry/";
Path = Config::GetDataDirectory(false) + "Telemetry/";
}
}
return Path;
}
const fextl::string& GetDataDirectory() {
return Paths[PATH_DATA_DIR];
const fextl::string& GetDataDirectory(bool Global) {
return Paths[PATH_DATA_DIR_LOCAL + Global];
}
const fextl::string& GetConfigDirectory(bool Global) {
@@ -230,27 +231,26 @@ void Load() {
}
}
fextl::string ExpandPath(const fextl::string& ContainerPrefix, fextl::string PathName) {
fextl::string ExpandPath(const fextl::string& ContainerPrefix, const fextl::string& PathName) {
if (PathName.empty()) {
return {};
}
// Expand home if it exists
if (FHU::Filesystem::IsRelative(PathName)) {
fextl::string Home = getenv("HOME") ?: "";
// Home expansion only works if it is the first character
// This matches bash behaviour
if (PathName.at(0) == '~') {
PathName.replace(0, 1, Home);
return PathName;
if (PathName.starts_with("~/")) {
Home.append(PathName.begin() + 1, PathName.end());
return Home;
}
// Expand relative path to absolute
char ExistsTempPath[PATH_MAX];
char* RealPath = FHU::Filesystem::Absolute(PathName.c_str(), ExistsTempPath);
if (RealPath) {
PathName = RealPath;
if (RealPath && FHU::Filesystem::Exists(RealPath)) {
return RealPath;
}
// Only return if it exists
@@ -318,81 +318,71 @@ fextl::string FindContainerPrefix() {
void ReloadMetaLayer() {
Meta->Load();
// Do configuration option fix ups after everything is reloaded
if (FEXCore::Config::Exists(FEXCore::Config::CONFIG_CORE)) {
// Sanitize Core option
FEX_CONFIG_OPT(Core, CORE);
#if (_M_X86_64)
constexpr uint32_t MaxCoreNumber = 1;
#else
constexpr uint32_t MaxCoreNumber = 0;
#endif
if (Core > MaxCoreNumber) {
// Sanitize the core option by setting the core to the JIT if invalid
FEXCore::Config::EraseSet(FEXCore::Config::CONFIG_CORE, fextl::fmt::format("{}", static_cast<uint32_t>(FEXCore::Config::CONFIG_IRJIT)));
}
}
if (FEXCore::Config::Exists(FEXCore::Config::CONFIG_CACHEOBJECTCODECOMPILATION)) {
FEX_CONFIG_OPT(CacheObjectCodeCompilation, CACHEOBJECTCODECOMPILATION);
FEX_CONFIG_OPT(Core, CORE);
}
fextl::string ContainerPrefix {FindContainerPrefix()};
auto ExpandPathIfExists = [&ContainerPrefix](FEXCore::Config::ConfigOption Config, fextl::string PathName) {
auto NewPath = ExpandPath(ContainerPrefix, PathName);
const fextl::string ContainerPrefix {FindContainerPrefix()};
auto ExpandPathIfExists = [&ContainerPrefix](FEXCore::Config::ConfigOption Config, const fextl::string& PathName) {
const auto NewPath = ExpandPath(ContainerPrefix, PathName);
if (!NewPath.empty()) {
FEXCore::Config::EraseSet(Config, NewPath);
}
};
if (FEXCore::Config::Exists(FEXCore::Config::CONFIG_ROOTFS)) {
FEX_CONFIG_OPT(PathName, ROOTFS);
auto ExpandedString = ExpandPath(ContainerPrefix, PathName());
const auto PathName = *Meta->Get(FEXCore::Config::CONFIG_ROOTFS);
const auto ExpandedString = ExpandPath(ContainerPrefix, *PathName);
if (!ExpandedString.empty()) {
// Adjust the path if it ended up being relative
FEXCore::Config::EraseSet(FEXCore::Config::CONFIG_ROOTFS, ExpandedString);
} else if (!PathName().empty()) {
} else if (!PathName->empty()) {
// If the filesystem doesn't exist then let's see if it exists in the fex-emu folder
fextl::string NamedRootFS = GetDataDirectory() + "RootFS/" + PathName();
if (FHU::Filesystem::Exists(NamedRootFS)) {
FEXCore::Config::EraseSet(FEXCore::Config::CONFIG_ROOTFS, NamedRootFS);
const auto PathNameCopy = *PathName;
for (auto Global : {true, false}) {
for (auto DirectoryFetchers : {GetDataDirectory, GetConfigDirectory}) {
fextl::string NamedRootFS = DirectoryFetchers(Global) + "RootFS/" + PathNameCopy;
if (FHU::Filesystem::Exists(NamedRootFS)) {
FEXCore::Config::EraseSet(FEXCore::Config::CONFIG_ROOTFS, NamedRootFS);
}
}
}
}
}
if (FEXCore::Config::Exists(FEXCore::Config::CONFIG_THUNKHOSTLIBS)) {
FEX_CONFIG_OPT(PathName, THUNKHOSTLIBS);
ExpandPathIfExists(FEXCore::Config::CONFIG_THUNKHOSTLIBS, PathName());
const auto PathName = *Meta->Get(FEXCore::Config::CONFIG_THUNKHOSTLIBS);
ExpandPathIfExists(FEXCore::Config::CONFIG_THUNKHOSTLIBS, *PathName);
}
if (FEXCore::Config::Exists(FEXCore::Config::CONFIG_THUNKGUESTLIBS)) {
FEX_CONFIG_OPT(PathName, THUNKGUESTLIBS);
ExpandPathIfExists(FEXCore::Config::CONFIG_THUNKGUESTLIBS, PathName());
const auto PathName = *Meta->Get(FEXCore::Config::CONFIG_THUNKGUESTLIBS);
ExpandPathIfExists(FEXCore::Config::CONFIG_THUNKGUESTLIBS, *PathName);
}
if (FEXCore::Config::Exists(FEXCore::Config::CONFIG_THUNKCONFIG)) {
FEX_CONFIG_OPT(PathName, THUNKCONFIG);
auto ExpandedString = ExpandPath(ContainerPrefix, PathName());
const auto PathName = *Meta->Get(FEXCore::Config::CONFIG_THUNKCONFIG);
const auto ExpandedString = ExpandPath(ContainerPrefix, *PathName);
if (!ExpandedString.empty()) {
// Adjust the path if it ended up being relative
FEXCore::Config::EraseSet(FEXCore::Config::CONFIG_THUNKCONFIG, ExpandedString);
} else if (!PathName().empty()) {
} else if (!PathName->empty()) {
// If the filesystem doesn't exist then let's see if it exists in the fex-emu folder
fextl::string NamedConfig = GetDataDirectory() + "ThunkConfigs/" + PathName();
if (FHU::Filesystem::Exists(NamedConfig)) {
FEXCore::Config::EraseSet(FEXCore::Config::CONFIG_THUNKCONFIG, NamedConfig);
const auto PathNameCopy = *PathName;
for (auto Global : {true, false}) {
for (auto DirectoryFetchers : {GetDataDirectory, GetConfigDirectory}) {
fextl::string NamedConfig = DirectoryFetchers(Global) + "ThunkConfigs/" + PathNameCopy;
if (FHU::Filesystem::Exists(NamedConfig)) {
FEXCore::Config::EraseSet(FEXCore::Config::CONFIG_THUNKCONFIG, NamedConfig);
}
}
}
}
}
if (FEXCore::Config::Exists(FEXCore::Config::CONFIG_OUTPUTLOG)) {
FEX_CONFIG_OPT(PathName, OUTPUTLOG);
if (PathName() != "stdout" && PathName() != "stderr" && PathName() != "server") {
ExpandPathIfExists(FEXCore::Config::CONFIG_OUTPUTLOG, PathName());
const auto PathName = *Meta->Get(FEXCore::Config::CONFIG_OUTPUTLOG);
if (*PathName != "stdout" && *PathName != "stderr" && *PathName != "server") {
ExpandPathIfExists(FEXCore::Config::CONFIG_OUTPUTLOG, *PathName);
}
}
if (FEXCore::Config::Exists(FEXCore::Config::CONFIG_DUMPIR) && !FEXCore::Config::Exists(FEXCore::Config::CONFIG_PASSMANAGERDUMPIR)) {
// If DumpIR is set but no PassManagerDumpIR configuration is set, then default to `afteropt`
FEX_CONFIG_OPT(PathName, DUMPIR);
if (PathName() != "no") {
const auto PathName = *Meta->Get(FEXCore::Config::CONFIG_DUMPIR);
if (*PathName != "no") {
EraseSet(FEXCore::Config::ConfigOption::CONFIG_PASSMANAGERDUMPIR,
fextl::fmt::format("{}", static_cast<uint64_t>(FEXCore::Config::PassManagerDumpIR::AFTEROPT)));
}
+7 -13
View File
@@ -1,19 +1,6 @@
{
"Options": {
"CPU": {
"Core": {
"Type": "uint32",
"Default": "FEXCore::Config::ConfigCore::CONFIG_IRJIT",
"TextDefault": "irjit",
"ShortArg": "c",
"Choices": [ "irjit", "host" ],
"ArgumentHandler": "CoreHandler",
"Desc": [
"Which CPU core to use",
"host only exists on x86_64",
"[irjit, host]"
]
},
"Multiblock": {
"Type": "bool",
"Default": "false",
@@ -485,6 +472,13 @@
"Sleeps the process at startup for a duration of seconds.",
"Useful if an application crashes too quickly to attach a debugger."
]
},
"StartupSleepProcName": {
"Type": "str",
"Default": "",
"Desc": [
"Contrains the startup sleep to only apply to processes that match this name."
]
}
},
"Misc": {
+5 -12
View File
@@ -8,6 +8,7 @@
#include <FEXCore/Core/CPUID.h>
#include <FEXCore/Core/HostFeatures.h>
#include <FEXCore/Core/SignalDelegator.h>
#include <FEXCore/Core/Thunks.h>
#include "FEXCore/Debug/InternalThreadState.h"
#include <string.h>
@@ -23,14 +24,6 @@ fextl::unique_ptr<FEXCore::Context::Context> FEXCore::Context::Context::CreateNe
return fextl::make_unique<FEXCore::Context::ContextImpl>(Features);
}
void FEXCore::Context::ContextImpl::SetExitHandler(ExitHandler handler) {
CustomExitHandler = std::move(handler);
}
ExitHandler FEXCore::Context::ContextImpl::GetExitHandler() const {
return CustomExitHandler;
}
void FEXCore::Context::ContextImpl::CompileRIP(FEXCore::Core::InternalThreadState* Thread, uint64_t GuestRIP) {
CompileBlock(Thread->CurrentFrame, GuestRIP);
}
@@ -39,10 +32,6 @@ void FEXCore::Context::ContextImpl::CompileRIPCount(FEXCore::Core::InternalThrea
CompileBlock(Thread->CurrentFrame, GuestRIP, MaxInst);
}
void FEXCore::Context::ContextImpl::SetCustomCPUBackendFactory(CustomCPUFactoryType Factory) {
CustomCPUFactory = std::move(Factory);
}
void FEXCore::Context::ContextImpl::SetSignalDelegator(FEXCore::SignalDelegator* _SignalDelegation) {
SignalDelegation = _SignalDelegation;
}
@@ -52,6 +41,10 @@ void FEXCore::Context::ContextImpl::SetSyscallHandler(FEXCore::HLE::SyscallHandl
SourcecodeResolver = Handler->GetSourcecodeResolver();
}
void FEXCore::Context::ContextImpl::SetThunkHandler(FEXCore::ThunkHandler* Handler) {
ThunkHandler = Handler;
}
FEXCore::CPUID::FunctionResults FEXCore::Context::ContextImpl::RunCPUIDFunction(uint32_t Function, uint32_t Leaf) {
return CPUID.RunFunction(Function, Leaf);
}
+33 -62
View File
@@ -26,14 +26,8 @@
#include <stdint.h>
#include <atomic>
#include <condition_variable>
#include <functional>
#include <istream>
#include <memory>
#include <mutex>
#include <shared_mutex>
#include <stddef.h>
#include <queue>
namespace FEXCore {
class CodeLoader;
@@ -65,16 +59,20 @@ namespace Validation {
} // namespace FEXCore::IR
namespace FEXCore::Context {
enum CoreRunningMode {
MODE_RUN = 0,
MODE_SINGLESTEP = 1,
};
struct ExitFunctionLinkData {
uint64_t HostBranch;
uint64_t GuestRIP;
};
struct CustomIRResult {
void* Creator;
void* Data;
CustomIRResult(void* Creator, void* Data)
: Creator(Creator)
, Data(Data) {}
};
using BlockDelinkerFunc = void (*)(FEXCore::Core::CpuStateFrame* Frame, FEXCore::Context::ExitFunctionLinkData* Record);
constexpr uint32_t TSC_SCALE_MAXIMUM = 1'000'000'000; ///< 1Ghz
@@ -83,22 +81,18 @@ public:
// Context base class implementation.
bool InitCore() override;
void SetExitHandler(ExitHandler handler) override;
ExitHandler GetExitHandler() const override;
ExitReason RunUntilExit(FEXCore::Core::InternalThreadState* Thread) override;
void ExecuteThread(FEXCore::Core::InternalThreadState* Thread) override;
void CompileRIP(FEXCore::Core::InternalThreadState* Thread, uint64_t GuestRIP) override;
void CompileRIPCount(FEXCore::Core::InternalThreadState* Thread, uint64_t GuestRIP, uint64_t MaxInst) override;
void SetCustomCPUBackendFactory(CustomCPUFactoryType Factory) override;
void HandleCallback(FEXCore::Core::InternalThreadState* Thread, uint64_t RIP) override;
bool IsAddressInCurrentBlock(FEXCore::Core::InternalThreadState* Thread, uint64_t Address, uint64_t Size) override;
bool IsCurrentBlockSingleInst(FEXCore::Core::InternalThreadState* Thread) override;
uint64_t RestoreRIPFromHostPC(FEXCore::Core::InternalThreadState* Thread, uint64_t HostPC) override;
uint32_t ReconstructCompactedEFLAGS(FEXCore::Core::InternalThreadState* Thread, bool WasInJIT, uint64_t* HostGPRs, uint64_t PSTATE) override;
uint32_t ReconstructCompactedEFLAGS(FEXCore::Core::InternalThreadState* Thread, bool WasInJIT, const uint64_t* HostGPRs, uint64_t PSTATE) override;
void SetFlagsFromCompactedEFLAGS(FEXCore::Core::InternalThreadState* Thread, uint32_t EFLAGS) override;
void ReconstructXMMRegisters(const FEXCore::Core::InternalThreadState* Thread, __uint128_t* XMM_Low, __uint128_t* YMM_High) override;
@@ -117,33 +111,29 @@ public:
* Usecases:
* Parent thread Creation:
* - Thread = CreateThread(InitialRIP, InitialStack, nullptr, 0);
* - CTX->RunUntilExit(Thread);
* - CTX->ExecuteThread(Thread);
* OS thread Creation:
* - Thread = CreateThread(0, 0, NewState, PPID);
* - Thread->ExecutionThread = FEXCore::Threads::Thread::Create(ThreadHandler, Arg);
* - ThreadHandler calls `CTX->ExecutionThread(Thread)`
* - ThreadHandler calls `CTX->ExecuteThread(Thread)`
* OS fork (New thread created with a clone of thread state):
* - clone{2, 3}
* - Thread = CreateThread(0, 0, CopyOfThreadState, PPID);
* - ExecutionThread(Thread); // Starts executing without creating another host thread
* - ExecuteThread(Thread); // Starts executing without creating another host thread
* Thunk callback executing guest code from native host thread
* - Thread = CreateThread(0, 0, NewState, PPID);
* - InitializeThreadTLSData(Thread);
* - HandleCallback(Thread, RIP);
*/
FEXCore::Core::InternalThreadState*
CreateThread(uint64_t InitialRIP, uint64_t StackPointer, FEXCore::Core::CPUState* NewThreadState, uint64_t ParentTID) override;
// Public for threading
void ExecutionThread(FEXCore::Core::InternalThreadState* Thread) override;
CreateThread(uint64_t InitialRIP, uint64_t StackPointer, const FEXCore::Core::CPUState* NewThreadState, uint64_t ParentTID) override;
/**
* @brief Destroys this FEX thread object and stops tracking it internally
*
* @param Thread The internal FEX thread state object
*/
void DestroyThread(FEXCore::Core::InternalThreadState* Thread, bool NeedsTLSUninstall) override;
void DestroyThread(FEXCore::Core::InternalThreadState* Thread) override;
#ifndef _WIN32
void LockBeforeFork(FEXCore::Core::InternalThreadState* Thread) override;
@@ -151,6 +141,7 @@ public:
#endif
void SetSignalDelegator(FEXCore::SignalDelegator* SignalDelegation) override;
void SetSyscallHandler(FEXCore::HLE::SyscallHandler* Handler) override;
void SetThunkHandler(FEXCore::ThunkHandler* Handler) override;
FEXCore::CPUID::FunctionResults RunCPUIDFunction(uint32_t Function, uint32_t Leaf) override;
FEXCore::CPUID::XCRResults RunXCRFunction(uint32_t Function) override;
@@ -190,9 +181,10 @@ public:
bool IsAddressInCodeBuffer(FEXCore::Core::InternalThreadState* Thread, uintptr_t Address) const override;
// returns false if a handler was already registered
CustomIRResult AddCustomIREntrypoint(uintptr_t Entrypoint, CustomIREntrypointHandler Handler, void* Creator = nullptr, void* Data = nullptr);
std::optional<CustomIRResult>
AddCustomIREntrypoint(uintptr_t Entrypoint, CustomIREntrypointHandler Handler, void* Creator = nullptr, void* Data = nullptr);
void AppendThunkDefinitions(std::span<const FEXCore::IR::ThunkDefinition> Definitions) override;
void AddThunkTrampolineIRHandler(uintptr_t Entrypoint, uintptr_t GuestThunkEntrypoint) override;
public:
friend class FEXCore::HLE::SyscallHandler;
@@ -203,7 +195,6 @@ public:
friend class FEXCore::IR::Validation::IRValidation;
struct {
CoreRunningMode RunningMode {CoreRunningMode::MODE_RUN};
uint64_t VirtualMemSize {1ULL << 36};
uint64_t TSCScale = 0;
@@ -223,12 +214,8 @@ public:
FEX_CONFIG_OPT(AOTIRGenerate, AOTIRGENERATE);
FEX_CONFIG_OPT(AOTIRLoad, AOTIRLOAD);
FEX_CONFIG_OPT(SMCChecks, SMCCHECKS);
FEX_CONFIG_OPT(Core, CORE);
FEX_CONFIG_OPT(MaxInstPerBlock, MAXINST);
FEX_CONFIG_OPT(RootFSPath, ROOTFS);
FEX_CONFIG_OPT(ThunkHostLibsPath, THUNKHOSTLIBS);
FEX_CONFIG_OPT(ThunkHostLibsPath32, THUNKHOSTLIBS32);
FEX_CONFIG_OPT(ThunkConfigFile, THUNKCONFIG);
FEX_CONFIG_OPT(GlobalJITNaming, GLOBALJITNAMING);
FEX_CONFIG_OPT(LibraryJITNaming, LIBRARYJITNAMING);
FEX_CONFIG_OPT(BlockJITNaming, BLOCKJITNAMING);
@@ -242,8 +229,6 @@ public:
FEX_CONFIG_OPT(StrictInProcessSplitLocks, STRICTINPROCESSSPLITLOCKS);
} Config;
std::atomic_bool CoreShuttingDown {false};
FEXCore::ForkableSharedMutex CodeInvalidationMutex;
uint32_t StrictSplitLockMutex {};
@@ -253,16 +238,9 @@ public:
FEXCore::CPUIDEmu CPUID;
FEXCore::HLE::SyscallHandler* SyscallHandler {};
FEXCore::HLE::SourcecodeResolver* SourcecodeResolver {};
fextl::unique_ptr<FEXCore::ThunkHandler> ThunkHandler;
FEXCore::ThunkHandler* ThunkHandler {};
fextl::unique_ptr<FEXCore::CPU::Dispatcher> Dispatcher;
CustomCPUFactoryType CustomCPUFactory;
FEXCore::Context::ExitHandler CustomExitHandler;
#ifdef BLOCKSTATS
fextl::unique_ptr<FEXCore::BlockSamplingData> BlockData;
#endif
SignalDelegator* SignalDelegation {};
X86GeneratedCode X86CodeGen;
@@ -293,7 +271,8 @@ public:
void RemoveCustomIREntrypoint(uintptr_t Entrypoint);
struct GenerateIRResult {
fextl::unique_ptr<FEXCore::IR::IRStorageBase> IR;
std::optional<IR::IRListView> IRView;
IR::RegisterAllocationData* RAData;
uint64_t TotalInstructions;
uint64_t TotalInstructionsLength;
uint64_t StartAddr;
@@ -304,28 +283,17 @@ public:
struct CompileCodeResult {
void* CompiledCode;
fextl::unique_ptr<FEXCore::IR::IRStorageBase> IR;
FEXCore::Core::DebugData* DebugData;
bool GeneratedIR;
fextl::unique_ptr<FEXCore::Core::DebugData> DebugData;
uint64_t StartAddr;
uint64_t Length;
};
[[nodiscard]]
CompileCodeResult CompileCode(FEXCore::Core::InternalThreadState* Thread, uint64_t GuestRIP, uint64_t MaxInst = 0);
uintptr_t CompileBlock(FEXCore::Core::CpuStateFrame* Frame, uint64_t GuestRIP, uint64_t MaxInst = 0);
uintptr_t CompileSingleStep(FEXCore::Core::CpuStateFrame* Frame, uint64_t GuestRIP);
// Used for thread creation from syscalls
/**
* @brief Initializes TID, PID and TLS data for a thread
*
* @param Thread The internal FEX thread state object
*/
void InitializeThreadTLSData(FEXCore::Core::InternalThreadState* Thread);
void CopyMemoryMapping(FEXCore::Core::InternalThreadState* ParentThread, FEXCore::Core::InternalThreadState* ChildThread);
uint8_t GetGPRSize() const {
return Config.Is64BitMode ? 8 : 4;
IR::OpSize GetGPROpSize() const {
return Config.Is64BitMode ? IR::OpSize::i64Bit : IR::OpSize::i32Bit;
}
FEXCore::JITSymbols Symbols;
@@ -368,6 +336,10 @@ protected:
AtomicTSOEmulationEnabled = false;
VectorAtomicTSOEmulationEnabled = false;
MemcpyAtomicTSOEmulationEnabled = false;
} else if (Config.ParanoidTSO) {
AtomicTSOEmulationEnabled = true;
VectorAtomicTSOEmulationEnabled = true;
MemcpyAtomicTSOEmulationEnabled = true;
} else {
// Atomic TSO emulation only enabled if the config option is enabled.
AtomicTSOEmulationEnabled = (IsMemoryShared || !Config.TSOAutoMigration) && Config.TSOEnabled;
@@ -393,7 +365,6 @@ private:
IR::AOTIRCaptureCache IRCaptureCache;
fextl::unique_ptr<FEXCore::CodeSerialize::CodeObjectSerializeService> CodeObjectCacheService;
bool StartPaused = false;
bool IsMemoryShared = false;
bool SupportsHardwareTSO = false;
bool AtomicTSOEmulationEnabled = true;
@@ -1,10 +1,8 @@
// SPDX-License-Identifier: MIT
#include "Interface/Core/ArchHelpers/Arm64Emitter.h"
#include "FEXCore/Core/X86Enums.h"
#include "FEXCore/Utils/AllocatorHooks.h"
#include "Interface/Core/Dispatcher/Dispatcher.h"
#include "Interface/Context/Context.h"
#include "Interface/HLE/Thunks/Thunks.h"
#include <FEXCore/Core/CoreState.h>
#include <FEXCore/Utils/LogManager.h>
@@ -95,6 +93,7 @@ namespace x64 {
ARMEmitter::Reg::r20,
ARMEmitter::Reg::r21,
ARMEmitter::Reg::r22,
// PF/AF must be last.
REG_PF,
REG_AF,
};
@@ -611,7 +610,7 @@ void Arm64Emitter::FillSpecialRegs(ARMEmitter::Register TmpReg, ARMEmitter::Regi
}
#endif
if (SetPredRegs) {
if (SetPredRegs && (EmitterCTX->HostFeatures.SupportsSVE256 || EmitterCTX->HostFeatures.SupportsSVE128)) {
// Set up predicate registers.
// We don't bother spilling these in SpillStaticRegs,
// since all that matters is we restore them on a fill.
@@ -623,6 +622,9 @@ void Arm64Emitter::FillSpecialRegs(ARMEmitter::Register TmpReg, ARMEmitter::Regi
if (EmitterCTX->HostFeatures.SupportsSVE128) {
ptrue(ARMEmitter::SubRegSize::i8Bit, PRED_TMP_16B, ARMEmitter::PredicatePattern::SVE_VL16);
}
// Fill in the predicate register for the x87 ldst SVE optimization.
ptrue(ARMEmitter::SubRegSize::i16Bit, PRED_X87_SVEOPT, ARMEmitter::PredicatePattern::SVE_VL5);
}
}
@@ -1047,7 +1049,7 @@ void Arm64Emitter::FillForPreserveAllABICall(bool FPRs) {
}
// Fill the static registers.
FillStaticRegs(true, PreserveSRAMask, PreserveSRAFPRMask);
FillStaticRegs(FPRs, PreserveSRAMask, PreserveSRAFPRMask);
// Pop the vector registers.
PopVectorRegisters(CanUseSVE256, DynamicFPRs);
@@ -18,10 +18,8 @@
#include <CodeEmitter/Emitter.h>
#include <CodeEmitter/Registers.h>
#include <array>
#include <cstddef>
#include <cstdint>
#include <utility>
#include <span>
namespace FEXCore::Context {
@@ -48,6 +46,10 @@ constexpr auto REG_AF = ARMEmitter::Reg::r27;
// Vector temporaries
constexpr auto VTMP1 = ARMEmitter::VReg::v0;
constexpr auto VTMP2 = ARMEmitter::VReg::v1;
// Predicate register for X87 SVE Optimization
constexpr auto SVE_OPT_PRED = ARMEmitter::PReg::p2;
#else
constexpr auto TMP1 = ARMEmitter::XReg::x10;
constexpr auto TMP2 = ARMEmitter::XReg::x11;
@@ -67,6 +69,9 @@ constexpr auto VTMP2 = ARMEmitter::VReg::v17;
constexpr auto EC_CALL_CHECKER_PC_REG = ARMEmitter::XReg::x9;
constexpr auto EC_ENTRY_CPUAREA_REG = ARMEmitter::XReg::x17;
// Predicate register for X87 SVE Optimization
constexpr auto SVE_OPT_PRED = ARMEmitter::PReg::p2;
// These structures are not included in the standard Windows headers, define the offsets of members we care about for EC here.
constexpr size_t TEB_CPU_AREA_OFFSET = 0x1788;
constexpr size_t TEB_PEB_OFFSET = 0x60;
@@ -74,8 +79,16 @@ constexpr size_t PEB_EC_CODE_BITMAP_OFFSET = 0x368;
constexpr size_t CPU_AREA_IN_SYSCALL_CALLBACK_OFFSET = 0x1;
constexpr size_t CPU_AREA_EMULATOR_STACK_BASE_OFFSET = 0x8;
constexpr size_t CPU_AREA_EMULATOR_DATA_OFFSET = 0x30;
constexpr uint64_t EC_CODE_BITMAP_MAX_ADDRESS = 1ULL << 47;
#endif
// Will force one single instruction block to be generated first if set when entering the JIT filling SRA.
constexpr auto ENTRY_FILL_SRA_SINGLE_INST_REG = TMP1;
// Predicate to use in the X87 SVE optimization
constexpr ARMEmitter::PRegister PRED_X87_SVEOPT = ARMEmitter::PReg::p2;
// Predicate register temporaries (used when AVX support is enabled)
// PRED_TMP_16B indicates a predicate register that indicates the first 16 bytes set to 1.
// PRED_TMP_32B indicates a predicate register that indicates the first 32 bytes set to 1.
@@ -1,51 +0,0 @@
// SPDX-License-Identifier: MIT
#include "Interface/Core/BlockSamplingData.h"
#include <FEXCore/Utils/LogManager.h>
#include <cstring>
#include <fstream>
#include <utility>
namespace FEXCore {
void BlockSamplingData::DumpBlockData() {
std::fstream Output;
Output.open("output.csv", std::fstream::out | std::fstream::binary);
if (!Output.is_open()) {
return;
}
Output << "Entry, Min, Max, Total, Calls, Average" << std::endl;
for (auto it : SamplingMap) {
if (!it.second->TotalCalls) {
continue;
}
Output << "0x" << std::hex << it.first << ", " << std::dec << it.second->Min << ", " << std::dec << it.second->Max << ", " << std::dec
<< it.second->TotalTime << ", " << std::dec << it.second->TotalCalls << ", " << std::dec
<< ((double)it.second->TotalTime / (double)it.second->TotalCalls) << std::endl;
}
Output.close();
LogMan::Msg::DFmt("Dumped {} blocks of sampling data", SamplingMap.size());
}
BlockSamplingData::BlockData* BlockSamplingData::GetBlockData(uint64_t RIP) {
auto it = SamplingMap.find(RIP);
if (it != SamplingMap.end()) {
return it->second;
}
BlockData* NewData = new BlockData {};
memset(NewData, 0, sizeof(BlockData));
NewData->Min = ~0ULL;
SamplingMap[RIP] = NewData;
return NewData;
}
BlockSamplingData::~BlockSamplingData() {
DumpBlockData();
for (auto it : SamplingMap) {
delete it.second;
}
SamplingMap.clear();
}
} // namespace FEXCore
@@ -1,25 +0,0 @@
// SPDX-License-Identifier: MIT
#pragma once
#include <cstdint>
#include <unordered_map>
namespace FEXCore {
class BlockSamplingData {
public:
struct BlockData {
uint64_t Start, End;
uint64_t Min, Max;
uint64_t TotalTime;
uint64_t TotalCalls;
};
BlockData* GetBlockData(uint64_t RIP);
~BlockSamplingData();
void DumpBlockData();
private:
std::unordered_map<uint64_t, BlockData*> SamplingMap;
};
} // namespace FEXCore
+10 -1
View File
@@ -39,6 +39,15 @@ namespace CPU {
{0xC90F'DAA2'2168'C235ULL, 0x0000'0000'0000'4000ULL}, // NAMED_VECTOR_X87_PI
{0x9A20'9A84'FBCF'F799ULL, 0x0000'0000'0000'3FFDULL}, // NAMED_VECTOR_X87_LOG10_2
{0xB172'17F7'D1CF'79ACULL, 0x0000'0000'0000'3FFEULL}, // NAMED_VECTOR_X87_LOG_2
{0x4F00'0000'4F00'0000ULL, 0x4F00'0000'4F00'0000ULL}, // NAMED_VECTOR_CVTMAX_F32_I32
{0x4F00'0000'4F00'0000ULL, 0x4F00'0000'4F00'0000ULL}, // NAMED_VECTOR_CVTMAX_F32_I32_UPPER
{0x5F00'0000'5F00'0000ULL, 0x5F00'0000'5F00'0000ULL}, // NAMED_VECTOR_CVTMAX_F32_I64
{0x41E0'0000'0000'0000ULL, 0x41E0'0000'0000'0000ULL}, // NAMED_VECTOR_CVTMAX_F64_I32
{0x41E0'0000'0000'0000ULL, 0x41E0'0000'0000'0000ULL}, // NAMED_VECTOR_CVTMAX_F64_I32_UPPER
{0x43E0'0000'0000'0000ULL, 0x43E0'0000'0000'0000ULL}, // NAMED_VECTOR_CVTMAX_F64_I64
{0x8000'0000'8000'0000ULL, 0x8000'0000'8000'0000ULL}, // NAMED_VECTOR_CVTMAX_I32
{0x8000'0000'0000'0000ULL, 0x8000'0000'0000'0000ULL}, // NAMED_VECTOR_CVTMAX_I64
{0x0000'0000'0000'0000ULL, 0x0000'0000'0000'8000ULL}, // NAMED_VECTOR_F80_SIGN_MASK
};
constexpr static auto PSHUFLW_LUT {[]() consteval {
@@ -364,7 +373,7 @@ namespace CPU {
CodeBuffer Buffer;
Buffer.Size = Size;
Buffer.Ptr = static_cast<uint8_t*>(FEXCore::Allocator::VirtualAlloc(Buffer.Size, true));
LOGMAN_THROW_AA_FMT(!!Buffer.Ptr, "Couldn't allocate code buffer");
LOGMAN_THROW_A_FMT(!!Buffer.Ptr, "Couldn't allocate code buffer");
if (static_cast<Context::ContextImpl*>(ThreadState->CTX)->Config.GlobalJITNaming()) {
static_cast<Context::ContextImpl*>(ThreadState->CTX)->Symbols.RegisterJITSpace(Buffer.Ptr, Buffer.Size);
+12 -46
View File
@@ -48,11 +48,6 @@ namespace CPU {
CPUBackend(FEXCore::Core::InternalThreadState* ThreadState, size_t InitialCodeSize, size_t MaxCodeSize);
virtual ~CPUBackend();
/**
* @return The name of this backend
*/
[[nodiscard]]
virtual fextl::string GetName() = 0;
struct CompiledCode {
// Where this code block begins.
@@ -85,9 +80,16 @@ namespace CPU {
struct JITCodeTail {
// The total size of the codeblock from [BlockBegin, BlockBegin+Size).
size_t Size;
// RIP that the block's entry comes from.
uint64_t RIP;
// The length of the guest code for this block.
size_t GuestSize;
// If this block represents a single guest instruction.
bool SingleInst;
// Number of RIP entries for this JIT Code section.
uint32_t NumberOfRIPEntries;
@@ -100,22 +102,6 @@ namespace CPU {
uint32_t _Pad;
};
// Entries that live after the JITCodeTail.
// These entries correlate JIT code regions with guest RIP regions.
// Using these entries FEX is able to reconstruct the guest RIP accurately when an instruction cause a signal fault.
// Packed using 16-bit entries to ensure the size isn't too large.
// These smaller sizes means that each entry is relative to each other instead of absolute offset from the start of the JIT block.
// When reconstructing the RIP, each entry must be walked linearly and accumulated with the previous entries.
// This is a trade-off between compression inside the JIT code space and execution time when reconstruction the RIP.
// RIP reconstruction when faulting is less likely so we are requiring the accumulation.
struct JITRIPReconstructEntries {
// The Host PC offset from the previous entry.
uint16_t HostPCOffset;
// How much to offset the RIP from the previous entry.
uint16_t GuestRIPOffset;
};
/**
* @brief Tells this CPUBackend to compile code for the provided IR and DebugData
*
@@ -124,17 +110,17 @@ namespace CPU {
*
* This is a thread specific compilation unit since there is one CPUBackend per guest thread
*
* If NeedsOpDispatch is returning false then IR and DebugData may be null and the expectation is that the code will still compile
* FEXCore::Core::ThreadState* is valid at the time of compilation.
*
* @param Size - The byte size of the guest code for this block
* @param SingleInst - If this block represents a single guest instruction
* @param IR - IR that maps to the IR for this RIP
* @param DebugData - Debug data that is available for this IR indirectly
* @param CheckTF - If EFLAGS.TF checks should be emitted at the start of the block
*
* @return Information about the compiled code block.
*/
[[nodiscard]]
virtual CompiledCode CompileCode(uint64_t Entry, const FEXCore::IR::IRListView* IR, FEXCore::Core::DebugData* DebugData,
const FEXCore::IR::RegisterAllocationData* RAData) = 0;
virtual CompiledCode CompileCode(uint64_t Entry, uint64_t Size, bool SingleInst, const FEXCore::IR::IRListView* IR,
FEXCore::Core::DebugData* DebugData, const FEXCore::IR::RegisterAllocationData* RAData, bool CheckTF) = 0;
/**
* @brief Relocates a block of code from the JIT code object cache
@@ -149,26 +135,6 @@ namespace CPU {
return nullptr;
}
/**
* @brief Function for mapping memory in to the CPUBackend's visible space. Allows setting up virtual mappings if required
*
* @return Currently unused
*/
[[nodiscard]]
virtual void* MapRegion(void* HostPtr, uint64_t GuestPtr, uint64_t Size) = 0;
/**
* @brief Lets FEXCore know if this CPUBackend needs IR and DebugData for CompileCode
*
* This is useful if the FEXCore Frontend hits an x86-64 instruction that isn't understood but can continue regardless
*
* This is useful for example, a VM based CPUbackend
*
* @return true if it needs the IR
*/
[[nodiscard]]
virtual bool NeedsOpDispatch() = 0;
virtual void ClearCache() {}
/**
+49 -17
View File
@@ -90,26 +90,45 @@ namespace ProductNames {
#endif
} // namespace ProductNames
static uint32_t GetCPUID() {
uint32_t GetCPUID_Syscall() {
uint32_t CPU {};
FHU::Syscalls::getcpu(&CPU, nullptr);
return CPU;
}
struct CPUFamily {
uint32_t Stepping : 4;
uint32_t Model : 4;
uint32_t ExtendedModel : 4;
uint32_t FamilyID : 4;
uint32_t ExtendedFamilyID : 8;
uint32_t ProcessorType : 4;
};
constexpr static uint32_t GenerateFamily(const CPUFamily Family) {
return Family.Stepping | (Family.Model << 4) | (Family.FamilyID << 8) | (Family.ProcessorType << 12) | (Family.ExtendedModel << 16) |
(Family.ExtendedFamilyID << 20);
}
#ifdef CPUID_AMD
constexpr uint32_t FAMILY_IDENTIFIER = 0 | // Stepping
(0xA << 4) | // Model
(0xF << 8) | // Family ID
(0 << 12) | // Processor type
(0 << 16) | // Extended model ID
(1 << 20); // Extended family ID
constexpr uint32_t FAMILY_IDENTIFIER = GenerateFamily(CPUFamily {
.Stepping = 0,
.Model = 0xA,
.ExtendedModel = 0,
.FamilyID = 0xF,
.ExtendedFamilyID = 1,
.ProcessorType = 0,
});
#else
constexpr uint32_t FAMILY_IDENTIFIER = 0 | // Stepping
(0x7 << 4) | // Model
(0x6 << 8) | // Family ID
(0 << 12) | // Processor type
(1 << 16) | // Extended model ID
(0x0 << 20); // Extended family ID
constexpr uint32_t FAMILY_IDENTIFIER = GenerateFamily(CPUFamily {
.Stepping = 1,
.Model = 6,
.ExtendedModel = 0xA,
.FamilyID = 6,
.ExtendedFamilyID = 0,
.ProcessorType = 0,
});
#endif
#ifdef _M_ARM_64
@@ -119,6 +138,12 @@ uint32_t GetCycleCounterFrequency() {
return Result;
}
uint32_t GetCPUID_TPIDRRO() {
uint64_t Result {};
__asm("mrs %[Res], TPIDRRO_EL0" : [Res] "=r"(Result));
return Result;
}
void CPUIDEmu::SetupHostHybridFlag() {
PerCPUData.resize(Cores);
@@ -167,7 +192,6 @@ void CPUIDEmu::SetupHostHybridFlag() {
{0x41, 0xd4e, 1, ProductNames::ARM_X3}, // X3
{0x41, 0xd4d, 1, ProductNames::ARM_A715}, // A715
{0x41, 0xd4f, 1, ProductNames::ARM_V2}, // V2
{0x41, 0xd49, 1, ProductNames::ARM_N2}, // N2
{0x41, 0xd4b, 1, ProductNames::ARM_A78C}, // A78C
{0x41, 0xd4a, 1, ProductNames::ARM_E1}, // E1
{0x41, 0xd49, 1, ProductNames::ARM_N2}, // N2
@@ -876,11 +900,11 @@ FEXCore::CPUID::FunctionResults CPUIDEmu::Function_8000_0000h(uint32_t Leaf) con
// Extended processor and feature bits
FEXCore::CPUID::FunctionResults CPUIDEmu::Function_8000_0001h(uint32_t Leaf) const {
// RDTSCP is disabled on WIN32/Wine because there is no sane way to query processor ID.
#ifndef _WIN32
constexpr uint32_t SUPPORTS_RDTSCP = 1;
#else
constexpr uint32_t SUPPORTS_RDTSCP = 0;
// RDTSCP under WIN32 is only supported if CPUIndex is available in TPIDRRO.
const uint32_t SUPPORTS_RDTSCP = SupportsCPUIndexInTPIDRRO;
#endif
FEXCore::CPUID::FunctionResults Res {};
@@ -1194,12 +1218,20 @@ FEXCore::CPUID::XCRResults CPUIDEmu::XCRFunction_0h() const {
}
CPUIDEmu::CPUIDEmu(const FEXCore::Context::ContextImpl* ctx)
: CTX {ctx} {
: CTX {ctx}
, SupportsCPUIndexInTPIDRRO {CTX->HostFeatures.SupportsCPUIndexInTPIDRRO}
, GetCPUID {GetCPUID_Syscall} {
Cores = CTX->HostFeatures.CPUMIDRs.size();
// Setup some state tracking
SetupHostHybridFlag();
SetupFeatures();
#ifdef _M_ARM_64
if (SupportsCPUIndexInTPIDRRO) {
GetCPUID = GetCPUID_TPIDRRO;
}
#endif
}
} // namespace FEXCore
+4
View File
@@ -115,6 +115,7 @@ public:
private:
const FEXCore::Context::ContextImpl* CTX;
bool SupportsCPUIndexInTPIDRRO {};
bool Hybrid {};
uint32_t Cores {};
FEX_CONFIG_OPT(HideHypervisorBit, HIDEHYPERVISORBIT);
@@ -510,5 +511,8 @@ private:
// 0x8000'001F: AMD Secure Encryption
{SupportsConstant::CONSTANT, NeedsLeafConstant::NOLEAFCONSTANT},
}};
using GetCPUIDPtr = uint32_t (*)();
GetCPUIDPtr GetCPUID;
};
} // namespace FEXCore
+166 -201
View File
@@ -9,18 +9,16 @@ $end_info$
*/
#include <cstdint>
#include "Interface/Context/Context.h"
#include "Interface/Core/ArchHelpers//Arm64Emitter.h"
#include "Interface/Core/ArchHelpers/Arm64Emitter.h"
#include "Interface/Core/LookupCache.h"
#include "Interface/Core/CPUBackend.h"
#include "Interface/Core/CPUID.h"
#include "Interface/Core/Frontend.h"
#include "Interface/Core/ObjectCache/ObjectCacheService.h"
#include "Interface/Core/OpcodeDispatcher.h"
#include "Interface/Core/JIT/JITCore.h"
#include "Interface/Core/JIT/JITClass.h"
#include "Interface/Core/Dispatcher/Dispatcher.h"
#include "Interface/Core/X86Tables/X86Tables.h"
#include "Interface/HLE/Thunks/Thunks.h"
#include "Interface/IR/IR.h"
#include "Interface/IR/IREmitter.h"
#include "Interface/IR/Passes/RegisterAllocationPass.h"
@@ -30,11 +28,13 @@ $end_info$
#include "Utils/Allocator.h"
#include "Utils/Allocator/HostAllocator.h"
#include "Utils/SpinWaitLock.h"
#include "Utils/variable_length_integer.h"
#include <FEXCore/Config/Config.h>
#include <FEXCore/Core/Context.h>
#include <FEXCore/Core/CoreState.h>
#include <FEXCore/Core/SignalDelegator.h>
#include <FEXCore/Core/Thunks.h>
#include <FEXCore/Core/X86Enums.h>
#include <FEXCore/Debug/InternalThreadState.h>
#include <FEXCore/HLE/SyscallHandler.h>
@@ -79,9 +79,6 @@ ContextImpl::ContextImpl(const FEXCore::HostFeatures& Features)
: HostFeatures {Features}
, CPUID {this}
, IRCaptureCache {this} {
#ifdef BLOCKSTATS
BlockData = std::make_unique<FEXCore::BlockSamplingData>();
#endif
if (Config.CacheObjectCodeCompilation() != FEXCore::Config::ConfigObjectCodeHandler::CONFIG_NONE) {
CodeObjectCacheService = fextl::make_unique<FEXCore::CodeSerialize::CodeObjectSerializeService>(this);
}
@@ -116,31 +113,59 @@ ContextImpl::~ContextImpl() {
}
}
uint64_t ContextImpl::RestoreRIPFromHostPC(FEXCore::Core::InternalThreadState* Thread, uint64_t HostPC) {
const auto Frame = Thread->CurrentFrame;
struct GetFrameBlockInfoResult {
const CPU::CPUBackend::JITCodeHeader* InlineHeader;
const CPU::CPUBackend::JITCodeTail* InlineTail;
};
static GetFrameBlockInfoResult GetFrameBlockInfo(FEXCore::Core::CpuStateFrame* Frame) {
const uint64_t BlockBegin = Frame->State.InlineJITBlockHeader;
auto InlineHeader = reinterpret_cast<const CPU::CPUBackend::JITCodeHeader*>(BlockBegin);
if (InlineHeader) {
auto InlineTail = reinterpret_cast<const CPU::CPUBackend::JITCodeTail*>(Frame->State.InlineJITBlockHeader + InlineHeader->OffsetToBlockTail);
auto RIPEntries = reinterpret_cast<const CPU::CPUBackend::JITRIPReconstructEntries*>(
Frame->State.InlineJITBlockHeader + InlineHeader->OffsetToBlockTail + InlineTail->OffsetToRIPEntries);
return {InlineHeader, InlineTail};
}
return {InlineHeader, nullptr};
}
bool ContextImpl::IsAddressInCurrentBlock(FEXCore::Core::InternalThreadState* Thread, uint64_t Address, uint64_t Size) {
auto [_, InlineTail] = GetFrameBlockInfo(Thread->CurrentFrame);
return InlineTail && (Address + Size > InlineTail->RIP && Address < InlineTail->RIP + InlineTail->GuestSize);
}
bool ContextImpl::IsCurrentBlockSingleInst(FEXCore::Core::InternalThreadState* Thread) {
auto [_, InlineTail] = GetFrameBlockInfo(Thread->CurrentFrame);
return InlineTail && InlineTail->SingleInst;
}
uint64_t ContextImpl::RestoreRIPFromHostPC(FEXCore::Core::InternalThreadState* Thread, uint64_t HostPC) {
const auto Frame = Thread->CurrentFrame;
const uint64_t BlockBegin = Frame->State.InlineJITBlockHeader;
auto [InlineHeader, InlineTail] = GetFrameBlockInfo(Thread->CurrentFrame);
if (InlineHeader) {
// Check if the host PC is currently within a code block.
// If it is then RIP can be reconstructed from the beginning of the code block.
// This is currently as close as FEX can get RIP reconstructions.
if (HostPC >= reinterpret_cast<uint64_t>(BlockBegin) && HostPC < reinterpret_cast<uint64_t>(BlockBegin + InlineTail->Size)) {
auto RIPEntry =
reinterpret_cast<const uint8_t*>(Frame->State.InlineJITBlockHeader + InlineHeader->OffsetToBlockTail + InlineTail->OffsetToRIPEntries);
// Reconstruct RIP from JIT entries for this block.
uint64_t StartingHostPC = BlockBegin;
uint64_t StartingGuestRIP = InlineTail->RIP;
for (uint32_t i = 0; i < InlineTail->NumberOfRIPEntries; ++i) {
const auto& RIPEntry = RIPEntries[i];
if (HostPC >= (StartingHostPC + RIPEntry.HostPCOffset)) {
auto HostPCOffset = FEXCore::Utils::vl64::Decode(RIPEntry);
RIPEntry += HostPCOffset.Size;
auto GuestRIPOffset = FEXCore::Utils::vl64::Decode(RIPEntry);
RIPEntry += GuestRIPOffset.Size;
if (HostPC >= (StartingHostPC + HostPCOffset.Integer)) {
// We are beyond this entry, keep going forward.
StartingHostPC += RIPEntry.HostPCOffset;
StartingGuestRIP += RIPEntry.GuestRIPOffset;
StartingHostPC += HostPCOffset.Integer;
StartingGuestRIP += GuestRIPOffset.Integer;
} else {
// Passed where the Host PC is at. Break now.
break;
@@ -154,7 +179,8 @@ uint64_t ContextImpl::RestoreRIPFromHostPC(FEXCore::Core::InternalThreadState* T
return Frame->State.rip;
}
uint32_t ContextImpl::ReconstructCompactedEFLAGS(FEXCore::Core::InternalThreadState* Thread, bool WasInJIT, uint64_t* HostGPRs, uint64_t PSTATE) {
uint32_t ContextImpl::ReconstructCompactedEFLAGS(FEXCore::Core::InternalThreadState* Thread, bool WasInJIT, const uint64_t* HostGPRs,
uint64_t PSTATE) {
const auto Frame = Thread->CurrentFrame;
uint32_t EFLAGS {};
@@ -164,6 +190,7 @@ uint32_t ContextImpl::ReconstructCompactedEFLAGS(FEXCore::Core::InternalThreadSt
case X86State::RFLAG_CF_RAW_LOC:
case X86State::RFLAG_PF_RAW_LOC:
case X86State::RFLAG_AF_RAW_LOC:
case X86State::RFLAG_TF_RAW_LOC:
case X86State::RFLAG_ZF_RAW_LOC:
case X86State::RFLAG_SF_RAW_LOC:
case X86State::RFLAG_OF_RAW_LOC:
@@ -216,6 +243,9 @@ uint32_t ContextImpl::ReconstructCompactedEFLAGS(FEXCore::Core::InternalThreadSt
uint32_t AF = ((Frame->State.af_raw ^ PFByte) & (1 << 4)) ? 1 : 0;
EFLAGS |= AF << X86State::RFLAG_AF_RAW_LOC;
uint8_t TFByte = Frame->State.flags[X86State::RFLAG_TF_RAW_LOC];
EFLAGS |= (TFByte & 1) << X86State::RFLAG_TF_RAW_LOC;
// DF is pretransformed, undo the transform from 1/-1 back to 0/1
uint8_t DFByte = Frame->State.flags[X86State::RFLAG_DF_RAW_LOC];
if (DFByte & 0x80) {
@@ -322,8 +352,6 @@ bool ContextImpl::InitCore() {
// Set up the SignalDelegator config since core is initialized.
FEXCore::SignalDelegator::SignalDelegatorConfig SignalConfig {
.SupportsAVX = HostFeatures.SupportsAVX,
.DispatcherBegin = Dispatcher->Start,
.DispatcherEnd = Dispatcher->End,
@@ -352,7 +380,6 @@ bool ContextImpl::InitCore() {
SignalDelegation->SetConfig(SignalConfig);
#ifndef _WIN32
ThunkHandler = FEXCore::ThunkHandler::Create();
#elif !defined(_M_ARM64EC)
// WOW64 always needs the interrupt fault check to be enabled.
Config.NeedsPendingInterruptFaultCheck = true;
@@ -361,8 +388,6 @@ bool ContextImpl::InitCore() {
if (Config.GdbServer) {
// If gdbserver is enabled then this needs to be enabled.
Config.NeedsPendingInterruptFaultCheck = true;
// FEX needs to start paused when gdb is enabled.
StartPaused = true;
}
return true;
@@ -372,32 +397,17 @@ void ContextImpl::HandleCallback(FEXCore::Core::InternalThreadState* Thread, uin
static_cast<ContextImpl*>(Thread->CTX)->Dispatcher->ExecuteJITCallback(Thread->CurrentFrame, RIP);
}
FEXCore::Context::ExitReason ContextImpl::RunUntilExit(FEXCore::Core::InternalThreadState* Thread) {
ExecutionThread(Thread);
CoreShuttingDown.store(true);
if (CustomExitHandler) {
CustomExitHandler(Thread, FEXCore::Context::ExitReason::EXIT_SHUTDOWN);
return Thread->ExitReason;
}
return FEXCore::Context::ExitReason::EXIT_SHUTDOWN;
}
void ContextImpl::ExecuteThread(FEXCore::Core::InternalThreadState* Thread) {
Dispatcher->ExecuteDispatch(Thread->CurrentFrame);
}
void ContextImpl::InitializeThreadTLSData(FEXCore::Core::InternalThreadState* Thread) {
// Let's do some initial bookkeeping here
if (ThunkHandler) {
ThunkHandler->RegisterTLSState(Thread);
if (CodeObjectCacheService) {
// Ensure the Code Object Serialization service has fully serialized this thread's data before clearing the cache
// Use the thread's object cache ref counter for this
CodeSerialize::CodeObjectSerializeService::WaitForEmptyJobQueue(&Thread->ObjectCacheRefCounter);
}
#ifndef _WIN32
Alloc::OSAllocator::RegisterTLSData(Thread);
#endif
// If it is the parent thread that died then just leave
FEX_TODO("This doesn't make sense when the parent thread doesn't outlive its children");
}
void ContextImpl::InitializeCompiler(FEXCore::Core::InternalThreadState* Thread) {
@@ -412,29 +422,23 @@ void ContextImpl::InitializeCompiler(FEXCore::Core::InternalThreadState* Thread)
Dispatcher->InitThreadPointers(Thread);
Thread->CTX = this;
Thread->PassManager->AddDefaultPasses(this);
Thread->PassManager->AddDefaultValidationPasses();
Thread->PassManager->RegisterSyscallHandler(SyscallHandler);
// Create CPU backend
switch (Config.Core) {
case FEXCore::Config::CONFIG_IRJIT:
Thread->PassManager->InsertRegisterAllocationPass();
Thread->CPUBackend = FEXCore::CPU::CreateArm64JITCore(this, Thread);
break;
case FEXCore::Config::CONFIG_CUSTOM: Thread->CPUBackend = CustomCPUFactory(this, Thread); break;
default: ERROR_AND_DIE_FMT("Unknown core configuration"); break;
}
Thread->PassManager->InsertRegisterAllocationPass();
Thread->CPUBackend = FEXCore::CPU::CreateArm64JITCore(this, Thread);
Thread->PassManager->Finalize();
}
FEXCore::Core::InternalThreadState*
ContextImpl::CreateThread(uint64_t InitialRIP, uint64_t StackPointer, FEXCore::Core::CPUState* NewThreadState, uint64_t ParentTID) {
FEXCore::Core::InternalThreadState* Thread = new FEXCore::Core::InternalThreadState {};
ContextImpl::CreateThread(uint64_t InitialRIP, uint64_t StackPointer, const FEXCore::Core::CPUState* NewThreadState, uint64_t ParentTID) {
FEXCore::Core::InternalThreadState* Thread = new FEXCore::Core::InternalThreadState {
.CTX = this,
};
Thread->CurrentFrame->State.gregs[X86State::REG_RSP] = StackPointer;
Thread->CurrentFrame->State.rip = InitialRIP;
@@ -459,13 +463,7 @@ ContextImpl::CreateThread(uint64_t InitialRIP, uint64_t StackPointer, FEXCore::C
return Thread;
}
void ContextImpl::DestroyThread(FEXCore::Core::InternalThreadState* Thread, bool NeedsTLSUninstall) {
if (NeedsTLSUninstall) {
#ifndef _WIN32
Alloc::OSAllocator::UninstallTLSData(Thread);
#endif
}
void ContextImpl::DestroyThread(FEXCore::Core::InternalThreadState* Thread) {
FEXCore::Allocator::VirtualProtect(&Thread->InterruptFaultPage, sizeof(Thread->InterruptFaultPage),
Allocator::ProtectOptions::Read | Allocator::ProtectOptions::Write);
delete Thread;
@@ -505,7 +503,7 @@ void ContextImpl::AddBlockMapping(FEXCore::Core::InternalThreadState* Thread, ui
void ContextImpl::ClearCodeCache(FEXCore::Core::InternalThreadState* Thread) {
FEXCORE_PROFILE_INSTANT("ClearCodeCache");
{
if (CodeObjectCacheService) {
// Ensure the Code Object Serialization service has fully serialized this thread's data before clearing the cache
// Use the thread's object cache ref counter for this
CodeSerialize::CodeObjectSerializeService::WaitForEmptyJobQueue(&Thread->ObjectCacheRefCounter);
@@ -524,40 +522,6 @@ static void IRDumper(FEXCore::Core::InternalThreadState* Thread, IR::IREmitter*
fextl::fmt::print(FD, "IR-ShouldDump-{} 0x{:x}:\n{}\n@@@@@\n", RA ? "post" : "pre", GuestRIP, out.str());
};
// IRStorageBase with fully owned memory
struct IRListCopy : public IR::IRStorageBase {
std::span<std::byte> IRData;
std::span<std::byte> ListData;
// TODO: Consider defaulting to empty RAData instead?
IR::RegisterAllocationData::UniquePtr RADataInternal;
IRListCopy(const IR::IRListView& view, IR::RegisterAllocationData::UniquePtr RAData)
: RADataInternal(std::move(RAData)) {
std::byte* Storage = reinterpret_cast<std::byte*>(FEXCore::Allocator::malloc(view.GetDataSize() + view.GetListSize()));
IRData = {Storage, Storage + view.GetDataSize()};
ListData = {Storage + view.GetDataSize(), Storage + view.GetDataSize() + view.GetListSize()};
memcpy(IRData.data(), (char*)view.GetData(), IRData.size());
memcpy(ListData.data(), (char*)view.GetListData(), ListData.size());
}
IRListCopy(const IRListCopy& other) = delete;
IRListCopy(IRListCopy&& other) = delete;
~IRListCopy() {
FEXCore::Allocator::free(IRData.data());
}
const IR::RegisterAllocationData* RAData() override {
return RADataInternal.get();
}
IR::IRListView GetIRView() override {
return IR::IRListView {IRData.data(), ListData.data(), IRData.size(), ListData.size()};
}
};
ContextImpl::GenerateIRResult
ContextImpl::GenerateIR(FEXCore::Core::InternalThreadState* Thread, uint64_t GuestRIP, bool ExtendedDebugInfo, uint64_t MaxInst) {
FEXCORE_PROFILE_SCOPED("GenerateIR");
@@ -586,6 +550,7 @@ ContextImpl::GenerateIR(FEXCore::Core::InternalThreadState* Thread, uint64_t Gue
GuestCode = reinterpret_cast<const uint8_t*>(GuestRIP);
bool HadDispatchError {false};
bool HadInvalidInst {false};
Thread->FrontendDecoder->DecodeInstructionsAtEntry(GuestCode, GuestRIP, MaxInst,
[Thread](uint64_t BlockEntry, uint64_t Start, uint64_t Length) {
@@ -599,7 +564,7 @@ ContextImpl::GenerateIR(FEXCore::Core::InternalThreadState* Thread, uint64_t Gue
Thread->OpDispatcher->BeginFunction(GuestRIP, CodeBlocks, BlockInfo->TotalInstructionCount);
const uint8_t GPRSize = GetGPRSize();
const auto GPRSize = GetGPROpSize();
for (size_t j = 0; j < CodeBlocks->size(); ++j) {
const FEXCore::Frontend::Decoder::DecodedBlocks& Block = CodeBlocks->at(j);
@@ -615,7 +580,7 @@ ContextImpl::GenerateIR(FEXCore::Core::InternalThreadState* Thread, uint64_t Gue
if (InstsInBlock == 0) {
// Special case for an empty instruction block.
Thread->OpDispatcher->ExitFunction(Thread->OpDispatcher->_EntrypointOffset(IR::SizeToOpSize(GPRSize), Block.Entry - GuestRIP));
Thread->OpDispatcher->ExitFunction(Thread->OpDispatcher->_EntrypointOffset(GPRSize, Block.Entry - GuestRIP));
}
for (size_t i = 0; i < InstsInBlock; ++i) {
@@ -658,8 +623,7 @@ ContextImpl::GenerateIR(FEXCore::Core::InternalThreadState* Thread, uint64_t Gue
Thread->OpDispatcher->SetCurrentCodeBlock(CodeWasChangedBlock);
Thread->OpDispatcher->_ThreadRemoveCodeEntry();
Thread->OpDispatcher->ExitFunction(
Thread->OpDispatcher->_EntrypointOffset(IR::SizeToOpSize(GPRSize), Block.Entry + BlockInstructionsLength - GuestRIP));
Thread->OpDispatcher->ExitFunction(Thread->OpDispatcher->_EntrypointOffset(GPRSize, Block.Entry + BlockInstructionsLength - GuestRIP));
auto NextOpBlock = Thread->OpDispatcher->CreateNewCodeBlockAfter(CurrentBlock);
@@ -684,30 +648,34 @@ ContextImpl::GenerateIR(FEXCore::Core::InternalThreadState* Thread, uint64_t Gue
++TotalInstructions;
}
} else {
if (TableInfo) {
LogMan::Msg::EFmt("Invalid or Unknown instruction: {} 0x{:x}", TableInfo->Name ?: "UND", Block.Entry - GuestRIP);
}
// Invalid instruction
Thread->OpDispatcher->InvalidOp(DecodedInfo);
Thread->OpDispatcher->ExitFunction(Thread->OpDispatcher->_EntrypointOffset(IR::SizeToOpSize(GPRSize), Block.Entry - GuestRIP));
if (!BlockInstructionsLength) {
// SMC can modify block contents and patch invalid instructions to valid ones inline.
// End blocks upon encountering them and only emit an invalid opcode exception if there are no prior instructions in the block (that could have modified it to be valid).
if (TableInfo) {
LogMan::Msg::EFmt("Invalid or Unknown instruction: {} 0x{:x}", TableInfo->Name ?: "UND", Block.Entry - GuestRIP);
}
Thread->OpDispatcher->InvalidOp(DecodedInfo);
}
HadInvalidInst = true;
}
const bool NeedsBlockEnd =
(HadDispatchError && TotalInstructions > 0) || (Thread->OpDispatcher->NeedsBlockEnder() && i + 1 == InstsInBlock);
const bool NeedsBlockEnd = (HadDispatchError && TotalInstructions > 0) ||
(Thread->OpDispatcher->NeedsBlockEnder() && i + 1 == InstsInBlock) || HadInvalidInst;
// If we had a dispatch error then leave early
if (HadDispatchError && TotalInstructions == 0) {
// Couldn't handle any instruction in op dispatcher
Thread->OpDispatcher->ResetWorkingList();
return {nullptr, 0, 0, 0, 0};
return {{}, nullptr, 0, 0, 0, 0};
}
if (NeedsBlockEnd) {
const uint8_t GPRSize = GetGPRSize();
// We had some instructions. Early exit
Thread->OpDispatcher->ExitFunction(
Thread->OpDispatcher->_EntrypointOffset(IR::SizeToOpSize(GPRSize), Block.Entry + BlockInstructionsLength - GuestRIP));
Thread->OpDispatcher->ExitFunction(Thread->OpDispatcher->_EntrypointOffset(GPRSize, Block.Entry + BlockInstructionsLength - GuestRIP));
break;
}
@@ -734,19 +702,16 @@ ContextImpl::GenerateIR(FEXCore::Core::InternalThreadState* Thread, uint64_t Gue
// Run the passmanager over the IR from the dispatcher
Thread->PassManager->Run(IREmitter);
auto RAData = Thread->PassManager->HasPass("RA") ? Thread->PassManager->GetPass<IR::RegisterAllocationPass>("RA")->GetAllocationData() : nullptr;
// Debug
if (ShouldDump) {
IRDumper(Thread, IREmitter, GuestRIP,
Thread->PassManager->HasPass("RA") ? Thread->PassManager->GetPass<IR::RegisterAllocationPass>("RA")->GetAllocationData() : nullptr);
IRDumper(Thread, IREmitter, GuestRIP, RAData);
}
auto RAData = Thread->PassManager->HasPass("RA") ? Thread->PassManager->GetPass<IR::RegisterAllocationPass>("RA")->PullAllocationData() : nullptr;
auto IRList = fextl::make_unique<IRListCopy>(IREmitter->ViewIR(), std::move(RAData));
IREmitter->DelayedDisownBuffer();
return {
.IR = std::move(IRList),
.IRView = IREmitter->ViewIR(),
.RAData = RAData,
.TotalInstructions = TotalInstructions,
.TotalInstructionsLength = TotalInstructionsLength,
.StartAddr = Thread->FrontendDecoder->DecodedMinAddress,
@@ -763,9 +728,7 @@ ContextImpl::CompileCodeResult ContextImpl::CompileCode(FEXCore::Core::InternalT
if (CompiledCode) {
return {
.CompiledCode = CompiledCode,
.IR = nullptr, // No IR/RA data generated
.DebugData = nullptr, // nullptr here ensures that code serialization doesn't occur on from cache read
.GeneratedIR = false, // nullptr here ensures IR cache mechanisms won't run
.StartAddr = 0, // Unused
.Length = 0, // Unused
};
@@ -780,47 +743,30 @@ ContextImpl::CompileCodeResult ContextImpl::CompileCode(FEXCore::Core::InternalT
}
}
fextl::unique_ptr<FEXCore::IR::IRStorageBase> IR;
FEXCore::Core::DebugData* DebugData {};
uint64_t StartAddr {};
uint64_t Length {};
// AOT IR bookkeeping and cache
{
auto IRFromAOT = IRCaptureCache.PreGenerateIRFetch(Thread, GuestRIP);
if (IRFromAOT) {
// Setup pointers to internal structures
IR = std::move(IRFromAOT->IR);
DebugData = IRFromAOT->DebugData;
StartAddr = IRFromAOT->StartAddr;
Length = IRFromAOT->Length;
}
// Generate IR + Meta Info
auto [IRView, RAData, TotalInstructions, TotalInstructionsLength, StartAddr, Length] =
GenerateIR(Thread, GuestRIP, Config.GDBSymbols(), MaxInst);
if (!IRView) {
return {nullptr, nullptr, 0, 0};
}
auto DebugData = fextl::make_unique<FEXCore::Core::DebugData>();
if (!IR) {
// Generate IR + Meta Info
auto [IRCopy, TotalInstructions, TotalInstructionsLength, _StartAddr, _Length] = GenerateIR(Thread, GuestRIP, Config.GDBSymbols(), MaxInst);
// If the trap flag is set we generate single instruction blocks that each check to generate a single step exception.
bool TFSet = Thread->CurrentFrame->State.flags[X86State::RFLAG_TF_RAW_LOC];
// Setup pointers to internal structures
IR = std::move(IRCopy);
DebugData = new FEXCore::Core::DebugData();
StartAddr = _StartAddr;
Length = _Length;
}
if (!IR) {
return {};
}
// Attempt to get the CPU backend to compile this code
auto IRView = IR->GetIRView();
auto CompiledCode = Thread->CPUBackend->CompileCode(GuestRIP, Length, TotalInstructions == 1, &*IRView, DebugData.get(), RAData, TFSet);
// Release the IR
Thread->OpDispatcher->DelayedDisownBuffer();
return {
// FEX currently throws away the CPUBackend::CompiledCode object other than the entrypoint
// In the future with code caching getting wired up, we will pass the rest of the data forward.
// TODO: Pass the data forward when code caching is wired up to this.
.CompiledCode = Thread->CPUBackend->CompileCode(GuestRIP, &IRView, DebugData, IR->RAData()).BlockEntry,
.IR = std::move(IR),
.DebugData = DebugData,
.GeneratedIR = true,
.CompiledCode = CompiledCode.BlockEntry,
.DebugData = std::move(DebugData),
.StartAddr = StartAddr,
.Length = Length,
};
@@ -839,7 +785,7 @@ uintptr_t ContextImpl::CompileBlock(FEXCore::Core::CpuStateFrame* Frame, uint64_
return HostCode;
}
auto [CodePtr, IR, DebugData, GeneratedIR, StartAddr, Length] = CompileCode(Thread, GuestRIP, MaxInst);
auto [CodePtr, DebugData, StartAddr, Length] = CompileCode(Thread, GuestRIP, MaxInst);
if (CodePtr == nullptr) {
return 0;
}
@@ -890,7 +836,7 @@ uintptr_t ContextImpl::CompileBlock(FEXCore::Core::CpuStateFrame* Frame, uint64_
// Clear any relocations that might have been generated
Thread->CPUBackend->ClearRelocations();
if (IRCaptureCache.PostCompileCode(Thread, CodePtr, GuestRIP, StartAddr, Length, std::move(IR), DebugData, GeneratedIR)) {
if (IRCaptureCache.PostCompileCode(Thread, CodePtr, GuestRIP, StartAddr, Length, {}, DebugData.get(), false)) {
// Early exit
return (uintptr_t)CodePtr;
}
@@ -902,43 +848,22 @@ uintptr_t ContextImpl::CompileBlock(FEXCore::Core::CpuStateFrame* Frame, uint64_
return (uintptr_t)CodePtr;
}
void ContextImpl::ExecutionThread(FEXCore::Core::InternalThreadState* Thread) {
Thread->ExitReason = FEXCore::Context::ExitReason::EXIT_WAITING;
uintptr_t ContextImpl::CompileSingleStep(FEXCore::Core::CpuStateFrame* Frame, uint64_t GuestRIP) {
FEXCORE_PROFILE_SCOPED("CompileSingleStep");
auto Thread = Frame->Thread;
InitializeThreadTLSData(Thread);
// Invalidate might take a unique lock on this, to guarantee that during invalidation no code gets compiled
auto lk = GuardSignalDeferringSection<std::shared_lock>(CodeInvalidationMutex, Thread);
// Now notify the thread that we are initialized
Thread->ThreadWaiting.NotifyAll();
if (StartPaused || Thread->StartPaused) {
// Parent thread doesn't need to wait to run
Thread->StartRunning.Wait();
auto [CodePtr, DebugData, StartAddr, Length] = CompileCode(Thread, GuestRIP, 1);
if (CodePtr == nullptr) {
return 0;
}
if (!Thread->RunningEvents.EarlyExit.load()) {
Thread->RunningEvents.WaitingToStart = false;
// Clear any relocations that might have been generated
Thread->CPUBackend->ClearRelocations();
Thread->ExitReason = FEXCore::Context::ExitReason::EXIT_NONE;
Thread->RunningEvents.Running = true;
static_cast<ContextImpl*>(Thread->CTX)->Dispatcher->ExecuteDispatch(Thread->CurrentFrame);
Thread->RunningEvents.Running = false;
}
{
// Ensure the Code Object Serialization service has fully serialized this thread's data before clearing the cache
// Use the thread's object cache ref counter for this
CodeSerialize::CodeObjectSerializeService::WaitForEmptyJobQueue(&Thread->ObjectCacheRefCounter);
}
// If it is the parent thread that died then just leave
FEX_TODO("This doesn't make sense when the parent thread doesn't outlive its children");
#ifndef _WIN32
Alloc::OSAllocator::UninstallTLSData(Thread);
#endif
return (uintptr_t)CodePtr;
}
static void InvalidateGuestThreadCodeRange(FEXCore::Core::InternalThreadState* Thread, uint64_t Start, uint64_t Length) {
@@ -966,6 +891,10 @@ void ContextImpl::InvalidateGuestCodeRange(FEXCore::Core::InternalThreadState* T
}
void ContextImpl::MarkMemoryShared(FEXCore::Core::InternalThreadState* Thread) {
if (!Thread) {
return;
}
if (!IsMemoryShared) {
IsMemoryShared = true;
UpdateAtomicTSOEmulationConfig();
@@ -994,7 +923,8 @@ void ContextImpl::ThreadRemoveCodeEntry(FEXCore::Core::InternalThreadState* Thre
Thread->LookupCache->Erase(Thread->CurrentFrame, GuestRIP);
}
CustomIRResult ContextImpl::AddCustomIREntrypoint(uintptr_t Entrypoint, CustomIREntrypointHandler Handler, void* Creator, void* Data) {
std::optional<CustomIRResult>
ContextImpl::AddCustomIREntrypoint(uintptr_t Entrypoint, CustomIREntrypointHandler Handler, void* Creator, void* Data) {
LOGMAN_THROW_A_FMT(Config.Is64BitMode || !(Entrypoint >> 32), "64-bit Entrypoint in 32-bit mode {:x}", Entrypoint);
std::unique_lock lk(CustomIRMutex);
@@ -1004,10 +934,51 @@ CustomIRResult ContextImpl::AddCustomIREntrypoint(uintptr_t Entrypoint, CustomIR
if (!InsertedIterator.second) {
const auto& [fn, Creator, Data] = InsertedIterator.first->second;
return CustomIRResult(std::move(lk), Creator, Data);
} else {
lk.unlock();
return CustomIRResult(std::move(lk), 0, 0);
return CustomIRResult(Creator, Data);
}
return std::nullopt;
}
void ContextImpl::AddThunkTrampolineIRHandler(uintptr_t Entrypoint, uintptr_t GuestThunkEntrypoint) {
LOGMAN_THROW_A_FMT(Entrypoint, "Tried to link null pointer address to guest function");
LOGMAN_THROW_A_FMT(GuestThunkEntrypoint, "Tried to link address to null pointer guest function");
if (!Config.Is64BitMode) {
LOGMAN_THROW_A_FMT((Entrypoint >> 32) == 0, "Tried to link 64-bit address in 32-bit mode");
LOGMAN_THROW_A_FMT((GuestThunkEntrypoint >> 32) == 0, "Tried to link 64-bit address in 32-bit mode");
}
LogMan::Msg::DFmt("Thunks: Adding guest trampoline from address {:#x} to guest function {:#x}", Entrypoint, GuestThunkEntrypoint);
auto Result = AddCustomIREntrypoint(
Entrypoint,
[this, GuestThunkEntrypoint](uintptr_t Entrypoint, FEXCore::IR::IREmitter* emit) {
auto IRHeader = emit->_IRHeader(emit->Invalid(), Entrypoint, 0, 0);
auto Block = emit->CreateCodeNode();
IRHeader.first->Blocks = emit->WrapNode(Block);
emit->SetCurrentCodeBlock(Block);
const auto GPRSize = GetGPROpSize();
if (GPRSize == IR::OpSize::i64Bit) {
emit->_StoreRegister(emit->_Constant(Entrypoint), X86State::REG_R11, IR::GPRClass, GPRSize);
} else {
emit->_StoreContext(GPRSize, IR::FPRClass, emit->_VCastFromGPR(IR::OpSize::i64Bit, IR::OpSize::i64Bit, emit->_Constant(Entrypoint)),
offsetof(Core::CPUState, mm[0][0]));
}
emit->_ExitFunction(emit->_Constant(GuestThunkEntrypoint));
},
ThunkHandler, (void*)GuestThunkEntrypoint);
if (Result.has_value()) {
if (Result->Creator != ThunkHandler) {
ERROR_AND_DIE_FMT("Input address for AddThunkTrampoline is already linked by another module");
}
if (Result->Data != (void*)GuestThunkEntrypoint) {
// NOTE: This may happen in Vulkan thunks if the Vulkan driver resolves two different symbols
// to the same function (e.g. vkGetPhysicalDeviceFeatures2/vkGetPhysicalDeviceFeatures2KHR)
LogMan::Msg::EFmt("Input address for AddThunkTrampoline is already linked elsewhere");
}
}
}
@@ -1030,12 +1001,6 @@ void ContextImpl::UnloadAOTIRCacheEntry(IR::AOTIRCacheEntry* Entry) {
IRCaptureCache.UnloadAOTIRCacheEntry(Entry);
}
void ContextImpl::AppendThunkDefinitions(std::span<const FEXCore::IR::ThunkDefinition> Definitions) {
if (ThunkHandler) {
ThunkHandler->AppendThunkDefinitions(Definitions);
}
}
void ContextImpl::ConfigureAOTGen(FEXCore::Core::InternalThreadState* Thread, fextl::set<uint64_t>* ExternalBranches, uint64_t SectionMaxAddress) {
Thread->FrontendDecoder->SetExternalBranches(ExternalBranches);
Thread->FrontendDecoder->SetSectionMaxAddress(SectionMaxAddress);
@@ -46,6 +46,8 @@ Dispatcher::~Dispatcher() {
}
void Dispatcher::EmitDispatcher() {
// Don't modify TMP3 since it contains our RIP once the block doesn't exist
auto RipReg = TMP3;
#ifdef VIXL_DISASSEMBLER
const auto DisasmBegin = GetCursorAddress<const vixl::aarch64::Instruction*>();
#endif
@@ -61,8 +63,9 @@ void Dispatcher::EmitDispatcher() {
// }
ARMEmitter::ForwardLabel l_CTX;
ARMEmitter::SingleUseForwardLabel l_Sleep;
ARMEmitter::SingleUseForwardLabel l_CompileBlock;
ARMEmitter::ForwardLabel l_Sleep;
ARMEmitter::ForwardLabel l_CompileBlock;
ARMEmitter::ForwardLabel l_CompileSingleStep;
// Push all the register we need to save
PushCalleeSavedRegisters();
@@ -81,6 +84,7 @@ void Dispatcher::EmitDispatcher() {
FillStaticRegs();
ARMEmitter::BiDirectionalLabel LoopTop {};
ARMEmitter::ForwardLabel CompileSingleStep;
#ifdef _M_ARM_64EC
b(&LoopTop);
@@ -89,6 +93,10 @@ void Dispatcher::EmitDispatcher() {
ldr(STATE, EC_ENTRY_CPUAREA_REG, CPU_AREA_EMULATOR_DATA_OFFSET);
FillStaticRegs();
ldr(RipReg, STATE_PTR(CpuStateFrame, State.rip));
// Force a single instruction block if ENTRY_FILL_SRA_SINGLE_INST_REG is nonzero entering the JIT, used for inline SMC handling.
cbnz(ARMEmitter::Size::i32Bit, ENTRY_FILL_SRA_SINGLE_INST_REG, &CompileSingleStep);
// Enter JIT
b(&LoopTop);
@@ -116,10 +124,11 @@ void Dispatcher::EmitDispatcher() {
AbsoluteLoopTopAddress = GetCursorAddress<uint64_t>();
// Load in our RIP
// Don't modify TMP3 since it contains our RIP once the block doesn't exist
auto RipReg = TMP3;
ldr(RipReg, STATE_PTR(CpuStateFrame, State.rip));
ldrb(TMP1, STATE_PTR(CpuStateFrame, State.flags[X86State::RFLAG_TF_RAW_LOC]));
cbnz(ARMEmitter::Size::i32Bit, TMP1, &CompileSingleStep);
// L1 Cache
ldr(TMP1, STATE_PTR(CpuStateFrame, Pointers.Common.L1Pointer));
@@ -204,37 +213,21 @@ void Dispatcher::EmitDispatcher() {
ret();
}
{
ExitFunctionLinkerAddress = GetCursorAddress<uint64_t>();
SpillStaticRegs(TMP1);
// Clobbers TMP1/2
auto EmitSignalGuardedRegion = [&](auto Body) {
#ifndef _WIN32
ldr(ARMEmitter::XReg::x0, STATE, offsetof(FEXCore::Core::CPUState, DeferredSignalRefCount));
add(ARMEmitter::Size::i64Bit, ARMEmitter::XReg::x0, ARMEmitter::XReg::x0, 1);
str(ARMEmitter::XReg::x0, STATE, offsetof(FEXCore::Core::CPUState, DeferredSignalRefCount));
ldr(TMP2, STATE, offsetof(FEXCore::Core::CPUState, DeferredSignalRefCount));
add(ARMEmitter::Size::i64Bit, TMP2, TMP2, 1);
str(TMP2, STATE, offsetof(FEXCore::Core::CPUState, DeferredSignalRefCount));
#endif
#ifdef _M_ARM_64EC
ldr(ARMEmitter::XReg::x0, ARMEmitter::XReg::x18, TEB_CPU_AREA_OFFSET);
LoadConstant(ARMEmitter::Size::i32Bit, ARMEmitter::Reg::r1, 1);
strb(ARMEmitter::WReg::w1, ARMEmitter::XReg::x0, CPU_AREA_IN_SYSCALL_CALLBACK_OFFSET);
ldr(TMP2, ARMEmitter::XReg::x18, TEB_CPU_AREA_OFFSET);
LoadConstant(ARMEmitter::Size::i32Bit, TMP1, 1);
strb(TMP1.W(), TMP2, CPU_AREA_IN_SYSCALL_CALLBACK_OFFSET);
#endif
mov(ARMEmitter::XReg::x0, STATE);
mov(ARMEmitter::XReg::x1, ARMEmitter::XReg::lr);
ldr(ARMEmitter::XReg::x2, STATE_PTR(CpuStateFrame, Pointers.Common.ExitFunctionLink));
if (!CTX->Config.DisableVixlIndirectCalls) [[unlikely]] {
GenerateIndirectRuntimeCall<uintptr_t, void*, void*>(ARMEmitter::Reg::r2);
} else {
blr(ARMEmitter::Reg::r2);
}
if (!TMP_ABIARGS) {
mov(TMP1, ARMEmitter::XReg::x0);
}
FillStaticRegs();
Body();
#ifdef _M_ARM_64EC
ldr(TMP2, ARMEmitter::XReg::x18, TEB_CPU_AREA_OFFSET);
@@ -250,17 +243,38 @@ void Dispatcher::EmitDispatcher() {
strb(ARMEmitter::XReg::zr, STATE,
offsetof(FEXCore::Core::InternalThreadState, InterruptFaultPage) - offsetof(FEXCore::Core::InternalThreadState, BaseFrameState));
#endif
};
{
ExitFunctionLinkerAddress = GetCursorAddress<uint64_t>();
EmitSignalGuardedRegion([&]() {
SpillStaticRegs(TMP1);
mov(ARMEmitter::XReg::x0, STATE);
mov(ARMEmitter::XReg::x1, ARMEmitter::XReg::lr);
ldr(ARMEmitter::XReg::x2, STATE_PTR(CpuStateFrame, Pointers.Common.ExitFunctionLink));
if (!CTX->Config.DisableVixlIndirectCalls) [[unlikely]] {
GenerateIndirectRuntimeCall<uintptr_t, void*, void*>(ARMEmitter::Reg::r2);
} else {
blr(ARMEmitter::Reg::r2);
}
if (!TMP_ABIARGS) {
mov(TMP1, ARMEmitter::XReg::x0);
}
FillStaticRegs();
});
br(TMP1);
}
// Need to create the block
{
Bind(&NoBlock);
#ifdef _M_ARM_64EC
// Clobbers TMP1/2
auto EmitECExitCheck = [&]() {
// Check the EC code bitmap incase we need to exit the JIT to call into native code.
ARMEmitter::SingleUseForwardLabel l_NotECCode;
ARMEmitter::ForwardLabel l_NotECCode;
ldr(TMP1, ARMEmitter::XReg::x18, TEB_PEB_OFFSET);
ldr(TMP1, TMP1, PEB_EC_CODE_BITMAP_OFFSET);
@@ -277,56 +291,83 @@ void Dispatcher::EmitDispatcher() {
br(TMP2);
Bind(&l_NotECCode);
};
#endif
SpillStaticRegs(TMP1);
if (!TMP_ABIARGS) {
mov(ARMEmitter::XReg::x2, RipReg);
}
#ifndef _WIN32
ldr(ARMEmitter::XReg::x0, STATE, offsetof(FEXCore::Core::CPUState, DeferredSignalRefCount));
add(ARMEmitter::Size::i64Bit, ARMEmitter::XReg::x0, ARMEmitter::XReg::x0, 1);
str(ARMEmitter::XReg::x0, STATE, offsetof(FEXCore::Core::CPUState, DeferredSignalRefCount));
#endif
// Need to create the block
{
Bind(&NoBlock);
#ifdef _M_ARM_64EC
ldr(ARMEmitter::XReg::x0, ARMEmitter::XReg::x18, TEB_CPU_AREA_OFFSET);
LoadConstant(ARMEmitter::Size::i32Bit, ARMEmitter::Reg::r1, 1);
strb(ARMEmitter::WReg::w1, ARMEmitter::XReg::x0, CPU_AREA_IN_SYSCALL_CALLBACK_OFFSET);
EmitECExitCheck();
#endif
ldr(ARMEmitter::XReg::x0, &l_CTX);
mov(ARMEmitter::XReg::x1, STATE);
// x2 contains guest RIP
mov(ARMEmitter::XReg::x3, 0);
ldr(ARMEmitter::XReg::x4, &l_CompileBlock);
EmitSignalGuardedRegion([&]() {
SpillStaticRegs(TMP1);
if (!CTX->Config.DisableVixlIndirectCalls) [[unlikely]] {
GenerateIndirectRuntimeCall<uintptr_t, void*, void*, uint64_t, uint64_t>(ARMEmitter::Reg::r4);
} else {
blr(ARMEmitter::Reg::r4); // { CTX, Frame, RIP, MaxInst }
}
if (!TMP_ABIARGS) {
mov(ARMEmitter::XReg::x2, RipReg);
}
FillStaticRegs();
ldr(ARMEmitter::XReg::x0, &l_CTX);
mov(ARMEmitter::XReg::x1, STATE);
// x2 contains guest RIP
mov(ARMEmitter::XReg::x3, 0);
ldr(ARMEmitter::XReg::x4, &l_CompileBlock);
if (!CTX->Config.DisableVixlIndirectCalls) [[unlikely]] {
GenerateIndirectRuntimeCall<uintptr_t, void*, void*, uint64_t, uint64_t>(ARMEmitter::Reg::r4);
} else {
blr(ARMEmitter::Reg::r4); // { CTX, Frame, RIP, MaxInst }
}
// Result is now in x0
if (!TMP_ABIARGS) {
mov(TMP1, ARMEmitter::XReg::x0);
}
FillStaticRegs();
});
// Jump to the compiled block
br(TMP1);
}
{
Bind(&CompileSingleStep);
#ifdef _M_ARM_64EC
ldr(TMP1, ARMEmitter::XReg::x18, TEB_CPU_AREA_OFFSET);
strb(ARMEmitter::WReg::zr, TMP1, CPU_AREA_IN_SYSCALL_CALLBACK_OFFSET);
EmitECExitCheck();
#endif
#ifndef _WIN32
ldr(TMP1, STATE, offsetof(FEXCore::Core::CPUState, DeferredSignalRefCount));
sub(ARMEmitter::Size::i64Bit, TMP1, TMP1, 1);
str(TMP1, STATE, offsetof(FEXCore::Core::CPUState, DeferredSignalRefCount));
EmitSignalGuardedRegion([&]() {
SpillStaticRegs(TMP1);
// Trigger segfault if any deferred signals are pending
strb(ARMEmitter::XReg::zr, STATE,
offsetof(FEXCore::Core::InternalThreadState, InterruptFaultPage) - offsetof(FEXCore::Core::InternalThreadState, BaseFrameState));
#endif
if (!TMP_ABIARGS) {
mov(ARMEmitter::XReg::x2, RipReg);
}
b(&LoopTop);
ldr(ARMEmitter::XReg::x0, &l_CTX);
mov(ARMEmitter::XReg::x1, STATE);
// x2 contains guest RIP
ldr(ARMEmitter::XReg::x4, &l_CompileSingleStep);
if (!CTX->Config.DisableVixlIndirectCalls) [[unlikely]] {
GenerateIndirectRuntimeCall<uintptr_t, void*, void*, uint64_t, uint64_t>(ARMEmitter::Reg::r4);
} else {
blr(ARMEmitter::Reg::r4); // { CTX, Frame, RIP }
}
// Result is now in x0
if (!TMP_ABIARGS) {
mov(TMP1, ARMEmitter::XReg::x0);
}
FillStaticRegs();
});
// Jump to the compiled block
br(TMP1);
}
{
@@ -505,8 +546,11 @@ void Dispatcher::EmitDispatcher() {
Bind(&l_Sleep);
dc64(reinterpret_cast<uint64_t>(SleepThread));
Bind(&l_CompileBlock);
FEXCore::Utils::MemberFunctionToPointerCast PMF(&FEXCore::Context::ContextImpl::CompileBlock);
dc64(PMF.GetConvertedPointer());
FEXCore::Utils::MemberFunctionToPointerCast PMFCompileBlock(&FEXCore::Context::ContextImpl::CompileBlock);
dc64(PMFCompileBlock.GetConvertedPointer());
Bind(&l_CompileSingleStep);
FEXCore::Utils::MemberFunctionToPointerCast PMFCompileSingleStep(&FEXCore::Context::ContextImpl::CompileSingleStep);
dc64(PMFCompileSingleStep.GetConvertedPointer());
Start = reinterpret_cast<uint64_t>(DispatchPtr);
End = GetCursorAddress<uint64_t>();
+171 -93
View File
@@ -75,7 +75,7 @@ Decoder::~Decoder() {
uint8_t Decoder::ReadByte() {
uint8_t Byte = InstStream[InstructionSize];
LOGMAN_THROW_AA_FMT(InstructionSize < MAX_INST_SIZE, "Max instruction size exceeded!");
LOGMAN_THROW_A_FMT(InstructionSize < MAX_INST_SIZE, "Max instruction size exceeded!");
Instruction[InstructionSize] = Byte;
InstructionSize++;
return Byte;
@@ -87,7 +87,7 @@ uint8_t Decoder::PeekByte(uint8_t Offset) const {
}
uint64_t Decoder::ReadData(uint8_t Size) {
LOGMAN_THROW_AA_FMT(Size != 0 && Size <= sizeof(uint64_t), "Unknown data size to read");
LOGMAN_THROW_A_FMT(Size != 0 && Size <= sizeof(uint64_t), "Unknown data size to read");
uint64_t Res = 0;
std::memcpy(&Res, &InstStream[InstructionSize], Size);
@@ -220,7 +220,8 @@ void Decoder::DecodeModRM_64(X86Tables::DecodedOperand* Operand, X86Tables::ModR
// The invalid encoding types are described at Table 1-12. "promoted nsigned is always non-zero"
{
// If we have a VSIB byte (as opposed to SIB), then the index register is a vector.
const bool IsIndexVector = (DecodeInst->TableInfo->Flags & InstFlags::FLAGS_VEX_VSIB) != 0;
// DecodeInst->TableInfo may be null in the case of 3DNow! ModRM decoding.
const bool IsIndexVector = DecodeInst->TableInfo && (DecodeInst->TableInfo->Flags & InstFlags::FLAGS_VEX_VSIB) != 0;
uint8_t InvalidSIBIndex = 0b100; ///< SIB Index where there is no register encoding.
if (IsIndexVector) {
DecodeInst->Flags |= X86Tables::DecodeFlags::FLAG_VSIB_BYTE;
@@ -234,7 +235,7 @@ void Decoder::DecodeModRM_64(X86Tables::DecodedOperand* Operand, X86Tables::ModR
Operand->Data.SIB.Base = MapModRMToReg(BaseREX, SIB.base, false, false, false, false, ModRM.mod == 0 ? 0b101 : 16);
}
LOGMAN_THROW_AA_FMT(Displacement <= 4, "Number of bytes should be <= 4 for literal src");
LOGMAN_THROW_A_FMT(Displacement <= 4, "Number of bytes should be <= 4 for literal src");
if (Displacement) {
uint64_t Literal = ReadData(Displacement);
@@ -281,10 +282,10 @@ bool Decoder::NormalOp(const FEXCore::X86Tables::X86InstInfo* Info, uint16_t Op,
return false;
}
LOGMAN_THROW_AA_FMT(!(Info->Type >= FEXCore::X86Tables::TYPE_GROUP_1 && Info->Type <= FEXCore::X86Tables::TYPE_GROUP_P), "Group Ops "
"should have "
"been decoded "
"before this!");
LOGMAN_THROW_A_FMT(!(Info->Type >= FEXCore::X86Tables::TYPE_GROUP_1 && Info->Type <= FEXCore::X86Tables::TYPE_GROUP_P), "Group Ops "
"should have "
"been decoded "
"before this!");
uint8_t DestSize {};
const bool HasWideningDisplacement =
@@ -403,7 +404,7 @@ bool Decoder::NormalOp(const FEXCore::X86Tables::X86InstInfo* Info, uint16_t Op,
HAS_NON_XMM_SUBFLAG(Info->Flags, FEXCore::X86Tables::InstFlags::FLAGS_SF_DST_RAX) ? FEXCore::X86State::REG_RAX : FEXCore::X86State::REG_RDX;
CurrentDest = &DecodeInst->Src[0];
} else if (HAS_NON_XMM_SUBFLAG(Info->Flags, FEXCore::X86Tables::InstFlags::FLAGS_SF_REX_IN_BYTE)) {
LOGMAN_THROW_AA_FMT(!HasMODRM, "This instruction shouldn't have ModRM!");
LOGMAN_THROW_A_FMT(!HasMODRM, "This instruction shouldn't have ModRM!");
// If the REX is in the byte that means the lower nibble of the OP contains the destination GPR
// This also means that the destination is always a GPR on these ones
@@ -521,7 +522,7 @@ bool Decoder::NormalOp(const FEXCore::X86Tables::X86InstInfo* Info, uint16_t Op,
}
if (Bytes != 0) {
LOGMAN_THROW_AA_FMT(Bytes <= 8, "Number of bytes should be <= 8 for literal src");
LOGMAN_THROW_A_FMT(Bytes <= 8, "Number of bytes should be <= 8 for literal src");
DecodeInst->Src[CurrentSrc].Data.Literal.Size = Bytes;
@@ -544,8 +545,8 @@ bool Decoder::NormalOp(const FEXCore::X86Tables::X86InstInfo* Info, uint16_t Op,
DecodeInst->Src[CurrentSrc].Data.Literal.Value = Literal;
}
LOGMAN_THROW_AA_FMT(Bytes == 0, "Inst at 0x{:x}: 0x{:04x} '{}' Had an instruction of size {} with {} remaining", DecodeInst->PC,
DecodeInst->OP, DecodeInst->TableInfo->Name ?: "UND", InstructionSize, Bytes);
LOGMAN_THROW_A_FMT(Bytes == 0, "Inst at 0x{:x}: 0x{:04x} '{}' Had an instruction of size {} with {} remaining", DecodeInst->PC,
DecodeInst->OP, DecodeInst->TableInfo->Name ?: "UND", InstructionSize, Bytes);
DecodeInst->InstSize = InstructionSize;
return true;
}
@@ -562,7 +563,7 @@ bool Decoder::NormalOpHeader(const FEXCore::X86Tables::X86InstInfo* Info, uint16
return false;
}
LOGMAN_THROW_AA_FMT(Info->Type != FEXCore::X86Tables::TYPE_REX_PREFIX, "REX PREFIX should have been decoded before this!");
LOGMAN_THROW_A_FMT(Info->Type != FEXCore::X86Tables::TYPE_REX_PREFIX, "REX PREFIX should have been decoded before this!");
// A normal instruction is the most likely.
if (Info->Type == FEXCore::X86Tables::TYPE_INST) [[likely]] {
@@ -612,7 +613,7 @@ bool Decoder::NormalOpHeader(const FEXCore::X86Tables::X86InstInfo* Info, uint16
255, 0, 1, 2, 255, 255, 255, 3,
};
uint8_t Field = RegToField[ModRM.reg];
LOGMAN_THROW_AA_FMT(Field != 255, "Invalid field selected!");
LOGMAN_THROW_A_FMT(Field != 255, "Invalid field selected!");
LocalOp = (Field << 3) | ModRM.rm;
return NormalOp(&SecondModRMTableOps[LocalOp], LocalOp);
@@ -690,12 +691,8 @@ bool Decoder::NormalOpHeader(const FEXCore::X86Tables::X86InstInfo* Info, uint16
}
} else if (Info->Type == FEXCore::X86Tables::TYPE_GROUP_EVEX) {
FEXCORE_TELEMETRY_SET(EVEXOpTelem, 1);
/* uint8_t P1 = */ ReadByte();
/* uint8_t P2 = */ ReadByte();
/* uint8_t P3 = */ ReadByte();
uint8_t EVEXOp = ReadByte();
return NormalOp(&EVEXTableOps[EVEXOp], EVEXOp);
// EVEX unsupported
return false;
}
LOGMAN_MSG_A_FMT("Invalid instruction decoding type");
@@ -930,8 +927,9 @@ void Decoder::BranchTargetInMultiblockRange() {
// If the RIP setting is conditional AND within our symbol range then it can be considered for multiblock
uint64_t TargetRIP = 0;
const uint8_t GPRSize = CTX->GetGPRSize();
const auto GPRSize = CTX->GetGPROpSize();
bool Conditional = true;
const auto InstEnd = DecodeInst->PC + DecodeInst->InstSize;
switch (DecodeInst->OP) {
case 0x70 ... 0x7F: // Conditional JUMP
@@ -940,17 +938,17 @@ void Decoder::BranchTargetInMultiblockRange() {
// auto RIPOffset = LoadSource(Op, Op->Src[0], Op->Flags);
// auto RIPTargetConst = _Constant(Op->PC + Op->InstSize);
// Target offset is PC + InstSize + Literal
TargetRIP = DecodeInst->PC + DecodeInst->InstSize + DecodeInst->Src[0].Literal();
TargetRIP = InstEnd + DecodeInst->Src[0].Literal();
break;
}
case 0xE9:
case 0xEB: // Both are unconditional JMP instructions
TargetRIP = DecodeInst->PC + DecodeInst->InstSize + DecodeInst->Src[0].Literal();
TargetRIP = InstEnd + DecodeInst->Src[0].Literal();
Conditional = false;
break;
case 0xE8: // Call - Immediate target, We don't want to inline calls
if (ExternalBranches) {
ExternalBranches->insert(DecodeInst->PC + DecodeInst->InstSize);
ExternalBranches->insert(InstEnd);
}
[[fallthrough]];
case 0xC2: // RET imm
@@ -958,13 +956,15 @@ void Decoder::BranchTargetInMultiblockRange() {
default: return; break;
}
if (GPRSize == 4) {
if (GPRSize == IR::OpSize::i32Bit) {
// If we are running a 32bit guest then wrap around addresses that go above 32bit
TargetRIP &= 0xFFFFFFFFU;
}
// If the target RIP is x86 code within the symbol ranges then we are golden
bool ValidMultiblockMember = TargetRIP >= SymbolMinAddress && TargetRIP < SymbolMaxAddress;
// Forbid cross-page branches to both avoid massive (range-wise) code blocks in highly fragmented code and trying to decode unmapped branch targets
bool ValidMultiblockMember =
TargetRIP >= SymbolMinAddress && TargetRIP < std::min(FEXCore::AlignUp(InstEnd, FEXCore::Utils::FEX_PAGE_SIZE), SymbolMaxAddress);
#ifdef _M_ARM_64EC
ValidMultiblockMember = ValidMultiblockMember && !RtlIsEcCode(TargetRIP);
@@ -977,15 +977,10 @@ void Decoder::BranchTargetInMultiblockRange() {
MaxCondBranchBackwards = std::min(MaxCondBranchBackwards, TargetRIP);
// If we are conditional then a target can be the instruction past the conditional instruction
uint64_t FallthroughRIP = DecodeInst->PC + DecodeInst->InstSize;
if (!HasBlocks.contains(FallthroughRIP)) {
CurrentBlockTargets.insert(FallthroughRIP);
}
AddBranchTarget(InstEnd);
}
if (!HasBlocks.contains(TargetRIP)) {
CurrentBlockTargets.insert(TargetRIP);
}
AddBranchTarget(TargetRIP);
} else {
if (ExternalBranches) {
ExternalBranches->insert(TargetRIP);
@@ -993,19 +988,23 @@ void Decoder::BranchTargetInMultiblockRange() {
}
}
bool Decoder::BranchTargetCanContinue(bool FinalInstruction) const {
if (FinalInstruction) {
bool Decoder::InstCanContinue() const {
if (DecodeInst->PC + DecodeInst->InstSize == NextBlockStartAddress) {
return false;
}
if (!(DecodeInst->TableInfo->Flags & (FEXCore::X86Tables::InstFlags::FLAGS_BLOCK_END | FEXCore::X86Tables::InstFlags::FLAGS_SETS_RIP))) {
return true;
}
uint64_t TargetRIP = 0;
const uint8_t GPRSize = CTX->GetGPRSize();
const auto GPRSize = CTX->GetGPROpSize();
if (DecodeInst->OP == 0xE8) { // Call - immediate target
const uint64_t NextRIP = DecodeInst->PC + DecodeInst->InstSize;
TargetRIP = DecodeInst->PC + DecodeInst->InstSize + DecodeInst->Src[0].Literal();
if (GPRSize == 4) {
if (GPRSize == IR::OpSize::i32Bit) {
// If we are running a 32bit guest then wrap around addresses that go above 32bit
TargetRIP &= 0xFFFFFFFFU;
}
@@ -1021,6 +1020,59 @@ bool Decoder::BranchTargetCanContinue(bool FinalInstruction) const {
return false;
}
void Decoder::AddBranchTarget(uint64_t Target) {
if (VisitedBlocks.contains(Target)) {
return;
}
auto BlockSuccIt = std::lower_bound(BlockInfo.Blocks.begin(), BlockInfo.Blocks.end(), Target,
[](const auto& a, uint64_t Address) { return a.Entry < Address; });
LOGMAN_THROW_A_FMT(BlockSuccIt == BlockInfo.Blocks.end() || BlockSuccIt->Entry != Target, "unexpected");
if (BlockSuccIt != BlockInfo.Blocks.begin()) {
auto BlockIt = std::prev(BlockSuccIt);
if (BlockIt->Entry + BlockIt->Size > Target) {
uint64_t SplitIdx = 0;
uint64_t SplitAddr = BlockIt->Entry;
// Find the instruction boundary of the split
for (; SplitIdx < BlockIt->NumInstructions && SplitAddr < Target; SplitIdx++) {
SplitAddr += BlockIt->DecodedInstructions[SplitIdx].InstSize;
}
uint64_t SplitOffset = SplitAddr - BlockIt->Entry;
LOGMAN_THROW_A_FMT(SplitIdx != 0, "unexpected");
if (SplitAddr == Target) {
// Split at the boundary
DecodedBlocks SplitBlock {
.Entry = SplitAddr,
.Size = BlockIt->Size - SplitOffset,
.NumInstructions = BlockIt->NumInstructions - SplitIdx,
.DecodedInstructions = BlockIt->DecodedInstructions + SplitIdx,
.HasInvalidInstruction = BlockIt->HasInvalidInstruction,
};
BlockIt->Size = SplitOffset;
BlockIt->NumInstructions = SplitIdx;
BlockInfo.Blocks.insert(BlockSuccIt, SplitBlock);
} // else misaligned, leave as a branch out of the block
// If we split a block then the target has already been visited as part of that, if it was
// misaligned the jump will just leave the multiblock, mark it as visited to avoid running
// this code path again and just bail out early.
VisitedBlocks.insert(Target);
return;
}
}
CurrentBlockTargets.insert(Target);
if (Target >= DecodeInst->PC + DecodeInst->InstSize && Target < NextBlockStartAddress) {
NextBlockStartAddress = Target;
}
}
const uint8_t* Decoder::AdjustAddrForSpecialRegion(const uint8_t* _InstStream, uint64_t EntryPoint, uint64_t RIP) {
constexpr uint64_t VSyscall_Base = 0xFFFF'FFFF'FF60'0000ULL;
constexpr uint64_t VSyscall_End = VSyscall_Base + 0x1000;
@@ -1044,7 +1096,7 @@ void Decoder::DecodeInstructionsAtEntry(const uint8_t* _InstStream, uint64_t PC,
BlockInfo.TotalInstructionCount = 0;
BlockInfo.Blocks.clear();
BlocksToDecode.clear();
HasBlocks.clear();
VisitedBlocks.clear();
// Reset internal state management
DecodedSize = 0;
MaxCondBranchForward = 0;
@@ -1082,30 +1134,61 @@ void Decoder::DecodeInstructionsAtEntry(const uint8_t* _InstStream, uint64_t PC,
}
bool EntryBlock {true};
bool FinalInstruction {false};
while (!BlocksToDecode.empty()) {
while (!FinalInstruction && !BlocksToDecode.empty()) {
auto BlockDecodeIt = BlocksToDecode.begin();
uint64_t RIPToDecode = *BlockDecodeIt;
BlockInfo.Blocks.emplace_back();
DecodedBlocks& CurrentBlockDecoding = BlockInfo.Blocks.back();
BlocksToDecode.erase(BlockDecodeIt);
VisitedBlocks.emplace(RIPToDecode);
CurrentBlockDecoding.Entry = RIPToDecode;
auto BlockSuccIt = std::lower_bound(BlockInfo.Blocks.begin(), BlockInfo.Blocks.end(), RIPToDecode,
[](const auto& a, uint64_t Address) { return a.Entry < Address; });
LOGMAN_THROW_A_FMT(BlockSuccIt == BlockInfo.Blocks.end() || BlockSuccIt->Entry != RIPToDecode, "unexpected");
NextBlockStartAddress = ~0ULL;
if (!BlocksToDecode.empty()) {
// We just erased the lowest, the front is then the second lowest
NextBlockStartAddress = *BlocksToDecode.begin();
}
if (BlockSuccIt != BlockInfo.Blocks.end() && BlockSuccIt->Entry < NextBlockStartAddress) {
NextBlockStartAddress = BlockSuccIt->Entry;
}
LOGMAN_THROW_A_FMT(NextBlockStartAddress > RIPToDecode, "unexpected");
// Insert the block now so it can be looked up and split if necessary on a backward edge
auto BlockIt = BlockInfo.Blocks.emplace(BlockSuccIt);
BlockIt->Entry = RIPToDecode;
BlockIt->Size = 0;
uint64_t PCOffset = 0;
uint64_t BlockNumberOfInstructions {};
uint64_t BlockStartOffset = DecodedSize;
bool EraseBlock = true; // Unset once the block contains an instruction
BlockIt->DecodedInstructions = &DecodedBuffer[BlockStartOffset];
BlockIt->NumInstructions = 0;
// Do a bit of pointer math to figure out where we are in code
InstStream = AdjustAddrForSpecialRegion(_InstStream, EntryPoint, RIPToDecode);
while (1) {
// MAX_INST_SIZE assumes worst case
auto OpMinAddress = RIPToDecode + PCOffset;
auto OpMaxAddress = OpMinAddress + MAX_INST_SIZE;
InstructionSize = 0;
auto OpMinPage = OpMinAddress & FEXCore::Utils::FEX_PAGE_MASK;
// MAX_INST_SIZE assumes worst case
auto OpAddress = RIPToDecode + PCOffset;
auto OpMaxAddress = OpAddress + MAX_INST_SIZE;
auto OpMinPage = OpAddress & FEXCore::Utils::FEX_PAGE_MASK;
auto OpMaxPage = OpMaxAddress & FEXCore::Utils::FEX_PAGE_MASK;
if (!EntryBlock && OpMinPage == OpMaxPage && PeekByte(0) == 0 && PeekByte(1) == 0) [[unlikely]] {
// End the multiblock early if we hit 2 consecutive null bytes (add [rax], al) in the same page with the
// assumption we are most likely trying to explore garbage code.
break;
}
if (OpMinPage != CurrentCodePage) {
CurrentCodePage = OpMinPage;
CodePages.insert(CurrentCodePage);
@@ -1116,64 +1199,66 @@ void Decoder::DecodeInstructionsAtEntry(const uint8_t* _InstStream, uint64_t PC,
CodePages.insert(CurrentCodePage);
}
bool ErrorDuringDecoding = !DecodeInstruction(RIPToDecode + PCOffset);
bool ErrorDuringDecoding = !DecodeInstruction(OpAddress);
uint64_t OpEndAddress = OpAddress + DecodeInst->InstSize;
if (ErrorDuringDecoding) [[unlikely]] {
// Put an invalid instruction in the stream so the core can raise SIGILL if hit
CurrentBlockDecoding.HasInvalidInstruction = true;
BlockIt->HasInvalidInstruction = true;
// Error while decoding instruction. We don't know the table or instruction size
DecodeInst->TableInfo = nullptr;
DecodeInst->InstSize = 0;
}
if (!ErrorDuringDecoding) {
} else {
// If there wasn't an error during decoding but we have no dispatcher for the instruction then claim invalid instruction.
auto TableInfo = DecodedBuffer[BlockStartOffset + BlockNumberOfInstructions].TableInfo;
auto TableInfo = DecodeInst->TableInfo;
if (!TableInfo || !TableInfo->OpcodeDispatcher) {
CurrentBlockDecoding.HasInvalidInstruction = true;
BlockIt->HasInvalidInstruction = true;
}
}
DecodedMinAddress = std::min(DecodedMinAddress, RIPToDecode + PCOffset);
DecodedMaxAddress = std::max(DecodedMaxAddress, RIPToDecode + PCOffset + DecodeInst->InstSize);
DecodedMinAddress = std::min(DecodedMinAddress, OpAddress);
DecodedMaxAddress = std::max(DecodedMaxAddress, OpEndAddress);
if (OpEndAddress > NextBlockStartAddress) {
// This instruction would overlap with another so skip adding it to the multiblock
break;
}
EraseBlock = false; // Block contains at least one valid instruction, so unset erase
++TotalInstructions;
++BlockNumberOfInstructions;
++DecodedSize;
++BlockIt->NumInstructions;
BlockIt->Size += DecodeInst->InstSize;
// Can not continue this block at all on invalid instruction
if (CurrentBlockDecoding.HasInvalidInstruction) [[unlikely]] {
if (BlockIt->HasInvalidInstruction) [[unlikely]] {
if (!EntryBlock) {
// In multiblock configurations, we can early terminate any non-entrypoint blocks with the expectation that this won't get hit.
// Improves compile-times.
// Just need to undo additions that this block decoding has caused.
TotalInstructions -= CurrentBlockDecoding.NumInstructions;
TotalInstructions -= BlockIt->NumInstructions;
DecodedSize = BlockStartOffset;
BlockNumberOfInstructions = 0;
InstStream -= PCOffset;
CurrentBlockTargets.clear();
EraseBlock = true;
}
break;
}
bool CanContinue = false;
if (!(DecodeInst->TableInfo->Flags & (FEXCore::X86Tables::InstFlags::FLAGS_BLOCK_END | FEXCore::X86Tables::InstFlags::FLAGS_SETS_RIP))) {
// If this isn't a block ender then we can keep going regardless
CanContinue = true;
// Check if we need to end the entire multiblock
FinalInstruction = DecodedSize >= MaxInst || DecodedSize >= DefaultDecodedBufferSize || TotalInstructions >= MaxInst;
if (FinalInstruction) {
break;
}
bool FinalInstruction = DecodedSize >= MaxInst || DecodedSize >= DefaultDecodedBufferSize || TotalInstructions >= MaxInst;
if (!InstCanContinue()) {
if (DecodeInst->TableInfo->Flags & FEXCore::X86Tables::InstFlags::FLAGS_SETS_RIP) {
// If we have multiblock enabled
// If the branch target is within our multiblock range then we can keep going on
// We don't want to short circuit this since we want to calculate our ranges still
// NOTE: This will invalidate BlockIt, this is fine as we immediately break from the loop and EraseBlock cannot be true
BranchTargetInMultiblockRange();
}
if (DecodeInst->TableInfo->Flags & FEXCore::X86Tables::InstFlags::FLAGS_SETS_RIP) {
// If we have multiblock enabled
// If the branch target is within our multiblock range then we can keep going on
// We don't want to short circuit this since we want to calculate our ranges still
BranchTargetInMultiblockRange();
// Bypass branches if we can continue through them in some cases.
CanContinue |= BranchTargetCanContinue(FinalInstruction);
}
if (FinalInstruction || !CanContinue) {
break;
}
@@ -1181,29 +1266,22 @@ void Decoder::DecodeInstructionsAtEntry(const uint8_t* _InstStream, uint64_t PC,
InstStream += DecodeInst->InstSize;
}
BlocksToDecode.merge(CurrentBlockTargets);
// NOTE: BlockIt is only valid here in the EraseBlock case
if (EraseBlock) {
BlockInfo.Blocks.erase(BlockIt);
} else {
BlocksToDecode.merge(CurrentBlockTargets);
}
CurrentBlockTargets.clear();
BlocksToDecode.erase(BlockDecodeIt);
HasBlocks.emplace(RIPToDecode);
// Copy over only the number of instructions we decoded
CurrentBlockDecoding.NumInstructions = BlockNumberOfInstructions;
CurrentBlockDecoding.DecodedInstructions = &DecodedBuffer[BlockStartOffset];
BlockInfo.TotalInstructionCount += BlockNumberOfInstructions;
EntryBlock = false;
}
BlockInfo.TotalInstructionCount = TotalInstructions;
for (auto CodePage : CodePages) {
AddContainedCodePage(PC, CodePage, FEXCore::Utils::FEX_PAGE_SIZE);
}
// sort for better branching
std::sort(BlockInfo.Blocks.begin(), BlockInfo.Blocks.end(),
[](const FEXCore::Frontend::Decoder::DecodedBlocks& a, const FEXCore::Frontend::Decoder::DecodedBlocks& b) {
return a.Entry < b.Entry;
});
}
} // namespace FEXCore::Frontend
+6 -2
View File
@@ -22,6 +22,7 @@ public:
// New Frontend decoding
struct DecodedBlocks final {
uint64_t Entry {};
uint64_t Size {};
uint64_t NumInstructions {};
FEXCore::X86Tables::DecodedInst* DecodedInstructions;
bool HasInvalidInstruction {};
@@ -70,7 +71,9 @@ private:
bool DecodeInstruction(uint64_t PC);
void BranchTargetInMultiblockRange();
bool BranchTargetCanContinue(bool FinalInstruction) const;
bool InstCanContinue() const;
void AddBranchTarget(uint64_t Target);
uint8_t ReadByte();
uint8_t PeekByte(uint8_t Offset) const;
@@ -102,11 +105,12 @@ private:
uint64_t SymbolMaxAddress {};
uint64_t SymbolMinAddress {~0ULL};
uint64_t SectionMaxAddress {~0ULL};
uint64_t NextBlockStartAddress {~0ULL};
DecodedBlockInformation BlockInfo;
fextl::set<uint64_t> CurrentBlockTargets;
fextl::set<uint64_t> BlocksToDecode;
fextl::set<uint64_t> HasBlocks;
fextl::set<uint64_t> VisitedBlocks;
fextl::set<uint64_t>* ExternalBranches {nullptr};
// ModRM rm decoding
@@ -6,17 +6,20 @@
#include "Interface/IR/IR.h"
namespace FEXCore::CPU {
FEXCORE_PRESERVE_ALL_ATTR static softfloat_state SoftFloatStateFromFCW(uint16_t FCW) {
softfloat_state State;
FEXCORE_PRESERVE_ALL_ATTR static softfloat_state SoftFloatStateFromFCW(uint16_t FCW, bool Force80BitPrecision = false) {
softfloat_state State {};
State.detectTininess = softfloat_tininess_afterRounding;
State.exceptionFlags = 0;
State.roundingPrecision = 80;
auto PC = (FCW >> 8) & 3;
switch (PC) {
case 0: State.roundingPrecision = 32; break;
case 2: State.roundingPrecision = 64; break;
case 3: State.roundingPrecision = 80; break;
case 1: LOGMAN_MSG_A_FMT("Invalid x87 precision mode, {}", PC);
if (!Force80BitPrecision) {
auto PC = (FCW >> 8) & 3;
switch (PC) {
case 0: State.roundingPrecision = 32; break;
case 2: State.roundingPrecision = 64; break;
case 3: State.roundingPrecision = 80; break;
case 1: LOGMAN_MSG_A_FMT("Invalid x87 precision mode, {}", PC);
}
}
auto RC = (FCW >> 10) & 3;
@@ -132,7 +135,7 @@ struct OpHandlers<IR::OP_F80CVTTOINT> {
template<>
struct OpHandlers<IR::OP_F80ROUND> {
FEXCORE_PRESERVE_ALL_ATTR static X80SoftFloat handle(uint16_t FCW, X80SoftFloat Src1) {
softfloat_state State = SoftFloatStateFromFCW(FCW);
softfloat_state State = SoftFloatStateFromFCW(FCW, true);
return X80SoftFloat::FRNDINT(&State, Src1);
}
};
@@ -140,7 +143,7 @@ struct OpHandlers<IR::OP_F80ROUND> {
template<>
struct OpHandlers<IR::OP_F80F2XM1> {
FEXCORE_PRESERVE_ALL_ATTR static X80SoftFloat handle(uint16_t FCW, X80SoftFloat Src1) {
softfloat_state State = SoftFloatStateFromFCW(FCW);
softfloat_state State = SoftFloatStateFromFCW(FCW, true);
return X80SoftFloat::F2XM1(&State, Src1);
}
};
@@ -148,7 +151,7 @@ struct OpHandlers<IR::OP_F80F2XM1> {
template<>
struct OpHandlers<IR::OP_F80TAN> {
FEXCORE_PRESERVE_ALL_ATTR static X80SoftFloat handle(uint16_t FCW, X80SoftFloat Src1) {
softfloat_state State = SoftFloatStateFromFCW(FCW);
softfloat_state State = SoftFloatStateFromFCW(FCW, true);
return X80SoftFloat::FTAN(&State, Src1);
}
};
@@ -164,7 +167,7 @@ struct OpHandlers<IR::OP_F80SQRT> {
template<>
struct OpHandlers<IR::OP_F80SIN> {
FEXCORE_PRESERVE_ALL_ATTR static X80SoftFloat handle(uint16_t FCW, X80SoftFloat Src1) {
softfloat_state State = SoftFloatStateFromFCW(FCW);
softfloat_state State = SoftFloatStateFromFCW(FCW, true);
return X80SoftFloat::FSIN(&State, Src1);
}
};
@@ -172,7 +175,7 @@ struct OpHandlers<IR::OP_F80SIN> {
template<>
struct OpHandlers<IR::OP_F80COS> {
FEXCORE_PRESERVE_ALL_ATTR static X80SoftFloat handle(uint16_t FCW, X80SoftFloat Src1) {
softfloat_state State = SoftFloatStateFromFCW(FCW);
softfloat_state State = SoftFloatStateFromFCW(FCW, true);
return X80SoftFloat::FCOS(&State, Src1);
}
};
@@ -226,7 +229,7 @@ struct OpHandlers<IR::OP_F80DIV> {
template<>
struct OpHandlers<IR::OP_F80FYL2X> {
FEXCORE_PRESERVE_ALL_ATTR static X80SoftFloat handle(uint16_t FCW, X80SoftFloat Src1, X80SoftFloat Src2) {
softfloat_state State = SoftFloatStateFromFCW(FCW);
softfloat_state State = SoftFloatStateFromFCW(FCW, true);
return X80SoftFloat::FYL2X(&State, Src1, Src2);
}
};
@@ -234,7 +237,7 @@ struct OpHandlers<IR::OP_F80FYL2X> {
template<>
struct OpHandlers<IR::OP_F80ATAN> {
FEXCORE_PRESERVE_ALL_ATTR static X80SoftFloat handle(uint16_t FCW, X80SoftFloat Src1, X80SoftFloat Src2) {
softfloat_state State = SoftFloatStateFromFCW(FCW);
softfloat_state State = SoftFloatStateFromFCW(FCW, true);
return X80SoftFloat::FATAN(&State, Src1, Src2);
}
};
@@ -242,7 +245,7 @@ struct OpHandlers<IR::OP_F80ATAN> {
template<>
struct OpHandlers<IR::OP_F80FPREM1> {
FEXCORE_PRESERVE_ALL_ATTR static X80SoftFloat handle(uint16_t FCW, X80SoftFloat Src1, X80SoftFloat Src2) {
softfloat_state State = SoftFloatStateFromFCW(FCW);
softfloat_state State = SoftFloatStateFromFCW(FCW, true);
return X80SoftFloat::FREM1(&State, Src1, Src2);
}
};
@@ -250,7 +253,7 @@ struct OpHandlers<IR::OP_F80FPREM1> {
template<>
struct OpHandlers<IR::OP_F80FPREM> {
FEXCORE_PRESERVE_ALL_ATTR static X80SoftFloat handle(uint16_t FCW, X80SoftFloat Src1, X80SoftFloat Src2) {
softfloat_state State = SoftFloatStateFromFCW(FCW);
softfloat_state State = SoftFloatStateFromFCW(FCW, true);
return X80SoftFloat::FREM(&State, Src1, Src2);
}
};
@@ -258,7 +261,7 @@ struct OpHandlers<IR::OP_F80FPREM> {
template<>
struct OpHandlers<IR::OP_F80SCALE> {
FEXCORE_PRESERVE_ALL_ATTR static X80SoftFloat handle(uint16_t FCW, X80SoftFloat Src1, X80SoftFloat Src2) {
softfloat_state State = SoftFloatStateFromFCW(FCW);
softfloat_state State = SoftFloatStateFromFCW(FCW, true);
return X80SoftFloat::FSCALE(&State, Src1, Src2);
}
};
@@ -322,8 +325,11 @@ struct OpHandlers<IR::OP_F64FYL2X> {
template<>
struct OpHandlers<IR::OP_F64SCALE> {
static double handle(uint16_t FCW, double src1, double src2) {
double trunc = (double)(int64_t)(src2); // truncate
return src1 * exp2(trunc);
if (src1 == 0.0) { // src1 might be +/- zero
return src1; // this will return negative or positive zero if when appropriate
}
double trun = trunc(src2);
return src1 * exp2(trun);
}
};
@@ -79,17 +79,17 @@ void InterpreterOps::FillFallbackIndexPointers(uint64_t* Info) {
}
bool InterpreterOps::GetFallbackHandler(bool SupportsPreserveAllABI, const IR::IROp_Header* IROp, FallbackInfo* Info) {
uint8_t OpSize = IROp->Size;
const auto OpSize = IROp->Size;
switch (IROp->Op) {
case IR::OP_F80CVTTO: {
auto Op = IROp->C<IR::IROp_F80CVTTo>();
switch (Op->SrcSize) {
case 4: {
case IR::OpSize::i32Bit: {
*Info = {FABI_F80_I16_F32, (void*)&FEXCore::CPU::OpHandlers<IR::OP_F80CVTTO>::handle4, Core::OPINDEX_F80CVTTO_4, SupportsPreserveAllABI};
return true;
}
case 8: {
case IR::OpSize::i64Bit: {
*Info = {FABI_F80_I16_F64, (void*)&FEXCore::CPU::OpHandlers<IR::OP_F80CVTTO>::handle8, Core::OPINDEX_F80CVTTO_8, SupportsPreserveAllABI};
return true;
}
@@ -99,11 +99,11 @@ bool InterpreterOps::GetFallbackHandler(bool SupportsPreserveAllABI, const IR::I
}
case IR::OP_F80CVT: {
switch (OpSize) {
case 4: {
case IR::OpSize::i32Bit: {
*Info = {FABI_F32_I16_F80, (void*)&FEXCore::CPU::OpHandlers<IR::OP_F80CVT>::handle4, Core::OPINDEX_F80CVT_4, SupportsPreserveAllABI};
return true;
}
case 8: {
case IR::OpSize::i64Bit: {
*Info = {FABI_F64_I16_F80, (void*)&FEXCore::CPU::OpHandlers<IR::OP_F80CVT>::handle8, Core::OPINDEX_F80CVT_8, SupportsPreserveAllABI};
return true;
}
@@ -115,7 +115,7 @@ bool InterpreterOps::GetFallbackHandler(bool SupportsPreserveAllABI, const IR::I
auto Op = IROp->C<IR::IROp_F80CVTInt>();
switch (OpSize) {
case 2: {
case IR::OpSize::i16Bit: {
if (Op->Truncate) {
*Info = {FABI_I16_I16_F80, (void*)&FEXCore::CPU::OpHandlers<IR::OP_F80CVTINT>::handle2t, Core::OPINDEX_F80CVTINT_TRUNC2,
SupportsPreserveAllABI};
@@ -124,7 +124,7 @@ bool InterpreterOps::GetFallbackHandler(bool SupportsPreserveAllABI, const IR::I
}
return true;
}
case 4: {
case IR::OpSize::i32Bit: {
if (Op->Truncate) {
*Info = {FABI_I32_I16_F80, (void*)&FEXCore::CPU::OpHandlers<IR::OP_F80CVTINT>::handle4t, Core::OPINDEX_F80CVTINT_TRUNC4,
SupportsPreserveAllABI};
@@ -133,7 +133,7 @@ bool InterpreterOps::GetFallbackHandler(bool SupportsPreserveAllABI, const IR::I
}
return true;
}
case 8: {
case IR::OpSize::i64Bit: {
if (Op->Truncate) {
*Info = {FABI_I64_I16_F80, (void*)&FEXCore::CPU::OpHandlers<IR::OP_F80CVTINT>::handle8t, Core::OPINDEX_F80CVTINT_TRUNC8,
SupportsPreserveAllABI};
@@ -156,11 +156,11 @@ bool InterpreterOps::GetFallbackHandler(bool SupportsPreserveAllABI, const IR::I
auto Op = IROp->C<IR::IROp_F80CVTToInt>();
switch (Op->SrcSize) {
case 2: {
case IR::OpSize::i16Bit: {
*Info = {FABI_F80_I16_I16, (void*)&FEXCore::CPU::OpHandlers<IR::OP_F80CVTTOINT>::handle2, Core::OPINDEX_F80CVTTOINT_2, SupportsPreserveAllABI};
return true;
}
case 4: {
case IR::OpSize::i32Bit: {
*Info = {FABI_F80_I16_I32, (void*)&FEXCore::CPU::OpHandlers<IR::OP_F80CVTTOINT>::handle4, Core::OPINDEX_F80CVTTOINT_4, SupportsPreserveAllABI};
return true;
}
@@ -5,9 +5,10 @@ tags: backend|arm64
$end_info$
*/
#include "CodeEmitter/Emitter.h"
#include "FEXCore/IR/IR.h"
#include "Interface/Context/Context.h"
#include "Interface/Core/JIT/Arm64/JITClass.h"
#include "Interface/Core/JIT/JITClass.h"
#include "Interface/IR/Passes/RegisterAllocationPass.h"
namespace FEXCore::CPU {
@@ -53,8 +54,8 @@ DEF_OP(EntrypointOffset) {
auto Constant = Entry + Op->Offset;
auto Dst = GetReg(Node);
uint64_t Mask = ~0ULL;
uint8_t OpSize = IROp->Size;
if (OpSize == 4) {
const auto OpSize = IROp->Size;
if (OpSize == IR::OpSize::i32Bit) {
Mask = 0xFFFF'FFFFULL;
}
@@ -91,10 +92,10 @@ DEF_OP(AddNZCV) {
uint64_t Const;
if (IsInlineConstant(Op->Src2, &Const)) {
LOGMAN_THROW_AA_FMT(IROp->Size >= 4, "Constant not allowed here");
LOGMAN_THROW_A_FMT(IROp->Size >= IR::OpSize::i32Bit, "Constant not allowed here");
cmn(EmitSize, Src1, Const);
} else if (IROp->Size < 4) {
unsigned Shift = 32 - (8 * IROp->Size);
} else if (IROp->Size < IR::OpSize::i32Bit) {
unsigned Shift = 32 - IR::OpSizeAsBits(IROp->Size);
lsl(ARMEmitter::Size::i32Bit, TMP1, Src1, Shift);
cmn(EmitSize, TMP1, GetReg(Op->Src2.ID()), ARMEmitter::ShiftType::LSL, Shift);
@@ -164,7 +165,7 @@ DEF_OP(TestNZ) {
// Shift the sign bit into place, clearing out the garbage in upper bits.
// Adding zero does an effective test, setting NZ according to the result and
// zeroing CV.
if (IROp->Size < 4) {
if (IROp->Size < IR::OpSize::i32Bit) {
// Cheaper to and+cmn than to lsl+lsl+tst, so do the and ourselves if
// needed.
if (Op->Src1 != Op->Src2) {
@@ -178,7 +179,7 @@ DEF_OP(TestNZ) {
Src1 = TMP1;
}
unsigned Shift = 32 - (IROp->Size * 8);
unsigned Shift = 32 - IR::OpSizeAsBits(IROp->Size);
cmn(EmitSize, ARMEmitter::Reg::zr, Src1, ARMEmitter::ShiftType::LSL, Shift);
} else {
if (IsInlineConstant(Op->Src2, &Const)) {
@@ -190,6 +191,30 @@ DEF_OP(TestNZ) {
}
}
DEF_OP(TestZ) {
auto Op = IROp->C<IR::IROp_TestZ>();
LOGMAN_THROW_A_FMT(IROp->Size < IR::OpSize::i32Bit, "TestNZ used at higher sizes");
const auto EmitSize = ARMEmitter::Size::i32Bit;
uint64_t Const;
uint64_t Mask = IROp->Size == IR::OpSize::i64Bit ? ~0ULL : ((1ull << IR::OpSizeAsBits(IROp->Size)) - 1);
auto Src1 = GetReg(Op->Src1.ID());
if (IsInlineConstant(Op->Src2, &Const)) {
// We can promote 8/16-bit tests to 32-bit since the constant is masked.
LOGMAN_THROW_A_FMT(!(Const & ~Mask), "constant is already masked");
tst(EmitSize, Src1, Const);
} else {
const auto Src2 = GetReg(Op->Src2.ID());
if (Src1 == Src2) {
tst(EmitSize, Src1 /* Src2 */, Mask);
} else {
and_(EmitSize, TMP1, Src1, Src2);
tst(EmitSize, TMP1, Mask);
}
}
}
DEF_OP(SubShift) {
auto Op = IROp->C<IR::IROp_SubShift>();
@@ -198,25 +223,25 @@ DEF_OP(SubShift) {
DEF_OP(SubNZCV) {
auto Op = IROp->C<IR::IROp_SubNZCV>();
const uint8_t OpSize = IROp->Size;
const auto OpSize = IROp->Size;
const auto EmitSize = ConvertSize(IROp);
uint64_t Const;
if (IsInlineConstant(Op->Src2, &Const)) {
LOGMAN_THROW_AA_FMT(OpSize >= 4, "Constant not allowed here");
LOGMAN_THROW_A_FMT(OpSize >= IR::OpSize::i32Bit, "Constant not allowed here");
cmp(EmitSize, GetReg(Op->Src1.ID()), Const);
} else {
unsigned Shift = OpSize < 4 ? (32 - (8 * OpSize)) : 0;
unsigned Shift = OpSize < IR::OpSize::i32Bit ? (32 - IR::OpSizeAsBits(OpSize)) : 0;
ARMEmitter::Register ShiftedSrc1 = GetZeroableReg(Op->Src1);
// Shift to fix flags for <32-bit ops.
// Any shift of zero is still zero so optimize out silly zero shifts.
if (OpSize < 4 && ShiftedSrc1 != ARMEmitter::Reg::zr) {
if (OpSize < IR::OpSize::i32Bit && ShiftedSrc1 != ARMEmitter::Reg::zr) {
lsl(ARMEmitter::Size::i32Bit, TMP1, ShiftedSrc1, Shift);
ShiftedSrc1 = TMP1;
}
if (OpSize < 4) {
if (OpSize < IR::OpSize::i32Bit) {
cmp(EmitSize, ShiftedSrc1, GetReg(Op->Src2.ID()), ARMEmitter::ShiftType::LSL, Shift);
} else {
cmp(EmitSize, ShiftedSrc1, GetReg(Op->Src2.ID()));
@@ -261,10 +286,10 @@ DEF_OP(SetSmallNZV) {
auto Op = IROp->C<IR::IROp_SetSmallNZV>();
LOGMAN_THROW_A_FMT(CTX->HostFeatures.SupportsFlagM, "Unsupported flagm op");
const uint8_t OpSize = IROp->Size;
LOGMAN_THROW_AA_FMT(OpSize == 1 || OpSize == 2, "Unsupported {} size: {}", __func__, OpSize);
const auto OpSize = IROp->Size;
LOGMAN_THROW_A_FMT(OpSize == IR::OpSize::i8Bit || OpSize == IR::OpSize::i16Bit, "Unsupported {} size: {}", __func__, OpSize);
if (OpSize == 1) {
if (OpSize == IR::OpSize::i8Bit) {
setf8(GetReg(Op->Src.ID()).W());
} else {
setf16(GetReg(Op->Src.ID()).W());
@@ -272,8 +297,43 @@ DEF_OP(SetSmallNZV) {
}
DEF_OP(AXFlag) {
LOGMAN_THROW_A_FMT(CTX->HostFeatures.SupportsFlagM2, "Unsupported flagm2 op");
axflag();
if (CTX->HostFeatures.SupportsFlagM2) {
axflag();
} else {
// AXFLAG is defined in the Arm spec as
//
// gt: nzCv -> nzCv
// lt: Nzcv -> nzcv <==> 1 + 0
// eq: nZCv -> nZCv <==> 1 + (~0)
// un: nzCV -> nZcv <==> 0 + 0
//
// For the latter 3 cases, we therefore get the right NZCV by adding V_inv
// to (eq ? ~0 : 0). The remaining case is forced with ccmn.
auto V_inv = GetReg(IROp->Args[0].ID());
csetm(ARMEmitter::Size::i64Bit, TMP1, ARMEmitter::Condition::CC_EQ);
ccmn(ARMEmitter::Size::i64Bit, V_inv, TMP1, ARMEmitter::StatusFlags {0x2} /* nzCv */, ARMEmitter::Condition::CC_LE);
}
}
DEF_OP(Parity) {
auto Op = IROp->C<IR::IROp_Parity>();
auto Raw = GetReg(Op->Raw.ID());
auto Dest = GetReg(Node);
// Cascade to calculate parity of bottom 8-bits to bottom bit.
eor(ARMEmitter::Size::i32Bit, TMP1, Raw, Raw, ARMEmitter::ShiftType::LSR, 4);
eor(ARMEmitter::Size::i32Bit, TMP1, TMP1, TMP1, ARMEmitter::ShiftType::LSR, 2);
if (Op->Invert) {
eon(ARMEmitter::Size::i32Bit, Dest, TMP1, TMP1, ARMEmitter::ShiftType::LSR, 1);
} else {
eor(ARMEmitter::Size::i32Bit, Dest, TMP1, TMP1, ARMEmitter::ShiftType::LSR, 1);
}
// The above sequence leaves garbage in the upper bits.
if (Op->Mask) {
and_(ARMEmitter::Size::i32Bit, Dest, Dest, 1);
}
}
DEF_OP(CondAddNZCV) {
@@ -341,20 +401,20 @@ DEF_OP(Div) {
// Each source is OpSize in size
// So you can have up to a 128bit divide from x86-64
const uint8_t OpSize = IROp->Size;
const auto OpSize = IROp->Size;
const auto EmitSize = ConvertSize(IROp);
const auto Dst = GetReg(Node);
auto Src1 = GetReg(Op->Src1.ID());
auto Src2 = GetReg(Op->Src2.ID());
if (OpSize == 1) {
if (OpSize == IR::OpSize::i8Bit) {
sxtb(EmitSize, TMP1, Src1);
sxtb(EmitSize, TMP2, Src2);
Src1 = TMP1;
Src2 = TMP2;
} else if (OpSize == 2) {
} else if (OpSize == IR::OpSize::i16Bit) {
sxth(EmitSize, TMP1, Src1);
sxth(EmitSize, TMP2, Src2);
@@ -370,20 +430,20 @@ DEF_OP(UDiv) {
// Each source is OpSize in size
// So you can have up to a 128bit divide from x86-64
const uint8_t OpSize = IROp->Size;
const auto OpSize = IROp->Size;
const auto EmitSize = ConvertSize(IROp);
const auto Dst = GetReg(Node);
auto Src1 = GetReg(Op->Src1.ID());
auto Src2 = GetReg(Op->Src2.ID());
if (OpSize == 1) {
if (OpSize == IR::OpSize::i8Bit) {
uxtb(EmitSize, TMP1, Src1);
uxtb(EmitSize, TMP2, Src2);
Src1 = TMP1;
Src2 = TMP2;
} else if (OpSize == 2) {
} else if (OpSize == IR::OpSize::i16Bit) {
uxth(EmitSize, TMP1, Src1);
uxth(EmitSize, TMP2, Src2);
@@ -398,20 +458,20 @@ DEF_OP(Rem) {
auto Op = IROp->C<IR::IROp_Rem>();
// Each source is OpSize in size
// So you can have up to a 128bit divide from x86-64
const uint8_t OpSize = IROp->Size;
const auto OpSize = IROp->Size;
const auto EmitSize = ConvertSize(IROp);
const auto Dst = GetReg(Node);
auto Src1 = GetReg(Op->Src1.ID());
auto Src2 = GetReg(Op->Src2.ID());
if (OpSize == 1) {
if (OpSize == IR::OpSize::i8Bit) {
sxtb(EmitSize, TMP1, Src1);
sxtb(EmitSize, TMP2, Src2);
Src1 = TMP1;
Src2 = TMP2;
} else if (OpSize == 2) {
} else if (OpSize == IR::OpSize::i16Bit) {
sxth(EmitSize, TMP1, Src1);
sxth(EmitSize, TMP2, Src2);
@@ -427,20 +487,20 @@ DEF_OP(URem) {
auto Op = IROp->C<IR::IROp_URem>();
// Each source is OpSize in size
// So you can have up to a 128bit divide from x86-64
const uint8_t OpSize = IROp->Size;
const auto OpSize = IROp->Size;
const auto EmitSize = ConvertSize(IROp);
const auto Dst = GetReg(Node);
auto Src1 = GetReg(Op->Src1.ID());
auto Src2 = GetReg(Op->Src2.ID());
if (OpSize == 1) {
if (OpSize == IR::OpSize::i8Bit) {
uxtb(EmitSize, TMP1, Src1);
uxtb(EmitSize, TMP2, Src2);
Src1 = TMP1;
Src2 = TMP2;
} else if (OpSize == 2) {
} else if (OpSize == IR::OpSize::i16Bit) {
uxth(EmitSize, TMP1, Src1);
uxth(EmitSize, TMP2, Src2);
@@ -454,15 +514,15 @@ DEF_OP(URem) {
DEF_OP(MulH) {
auto Op = IROp->C<IR::IROp_MulH>();
const uint8_t OpSize = IROp->Size;
const auto OpSize = IROp->Size;
LOGMAN_THROW_AA_FMT(OpSize == 4 || OpSize == 8, "Unsupported {} size: {}", __func__, OpSize);
LOGMAN_THROW_A_FMT(OpSize == IR::OpSize::i32Bit || OpSize == IR::OpSize::i64Bit, "Unsupported {} size: {}", __func__, OpSize);
const auto Dst = GetReg(Node);
const auto Src1 = GetReg(Op->Src1.ID());
const auto Src2 = GetReg(Op->Src2.ID());
if (OpSize == 4) {
if (OpSize == IR::OpSize::i32Bit) {
sxtw(TMP1, Src1.W());
sxtw(TMP2, Src2.W());
mul(ARMEmitter::Size::i32Bit, Dst, TMP1, TMP2);
@@ -474,15 +534,15 @@ DEF_OP(MulH) {
DEF_OP(UMulH) {
auto Op = IROp->C<IR::IROp_UMulH>();
const uint8_t OpSize = IROp->Size;
const auto OpSize = IROp->Size;
LOGMAN_THROW_AA_FMT(OpSize == 4 || OpSize == 8, "Unsupported {} size: {}", __func__, OpSize);
LOGMAN_THROW_A_FMT(OpSize == IR::OpSize::i32Bit || OpSize == IR::OpSize::i64Bit, "Unsupported {} size: {}", __func__, OpSize);
const auto Dst = GetReg(Node);
const auto Src1 = GetReg(Op->Src1.ID());
const auto Src2 = GetReg(Op->Src2.ID());
if (OpSize == 4) {
if (OpSize == IR::OpSize::i32Bit) {
uxtw(ARMEmitter::Size::i64Bit, TMP1, Src1);
uxtw(ARMEmitter::Size::i64Bit, TMP2, Src2);
mul(ARMEmitter::Size::i64Bit, Dst, TMP1, TMP2);
@@ -533,7 +593,7 @@ DEF_OP(Ornror) {
DEF_OP(AndWithFlags) {
auto Op = IROp->C<IR::IROp_AndWithFlags>();
const uint8_t OpSize = IROp->Size;
const auto OpSize = IROp->Size;
const auto EmitSize = ConvertSize(IROp);
uint64_t Const;
@@ -541,7 +601,7 @@ DEF_OP(AndWithFlags) {
auto Src1 = GetReg(Op->Src1.ID());
// See TestNZ
if (OpSize < 4) {
if (OpSize < IR::OpSize::i32Bit) {
if (IsInlineConstant(Op->Src2, &Const)) {
and_(EmitSize, Dst, Src1, Const);
} else {
@@ -554,7 +614,7 @@ DEF_OP(AndWithFlags) {
}
}
unsigned Shift = 32 - (OpSize * 8);
unsigned Shift = 32 - IR::OpSizeAsBits(OpSize);
cmn(EmitSize, ARMEmitter::Reg::zr, Dst, ARMEmitter::ShiftType::LSL, Shift);
} else {
if (IsInlineConstant(Op->Src2, &Const)) {
@@ -580,7 +640,7 @@ DEF_OP(XornShift) {
DEF_OP(Ashr) {
auto Op = IROp->C<IR::IROp_Ashr>();
const uint8_t OpSize = IROp->Size;
const auto OpSize = IROp->Size;
const auto EmitSize = ConvertSize(IROp);
const auto Dst = GetReg(Node);
@@ -588,29 +648,29 @@ DEF_OP(Ashr) {
uint64_t Const;
if (IsInlineConstant(Op->Src2, &Const)) {
if (OpSize >= 4) {
if (OpSize >= IR::OpSize::i32Bit) {
asr(EmitSize, Dst, Src1, (unsigned int)Const);
} else {
sbfx(EmitSize, TMP1, Src1, 0, OpSize * 8);
sbfx(EmitSize, TMP1, Src1, 0, IR::OpSizeAsBits(OpSize));
asr(EmitSize, Dst, TMP1, (unsigned int)Const);
ubfx(EmitSize, Dst, Dst, 0, OpSize * 8);
ubfx(EmitSize, Dst, Dst, 0, IR::OpSizeAsBits(OpSize));
}
} else {
const auto Src2 = GetReg(Op->Src2.ID());
if (OpSize >= 4) {
if (OpSize >= IR::OpSize::i32Bit) {
asrv(EmitSize, Dst, Src1, Src2);
} else {
sbfx(EmitSize, TMP1, Src1, 0, OpSize * 8);
sbfx(EmitSize, TMP1, Src1, 0, IR::OpSizeAsBits(OpSize));
asrv(EmitSize, Dst, TMP1, Src2);
ubfx(EmitSize, Dst, Dst, 0, OpSize * 8);
ubfx(EmitSize, Dst, Dst, 0, IR::OpSizeAsBits(OpSize));
}
}
}
DEF_OP(ShiftFlags) {
auto Op = IROp->C<IR::IROp_ShiftFlags>();
const uint8_t OpSize = Op->Size;
const auto EmitSize = OpSize == 8 ? ARMEmitter::Size::i64Bit : ARMEmitter::Size::i32Bit;
const auto OpSize = Op->Size;
const auto EmitSize = OpSize == IR::OpSize::i64Bit ? ARMEmitter::Size::i64Bit : ARMEmitter::Size::i32Bit;
const auto PFOutput = GetReg(Node);
const auto PFInput = GetReg(Op->PFInput.ID());
@@ -630,16 +690,16 @@ DEF_OP(ShiftFlags) {
// We need to mask the source before comparing it. We don't just skip flag
// updates for Src2=0 but anything that masks to zero.
and_(ARMEmitter::Size::i32Bit, TMP1, Src2, OpSize == 8 ? 0x3f : 0x1f);
and_(ARMEmitter::Size::i32Bit, TMP1, Src2, OpSize == IR::OpSize::i64Bit ? 0x3f : 0x1f);
ARMEmitter::SingleUseForwardLabel Done;
ARMEmitter::ForwardLabel Done;
cbz(EmitSize, TMP1, &Done);
{
// PF/SF/ZF/OF
if (OpSize >= 4) {
if (OpSize >= IR::OpSize::i32Bit) {
ands(EmitSize, PFTemp, Dst, Dst);
} else {
unsigned Shift = 32 - (OpSize * 8);
unsigned Shift = 32 - (IR::OpSizeToSize(OpSize) * 8);
cmn(EmitSize, ARMEmitter::Reg::zr, Dst, ARMEmitter::ShiftType::LSL, Shift);
mov(ARMEmitter::Size::i64Bit, PFTemp, Dst);
}
@@ -649,12 +709,12 @@ DEF_OP(ShiftFlags) {
// Extract the last bit shifted in to CF
if (Op->Shift == IR::ShiftType::LSL) {
if (OpSize >= 4) {
if (OpSize >= IR::OpSize::i32Bit) {
neg(EmitSize, CFWord, Src2);
lsrv(EmitSize, CFWord, Src1, CFWord);
} else {
CFWord = Dst.X();
CFBit = (OpSize * 8);
CFBit = IR::OpSizeToSize(OpSize) * 8;
}
} else {
sub(ARMEmitter::Size::i64Bit, CFWord, Src2, 1);
@@ -677,7 +737,7 @@ DEF_OP(ShiftFlags) {
rmif(CFWord, (CFBit - 1) % 64, (1 << 1) /* C */);
if (SetOF) {
rmif(TMP3, OpSize * 8 - 1, (1 << 0) /* V */);
rmif(TMP3, IR::OpSizeToSize(OpSize) * 8 - 1, (1 << 0) /* V */);
}
} else {
mrs(TMP2, ARMEmitter::SystemRegister::NZCV);
@@ -690,7 +750,7 @@ DEF_OP(ShiftFlags) {
bfi(ARMEmitter::Size::i32Bit, TMP2, CFWord, 29 /* C */, 1);
if (SetOF) {
lsr(EmitSize, TMP3, TMP3, OpSize * 8 - 1);
lsr(EmitSize, TMP3, TMP3, IR::OpSizeToSize(OpSize) * 8 - 1);
bfi(ARMEmitter::Size::i32Bit, TMP2, TMP3, 28 /* V */, 1);
}
@@ -710,14 +770,14 @@ DEF_OP(RotateFlags) {
const auto Result = GetReg(Op->Result.ID());
const auto Shift = GetReg(Op->Shift.ID());
const bool Left = Op->Left;
const auto EmitSize = Op->Size == 8 ? ARMEmitter::Size::i64Bit : ARMEmitter::Size::i32Bit;
const auto EmitSize = Op->Size == IR::OpSize::i64Bit ? ARMEmitter::Size::i64Bit : ARMEmitter::Size::i32Bit;
// If shift=0, flags are unaffected. Wrap the whole implementation in a cbz.
ARMEmitter::SingleUseForwardLabel Done;
ARMEmitter::ForwardLabel Done;
cbz(EmitSize, Shift, &Done);
{
// Extract the last bit shifted in to CF
const auto BitSize = Op->Size * 8;
const auto BitSize = IR::OpSizeToSize(Op->Size) * 8;
unsigned CFBit = Left ? 0 : BitSize - 1;
// For ROR, OF is the XOR of the new CF bit and the most significant bit of the result.
@@ -802,7 +862,7 @@ DEF_OP(PDep) {
const auto T1 = TMP4.R();
ARMEmitter::BackwardLabel NextBit;
ARMEmitter::SingleUseForwardLabel Done;
ARMEmitter::ForwardLabel Done;
// First, copy the input/mask, since we'll be clobbering. Copy as 64-bit to
// make this 0-uop on Firestorm.
@@ -837,7 +897,7 @@ DEF_OP(PDep) {
DEF_OP(PExt) {
auto Op = IROp->C<IR::IROp_PExt>();
const auto OpSize = IROp->Size;
const auto OpSizeBitsM1 = (OpSize * 8) - 1;
const auto OpSizeBitsM1 = IR::OpSizeAsBits(OpSize) - 1;
const auto EmitSize = ConvertSize48(IROp);
const auto Input = GetReg(Op->Input.ID());
@@ -862,9 +922,9 @@ DEF_OP(PExt) {
const auto BitReg = TMP2;
const auto ValueReg = TMP3;
ARMEmitter::SingleUseForwardLabel EarlyExit;
ARMEmitter::ForwardLabel EarlyExit;
ARMEmitter::BackwardLabel NextBit;
ARMEmitter::SingleUseForwardLabel Done;
ARMEmitter::ForwardLabel Done;
cbz(EmitSize, Mask, &EarlyExit);
mov(EmitSize, MaskReg, Mask);
@@ -892,8 +952,8 @@ DEF_OP(PExt) {
DEF_OP(LDiv) {
auto Op = IROp->C<IR::IROp_LDiv>();
const uint8_t OpSize = IROp->Size;
const auto EmitSize = OpSize >= 4 ? ARMEmitter::Size::i64Bit : ARMEmitter::Size::i32Bit;
const auto OpSize = IROp->Size;
const auto EmitSize = OpSize >= IR::OpSize::i32Bit ? ARMEmitter::Size::i64Bit : ARMEmitter::Size::i32Bit;
const auto Dst = GetReg(Node);
const auto Upper = GetReg(Op->Upper.ID());
@@ -903,14 +963,14 @@ DEF_OP(LDiv) {
// Each source is OpSize in size
// So you can have up to a 128bit divide from x86-64
switch (OpSize) {
case 2: {
case IR::OpSize::i16Bit: {
uxth(EmitSize, TMP1, Lower);
bfi(EmitSize, TMP1, Upper, 16, 16);
sxth(EmitSize, TMP2, Divisor);
sdiv(EmitSize, Dst, TMP1, TMP2);
break;
}
case 4: {
case IR::OpSize::i32Bit: {
// TODO: 32-bit operation should be guaranteed not to leave garbage in the upper bits.
mov(EmitSize, TMP1, Lower);
bfi(EmitSize, TMP1, Upper, 32, 32);
@@ -918,9 +978,9 @@ DEF_OP(LDiv) {
sdiv(EmitSize, Dst, TMP1, TMP2);
break;
}
case 8: {
ARMEmitter::SingleUseForwardLabel Only64Bit {};
ARMEmitter::SingleUseForwardLabel LongDIVRet {};
case IR::OpSize::i64Bit: {
ARMEmitter::ForwardLabel Only64Bit {};
ARMEmitter::ForwardLabel LongDIVRet {};
// Check if the upper bits match the top bit of the lower 64-bits
// Sign extend the top bit of lower bits
@@ -962,8 +1022,8 @@ DEF_OP(LDiv) {
DEF_OP(LUDiv) {
auto Op = IROp->C<IR::IROp_LUDiv>();
const uint8_t OpSize = IROp->Size;
const auto EmitSize = OpSize >= 4 ? ARMEmitter::Size::i64Bit : ARMEmitter::Size::i32Bit;
const auto OpSize = IROp->Size;
const auto EmitSize = OpSize >= IR::OpSize::i32Bit ? ARMEmitter::Size::i64Bit : ARMEmitter::Size::i32Bit;
const auto Dst = GetReg(Node);
const auto Upper = GetReg(Op->Upper.ID());
@@ -973,22 +1033,22 @@ DEF_OP(LUDiv) {
// Each source is OpSize in size
// So you can have up to a 128bit divide from x86-64=
switch (OpSize) {
case 2: {
case IR::OpSize::i16Bit: {
uxth(EmitSize, TMP1, Lower);
bfi(EmitSize, TMP1, Upper, 16, 16);
udiv(EmitSize, Dst, TMP1, Divisor);
break;
}
case 4: {
case IR::OpSize::i32Bit: {
// TODO: 32-bit operation should be guaranteed not to leave garbage in the upper bits.
mov(EmitSize, TMP1, Lower);
bfi(EmitSize, TMP1, Upper, 32, 32);
udiv(EmitSize, Dst, TMP1, Divisor);
break;
}
case 8: {
ARMEmitter::SingleUseForwardLabel Only64Bit {};
ARMEmitter::SingleUseForwardLabel LongDIVRet {};
case IR::OpSize::i64Bit: {
ARMEmitter::ForwardLabel Only64Bit {};
ARMEmitter::ForwardLabel LongDIVRet {};
// Check the upper bits for zero
// If the upper bits are zero then we can do a 64-bit divide
@@ -1026,8 +1086,8 @@ DEF_OP(LUDiv) {
DEF_OP(LRem) {
auto Op = IROp->C<IR::IROp_LRem>();
const uint8_t OpSize = IROp->Size;
const auto EmitSize = OpSize >= 4 ? ARMEmitter::Size::i64Bit : ARMEmitter::Size::i32Bit;
const auto OpSize = IROp->Size;
const auto EmitSize = OpSize >= IR::OpSize::i32Bit ? ARMEmitter::Size::i64Bit : ARMEmitter::Size::i32Bit;
const auto Dst = GetReg(Node);
const auto Upper = GetReg(Op->Upper.ID());
@@ -1037,7 +1097,7 @@ DEF_OP(LRem) {
// Each source is OpSize in size
// So you can have up to a 128bit divide from x86-64
switch (OpSize) {
case 2: {
case IR::OpSize::i16Bit: {
uxth(EmitSize, TMP1, Lower);
bfi(EmitSize, TMP1, Upper, 16, 16);
sxth(EmitSize, TMP2, Divisor);
@@ -1045,7 +1105,7 @@ DEF_OP(LRem) {
msub(EmitSize, Dst, TMP3, TMP2, TMP1);
break;
}
case 4: {
case IR::OpSize::i32Bit: {
// TODO: 32-bit operation should be guaranteed not to leave garbage in the upper bits.
mov(EmitSize, TMP1, Lower);
bfi(EmitSize, TMP1, Upper, 32, 32);
@@ -1054,9 +1114,9 @@ DEF_OP(LRem) {
msub(EmitSize, Dst, TMP2, TMP3, TMP1);
break;
}
case 8: {
ARMEmitter::SingleUseForwardLabel Only64Bit {};
ARMEmitter::SingleUseForwardLabel LongDIVRet {};
case IR::OpSize::i64Bit: {
ARMEmitter::ForwardLabel Only64Bit {};
ARMEmitter::ForwardLabel LongDIVRet {};
// Check if the upper bits match the top bit of the lower 64-bits
// Sign extend the top bit of lower bits
@@ -1100,8 +1160,8 @@ DEF_OP(LRem) {
DEF_OP(LURem) {
auto Op = IROp->C<IR::IROp_LURem>();
const uint8_t OpSize = IROp->Size;
const auto EmitSize = OpSize >= 4 ? ARMEmitter::Size::i64Bit : ARMEmitter::Size::i32Bit;
const auto OpSize = IROp->Size;
const auto EmitSize = OpSize >= IR::OpSize::i32Bit ? ARMEmitter::Size::i64Bit : ARMEmitter::Size::i32Bit;
const auto Dst = GetReg(Node);
const auto Upper = GetReg(Op->Upper.ID());
@@ -1111,14 +1171,14 @@ DEF_OP(LURem) {
// Each source is OpSize in size
// So you can have up to a 128bit divide from x86-64
switch (OpSize) {
case 2: {
case IR::OpSize::i16Bit: {
uxth(EmitSize, TMP1, Lower);
bfi(EmitSize, TMP1, Upper, 16, 16);
udiv(EmitSize, TMP2, TMP1, Divisor);
msub(EmitSize, Dst, TMP2, Divisor, TMP1);
break;
}
case 4: {
case IR::OpSize::i32Bit: {
// TODO: 32-bit operation should be guaranteed not to leave garbage in the upper bits.
mov(EmitSize, TMP1, Lower);
bfi(EmitSize, TMP1, Upper, 32, 32);
@@ -1126,9 +1186,9 @@ DEF_OP(LURem) {
msub(EmitSize, Dst, TMP2, Divisor, TMP1);
break;
}
case 8: {
ARMEmitter::SingleUseForwardLabel Only64Bit {};
ARMEmitter::SingleUseForwardLabel LongDIVRet {};
case IR::OpSize::i64Bit: {
ARMEmitter::ForwardLabel Only64Bit {};
ARMEmitter::ForwardLabel LongDIVRet {};
// Check the upper bits for zero
// If the upper bits are zero then we can do a 64-bit divide
@@ -1178,30 +1238,30 @@ DEF_OP(Not) {
DEF_OP(Popcount) {
auto Op = IROp->C<IR::IROp_Popcount>();
const uint8_t OpSize = IROp->Size;
const auto OpSize = IROp->Size;
const auto Dst = GetReg(Node);
const auto Src = GetReg(Op->Src.ID());
switch (OpSize) {
case 0x1:
case IR::OpSize::i8Bit:
fmov(ARMEmitter::Size::i32Bit, VTMP1.S(), Src);
// only use lowest byte
cnt(ARMEmitter::SubRegSize::i8Bit, VTMP1.D(), VTMP1.D());
break;
case 0x2:
case IR::OpSize::i16Bit:
fmov(ARMEmitter::Size::i32Bit, VTMP1.S(), Src);
cnt(ARMEmitter::SubRegSize::i8Bit, VTMP1.D(), VTMP1.D());
// only count two lowest bytes
addp(ARMEmitter::SubRegSize::i8Bit, VTMP1.D(), VTMP1.D(), VTMP1.D());
break;
case 0x4:
case IR::OpSize::i32Bit:
fmov(ARMEmitter::Size::i32Bit, VTMP1.S(), Src);
cnt(ARMEmitter::SubRegSize::i8Bit, VTMP1.D(), VTMP1.D());
// fmov has zero extended, unused bytes are zero
addv(ARMEmitter::SubRegSize::i8Bit, VTMP1.D(), VTMP1.D());
break;
case 0x8:
case IR::OpSize::i64Bit:
fmov(ARMEmitter::Size::i64Bit, VTMP1.D(), Src);
cnt(ARMEmitter::SubRegSize::i8Bit, VTMP1.D(), VTMP1.D());
// fmov has zero extended, unused bytes are zero
@@ -1220,34 +1280,27 @@ DEF_OP(FindLSB) {
const auto Dst = GetReg(Node);
const auto Src = GetReg(Op->Src.ID());
if (IROp->Size != 8) {
ubfx(EmitSize, TMP1, Src, 0, IROp->Size * 8);
cmp(EmitSize, TMP1, 0);
rbit(EmitSize, TMP1, TMP1);
} else {
rbit(EmitSize, TMP1, Src);
cmp(EmitSize, Src, 0);
}
// We assume the source is nonzero, so we can just rbit+clz without worrying
// about upper garbage for smaller types.
rbit(EmitSize, TMP1, Src);
clz(EmitSize, Dst, TMP1);
csinv(EmitSize, Dst, Dst, ARMEmitter::Reg::zr, ARMEmitter::Condition::CC_NE);
}
DEF_OP(FindMSB) {
auto Op = IROp->C<IR::IROp_FindMSB>();
const uint8_t OpSize = IROp->Size;
const auto OpSize = IROp->Size;
LOGMAN_THROW_AA_FMT(OpSize == 2 || OpSize == 4 || OpSize == 8, "Unsupported {} size: {}", __func__, OpSize);
LOGMAN_THROW_A_FMT(OpSize == IR::OpSize::i16Bit || OpSize == IR::OpSize::i32Bit || OpSize == IR::OpSize::i64Bit,
"Unsupported {} size: {}", __func__, OpSize);
const auto EmitSize = ConvertSize(IROp);
const auto Dst = GetReg(Node);
const auto Src = GetReg(Op->Src.ID());
movz(ARMEmitter::Size::i64Bit, TMP1, OpSize * 8 - 1);
movz(ARMEmitter::Size::i64Bit, TMP1, IR::OpSizeAsBits(OpSize) - 1);
if (OpSize == 2) {
if (OpSize == IR::OpSize::i16Bit) {
lsl(EmitSize, Dst, Src, 16);
orr(EmitSize, Dst, Dst, 0x8000);
clz(EmitSize, Dst, Dst);
} else {
clz(EmitSize, Dst, Src);
@@ -1258,9 +1311,10 @@ DEF_OP(FindMSB) {
DEF_OP(FindTrailingZeroes) {
auto Op = IROp->C<IR::IROp_FindTrailingZeroes>();
const uint8_t OpSize = IROp->Size;
const auto OpSize = IROp->Size;
LOGMAN_THROW_AA_FMT(OpSize == 2 || OpSize == 4 || OpSize == 8, "Unsupported {} size: {}", __func__, OpSize);
LOGMAN_THROW_A_FMT(OpSize == IR::OpSize::i16Bit || OpSize == IR::OpSize::i32Bit || OpSize == IR::OpSize::i64Bit,
"Unsupported {} size: {}", __func__, OpSize);
const auto EmitSize = ConvertSize(IROp);
const auto Dst = GetReg(Node);
@@ -1268,7 +1322,7 @@ DEF_OP(FindTrailingZeroes) {
rbit(EmitSize, Dst, Src);
if (OpSize == 2) {
if (OpSize == IR::OpSize::i16Bit) {
// This orr does two things. First, if the (masked) source is zero, it
// reverses to zero in the top so it forces clz to return 16. Second, it
// ensures garbage in the upper bits of the source don't affect clz, because
@@ -1282,15 +1336,16 @@ DEF_OP(FindTrailingZeroes) {
DEF_OP(CountLeadingZeroes) {
auto Op = IROp->C<IR::IROp_CountLeadingZeroes>();
const uint8_t OpSize = IROp->Size;
const auto OpSize = IROp->Size;
LOGMAN_THROW_AA_FMT(OpSize == 2 || OpSize == 4 || OpSize == 8, "Unsupported {} size: {}", __func__, OpSize);
LOGMAN_THROW_A_FMT(OpSize == IR::OpSize::i16Bit || OpSize == IR::OpSize::i32Bit || OpSize == IR::OpSize::i64Bit,
"Unsupported {} size: {}", __func__, OpSize);
const auto EmitSize = ConvertSize(IROp);
const auto Dst = GetReg(Node);
const auto Src = GetReg(Op->Src.ID());
if (OpSize == 2) {
if (OpSize == IR::OpSize::i16Bit) {
// Expressing as lsl+orr+clz clears away any garbage in the upper bits
// (alternatively could do uxth+clz+sub.. equal cost in total).
lsl(EmitSize, Dst, Src, 16);
@@ -1303,16 +1358,17 @@ DEF_OP(CountLeadingZeroes) {
DEF_OP(Rev) {
auto Op = IROp->C<IR::IROp_Rev>();
const uint8_t OpSize = IROp->Size;
const auto OpSize = IROp->Size;
LOGMAN_THROW_AA_FMT(OpSize == 2 || OpSize == 4 || OpSize == 8, "Unsupported {} size: {}", __func__, OpSize);
LOGMAN_THROW_A_FMT(OpSize == IR::OpSize::i16Bit || OpSize == IR::OpSize::i32Bit || OpSize == IR::OpSize::i64Bit,
"Unsupported {} size: {}", __func__, OpSize);
const auto EmitSize = ConvertSize(IROp);
const auto Dst = GetReg(Node);
const auto Src = GetReg(Op->Src.ID());
rev(EmitSize, Dst, Src);
if (OpSize == 2) {
if (OpSize == IR::OpSize::i16Bit) {
lsr(EmitSize, Dst, Dst, 16);
}
}
@@ -1338,10 +1394,10 @@ DEF_OP(Bfi) {
mov(EmitSize, TMP1, SrcDst);
bfi(EmitSize, TMP1, Src, Op->lsb, Op->Width);
if (IROp->Size >= 4) {
if (IROp->Size >= IR::OpSize::i32Bit) {
mov(EmitSize, Dst, TMP1.R());
} else {
ubfx(EmitSize, Dst, TMP1, 0, IROp->Size * 8);
ubfx(EmitSize, Dst, TMP1, 0, IR::OpSizeAsBits(IROp->Size));
}
}
}
@@ -1372,8 +1428,8 @@ DEF_OP(Bfxil) {
DEF_OP(Bfe) {
auto Op = IROp->C<IR::IROp_Bfe>();
LOGMAN_THROW_AA_FMT(IROp->Size <= 8, "OpSize is too large for BFE: {}", IROp->Size);
LOGMAN_THROW_AA_FMT(Op->Width != 0, "Invalid BFE width of 0");
LOGMAN_THROW_A_FMT(IROp->Size <= IR::OpSize::i64Bit, "OpSize is too large for BFE: {}", IROp->Size);
LOGMAN_THROW_A_FMT(Op->Width != 0, "Invalid BFE width of 0");
const auto EmitSize = ConvertSize(IROp);
const auto Dst = GetReg(Node);
@@ -1382,7 +1438,7 @@ DEF_OP(Bfe) {
if (Op->lsb == 0 && Op->Width == 32) {
mov(ARMEmitter::Size::i32Bit, Dst, Src);
} else if (Op->lsb == 0 && Op->Width == 64) {
LOGMAN_THROW_AA_FMT(IROp->Size == 8, "Must be 64-bit wide register");
LOGMAN_THROW_A_FMT(IROp->Size == IR::OpSize::i64Bit, "Must be 64-bit wide register");
mov(ARMEmitter::Size::i64Bit, Dst, Src);
} else {
ubfx(EmitSize, Dst, Src, Op->lsb, Op->Width);
@@ -1399,9 +1455,9 @@ DEF_OP(Sbfe) {
DEF_OP(Select) {
auto Op = IROp->C<IR::IROp_Select>();
const uint8_t OpSize = IROp->Size;
const auto OpSize = IROp->Size;
const auto EmitSize = ConvertSize(IROp);
const auto CompareEmitSize = Op->CompareSize == 8 ? ARMEmitter::Size::i64Bit : ARMEmitter::Size::i32Bit;
const auto CompareEmitSize = Op->CompareSize == IR::OpSize::i64Bit ? ARMEmitter::Size::i64Bit : ARMEmitter::Size::i32Bit;
uint64_t Const;
auto cc = MapCC(Op->Cond);
@@ -1418,7 +1474,7 @@ DEF_OP(Select) {
} else if (IsFPR(Op->Cmp1.ID())) {
const auto Src1 = GetVReg(Op->Cmp1.ID());
const auto Src2 = GetVReg(Op->Cmp2.ID());
fcmp(Op->CompareSize == 8 ? ARMEmitter::ScalarRegSize::i64Bit : ARMEmitter::ScalarRegSize::i32Bit, Src1, Src2);
fcmp(Op->CompareSize == IR::OpSize::i64Bit ? ARMEmitter::ScalarRegSize::i64Bit : ARMEmitter::ScalarRegSize::i32Bit, Src1, Src2);
} else {
LOGMAN_MSG_A_FMT("Select: Expected GPR or FPR");
}
@@ -1427,7 +1483,7 @@ DEF_OP(Select) {
bool is_const_true = IsInlineConstant(Op->TrueVal, &const_true);
bool is_const_false = IsInlineConstant(Op->FalseVal, &const_false);
uint64_t all_ones = OpSize == 8 ? 0xffff'ffff'ffff'ffffull : 0xffff'ffffull;
uint64_t all_ones = OpSize == IR::OpSize::i64Bit ? 0xffff'ffff'ffff'ffffull : 0xffff'ffffull;
ARMEmitter::Register Dst = GetReg(Node);
@@ -1456,7 +1512,7 @@ DEF_OP(NZCVSelect) {
bool is_const_true = IsInlineConstant(Op->TrueVal, &const_true);
bool is_const_false = IsInlineConstant(Op->FalseVal, &const_false);
uint64_t all_ones = IROp->Size == 8 ? 0xffff'ffff'ffff'ffffull : 0xffff'ffffull;
uint64_t all_ones = IROp->Size == IR::OpSize::i64Bit ? 0xffff'ffff'ffff'ffffull : 0xffff'ffffull;
ARMEmitter::Register Dst = GetReg(Node);
@@ -1475,6 +1531,14 @@ DEF_OP(NZCVSelect) {
}
}
DEF_OP(NZCVSelectV) {
auto Op = IROp->C<IR::IROp_NZCVSelectV>();
auto cc = MapCC(Op->Cond);
const auto SubRegSize = ConvertSubRegSizePair248(IROp);
fcsel(SubRegSize.Scalar, GetVReg(Node), GetVReg(Op->TrueVal.ID()), GetVReg(Op->FalseVal.ID()), cc);
}
DEF_OP(NZCVSelectIncrement) {
auto Op = IROp->C<IR::IROp_NZCVSelectIncrement>();
@@ -1485,12 +1549,12 @@ DEF_OP(VExtractToGPR) {
const auto Op = IROp->C<IR::IROp_VExtractToGPR>();
const auto OpSize = IROp->Size;
constexpr auto AVXRegBitSize = Core::CPUState::XMM_AVX_REG_SIZE * 8;
[[maybe_unused]] constexpr auto AVXRegBitSize = Core::CPUState::XMM_AVX_REG_SIZE * 8;
constexpr auto SSERegBitSize = Core::CPUState::XMM_SSE_REG_SIZE * 8;
const auto ElementSizeBits = Op->Header.ElementSize * 8;
const auto ElementSizeBits = IR::OpSizeAsBits(Op->Header.ElementSize);
const auto Offset = ElementSizeBits * Op->Index;
const auto Is256Bit = Offset >= SSERegBitSize;
[[maybe_unused]] const auto Is256Bit = Offset >= SSERegBitSize;
LOGMAN_THROW_A_FMT(!Is256Bit || (Is256Bit && HostSupportsSVE256), "Need SVE256 support in order to use {} with 256-bit operation", __func__);
const auto Dst = GetReg(Node);
@@ -1498,10 +1562,10 @@ DEF_OP(VExtractToGPR) {
const auto PerformMove = [&](const ARMEmitter::VRegister reg, int index) {
switch (OpSize) {
case 1: umov<ARMEmitter::SubRegSize::i8Bit>(Dst, Vector, index); break;
case 2: umov<ARMEmitter::SubRegSize::i16Bit>(Dst, Vector, index); break;
case 4: umov<ARMEmitter::SubRegSize::i32Bit>(Dst, Vector, index); break;
case 8: umov<ARMEmitter::SubRegSize::i64Bit>(Dst, Vector, index); break;
case IR::OpSize::i8Bit: umov<ARMEmitter::SubRegSize::i8Bit>(Dst, Vector, index); break;
case IR::OpSize::i16Bit: umov<ARMEmitter::SubRegSize::i16Bit>(Dst, Vector, index); break;
case IR::OpSize::i32Bit: umov<ARMEmitter::SubRegSize::i32Bit>(Dst, Vector, index); break;
case IR::OpSize::i64Bit: umov<ARMEmitter::SubRegSize::i64Bit>(Dst, Vector, index); break;
default: LOGMAN_MSG_A_FMT("Unhandled ExtractElementSize: {}", OpSize); break;
}
};
@@ -1512,8 +1576,8 @@ DEF_OP(VExtractToGPR) {
// when acting on larger register sizes.
PerformMove(Vector, Op->Index);
} else {
LOGMAN_THROW_AA_FMT(Is256Bit, "Can't perform 256-bit extraction with op side: {}", OpSize);
LOGMAN_THROW_AA_FMT(Offset < AVXRegBitSize, "Trying to extract element outside bounds of register. Offset={}, Index={}", Offset, Op->Index);
LOGMAN_THROW_A_FMT(Is256Bit, "Can't perform 256-bit extraction with op side: {}", OpSize);
LOGMAN_THROW_A_FMT(Offset < AVXRegBitSize, "Trying to extract element outside bounds of register. Offset={}, Index={}", Offset, Op->Index);
// We need to use the upper 128-bit lane, so lets move it down.
// Inverting our dedicated predicate for 128-bit operations selects
@@ -1526,10 +1590,10 @@ DEF_OP(VExtractToGPR) {
// upper half of the vector.
const auto SanitizedIndex = [OpSize, Op] {
switch (OpSize) {
case 1: return Op->Index - 16;
case 2: return Op->Index - 8;
case 4: return Op->Index - 4;
case 8: return Op->Index - 2;
case IR::OpSize::i8Bit: return Op->Index - 16;
case IR::OpSize::i16Bit: return Op->Index - 8;
case IR::OpSize::i32Bit: return Op->Index - 4;
case IR::OpSize::i64Bit: return Op->Index - 2;
default: LOGMAN_MSG_A_FMT("Unhandled OpSize: {}", OpSize); return 0;
}
}();
@@ -1545,7 +1609,7 @@ DEF_OP(Float_ToGPR_ZS) {
ARMEmitter::Register Dst = GetReg(Node);
ARMEmitter::VRegister Src = GetVReg(Op->Scalar.ID());
if (Op->SrcElementSize == 8) {
if (Op->SrcElementSize == IR::OpSize::i64Bit) {
fcvtzs(ConvertSize(IROp), Dst, Src.D());
} else {
fcvtzs(ConvertSize(IROp), Dst, Src.S());
@@ -1558,7 +1622,7 @@ DEF_OP(Float_ToGPR_S) {
ARMEmitter::Register Dst = GetReg(Node);
ARMEmitter::VRegister Src = GetVReg(Op->Scalar.ID());
if (Op->SrcElementSize == 8) {
if (Op->SrcElementSize == IR::OpSize::i64Bit) {
frinti(VTMP1.D(), Src.D());
fcvtzs(ConvertSize(IROp), Dst, VTMP1.D());
} else {
@@ -1569,7 +1633,7 @@ DEF_OP(Float_ToGPR_S) {
DEF_OP(FCmp) {
auto Op = IROp->C<IR::IROp_FCmp>();
const auto EmitSubSize = Op->ElementSize == 8 ? ARMEmitter::ScalarRegSize::i64Bit : ARMEmitter::ScalarRegSize::i32Bit;
const auto EmitSubSize = Op->ElementSize == IR::OpSize::i64Bit ? ARMEmitter::ScalarRegSize::i64Bit : ARMEmitter::ScalarRegSize::i32Bit;
ARMEmitter::VRegister Scalar1 = GetVReg(Op->Scalar1.ID());
ARMEmitter::VRegister Scalar2 = GetVReg(Op->Scalar2.ID());
@@ -6,8 +6,9 @@ desc: relocation logic of the arm64 splatter backend
$end_info$
*/
#include "Interface/Context/Context.h"
#include "Interface/Core/JIT/Arm64/JITClass.h"
#include "Interface/HLE/Thunks/Thunks.h"
#include "Interface/Core/JIT/JITClass.h"
#include <FEXCore/Core/Thunks.h>
namespace FEXCore::CPU {
@@ -85,7 +86,7 @@ bool Arm64JITCore::ApplyRelocations(uint64_t GuestEntry, uint64_t CodeEntry, uin
size_t DataIndex {};
for (size_t j = 0; j < NumRelocations; ++j) {
const FEXCore::CPU::Relocation* Reloc = reinterpret_cast<const FEXCore::CPU::Relocation*>(&EntryRelocations[DataIndex]);
LOGMAN_THROW_AA_FMT((DataIndex % alignof(Relocation)) == 0, "Alignment of relocation wasn't adhered to");
LOGMAN_THROW_A_FMT((DataIndex % alignof(Relocation)) == 0, "Alignment of relocation wasn't adhered to");
switch (Reloc->Header.Type) {
case FEXCore::CPU::RelocationTypes::RELOC_NAMED_SYMBOL_LITERAL: {
@@ -7,13 +7,13 @@ $end_info$
#include "Interface/Context/Context.h"
#include "Interface/Core/Dispatcher/Dispatcher.h"
#include "Interface/Core/JIT/Arm64/JITClass.h"
#include "Interface/Core/JIT/JITClass.h"
namespace FEXCore::CPU {
#define DEF_OP(x) void Arm64JITCore::Op_##x(IR::IROp_Header const* IROp, IR::NodeID Node)
DEF_OP(CASPair) {
auto Op = IROp->C<IR::IROp_CASPair>();
LOGMAN_THROW_AA_FMT(IROp->ElementSize == 4 || IROp->ElementSize == 8, "Wrong element size");
LOGMAN_THROW_A_FMT(IROp->ElementSize == IR::OpSize::i32Bit || IROp->ElementSize == IR::OpSize::i64Bit, "Wrong element size");
// Size is the size of each pair element
auto Dst0 = GetReg(Op->OutLo.ID());
auto Dst1 = GetReg(Op->OutHi.ID());
@@ -23,7 +23,7 @@ DEF_OP(CASPair) {
auto Desired1 = GetReg(Op->DesiredHi.ID());
auto MemSrc = GetReg(Op->Addr.ID());
const auto EmitSize = IROp->ElementSize == 8 ? ARMEmitter::Size::i64Bit : ARMEmitter::Size::i32Bit;
const auto EmitSize = IROp->ElementSize == IR::OpSize::i64Bit ? ARMEmitter::Size::i64Bit : ARMEmitter::Size::i32Bit;
if (CTX->HostFeatures.SupportsAtomics) {
// RA has heuristics to try to pair sources, but we need to handle the cases
// where they fail. We do so by moving to temporaries. Note we use 64-bit
@@ -61,8 +61,8 @@ DEF_OP(CASPair) {
mrs(TMP1, ARMEmitter::SystemRegister::NZCV);
ARMEmitter::BackwardLabel LoopTop;
ARMEmitter::SingleUseForwardLabel LoopNotExpected;
ARMEmitter::SingleUseForwardLabel LoopExpected;
ARMEmitter::ForwardLabel LoopNotExpected;
ARMEmitter::ForwardLabel LoopExpected;
Bind(&LoopTop);
// This instruction sequence must be synced with HandleCASPAL_Armv8.
@@ -108,13 +108,13 @@ DEF_OP(CAS) {
mov(EmitSize, GetReg(Node), TMP2.R());
} else {
ARMEmitter::BackwardLabel LoopTop;
ARMEmitter::SingleUseForwardLabel LoopNotExpected;
ARMEmitter::SingleUseForwardLabel LoopExpected;
ARMEmitter::ForwardLabel LoopNotExpected;
ARMEmitter::ForwardLabel LoopExpected;
Bind(&LoopTop);
ldaxr(SubEmitSize, TMP2, MemSrc);
if (IROp->Size == 1) {
if (IROp->Size == IR::OpSize::i8Bit) {
cmp(EmitSize, TMP2, Expected, ARMEmitter::ExtendedType::UXTB, 0);
} else if (IROp->Size == 2) {
} else if (IROp->Size == IR::OpSize::i16Bit) {
cmp(EmitSize, TMP2, Expected, ARMEmitter::ExtendedType::UXTH, 0);
} else {
cmp(EmitSize, TMP2, Expected);
@@ -273,18 +273,21 @@ DEF_OP(AtomicNeg) {
DEF_OP(AtomicSwap) {
auto Op = IROp->C<IR::IROp_AtomicSwap>();
uint8_t OpSize = IROp->Size;
LOGMAN_THROW_AA_FMT(OpSize == 8 || OpSize == 4 || OpSize == 2 || OpSize == 1, "Unexpected CAS size");
const auto OpSize = IROp->Size;
LOGMAN_THROW_A_FMT(
OpSize == IR::OpSize::i64Bit || OpSize == IR::OpSize::i32Bit || OpSize == IR::OpSize::i16Bit || OpSize == IR::OpSize::i8Bit, "Unexpecte"
"d CAS "
"size");
auto MemSrc = GetReg(Op->Addr.ID());
auto Src = GetReg(Op->Value.ID());
const auto EmitSize = ConvertSize(IROp);
const auto SubEmitSize = OpSize == 8 ? ARMEmitter::SubRegSize::i64Bit :
OpSize == 4 ? ARMEmitter::SubRegSize::i32Bit :
OpSize == 2 ? ARMEmitter::SubRegSize::i16Bit :
OpSize == 1 ? ARMEmitter::SubRegSize::i8Bit :
ARMEmitter::SubRegSize::i8Bit;
const auto SubEmitSize = OpSize == IR::OpSize::i64Bit ? ARMEmitter::SubRegSize::i64Bit :
OpSize == IR::OpSize::i32Bit ? ARMEmitter::SubRegSize::i32Bit :
OpSize == IR::OpSize::i16Bit ? ARMEmitter::SubRegSize::i16Bit :
OpSize == IR::OpSize::i8Bit ? ARMEmitter::SubRegSize::i8Bit :
ARMEmitter::SubRegSize::i8Bit;
if (CTX->HostFeatures.SupportsAtomics) {
ldswpal(SubEmitSize, Src, GetReg(Node), MemSrc);
@@ -294,7 +297,7 @@ DEF_OP(AtomicSwap) {
ldaxr(SubEmitSize, TMP2, MemSrc);
stlxr(SubEmitSize, TMP4, Src, MemSrc);
cbnz(EmitSize, TMP4, &LoopTop);
ubfm(EmitSize, GetReg(Node), TMP2, 0, OpSize * 8 - 1);
ubfm(EmitSize, GetReg(Node), TMP2, 0, IR::OpSizeAsBits(OpSize) - 1);
}
}
@@ -9,13 +9,13 @@ $end_info$
#include "FEXCore/IR/IR.h"
#include "Interface/Core/LookupCache.h"
#include "Interface/Core/JIT/Arm64/JITClass.h"
#include "Interface/Core/JIT/JITClass.h"
#include <FEXCore/Core/Thunks.h>
#include <FEXCore/Core/X86Enums.h>
#include <FEXCore/Debug/InternalThreadState.h>
#include <FEXCore/HLE/SyscallHandler.h>
#include <FEXCore/Utils/MathUtils.h>
#include <Interface/HLE/Thunks/Thunks.h>
namespace FEXCore::CPU {
#define DEF_OP(x) void Arm64JITCore::Op_##x(IR::IROp_Header const* IROp, IR::NodeID Node)
@@ -53,14 +53,14 @@ DEF_OP(ExitFunction) {
if (IsInlineConstant(Op->NewRIP, &NewRIP) || IsInlineEntrypointOffset(Op->NewRIP, &NewRIP)) {
#ifdef _M_ARM_64EC
if (RtlIsEcCode(NewRIP)) {
if (NewRIP < EC_CODE_BITMAP_MAX_ADDRESS && RtlIsEcCode(NewRIP)) {
add(ARMEmitter::Size::i64Bit, ARMEmitter::Reg::rsp, StaticRegisters[X86State::REG_RSP], 0);
LoadConstant(ARMEmitter::Size::i64Bit, EC_CALL_CHECKER_PC_REG, NewRIP);
ldr(TMP2, STATE_PTR(CpuStateFrame, Pointers.Common.ExitFunctionEC));
br(TMP2);
} else {
#endif
ARMEmitter::SingleUseForwardLabel l_BranchHost;
ARMEmitter::ForwardLabel l_BranchHost;
ldr(TMP1, &l_BranchHost);
blr(TMP1);
@@ -72,7 +72,7 @@ DEF_OP(ExitFunction) {
#endif
} else {
ARMEmitter::SingleUseForwardLabel FullLookup;
ARMEmitter::ForwardLabel FullLookup;
auto RipReg = GetReg(Op->NewRIP.ID());
// L1 Cache
@@ -117,7 +117,7 @@ DEF_OP(CondJump) {
[[maybe_unused]] const bool isConst = IsInlineConstant(Op->Cmp2, &Const);
auto Reg = GetReg(Op->Cmp1.ID());
const auto Size = Op->CompareSize == 4 ? ARMEmitter::Size::i32Bit : ARMEmitter::Size::i64Bit;
const auto Size = Op->CompareSize == IR::OpSize::i32Bit ? ARMEmitter::Size::i32Bit : ARMEmitter::Size::i64Bit;
LOGMAN_THROW_A_FMT(IsGPR(Op->Cmp1.ID()), "CondJump: Expected GPR");
LOGMAN_THROW_A_FMT(isConst, "CondJump: Expected constant source");
@@ -5,8 +5,7 @@ tags: backend|arm64
$end_info$
*/
#include "Interface/Context/Context.h"
#include "Interface/Core/JIT/Arm64/JITClass.h"
#include "Interface/Core/JIT/JITClass.h"
namespace FEXCore::CPU {
#define DEF_OP(x) void Arm64JITCore::Op_##x(IR::IROp_Header const* IROp, IR::NodeID Node)
@@ -16,18 +15,18 @@ DEF_OP(VInsGPR) {
const auto DestIdx = Op->DestIdx;
const auto ElementSize = Op->Header.ElementSize;
const auto Is256Bit = OpSize == Core::CPUState::XMM_AVX_REG_SIZE;
const auto Is256Bit = OpSize == IR::OpSize::i256Bit;
LOGMAN_THROW_A_FMT(!Is256Bit || (Is256Bit && HostSupportsSVE256), "Need SVE256 support in order to use {} with 256-bit operation", __func__);
const auto SubEmitSize = ConvertSubRegSize8(IROp);
const auto ElementsPer128Bit = 16 / ElementSize;
const auto ElementsPer128Bit = IR::NumElements(IR::OpSize::i128Bit, ElementSize);
const auto Dst = GetVReg(Node);
const auto DestVector = GetVReg(Op->DestVector.ID());
const auto Src = GetReg(Op->Src.ID());
if (HostSupportsSVE256 && Is256Bit) {
const auto ElementSizeBits = ElementSize * 8;
const auto ElementSizeBits = IR::OpSizeAsBits(ElementSize);
const auto Offset = ElementSizeBits * DestIdx;
const auto SSEBitSize = Core::CPUState::XMM_SSE_REG_SIZE * 8;
@@ -91,16 +90,16 @@ DEF_OP(VCastFromGPR) {
auto Src = GetReg(Op->Src.ID());
switch (Op->Header.ElementSize) {
case 1:
case IR::OpSize::i8Bit:
uxtb(ARMEmitter::Size::i32Bit, TMP1, Src);
fmov(ARMEmitter::Size::i32Bit, Dst.S(), TMP1);
break;
case 2:
case IR::OpSize::i16Bit:
uxth(ARMEmitter::Size::i32Bit, TMP1, Src);
fmov(ARMEmitter::Size::i32Bit, Dst.S(), TMP1);
break;
case 4: fmov(ARMEmitter::Size::i32Bit, Dst.S(), Src); break;
case 8: fmov(ARMEmitter::Size::i64Bit, Dst.D(), Src); break;
case IR::OpSize::i32Bit: fmov(ARMEmitter::Size::i32Bit, Dst.S(), Src); break;
case IR::OpSize::i64Bit: fmov(ARMEmitter::Size::i64Bit, Dst.D(), Src); break;
default: LOGMAN_MSG_A_FMT("Unknown castGPR element size: {}", Op->Header.ElementSize);
}
}
@@ -112,7 +111,7 @@ DEF_OP(VDupFromGPR) {
const auto Dst = GetVReg(Node);
const auto Src = GetReg(Op->Src.ID());
const auto Is256Bit = OpSize == Core::CPUState::XMM_AVX_REG_SIZE;
const auto Is256Bit = OpSize == IR::OpSize::i256Bit;
LOGMAN_THROW_A_FMT(!Is256Bit || (Is256Bit && HostSupportsSVE256), "Need SVE256 support in order to use {} with 256-bit operation", __func__);
const auto SubEmitSize = ConvertSubRegSize8(IROp);
@@ -127,8 +126,8 @@ DEF_OP(VDupFromGPR) {
DEF_OP(Float_FromGPR_S) {
const auto Op = IROp->C<IR::IROp_Float_FromGPR_S>();
const uint16_t ElementSize = Op->Header.ElementSize;
const uint16_t Conv = (ElementSize << 8) | Op->SrcElementSize;
const uint16_t ElementSize = IR::OpSizeToSize(Op->Header.ElementSize);
const uint16_t Conv = (ElementSize << 8) | IR::OpSizeToSize(Op->SrcElementSize);
auto Dst = GetVReg(Node);
auto Src = GetReg(Op->Src.ID());
@@ -166,7 +165,7 @@ DEF_OP(Float_FromGPR_S) {
DEF_OP(Float_FToF) {
auto Op = IROp->C<IR::IROp_Float_FToF>();
const uint16_t Conv = (Op->Header.ElementSize << 8) | Op->SrcElementSize;
const uint16_t Conv = (IR::OpSizeToSize(Op->Header.ElementSize) << 8) | IR::OpSizeToSize(Op->SrcElementSize);
auto Dst = GetVReg(Node);
auto Src = GetVReg(Op->Scalar.ID());
@@ -206,7 +205,7 @@ DEF_OP(Vector_SToF) {
const auto ElementSize = Op->Header.ElementSize;
const auto SubEmitSize = ConvertSubRegSize248(IROp);
const auto Is256Bit = OpSize == Core::CPUState::XMM_AVX_REG_SIZE;
const auto Is256Bit = OpSize == IR::OpSize::i256Bit;
LOGMAN_THROW_A_FMT(!Is256Bit || (Is256Bit && HostSupportsSVE256), "Need SVE256 support in order to use {} with 256-bit operation", __func__);
const auto Dst = GetVReg(Node);
@@ -216,15 +215,15 @@ DEF_OP(Vector_SToF) {
scvtf(Dst.Z(), SubEmitSize, Mask.Merging(), Vector.Z(), SubEmitSize);
} else {
if (OpSize == ElementSize) {
if (ElementSize == 8) {
if (ElementSize == IR::OpSize::i64Bit) {
scvtf(ARMEmitter::ScalarRegSize::i64Bit, Dst.D(), Vector.D());
} else if (ElementSize == 4) {
} else if (ElementSize == IR::OpSize::i32Bit) {
scvtf(ARMEmitter::ScalarRegSize::i32Bit, Dst.S(), Vector.S());
} else {
scvtf(ARMEmitter::ScalarRegSize::i16Bit, Dst.H(), Vector.H());
}
} else {
if (OpSize == 8) {
if (OpSize == IR::OpSize::i64Bit) {
scvtf(SubEmitSize, Dst.D(), Vector.D());
} else {
scvtf(SubEmitSize, Dst.Q(), Vector.Q());
@@ -239,7 +238,7 @@ DEF_OP(Vector_FToZS) {
const auto ElementSize = Op->Header.ElementSize;
const auto SubEmitSize = ConvertSubRegSize248(IROp);
const auto Is256Bit = OpSize == Core::CPUState::XMM_AVX_REG_SIZE;
const auto Is256Bit = OpSize == IR::OpSize::i256Bit;
LOGMAN_THROW_A_FMT(!Is256Bit || (Is256Bit && HostSupportsSVE256), "Need SVE256 support in order to use {} with 256-bit operation", __func__);
const auto Dst = GetVReg(Node);
@@ -249,15 +248,15 @@ DEF_OP(Vector_FToZS) {
fcvtzs(Dst.Z(), SubEmitSize, Mask.Merging(), Vector.Z(), SubEmitSize);
} else {
if (OpSize == ElementSize) {
if (ElementSize == 8) {
if (ElementSize == IR::OpSize::i64Bit) {
fcvtzs(ARMEmitter::ScalarRegSize::i64Bit, Dst.D(), Vector.D());
} else if (ElementSize == 4) {
} else if (ElementSize == IR::OpSize::i32Bit) {
fcvtzs(ARMEmitter::ScalarRegSize::i32Bit, Dst.S(), Vector.S());
} else {
fcvtzs(ARMEmitter::ScalarRegSize::i16Bit, Dst.H(), Vector.H());
}
} else {
if (OpSize == 8) {
if (OpSize == IR::OpSize::i64Bit) {
fcvtzs(SubEmitSize, Dst.D(), Vector.D());
} else {
fcvtzs(SubEmitSize, Dst.Q(), Vector.Q());
@@ -270,7 +269,7 @@ DEF_OP(Vector_FToS) {
const auto Op = IROp->C<IR::IROp_Vector_FToS>();
const auto OpSize = IROp->Size;
const auto Is256Bit = OpSize == Core::CPUState::XMM_AVX_REG_SIZE;
const auto Is256Bit = OpSize == IR::OpSize::i256Bit;
LOGMAN_THROW_A_FMT(!Is256Bit || (Is256Bit && HostSupportsSVE256), "Need SVE256 support in order to use {} with 256-bit operation", __func__);
const auto SubEmitSize = ConvertSubRegSize248(IROp);
@@ -285,7 +284,7 @@ DEF_OP(Vector_FToS) {
} else {
const auto Dst = GetVReg(Node);
const auto Vector = GetVReg(Op->Vector.ID());
if (OpSize == 8) {
if (OpSize == IR::OpSize::i64Bit) {
frinti(SubEmitSize, Dst.D(), Vector.D());
fcvtzs(SubEmitSize, Dst.D(), Dst.D());
} else {
@@ -301,10 +300,10 @@ DEF_OP(Vector_FToF) {
const auto ElementSize = Op->Header.ElementSize;
const auto SubEmitSize = ConvertSubRegSize248(IROp);
const auto Is256Bit = OpSize == Core::CPUState::XMM_AVX_REG_SIZE;
const auto Is256Bit = OpSize == IR::OpSize::i256Bit;
LOGMAN_THROW_A_FMT(!Is256Bit || (Is256Bit && HostSupportsSVE256), "Need SVE256 support in order to use {} with 256-bit operation", __func__);
const auto Conv = (ElementSize << 8) | Op->SrcElementSize;
const auto Conv = (IR::OpSizeToSize(ElementSize) << 8) | IR::OpSizeToSize(Op->SrcElementSize);
const auto Dst = GetVReg(Node);
const auto Vector = GetVReg(Op->Vector.ID());
@@ -404,7 +403,7 @@ DEF_OP(Vector_FToI) {
const auto ElementSize = Op->Header.ElementSize;
const auto SubEmitSize = ConvertSubRegSize248(IROp);
const auto Is256Bit = OpSize == Core::CPUState::XMM_AVX_REG_SIZE;
const auto Is256Bit = OpSize == IR::OpSize::i256Bit;
LOGMAN_THROW_A_FMT(!Is256Bit || (Is256Bit && HostSupportsSVE256), "Need SVE256 support in order to use {} with 256-bit operation", __func__);
const auto Dst = GetVReg(Node);
@@ -428,15 +427,15 @@ DEF_OP(Vector_FToI) {
// frinti having AdvSIMD, AdvSIMD scalar, and an SVE version),
// we can't just use a lambda without some seriously ugly casting.
// This is fairly self-contained otherwise.
#define ROUNDING_FN(name) \
if (ElementSize == 2) { \
name(Dst.H(), Vector.H()); \
} else if (ElementSize == 4) { \
name(Dst.S(), Vector.S()); \
} else if (ElementSize == 8) { \
name(Dst.D(), Vector.D()); \
} else { \
FEX_UNREACHABLE; \
#define ROUNDING_FN(name) \
if (ElementSize == IR::OpSize::i16Bit) { \
name(Dst.H(), Vector.H()); \
} else if (ElementSize == IR::OpSize::i32Bit) { \
name(Dst.S(), Vector.S()); \
} else if (ElementSize == IR::OpSize::i64Bit) { \
name(Dst.D(), Vector.D()); \
} else { \
FEX_UNREACHABLE; \
}
switch (Op->Round) {
@@ -465,7 +464,7 @@ DEF_OP(Vector_F64ToI32) {
const auto OpSize = IROp->Size;
const auto Round = Op->Round;
const auto Is256Bit = OpSize == Core::CPUState::XMM_AVX_REG_SIZE;
const auto Is256Bit = OpSize == IR::OpSize::i256Bit;
LOGMAN_THROW_A_FMT(!Is256Bit || (Is256Bit && HostSupportsSVE256), "Need SVE256 support in order to use {} with 256-bit operation", __func__);
const auto Dst = GetVReg(Node);
@@ -5,7 +5,7 @@ tags: backend|arm64
$end_info$
*/
#include "Interface/Core/JIT/Arm64/JITClass.h"
#include "Interface/Core/JIT/JITClass.h"
namespace FEXCore::CPU {
#define DEF_OP(x) void Arm64JITCore::Op_##x(IR::IROp_Header const* IROp, IR::NodeID Node)
@@ -17,14 +17,14 @@ DEF_OP(VAESImc) {
DEF_OP(VAESEnc) {
const auto Op = IROp->C<IR::IROp_VAESEnc>();
const auto OpSize = IROp->Size;
[[maybe_unused]] const auto OpSize = IROp->Size;
const auto Dst = GetVReg(Node);
const auto Key = GetVReg(Op->Key.ID());
const auto State = GetVReg(Op->State.ID());
const auto ZeroReg = GetVReg(Op->ZeroReg.ID());
LOGMAN_THROW_AA_FMT(OpSize == Core::CPUState::XMM_SSE_REG_SIZE, "Currently only supports 128-bit operations.");
LOGMAN_THROW_A_FMT(OpSize == IR::OpSize::i128Bit, "Currently only supports 128-bit operations.");
if (Dst == State && Dst != Key) {
// Optimal case in which Dst already contains the starting state.
@@ -42,14 +42,14 @@ DEF_OP(VAESEnc) {
DEF_OP(VAESEncLast) {
const auto Op = IROp->C<IR::IROp_VAESEncLast>();
const auto OpSize = IROp->Size;
[[maybe_unused]] const auto OpSize = IROp->Size;
const auto Dst = GetVReg(Node);
const auto Key = GetVReg(Op->Key.ID());
const auto State = GetVReg(Op->State.ID());
const auto ZeroReg = GetVReg(Op->ZeroReg.ID());
LOGMAN_THROW_AA_FMT(OpSize == Core::CPUState::XMM_SSE_REG_SIZE, "Currently only supports 128-bit operations.");
LOGMAN_THROW_A_FMT(OpSize == IR::OpSize::i128Bit, "Currently only supports 128-bit operations.");
if (Dst == State && Dst != Key) {
// Optimal case in which Dst already contains the starting state.
@@ -65,14 +65,14 @@ DEF_OP(VAESEncLast) {
DEF_OP(VAESDec) {
const auto Op = IROp->C<IR::IROp_VAESDec>();
const auto OpSize = IROp->Size;
[[maybe_unused]] const auto OpSize = IROp->Size;
const auto Dst = GetVReg(Node);
const auto Key = GetVReg(Op->Key.ID());
const auto State = GetVReg(Op->State.ID());
const auto ZeroReg = GetVReg(Op->ZeroReg.ID());
LOGMAN_THROW_AA_FMT(OpSize == Core::CPUState::XMM_SSE_REG_SIZE, "Currently only supports 128-bit operations.");
LOGMAN_THROW_A_FMT(OpSize == IR::OpSize::i128Bit, "Currently only supports 128-bit operations.");
if (Dst == State && Dst != Key) {
// Optimal case in which Dst already contains the starting state.
@@ -90,14 +90,14 @@ DEF_OP(VAESDec) {
DEF_OP(VAESDecLast) {
const auto Op = IROp->C<IR::IROp_VAESDecLast>();
const auto OpSize = IROp->Size;
[[maybe_unused]] const auto OpSize = IROp->Size;
const auto Dst = GetVReg(Node);
const auto Key = GetVReg(Op->Key.ID());
const auto State = GetVReg(Op->State.ID());
const auto ZeroReg = GetVReg(Op->ZeroReg.ID());
LOGMAN_THROW_AA_FMT(OpSize == Core::CPUState::XMM_SSE_REG_SIZE, "Currently only supports 128-bit operations.");
LOGMAN_THROW_A_FMT(OpSize == IR::OpSize::i128Bit, "Currently only supports 128-bit operations.");
if (Dst == State && Dst != Key) {
// Optimal case in which Dst already contains the starting state.
@@ -152,10 +152,10 @@ DEF_OP(CRC32) {
const auto Src2 = GetReg(Op->Src2.ID());
switch (Op->SrcSize) {
case 1: crc32cb(Dst.W(), Src1.W(), Src2.W()); break;
case 2: crc32ch(Dst.W(), Src1.W(), Src2.W()); break;
case 4: crc32cw(Dst.W(), Src1.W(), Src2.W()); break;
case 8: crc32cx(Dst.X(), Src1.X(), Src2.X()); break;
case IR::OpSize::i8Bit: crc32cb(Dst.W(), Src1.W(), Src2.W()); break;
case IR::OpSize::i16Bit: crc32ch(Dst.W(), Src1.W(), Src2.W()); break;
case IR::OpSize::i32Bit: crc32cw(Dst.W(), Src1.W(), Src2.W()); break;
case IR::OpSize::i64Bit: crc32cx(Dst.X(), Src1.X(), Src2.X()); break;
default: LOGMAN_MSG_A_FMT("Unknown CRC32 size: {}", Op->SrcSize);
}
}
@@ -187,13 +187,13 @@ DEF_OP(VSha256U0) {
DEF_OP(PCLMUL) {
const auto Op = IROp->C<IR::IROp_PCLMUL>();
const auto OpSize = IROp->Size;
[[maybe_unused]] const auto OpSize = IROp->Size;
const auto Dst = GetVReg(Node);
const auto Src1 = GetVReg(Op->Src1.ID());
const auto Src2 = GetVReg(Op->Src2.ID());
LOGMAN_THROW_AA_FMT(OpSize == Core::CPUState::XMM_SSE_REG_SIZE, "Currently only supports 128-bit operations.");
LOGMAN_THROW_A_FMT(OpSize == IR::OpSize::i128Bit, "Currently only supports 128-bit operations.");
switch (Op->Selector) {
case 0b00000000: pmull(ARMEmitter::SubRegSize::i128Bit, Dst.D(), Src1.D(), Src2.D()); break;
@@ -16,11 +16,12 @@ $end_info$
#include "Interface/Core/LookupCache.h"
#include "Interface/Core/Dispatcher/Dispatcher.h"
#include "Interface/Core/JIT/Arm64/JITClass.h"
#include "Interface/Core/JIT/JITClass.h"
#include "Interface/IR/Passes/RegisterAllocationPass.h"
#include "Utils/MemberFunctionToPointer.h"
#include "Utils/variable_length_integer.h"
#include <FEXCore/Core/X86Enums.h>
#include <FEXCore/Debug/InternalThreadState.h>
@@ -35,6 +36,7 @@ $end_info$
#include <stdio.h>
#include <unistd.h>
#include <string.h>
#include <limits>
static constexpr size_t INITIAL_CODE_SIZE = 1024 * 1024 * 16;
// We don't want to move above 128MB atm because that means we will have to encode longer jumps
@@ -469,7 +471,7 @@ static void DirectBlockDelinker(FEXCore::Core::CpuStateFrame* Frame, FEXCore::Co
auto LinkerAddress = Frame->Pointers.Common.ExitFunctionLinker;
uintptr_t branch = (uintptr_t)(Record)-8;
ARMEmitter::Emitter emit((uint8_t*)(branch), 8);
ARMEmitter::SingleUseForwardLabel l_BranchHost;
ARMEmitter::ForwardLabel l_BranchHost;
emit.ldr(TMP1, &l_BranchHost);
emit.blr(TMP1);
emit.Bind(&l_BranchHost);
@@ -484,11 +486,16 @@ static void IndirectBlockDelinker(FEXCore::Core::CpuStateFrame* Frame, FEXCore::
static uint64_t Arm64JITCore_ExitFunctionLink(FEXCore::Core::CpuStateFrame* Frame, FEXCore::Context::ExitFunctionLinkData* Record) {
auto Thread = Frame->Thread;
bool TFSet = Thread->CurrentFrame->State.flags[X86State::RFLAG_TF_RAW_LOC];
uintptr_t HostCode {};
auto GuestRip = Record->GuestRIP;
auto HostCode = Thread->LookupCache->FindBlock(GuestRip);
if (!TFSet) {
HostCode = Thread->LookupCache->FindBlock(GuestRip);
}
if (!HostCode) {
if (TFSet || !HostCode) {
// If TF is set, the cache must be skipped as different code needs to be generated.
Frame->State.rip = GuestRip;
return Frame->Pointers.Common.DispatcherLoopTop;
}
@@ -626,8 +633,8 @@ bool Arm64JITCore::IsInlineEntrypointOffset(const IR::OrderedNodeWrapper& WNode,
auto Op = OpHeader->C<IR::IROp_InlineEntrypointOffset>();
if (Value) {
uint64_t Mask = ~0ULL;
uint8_t OpSize = OpHeader->Size;
if (OpSize == 4) {
const auto Size = OpHeader->Size;
if (Size == IR::OpSize::i32Bit) {
Mask = 0xFFFF'FFFFULL;
}
*Value = (Entry + Op->Offset) & Mask;
@@ -654,8 +661,69 @@ bool Arm64JITCore::IsGPR(IR::NodeID Node) const {
return Class == IR::GPRClass || Class == IR::GPRFixedClass;
}
CPUBackend::CompiledCode Arm64JITCore::CompileCode(uint64_t Entry, const FEXCore::IR::IRListView* IR, FEXCore::Core::DebugData* DebugData,
const FEXCore::IR::RegisterAllocationData* RAData) {
void Arm64JITCore::EmitInterruptChecks(bool CheckTF) {
if (CheckTF) {
ARMEmitter::ForwardLabel l_TFUnset;
ARMEmitter::ForwardLabel l_TFBlocked;
// Note that this needs to be before the below suspend checks, as X86 checks this flag immediately after executing an instruction.
ldrb(TMP1, STATE_PTR(CpuStateFrame, State.flags[X86State::RFLAG_TF_RAW_LOC]));
cbz(ARMEmitter::Size::i32Bit, TMP1, &l_TFUnset);
// X86 semantically checks TF after executing each instruction, so e.g. setting a context with TF set will execute a single instruction
// and then raise an exception. However on the FEX side this is simpler to implement by checking at the start of each instruction, handle this by having bit 1 being unset in the flag state indicate that TF is blocked for a single instruction.
tbz(TMP1, 1, &l_TFBlocked);
// Block TF for a single instruction when the frontend jumps to a new context by unsetting bit 1.
ldrb(TMP1, STATE_PTR(CpuStateFrame, State.flags[X86State::RFLAG_TF_RAW_LOC]));
and_(ARMEmitter::Size::i32Bit, TMP1, TMP1, ~(1 << 1));
strb(TMP1, STATE_PTR(CpuStateFrame, State.flags[X86State::RFLAG_TF_RAW_LOC]));
Core::CpuStateFrame::SynchronousFaultDataStruct State = {
.FaultToTopAndGeneratedException = 1,
.Signal = Core::FAULT_SIGTRAP,
.TrapNo = X86State::X86_TRAPNO_DB,
.si_code = 2,
.err_code = 0,
};
uint64_t Constant {};
memcpy(&Constant, &State, sizeof(State));
LoadConstant(ARMEmitter::Size::i64Bit, TMP1, Constant);
str(TMP1, STATE, offsetof(FEXCore::Core::CpuStateFrame, SynchronousFaultData));
ldr(TMP1, STATE, offsetof(FEXCore::Core::CpuStateFrame, Pointers.Common.GuestSignal_SIGTRAP));
br(TMP1);
Bind(&l_TFBlocked);
// If TF was blocked for this instruction, unblock it for the next.
LoadConstant(ARMEmitter::Size::i32Bit, TMP1, 0b11);
strb(TMP1, STATE_PTR(CpuStateFrame, State.flags[X86State::RFLAG_TF_RAW_LOC]));
Bind(&l_TFUnset);
}
if (CTX->Config.NeedsPendingInterruptFaultCheck) {
// Trigger a fault if there are any pending interrupts
// Used only for suspend on WIN32 at the moment
strb(ARMEmitter::XReg::zr, STATE,
offsetof(FEXCore::Core::InternalThreadState, InterruptFaultPage) - offsetof(FEXCore::Core::InternalThreadState, BaseFrameState));
}
#ifdef _M_ARM_64EC
static constexpr uint16_t SuspendMagic {0xCAFE};
ldr(TMP2.W(), STATE_PTR(CpuStateFrame, SuspendDoorbell));
ARMEmitter::ForwardLabel l_NoSuspend;
cbz(ARMEmitter::Size::i32Bit, TMP2, &l_NoSuspend);
brk(SuspendMagic);
Bind(&l_NoSuspend);
#endif
}
CPUBackend::CompiledCode Arm64JITCore::CompileCode(uint64_t Entry, uint64_t Size, bool SingleInst, const FEXCore::IR::IRListView* IR,
FEXCore::Core::DebugData* DebugData, const FEXCore::IR::RegisterAllocationData* RAData,
bool CheckTF) {
FEXCORE_PROFILE_SCOPED("Arm64::CompileCode");
JumpTargets.clear();
@@ -711,22 +779,7 @@ CPUBackend::CompiledCode Arm64JITCore::CompileCode(uint64_t Entry, const FEXCore
adr(TMP1, &JITCodeHeaderLabel);
str(TMP1, STATE, offsetof(FEXCore::Core::CPUState, InlineJITBlockHeader));
if (CTX->Config.NeedsPendingInterruptFaultCheck) {
// Trigger a fault if there are any pending interrupts
// Used only for suspend on WIN32 at the moment
strb(ARMEmitter::XReg::zr, STATE,
offsetof(FEXCore::Core::InternalThreadState, InterruptFaultPage) - offsetof(FEXCore::Core::InternalThreadState, BaseFrameState));
}
#ifdef _M_ARM_64EC
static constexpr uint16_t SuspendMagic {0xCAFE};
ldr(TMP2.W(), STATE_PTR(CpuStateFrame, SuspendDoorbell));
ARMEmitter::SingleUseForwardLabel l_NoSuspend;
cbz(ARMEmitter::Size::i32Bit, TMP2, &l_NoSuspend);
brk(SuspendMagic);
Bind(&l_NoSuspend);
#endif
EmitInterruptChecks(CheckTF);
SpillSlots = RAData->SpillSlots();
@@ -747,7 +800,7 @@ CPUBackend::CompiledCode Arm64JITCore::CompileCode(uint64_t Entry, const FEXCore
using namespace FEXCore::IR;
#if defined(ASSERTIONS_ENABLED) && ASSERTIONS_ENABLED
auto BlockIROp = BlockHeader->CW<FEXCore::IR::IROp_CodeBlock>();
LOGMAN_THROW_AA_FMT(BlockIROp->Header.Op == IR::OP_CODEBLOCK, "IR type failed to be a code block");
LOGMAN_THROW_A_FMT(BlockIROp->Header.Op == IR::OP_CODEBLOCK, "IR type failed to be a code block");
#endif
auto BlockStartHostCode = GetCursorAddress<uint8_t*>();
@@ -800,34 +853,61 @@ CPUBackend::CompiledCode Arm64JITCore::CompileCode(uint64_t Entry, const FEXCore
auto JITBlockTail = GetCursorAddress<JITCodeTail*>();
CursorIncrement(sizeof(JITCodeTail));
auto JITRIPEntriesLocation = GetCursorAddress<uint8_t*>();
auto JITRIPEntries = GetCursorAddress<JITRIPReconstructEntries*>();
// Entries that live after the JITCodeTail.
// These entries correlate JIT code regions with guest RIP regions.
// Using these entries FEX is able to reconstruct the guest RIP accurately when an instruction cause a signal fault.
// Packed using two variable length integer entries to ensure the size isn't too large.
// These smaller sizes means that each entry is relative to each other instead of absolute offset from the start of the JIT block.
// When reconstructing the RIP, each entry must be walked linearly and accumulated with the previous entries.
// This is a trade-off between compression inside the JIT code space and execution time when reconstruction the RIP.
// RIP reconstruction when faulting is less likely so we are requiring the accumulation.
//
// struct {
// // The Host PC offset from the previous entry.
// FEXCore::Utils::vl64 HostPCOffset;
// // How much to offset the RIP from the previous entry.
// FEXCore::Utils::vl64 GuestRIPOffset;
// };
CursorIncrement(sizeof(JITRIPReconstructEntries) * DebugData->GuestOpcodes.size());
auto JITRIPEntriesBegin = GetCursorAddress<uint8_t*>();
// Put the block's RIP entry in the tail.
// This will be used for RIP reconstruction in the future.
// TODO: This needs to be a data RIP relocation once code caching works.
// Current relocation code doesn't support this feature yet.
JITBlockTail->RIP = Entry;
JITBlockTail->GuestSize = Size;
JITBlockTail->SingleInst = SingleInst;
JITBlockTail->SpinLockFutex = 0;
auto JITRIPEntriesLocation = JITRIPEntriesBegin;
{
// Store the RIP entries.
JITBlockTail->NumberOfRIPEntries = DebugData->GuestOpcodes.size();
JITBlockTail->OffsetToRIPEntries = JITRIPEntriesLocation - JITBlockTailLocation;
JITBlockTail->OffsetToRIPEntries = JITRIPEntriesBegin - JITBlockTailLocation;
uintptr_t CurrentRIPOffset = 0;
uint64_t CurrentPCOffset = 0;
for (size_t i = 0; i < DebugData->GuestOpcodes.size(); i++) {
const auto& GuestOpcode = DebugData->GuestOpcodes[i];
auto& RIPEntry = JITRIPEntries[i];
RIPEntry.HostPCOffset = GuestOpcode.HostEntryOffset - CurrentPCOffset;
RIPEntry.GuestRIPOffset = GuestOpcode.GuestEntryOffset - CurrentRIPOffset;
int64_t HostPCOffset = GuestOpcode.HostEntryOffset - CurrentPCOffset;
int64_t GuestRIPOffset = GuestOpcode.GuestEntryOffset - CurrentRIPOffset;
size_t Size = FEXCore::Utils::vl64::Encode(JITRIPEntriesLocation, HostPCOffset);
JITRIPEntriesLocation += Size;
Size = FEXCore::Utils::vl64::Encode(JITRIPEntriesLocation, GuestRIPOffset);
JITRIPEntriesLocation += Size;
CurrentPCOffset = GuestOpcode.HostEntryOffset;
CurrentRIPOffset = GuestOpcode.GuestEntryOffset;
}
}
CursorIncrement(JITRIPEntriesLocation - JITRIPEntriesBegin);
Align();
CodeHeader->OffsetToBlockTail = JITBlockTailLocation - CodeData.BlockBegin;
CodeData.Size = GetCursorAddress<uint8_t*>() - CodeData.BlockBegin;
@@ -839,7 +919,7 @@ CPUBackend::CompiledCode Arm64JITCore::CompileCode(uint64_t Entry, const FEXCore
#ifdef VIXL_DISASSEMBLER
if (Disassemble() & FEXCore::Config::Disassemble::STATS) {
auto HeaderOp = IR->GetHeader();
LOGMAN_THROW_AA_FMT(HeaderOp->Header.Op == IR::OP_IRHEADER, "First op wasn't IRHeader");
LOGMAN_THROW_A_FMT(HeaderOp->Header.Op == IR::OP_IRHEADER, "First op wasn't IRHeader");
LogMan::Msg::IFmt("RIP: 0x{:x}", Entry);
LogMan::Msg::IFmt("Guest Code instructions: {}", HeaderOp->NumHostInstructions);
@@ -38,23 +38,9 @@ public:
~Arm64JITCore() override;
[[nodiscard]]
fextl::string GetName() override {
return "JIT";
}
[[nodiscard]]
CPUBackend::CompiledCode CompileCode(uint64_t Entry, const FEXCore::IR::IRListView* IR, FEXCore::Core::DebugData* DebugData,
const FEXCore::IR::RegisterAllocationData* RAData) override;
[[nodiscard]]
void* MapRegion(void* HostPtr, uint64_t, uint64_t) override {
return HostPtr;
}
[[nodiscard]]
bool NeedsOpDispatch() override {
return true;
}
CPUBackend::CompiledCode
CompileCode(uint64_t Entry, uint64_t Size, bool SingleInst, const FEXCore::IR::IRListView* IR, FEXCore::Core::DebugData* DebugData,
const FEXCore::IR::RegisterAllocationData* RAData, bool CheckTF) override;
void ClearCache() override;
@@ -83,7 +69,7 @@ private:
ARMEmitter::Register GetReg(IR::NodeID Node) const {
const auto Reg = GetPhys(Node);
LOGMAN_THROW_AA_FMT(Reg.Class == IR::GPRFixedClass.Val || Reg.Class == IR::GPRClass.Val, "Unexpected Class: {}", Reg.Class);
LOGMAN_THROW_A_FMT(Reg.Class == IR::GPRFixedClass.Val || Reg.Class == IR::GPRClass.Val, "Unexpected Class: {}", Reg.Class);
if (Reg.Class == IR::GPRFixedClass.Val) {
return StaticRegisters[Reg.Reg];
@@ -98,7 +84,7 @@ private:
ARMEmitter::VRegister GetVReg(IR::NodeID Node) const {
const auto Reg = GetPhys(Node);
LOGMAN_THROW_AA_FMT(Reg.Class == IR::FPRFixedClass.Val || Reg.Class == IR::FPRClass.Val, "Unexpected Class: {}", Reg.Class);
LOGMAN_THROW_A_FMT(Reg.Class == IR::FPRFixedClass.Val || Reg.Class == IR::FPRClass.Val, "Unexpected Class: {}", Reg.Class);
if (Reg.Class == IR::FPRFixedClass.Val) {
return StaticFPRegisters[Reg.Reg];
@@ -125,7 +111,7 @@ private:
ARMEmitter::Register GetZeroableReg(IR::OrderedNodeWrapper Src) const {
uint64_t Const;
if (IsInlineConstant(Src, &Const)) {
LOGMAN_THROW_AA_FMT(Const == 0, "Only valid constant");
LOGMAN_THROW_A_FMT(Const == 0, "Only valid constant");
return ARMEmitter::Reg::zr;
} else {
return GetReg(Src.ID());
@@ -144,23 +130,25 @@ private:
[[nodiscard]]
ARMEmitter::Size ConvertSize(const IR::IROp_Header* Op) {
return Op->Size == 8 ? ARMEmitter::Size::i64Bit : ARMEmitter::Size::i32Bit;
return Op->Size == IR::OpSize::i64Bit ? ARMEmitter::Size::i64Bit : ARMEmitter::Size::i32Bit;
}
[[nodiscard]]
ARMEmitter::Size ConvertSize48(const IR::IROp_Header* Op) {
LOGMAN_THROW_AA_FMT(Op->Size == 4 || Op->Size == 8, "Invalid size");
LOGMAN_THROW_A_FMT(Op->Size == IR::OpSize::i32Bit || Op->Size == IR::OpSize::i64Bit, "Invalid size");
return ConvertSize(Op);
}
[[nodiscard]]
ARMEmitter::SubRegSize ConvertSubRegSize16(uint8_t ElementSize) {
LOGMAN_THROW_AA_FMT(ElementSize == 1 || ElementSize == 2 || ElementSize == 4 || ElementSize == 8 || ElementSize == 16, "Invalid size");
return ElementSize == 1 ? ARMEmitter::SubRegSize::i8Bit :
ElementSize == 2 ? ARMEmitter::SubRegSize::i16Bit :
ElementSize == 4 ? ARMEmitter::SubRegSize::i32Bit :
ElementSize == 8 ? ARMEmitter::SubRegSize::i64Bit :
ARMEmitter::SubRegSize::i128Bit;
ARMEmitter::SubRegSize ConvertSubRegSize16(IR::OpSize ElementSize) {
LOGMAN_THROW_A_FMT(ElementSize == IR::OpSize::i8Bit || ElementSize == IR::OpSize::i16Bit || ElementSize == IR::OpSize::i32Bit ||
ElementSize == IR::OpSize::i64Bit || ElementSize == IR::OpSize::i128Bit,
"Invalid size");
return ElementSize == IR::OpSize::i8Bit ? ARMEmitter::SubRegSize::i8Bit :
ElementSize == IR::OpSize::i16Bit ? ARMEmitter::SubRegSize::i16Bit :
ElementSize == IR::OpSize::i32Bit ? ARMEmitter::SubRegSize::i32Bit :
ElementSize == IR::OpSize::i64Bit ? ARMEmitter::SubRegSize::i64Bit :
ARMEmitter::SubRegSize::i128Bit;
}
[[nodiscard]]
@@ -169,8 +157,8 @@ private:
}
[[nodiscard]]
ARMEmitter::SubRegSize ConvertSubRegSize8(uint8_t ElementSize) {
LOGMAN_THROW_AA_FMT(ElementSize != 16, "Invalid size");
ARMEmitter::SubRegSize ConvertSubRegSize8(IR::OpSize ElementSize) {
LOGMAN_THROW_A_FMT(ElementSize != IR::OpSize::i128Bit, "Invalid size");
return ConvertSubRegSize16(ElementSize);
}
@@ -181,13 +169,13 @@ private:
[[nodiscard]]
ARMEmitter::SubRegSize ConvertSubRegSize4(const IR::IROp_Header* Op) {
LOGMAN_THROW_AA_FMT(Op->ElementSize != 8, "Invalid size");
LOGMAN_THROW_A_FMT(Op->ElementSize != IR::OpSize::i64Bit, "Invalid size");
return ConvertSubRegSize8(Op);
}
[[nodiscard]]
ARMEmitter::SubRegSize ConvertSubRegSize248(const IR::IROp_Header* Op) {
LOGMAN_THROW_AA_FMT(Op->ElementSize != 1, "Invalid size");
LOGMAN_THROW_A_FMT(Op->ElementSize != IR::OpSize::i8Bit, "Invalid size");
return ConvertSubRegSize8(Op);
}
@@ -198,13 +186,13 @@ private:
[[nodiscard]]
ARMEmitter::VectorRegSizePair ConvertSubRegSizePair8(const IR::IROp_Header* Op) {
LOGMAN_THROW_AA_FMT(Op->ElementSize != 16, "Invalid size");
LOGMAN_THROW_A_FMT(Op->ElementSize != IR::OpSize::i128Bit, "Invalid size");
return ConvertSubRegSizePair16(Op);
}
[[nodiscard]]
ARMEmitter::VectorRegSizePair ConvertSubRegSizePair248(const IR::IROp_Header* Op) {
LOGMAN_THROW_AA_FMT(Op->ElementSize != 1, "Invalid size");
LOGMAN_THROW_A_FMT(Op->ElementSize != IR::OpSize::i8Bit, "Invalid size");
return ConvertSubRegSizePair8(Op);
}
@@ -241,16 +229,20 @@ private:
bool IsGPR(IR::NodeID Node) const;
[[nodiscard]]
ARMEmitter::ExtendedMemOperand GenerateMemOperand(uint8_t AccessSize, ARMEmitter::Register Base, IR::OrderedNodeWrapper Offset,
ARMEmitter::ExtendedMemOperand GenerateMemOperand(IR::OpSize AccessSize, ARMEmitter::Register Base, IR::OrderedNodeWrapper Offset,
IR::MemOffsetType OffsetType, uint8_t OffsetScale);
[[nodiscard]]
ARMEmitter::Register ApplyMemOperand(IR::OpSize AccessSize, ARMEmitter::Register Base, ARMEmitter::Register Tmp,
IR::OrderedNodeWrapper Offset, IR::MemOffsetType OffsetType, uint8_t OffsetScale);
// NOTE: Will use TMP1 as a way to encode immediates that happen to fall outside
// the limits of the scalar plus immediate variant of SVE load/stores.
//
// TMP1 is safe to use again once this memory operand is used with its
// equivalent loads or stores that this was called for.
[[nodiscard]]
ARMEmitter::SVEMemOperand GenerateSVEMemOperand(uint8_t AccessSize, ARMEmitter::Register Base, IR::OrderedNodeWrapper Offset,
ARMEmitter::SVEMemOperand GenerateSVEMemOperand(IR::OpSize AccessSize, ARMEmitter::Register Base, IR::OrderedNodeWrapper Offset,
IR::MemOffsetType OffsetType, uint8_t OffsetScale);
[[nodiscard]]
@@ -331,20 +323,24 @@ private:
using ScalarFMAOpCaller =
std::function<void(ARMEmitter::VRegister Dst, ARMEmitter::VRegister Src1, ARMEmitter::VRegister Src2, ARMEmitter::VRegister Src3)>;
void VFScalarFMAOperation(uint8_t OpSize, uint8_t ElementSize, ScalarFMAOpCaller ScalarEmit, ARMEmitter::VRegister Dst,
void VFScalarFMAOperation(IR::OpSize OpSize, IR::OpSize ElementSize, ScalarFMAOpCaller ScalarEmit, ARMEmitter::VRegister Dst,
ARMEmitter::VRegister Upper, ARMEmitter::VRegister Vector1, ARMEmitter::VRegister Vector2,
ARMEmitter::VRegister Addend);
using ScalarBinaryOpCaller = std::function<void(ARMEmitter::VRegister Dst, ARMEmitter::VRegister Src1, ARMEmitter::VRegister Src2)>;
void VFScalarOperation(uint8_t OpSize, uint8_t ElementSize, bool ZeroUpperBits, ScalarBinaryOpCaller ScalarEmit,
void VFScalarOperation(IR::OpSize OpSize, IR::OpSize ElementSize, bool ZeroUpperBits, ScalarBinaryOpCaller ScalarEmit,
ARMEmitter::VRegister Dst, ARMEmitter::VRegister Vector1, ARMEmitter::VRegister Vector2);
using ScalarUnaryOpCaller = std::function<void(ARMEmitter::VRegister Dst, std::variant<ARMEmitter::VRegister, ARMEmitter::Register> SrcVar)>;
void VFScalarUnaryOperation(uint8_t OpSize, uint8_t ElementSize, bool ZeroUpperBits, ScalarUnaryOpCaller ScalarEmit, ARMEmitter::VRegister Dst,
ARMEmitter::VRegister Vector1, std::variant<ARMEmitter::VRegister, ARMEmitter::Register> Vector2);
void VFScalarUnaryOperation(IR::OpSize OpSize, IR::OpSize ElementSize, bool ZeroUpperBits, ScalarUnaryOpCaller ScalarEmit,
ARMEmitter::VRegister Dst, ARMEmitter::VRegister Vector1,
std::variant<ARMEmitter::VRegister, ARMEmitter::Register> Vector2);
void Emulate128BitGather(size_t Size, size_t ElementSize, ARMEmitter::VRegister Dst, ARMEmitter::VRegister IncomingDst,
void Emulate128BitGather(IR::OpSize Size, IR::OpSize ElementSize, ARMEmitter::VRegister Dst, ARMEmitter::VRegister IncomingDst,
std::optional<ARMEmitter::Register> BaseAddr, ARMEmitter::VRegister VectorIndexLow,
std::optional<ARMEmitter::VRegister> VectorIndexHigh, ARMEmitter::VRegister MaskReg, size_t VectorIndexSize,
std::optional<ARMEmitter::VRegister> VectorIndexHigh, ARMEmitter::VRegister MaskReg, IR::OpSize VectorIndexSize,
size_t DataElementOffsetStart, size_t IndexElementOffsetStart, uint8_t OffsetScale);
void EmitInterruptChecks(bool CheckTF);
// Runtime selection;
// Load and store TSO memory style
OpType RT_LoadMemTSO;
@@ -367,4 +363,7 @@ private:
#undef DEF_OP
};
[[nodiscard]]
fextl::unique_ptr<CPUBackend> CreateArm64JITCore(FEXCore::Context::ContextImpl* ctx, FEXCore::Core::InternalThreadState* Thread);
} // namespace FEXCore::CPU
@@ -1,21 +0,0 @@
// SPDX-License-Identifier: MIT
#pragma once
#include "Interface/Core/CPUBackend.h"
#include <FEXCore/fextl/memory.h>
namespace FEXCore::Context {
class ContextImpl;
}
namespace FEXCore::Core {
struct InternalThreadState;
}
namespace FEXCore::CPU {
class CPUBackend;
[[nodiscard]]
fextl::unique_ptr<CPUBackend> CreateArm64JITCore(FEXCore::Context::ContextImpl* ctx, FEXCore::Core::InternalThreadState* Thread);
} // namespace FEXCore::CPU
@@ -10,7 +10,7 @@ $end_info$
#endif
#include "Interface/Context/Context.h"
#include "Interface/Core/JIT/Arm64/JITClass.h"
#include "Interface/Core/JIT/JITClass.h"
#include "FEXCore/Debug/InternalThreadState.h"
#include <FEXCore/Core/SignalDelegator.h>
@@ -148,7 +148,7 @@ DEF_OP(PushRoundingMode) {
} else if (Op->RoundMode == 0) {
and_(ARMEmitter::Size::i64Bit, TMP1, Dest, ~(3 << 22));
} else {
LOGMAN_THROW_AA_FMT(Op->RoundMode == 1 || Op->RoundMode == 2, "expect a valid round mode");
LOGMAN_THROW_A_FMT(Op->RoundMode == 1 || Op->RoundMode == 2, "expect a valid round mode");
and_(ARMEmitter::Size::i64Bit, TMP1, Dest, ~(Op->RoundMode << 22));
orr(ARMEmitter::Size::i64Bit, TMP1, TMP1, (Op->RoundMode == 2 ? 1 : 2) << 22);
@@ -192,8 +192,17 @@ DEF_OP(Print) {
PopDynamicRegsAndLR();
}
#ifndef _WIN32
DEF_OP(ProcessorID) {
if (CTX->HostFeatures.SupportsCPUIndexInTPIDRRO) {
mrs(GetReg(Node), ARMEmitter::SystemRegister::TPIDRRO_EL0);
return;
}
#ifdef _WIN32
else {
// If on Windows and TPIDRRO isn't supported (like in wine), then this is a programming error.
ERROR_AND_DIE_FMT("Unsupported");
}
#else
// We always need to spill x8 since we can't know if it is live at this SSA location
uint32_t SpillMask = 1U << 8;
@@ -248,12 +257,8 @@ DEF_OP(ProcessorID) {
// CPU is in w0
// Node is in w1
orr(ARMEmitter::Size::i64Bit, GetReg(Node), ARMEmitter::Reg::r0, ARMEmitter::Reg::r1, ARMEmitter::ShiftType::LSL, 12);
}
#else
DEF_OP(ProcessorID) {
ERROR_AND_DIE_FMT("Unsupported");
}
#endif
}
DEF_OP(RDRAND) {
auto Op = IROp->C<IR::IROp_RDRAND>();
@@ -5,7 +5,7 @@ tags: backend|arm64
$end_info$
*/
#include "Interface/Core/JIT/Arm64/JITClass.h"
#include "Interface/Core/JIT/JITClass.h"
namespace FEXCore::CPU {
#define DEF_OP(x) void Arm64JITCore::Op_##x(IR::IROp_Header const* IROp, IR::NodeID Node)
@@ -8,6 +8,7 @@ $end_info$
#include <FEXCore/Utils/Allocator.h>
#include <FEXCore/Utils/LogManager.h>
#include <FEXCore/HLE/SyscallHandler.h>
#include "Interface/Context/Context.h"
#include "Interface/Core/LookupCache.h"
@@ -34,7 +35,8 @@ LookupCache::LookupCache(FEXCore::Context::ContextImpl* CTX)
// Allocate a region of memory that we can use to back our block pointers
// We need one pointer per page of virtual memory
// At 64GB of virtual memory this will allocate 128MB of virtual memory space
PagePointer = reinterpret_cast<uintptr_t>(FEXCore::Allocator::VirtualAlloc(TotalCacheSize));
PagePointer = reinterpret_cast<uintptr_t>(FEXCore::Allocator::VirtualAlloc(TotalCacheSize, false, false));
CTX->SyscallHandler->MarkOvercommitRange(PagePointer, TotalCacheSize);
// Allocate our memory backing our pages
// We need 32KB per guest page (One pointer per byte)
@@ -42,18 +44,18 @@ LookupCache::LookupCache(FEXCore::Context::ContextImpl* CTX)
// We currently limit to 128MB of real memory for caching for the total cache size.
// Can end up being inefficient if we compile a small number of blocks per page
PageMemory = PagePointer + ctx->Config.VirtualMemSize / 4096 * 8;
LOGMAN_THROW_AA_FMT(PageMemory != -1ULL, "Failed to allocate page memory");
LOGMAN_THROW_A_FMT(PageMemory != -1ULL, "Failed to allocate page memory");
// L1 Cache
L1Pointer = PageMemory + CODE_SIZE;
LOGMAN_THROW_AA_FMT(L1Pointer != -1ULL, "Failed to allocate L1Pointer");
LOGMAN_THROW_A_FMT(L1Pointer != -1ULL, "Failed to allocate L1Pointer");
VirtualMemSize = ctx->Config.VirtualMemSize;
}
LookupCache::~LookupCache() {
const size_t TotalCacheSize = ctx->Config.VirtualMemSize / 4096 * 8 + CODE_SIZE + L1_SIZE;
FEXCore::Allocator::VirtualFree(reinterpret_cast<void*>(PagePointer), TotalCacheSize);
ctx->SyscallHandler->UnmarkOvercommitRange(PagePointer, TotalCacheSize);
// No need to free BlockLinks map.
// These will get freed when their memory allocators are deallocated.
@@ -63,7 +65,7 @@ void LookupCache::ClearL2Cache() {
std::lock_guard<std::recursive_mutex> lk(WriteLock);
// Clear out the page memory
// PagePointer and PageMemory are sequential with each other. Clear both at once.
FEXCore::Allocator::VirtualDontNeed(reinterpret_cast<void*>(PagePointer), ctx->Config.VirtualMemSize / 4096 * 8 + CODE_SIZE);
FEXCore::Allocator::VirtualDontNeed(reinterpret_cast<void*>(PagePointer), ctx->Config.VirtualMemSize / 4096 * 8 + CODE_SIZE, false);
AllocateOffset = 0;
}
@@ -71,7 +73,7 @@ void LookupCache::ClearCache() {
std::lock_guard<std::recursive_mutex> lk(WriteLock);
// Clear L1 and L2 by clearing the full cache.
FEXCore::Allocator::VirtualDontNeed(reinterpret_cast<void*>(PagePointer), TotalCacheSize);
FEXCore::Allocator::VirtualDontNeed(reinterpret_cast<void*>(PagePointer), TotalCacheSize, false);
// Allocate a new pointer from the BlockLinks pma again.
BlockLinks = BlockLinks_pma->new_object<BlockLinksMapType>();
// All code is gone, clear the block list
+1 -1
View File
@@ -90,7 +90,7 @@ public:
std::lock_guard<std::recursive_mutex> lk(WriteLock);
[[maybe_unused]] auto Inserted = BlockList.emplace(Address, (uintptr_t)HostCode).second;
LOGMAN_THROW_AA_FMT(Inserted, "Duplicate block mapping added");
LOGMAN_THROW_A_FMT(Inserted, "Duplicate block mapping added");
// There is no need to update L1 or L2, they will get updated on first lookup
// However, adding to L1 here increases performance
@@ -1,5 +1,4 @@
// SPDX-License-Identifier: MIT
#include "Interface/Context/Context.h"
#include "Interface/Core/ObjectCache/ObjectCacheService.h"
#include <FEXCore/Config/Config.h>
@@ -1,8 +1,6 @@
// SPDX-License-Identifier: MIT
#include "Interface/Context/Context.h"
#include "Interface/Core/ObjectCache/ObjectCacheService.h"
#include <FEXCore/fextl/memory.h>
#include <FEXCore/fextl/string.h>
File diff suppressed because it is too large. Load diff
File diff suppressed because it is too large. Load diff
File diff suppressed because it is too large. Load diff
@@ -0,0 +1,107 @@
// SPDX-License-Identifier: MIT
#pragma once
#include "Interface/Core/OpcodeDispatcher.h"
namespace FEXCore::IR {
constexpr inline std::tuple<uint8_t, uint8_t, X86Tables::OpDispatchPtr> OpDispatch_BaseOpTable[] = {
// Instructions
{0x00, 6, &OpDispatchBuilder::Bind<&OpDispatchBuilder::ALUOp, FEXCore::IR::IROps::OP_ADD, FEXCore::IR::IROps::OP_ATOMICFETCHADD, 0>},
{0x08, 6, &OpDispatchBuilder::Bind<&OpDispatchBuilder::ALUOp, FEXCore::IR::IROps::OP_OR, FEXCore::IR::IROps::OP_ATOMICFETCHOR, 0>},
{0x10, 6, &OpDispatchBuilder::Bind<&OpDispatchBuilder::ADCOp, 0>},
{0x18, 6, &OpDispatchBuilder::Bind<&OpDispatchBuilder::SBBOp, 0>},
{0x20, 6, &OpDispatchBuilder::Bind<&OpDispatchBuilder::ALUOp, FEXCore::IR::IROps::OP_ANDWITHFLAGS, FEXCore::IR::IROps::OP_ATOMICFETCHAND, 0>},
{0x28, 6, &OpDispatchBuilder::Bind<&OpDispatchBuilder::ALUOp, FEXCore::IR::IROps::OP_SUB, FEXCore::IR::IROps::OP_ATOMICFETCHSUB, 0>},
{0x30, 6, &OpDispatchBuilder::Bind<&OpDispatchBuilder::ALUOp, FEXCore::IR::IROps::OP_XOR, FEXCore::IR::IROps::OP_ATOMICFETCHXOR, 0>},
{0x38, 6, &OpDispatchBuilder::Bind<&OpDispatchBuilder::CMPOp, 0>},
{0x50, 8, &OpDispatchBuilder::PUSHREGOp},
{0x58, 8, &OpDispatchBuilder::POPOp},
{0x68, 1, &OpDispatchBuilder::PUSHOp},
{0x69, 1, &OpDispatchBuilder::IMUL2SrcOp},
{0x6A, 1, &OpDispatchBuilder::PUSHOp},
{0x6B, 1, &OpDispatchBuilder::IMUL2SrcOp},
{0x6C, 4, &OpDispatchBuilder::PermissionRestrictedOp},
{0x70, 16, &OpDispatchBuilder::CondJUMPOp},
{0x84, 2, &OpDispatchBuilder::Bind<&OpDispatchBuilder::TESTOp, 0>},
{0x86, 2, &OpDispatchBuilder::XCHGOp},
{0x88, 4, &OpDispatchBuilder::Bind<&OpDispatchBuilder::MOVGPROp, 0>},
{0x8C, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::MOVSegOp, false>},
{0x8D, 1, &OpDispatchBuilder::LEAOp},
{0x8E, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::MOVSegOp, true>},
{0x8F, 1, &OpDispatchBuilder::POPOp},
{0x90, 8, &OpDispatchBuilder::XCHGOp},
{0x98, 1, &OpDispatchBuilder::CDQOp},
{0x99, 1, &OpDispatchBuilder::CQOOp},
{0x9B, 1, &OpDispatchBuilder::NOPOp},
{0x9C, 1, &OpDispatchBuilder::PUSHFOp},
{0x9D, 1, &OpDispatchBuilder::POPFOp},
{0x9E, 1, &OpDispatchBuilder::SAHFOp},
{0x9F, 1, &OpDispatchBuilder::LAHFOp},
{0xA4, 2, &OpDispatchBuilder::MOVSOp},
{0xA6, 2, &OpDispatchBuilder::CMPSOp},
{0xA8, 2, &OpDispatchBuilder::Bind<&OpDispatchBuilder::TESTOp, 0>},
{0xAA, 2, &OpDispatchBuilder::STOSOp},
{0xAC, 2, &OpDispatchBuilder::LODSOp},
{0xAE, 2, &OpDispatchBuilder::SCASOp},
{0xB0, 16, &OpDispatchBuilder::Bind<&OpDispatchBuilder::MOVGPROp, 0>},
{0xC2, 2, &OpDispatchBuilder::RETOp},
{0xC8, 1, &OpDispatchBuilder::EnterOp},
{0xC9, 1, &OpDispatchBuilder::LEAVEOp},
{0xCC, 2, &OpDispatchBuilder::INTOp},
{0xCF, 1, &OpDispatchBuilder::IRETOp},
{0xD7, 2, &OpDispatchBuilder::XLATOp},
{0xE0, 3, &OpDispatchBuilder::LoopOp},
{0xE3, 1, &OpDispatchBuilder::CondJUMPRCXOp},
{0xE4, 4, &OpDispatchBuilder::PermissionRestrictedOp},
{0xE8, 1, &OpDispatchBuilder::CALLOp},
{0xE9, 1, &OpDispatchBuilder::JUMPOp},
{0xEB, 1, &OpDispatchBuilder::JUMPOp},
{0xEC, 4, &OpDispatchBuilder::PermissionRestrictedOp},
{0xF1, 1, &OpDispatchBuilder::INTOp},
{0xF4, 1, &OpDispatchBuilder::INTOp},
{0xF5, 1, &OpDispatchBuilder::FLAGControlOp},
{0xF8, 2, &OpDispatchBuilder::FLAGControlOp},
{0xFA, 2, &OpDispatchBuilder::PermissionRestrictedOp},
{0xFC, 2, &OpDispatchBuilder::FLAGControlOp},
};
constexpr inline std::tuple<uint8_t, uint8_t, X86Tables::OpDispatchPtr> OpDispatch_BaseOpTable_64[] = {
{0x63, 1, &OpDispatchBuilder::MOVSXDOp},
{0xA0, 4, &OpDispatchBuilder::MOVOffsetOp},
};
constexpr inline std::tuple<uint8_t, uint8_t, X86Tables::OpDispatchPtr> OpDispatch_BaseOpTable_32[] = {
{0x06, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::PUSHSegmentOp, FEXCore::X86Tables::DecodeFlags::FLAG_ES_PREFIX>},
{0x07, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::POPSegmentOp, FEXCore::X86Tables::DecodeFlags::FLAG_ES_PREFIX>},
{0x0E, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::PUSHSegmentOp, FEXCore::X86Tables::DecodeFlags::FLAG_CS_PREFIX>},
{0x16, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::PUSHSegmentOp, FEXCore::X86Tables::DecodeFlags::FLAG_SS_PREFIX>},
{0x17, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::POPSegmentOp, FEXCore::X86Tables::DecodeFlags::FLAG_SS_PREFIX>},
{0x1E, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::PUSHSegmentOp, FEXCore::X86Tables::DecodeFlags::FLAG_DS_PREFIX>},
{0x1F, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::POPSegmentOp, FEXCore::X86Tables::DecodeFlags::FLAG_DS_PREFIX>},
{0x27, 1, &OpDispatchBuilder::DAAOp},
{0x2F, 1, &OpDispatchBuilder::DASOp},
{0x37, 1, &OpDispatchBuilder::AAAOp},
{0x3F, 1, &OpDispatchBuilder::AASOp},
{0x40, 8, &OpDispatchBuilder::INCOp},
{0x48, 8, &OpDispatchBuilder::DECOp},
{0x60, 1, &OpDispatchBuilder::PUSHAOp},
{0x61, 1, &OpDispatchBuilder::POPAOp},
{0xA0, 4, &OpDispatchBuilder::MOVOffsetOp},
{0xCE, 1, &OpDispatchBuilder::INTOp},
{0xD4, 1, &OpDispatchBuilder::AAMOp},
{0xD5, 1, &OpDispatchBuilder::AADOp},
{0xD6, 1, &OpDispatchBuilder::SALCOp},
};
} // namespace FEXCore::IR
@@ -43,19 +43,19 @@ void OpDispatchBuilder::SHA1NEXTEOp(OpcodeArgs) {
auto Tmp = _VAdd(OpSize::i128Bit, OpSize::i32Bit, Src, RotatedNode);
auto Result = _VInsElement(OpSize::i128Bit, OpSize::i32Bit, 3, 3, Src, Tmp);
StoreResult(FPRClass, Op, Result, -1);
StoreResult(FPRClass, Op, Result, OpSize::iInvalid);
}
void OpDispatchBuilder::SHA1MSG1Op(OpcodeArgs) {
Ref Dest = LoadSource(FPRClass, Op, Op->Dest, Op->Flags);
Ref Src = LoadSource(FPRClass, Op, Op->Src[0], Op->Flags);
Ref NewVec = _VExtr(16, 8, Dest, Src, 1);
Ref NewVec = _VExtr(OpSize::i128Bit, OpSize::i64Bit, Dest, Src, 1);
// [W0, W1, W2, W3] ^ [W2, W3, W4, W5]
Ref Result = _VXor(16, 1, Dest, NewVec);
Ref Result = _VXor(OpSize::i128Bit, OpSize::i8Bit, Dest, NewVec);
StoreResult(FPRClass, Op, Result, -1);
StoreResult(FPRClass, Op, Result, OpSize::iInvalid);
}
void OpDispatchBuilder::SHA1MSG2Op(OpcodeArgs) {
@@ -86,7 +86,7 @@ void OpDispatchBuilder::SHA1MSG2Op(OpcodeArgs) {
auto Result = _VInsElement(OpSize::i128Bit, OpSize::i32Bit, 0, 0, RotatedXor1, RotatedXorLower);
StoreResult(FPRClass, Op, Result, -1);
StoreResult(FPRClass, Op, Result, OpSize::iInvalid);
}
void OpDispatchBuilder::SHA1RNDS4Op(OpcodeArgs) {
@@ -121,25 +121,26 @@ void OpDispatchBuilder::SHA1RNDS4Op(OpcodeArgs) {
const uint64_t Imm8 = Op->Src[1].Literal() & 0b11;
const FnType Fn = fn_array[Imm8];
auto K = _Constant(32, k_array[Imm8]);
auto K = _Constant(OpSize::i32Bit, k_array[Imm8]);
Ref Dest = LoadSource(FPRClass, Op, Op->Dest, Op->Flags);
Ref Src = LoadSource(FPRClass, Op, Op->Src[0], Op->Flags);
auto W0E = _VExtractToGPR(16, 4, Src, 3);
auto W0E = _VExtractToGPR(OpSize::i128Bit, OpSize::i32Bit, Src, 3);
using RoundResult = std::tuple<Ref, Ref, Ref, Ref, Ref>;
const auto Round0 = [&]() -> RoundResult {
auto A = _VExtractToGPR(16, 4, Dest, 3);
auto B = _VExtractToGPR(16, 4, Dest, 2);
auto C = _VExtractToGPR(16, 4, Dest, 1);
auto D = _VExtractToGPR(16, 4, Dest, 0);
auto A = _VExtractToGPR(OpSize::i128Bit, OpSize::i32Bit, Dest, 3);
auto B = _VExtractToGPR(OpSize::i128Bit, OpSize::i32Bit, Dest, 2);
auto C = _VExtractToGPR(OpSize::i128Bit, OpSize::i32Bit, Dest, 1);
auto D = _VExtractToGPR(OpSize::i128Bit, OpSize::i32Bit, Dest, 0);
auto A1 =
_Add(OpSize::i32Bit, _Add(OpSize::i32Bit, _Add(OpSize::i32Bit, Fn(*this, B, C, D), _Ror(OpSize::i32Bit, A, _Constant(32, 27))), W0E), K);
_Add(OpSize::i32Bit,
_Add(OpSize::i32Bit, _Add(OpSize::i32Bit, Fn(*this, B, C, D), _Ror(OpSize::i32Bit, A, _Constant(OpSize::i32Bit, 27))), W0E), K);
auto B1 = A;
auto C1 = _Ror(OpSize::i32Bit, B, _Constant(32, 2));
auto C1 = _Ror(OpSize::i32Bit, B, _Constant(OpSize::i32Bit, 2));
auto D1 = C;
auto E1 = D;
@@ -147,13 +148,14 @@ void OpDispatchBuilder::SHA1RNDS4Op(OpcodeArgs) {
};
const auto Round1To3 = [&](Ref A, Ref B, Ref C, Ref D, Ref E, Ref Src, unsigned W_idx) -> RoundResult {
// Kill W and E at the beginning
auto W = _VExtractToGPR(16, 4, Src, W_idx);
auto W = _VExtractToGPR(OpSize::i128Bit, OpSize::i32Bit, Src, W_idx);
auto Q = _Add(OpSize::i32Bit, W, E);
auto ANext =
_Add(OpSize::i32Bit, _Add(OpSize::i32Bit, _Add(OpSize::i32Bit, Fn(*this, B, C, D), _Ror(OpSize::i32Bit, A, _Constant(32, 27))), Q), K);
_Add(OpSize::i32Bit,
_Add(OpSize::i32Bit, _Add(OpSize::i32Bit, Fn(*this, B, C, D), _Ror(OpSize::i32Bit, A, _Constant(OpSize::i32Bit, 27))), Q), K);
auto BNext = A;
auto CNext = _Ror(OpSize::i32Bit, B, _Constant(32, 2));
auto CNext = _Ror(OpSize::i32Bit, B, _Constant(OpSize::i32Bit, 2));
auto DNext = C;
auto ENext = D;
@@ -165,12 +167,12 @@ void OpDispatchBuilder::SHA1RNDS4Op(OpcodeArgs) {
auto [A3, B3, C3, D3, E3] = Round1To3(A2, B2, C2, D2, E2, Src, 1);
auto Final = Round1To3(A3, B3, C3, D3, E3, Src, 0);
auto Dest3 = _VInsGPR(16, 4, 3, Dest, std::get<0>(Final));
auto Dest2 = _VInsGPR(16, 4, 2, Dest3, std::get<1>(Final));
auto Dest1 = _VInsGPR(16, 4, 1, Dest2, std::get<2>(Final));
auto Dest0 = _VInsGPR(16, 4, 0, Dest1, std::get<3>(Final));
auto Dest3 = _VInsGPR(OpSize::i128Bit, OpSize::i32Bit, 3, Dest, std::get<0>(Final));
auto Dest2 = _VInsGPR(OpSize::i128Bit, OpSize::i32Bit, 2, Dest3, std::get<1>(Final));
auto Dest1 = _VInsGPR(OpSize::i128Bit, OpSize::i32Bit, 1, Dest2, std::get<2>(Final));
auto Dest0 = _VInsGPR(OpSize::i128Bit, OpSize::i32Bit, 0, Dest1, std::get<3>(Final));
StoreResult(FPRClass, Op, Dest0, -1);
StoreResult(FPRClass, Op, Dest0, OpSize::iInvalid);
}
void OpDispatchBuilder::SHA256MSG1Op(OpcodeArgs) {
@@ -183,52 +185,56 @@ void OpDispatchBuilder::SHA256MSG1Op(OpcodeArgs) {
Result = _VSha256U0(Dest, Src);
} else {
const auto Sigma0 = [this](Ref W) -> Ref {
return _Xor(OpSize::i32Bit, _Xor(OpSize::i32Bit, _Ror(OpSize::i32Bit, W, _Constant(32, 7)), _Ror(OpSize::i32Bit, W, _Constant(32, 18))),
_Lshr(OpSize::i32Bit, W, _Constant(32, 3)));
return _Xor(
OpSize::i32Bit,
_Xor(OpSize::i32Bit, _Ror(OpSize::i32Bit, W, _Constant(OpSize::i32Bit, 7)), _Ror(OpSize::i32Bit, W, _Constant(OpSize::i32Bit, 18))),
_Lshr(OpSize::i32Bit, W, _Constant(OpSize::i32Bit, 3)));
};
auto W4 = _VExtractToGPR(16, 4, Src, 0);
auto W3 = _VExtractToGPR(16, 4, Dest, 3);
auto W2 = _VExtractToGPR(16, 4, Dest, 2);
auto W1 = _VExtractToGPR(16, 4, Dest, 1);
auto W0 = _VExtractToGPR(16, 4, Dest, 0);
auto W4 = _VExtractToGPR(OpSize::i128Bit, OpSize::i32Bit, Src, 0);
auto W3 = _VExtractToGPR(OpSize::i128Bit, OpSize::i32Bit, Dest, 3);
auto W2 = _VExtractToGPR(OpSize::i128Bit, OpSize::i32Bit, Dest, 2);
auto W1 = _VExtractToGPR(OpSize::i128Bit, OpSize::i32Bit, Dest, 1);
auto W0 = _VExtractToGPR(OpSize::i128Bit, OpSize::i32Bit, Dest, 0);
auto Sig3 = _Add(OpSize::i32Bit, W3, Sigma0(W4));
auto Sig2 = _Add(OpSize::i32Bit, W2, Sigma0(W3));
auto Sig1 = _Add(OpSize::i32Bit, W1, Sigma0(W2));
auto Sig0 = _Add(OpSize::i32Bit, W0, Sigma0(W1));
auto D3 = _VInsGPR(16, 4, 3, Dest, Sig3);
auto D2 = _VInsGPR(16, 4, 2, D3, Sig2);
auto D1 = _VInsGPR(16, 4, 1, D2, Sig1);
Result = _VInsGPR(16, 4, 0, D1, Sig0);
auto D3 = _VInsGPR(OpSize::i128Bit, OpSize::i32Bit, 3, Dest, Sig3);
auto D2 = _VInsGPR(OpSize::i128Bit, OpSize::i32Bit, 2, D3, Sig2);
auto D1 = _VInsGPR(OpSize::i128Bit, OpSize::i32Bit, 1, D2, Sig1);
Result = _VInsGPR(OpSize::i128Bit, OpSize::i32Bit, 0, D1, Sig0);
}
StoreResult(FPRClass, Op, Result, -1);
StoreResult(FPRClass, Op, Result, OpSize::iInvalid);
}
void OpDispatchBuilder::SHA256MSG2Op(OpcodeArgs) {
const auto Sigma1 = [this](Ref W) -> Ref {
return _Xor(OpSize::i32Bit, _Xor(OpSize::i32Bit, _Ror(OpSize::i32Bit, W, _Constant(32, 17)), _Ror(OpSize::i32Bit, W, _Constant(32, 19))),
_Lshr(OpSize::i32Bit, W, _Constant(32, 10)));
return _Xor(
OpSize::i32Bit,
_Xor(OpSize::i32Bit, _Ror(OpSize::i32Bit, W, _Constant(OpSize::i32Bit, 17)), _Ror(OpSize::i32Bit, W, _Constant(OpSize::i32Bit, 19))),
_Lshr(OpSize::i32Bit, W, _Constant(OpSize::i32Bit, 10)));
};
Ref Dest = LoadSource(FPRClass, Op, Op->Dest, Op->Flags);
Ref Src = LoadSource(FPRClass, Op, Op->Src[0], Op->Flags);
auto W14 = _VExtractToGPR(16, 4, Src, 2);
auto W15 = _VExtractToGPR(16, 4, Src, 3);
auto W16 = _Add(OpSize::i32Bit, _VExtractToGPR(16, 4, Dest, 0), Sigma1(W14));
auto W17 = _Add(OpSize::i32Bit, _VExtractToGPR(16, 4, Dest, 1), Sigma1(W15));
auto W18 = _Add(OpSize::i32Bit, _VExtractToGPR(16, 4, Dest, 2), Sigma1(W16));
auto W19 = _Add(OpSize::i32Bit, _VExtractToGPR(16, 4, Dest, 3), Sigma1(W17));
auto W14 = _VExtractToGPR(OpSize::i128Bit, OpSize::i32Bit, Src, 2);
auto W15 = _VExtractToGPR(OpSize::i128Bit, OpSize::i32Bit, Src, 3);
auto W16 = _Add(OpSize::i32Bit, _VExtractToGPR(OpSize::i128Bit, OpSize::i32Bit, Dest, 0), Sigma1(W14));
auto W17 = _Add(OpSize::i32Bit, _VExtractToGPR(OpSize::i128Bit, OpSize::i32Bit, Dest, 1), Sigma1(W15));
auto W18 = _Add(OpSize::i32Bit, _VExtractToGPR(OpSize::i128Bit, OpSize::i32Bit, Dest, 2), Sigma1(W16));
auto W19 = _Add(OpSize::i32Bit, _VExtractToGPR(OpSize::i128Bit, OpSize::i32Bit, Dest, 3), Sigma1(W17));
auto D3 = _VInsGPR(16, 4, 3, Dest, W19);
auto D2 = _VInsGPR(16, 4, 2, D3, W18);
auto D1 = _VInsGPR(16, 4, 1, D2, W17);
auto D0 = _VInsGPR(16, 4, 0, D1, W16);
auto D3 = _VInsGPR(OpSize::i128Bit, OpSize::i32Bit, 3, Dest, W19);
auto D2 = _VInsGPR(OpSize::i128Bit, OpSize::i32Bit, 2, D3, W18);
auto D1 = _VInsGPR(OpSize::i128Bit, OpSize::i32Bit, 1, D2, W17);
auto D0 = _VInsGPR(OpSize::i128Bit, OpSize::i32Bit, 0, D1, W16);
StoreResult(FPRClass, Op, D0, -1);
StoreResult(FPRClass, Op, D0, OpSize::iInvalid);
}
Ref OpDispatchBuilder::BitwiseAtLeastTwo(Ref A, Ref B, Ref C) {
@@ -246,12 +252,12 @@ void OpDispatchBuilder::SHA256RNDS2Op(OpcodeArgs) {
return _Xor(OpSize::i32Bit, _And(OpSize::i32Bit, E, F), _Andn(OpSize::i32Bit, G, E));
};
const auto Sigma0 = [this](Ref A) -> Ref {
return _XorShift(OpSize::i32Bit, _XorShift(OpSize::i32Bit, _Ror(OpSize::i32Bit, A, _Constant(32, 2)), A, ShiftType::ROR, 13), A,
ShiftType::ROR, 22);
return _XorShift(OpSize::i32Bit, _XorShift(OpSize::i32Bit, _Ror(OpSize::i32Bit, A, _Constant(OpSize::i32Bit, 2)), A, ShiftType::ROR, 13),
A, ShiftType::ROR, 22);
};
const auto Sigma1 = [this](Ref E) -> Ref {
return _XorShift(OpSize::i32Bit, _XorShift(OpSize::i32Bit, _Ror(OpSize::i32Bit, E, _Constant(32, 6)), E, ShiftType::ROR, 11), E,
ShiftType::ROR, 25);
return _XorShift(OpSize::i32Bit, _XorShift(OpSize::i32Bit, _Ror(OpSize::i32Bit, E, _Constant(OpSize::i32Bit, 6)), E, ShiftType::ROR, 11),
E, ShiftType::ROR, 25);
};
Ref Dest = LoadSource(FPRClass, Op, Op->Dest, Op->Flags);
@@ -259,64 +265,64 @@ void OpDispatchBuilder::SHA256RNDS2Op(OpcodeArgs) {
// Hardcoded to XMM0
auto XMM0 = LoadXMMRegister(0);
auto E0 = _VExtractToGPR(16, 4, Src, 1);
auto F0 = _VExtractToGPR(16, 4, Src, 0);
auto G0 = _VExtractToGPR(16, 4, Dest, 1);
auto E0 = _VExtractToGPR(OpSize::i128Bit, OpSize::i32Bit, Src, 1);
auto F0 = _VExtractToGPR(OpSize::i128Bit, OpSize::i32Bit, Src, 0);
auto G0 = _VExtractToGPR(OpSize::i128Bit, OpSize::i32Bit, Dest, 1);
Ref Q0 = _Add(OpSize::i32Bit, Ch(E0, F0, G0), Sigma1(E0));
auto WK0 = _VExtractToGPR(16, 4, XMM0, 0);
auto WK0 = _VExtractToGPR(OpSize::i128Bit, OpSize::i32Bit, XMM0, 0);
Q0 = _Add(OpSize::i32Bit, Q0, WK0);
auto H0 = _VExtractToGPR(16, 4, Dest, 0);
auto H0 = _VExtractToGPR(OpSize::i128Bit, OpSize::i32Bit, Dest, 0);
Q0 = _Add(OpSize::i32Bit, Q0, H0);
auto A0 = _VExtractToGPR(16, 4, Src, 3);
auto B0 = _VExtractToGPR(16, 4, Src, 2);
auto C0 = _VExtractToGPR(16, 4, Dest, 3);
auto A0 = _VExtractToGPR(OpSize::i128Bit, OpSize::i32Bit, Src, 3);
auto B0 = _VExtractToGPR(OpSize::i128Bit, OpSize::i32Bit, Src, 2);
auto C0 = _VExtractToGPR(OpSize::i128Bit, OpSize::i32Bit, Dest, 3);
auto A1 = _Add(OpSize::i32Bit, _Add(OpSize::i32Bit, Q0, BitwiseAtLeastTwo(A0, B0, C0)), Sigma0(A0));
auto D0 = _VExtractToGPR(16, 4, Dest, 2);
auto D0 = _VExtractToGPR(OpSize::i128Bit, OpSize::i32Bit, Dest, 2);
auto E1 = _Add(OpSize::i32Bit, Q0, D0);
Ref Q1 = _Add(OpSize::i32Bit, Ch(E1, E0, F0), Sigma1(E1));
auto WK1 = _VExtractToGPR(16, 4, XMM0, 1);
auto WK1 = _VExtractToGPR(OpSize::i128Bit, OpSize::i32Bit, XMM0, 1);
Q1 = _Add(OpSize::i32Bit, Q1, WK1);
// Rematerialize G0. Costs a move but saves spilling, coming out ahead.
G0 = _VExtractToGPR(16, 4, Dest, 1);
G0 = _VExtractToGPR(OpSize::i128Bit, OpSize::i32Bit, Dest, 1);
Q1 = _Add(OpSize::i32Bit, Q1, G0);
auto A2 = _Add(OpSize::i32Bit, _Add(OpSize::i32Bit, Q1, BitwiseAtLeastTwo(A1, A0, B0)), Sigma0(A1));
// Rematerialize C0. As with G0.
C0 = _VExtractToGPR(16, 4, Dest, 3);
C0 = _VExtractToGPR(OpSize::i128Bit, OpSize::i32Bit, Dest, 3);
auto E2 = _Add(OpSize::i32Bit, Q1, C0);
auto Res3 = _VInsGPR(16, 4, 3, Dest, A2);
auto Res2 = _VInsGPR(16, 4, 2, Res3, A1);
auto Res1 = _VInsGPR(16, 4, 1, Res2, E2);
auto Res0 = _VInsGPR(16, 4, 0, Res1, E1);
auto Res3 = _VInsGPR(OpSize::i128Bit, OpSize::i32Bit, 3, Dest, A2);
auto Res2 = _VInsGPR(OpSize::i128Bit, OpSize::i32Bit, 2, Res3, A1);
auto Res1 = _VInsGPR(OpSize::i128Bit, OpSize::i32Bit, 1, Res2, E2);
auto Res0 = _VInsGPR(OpSize::i128Bit, OpSize::i32Bit, 0, Res1, E1);
StoreResult(FPRClass, Op, Res0, -1);
StoreResult(FPRClass, Op, Res0, OpSize::iInvalid);
}
void OpDispatchBuilder::AESImcOp(OpcodeArgs) {
Ref Src = LoadSource(FPRClass, Op, Op->Src[0], Op->Flags);
Ref Result = _VAESImc(Src);
StoreResult(FPRClass, Op, Result, -1);
StoreResult(FPRClass, Op, Result, OpSize::iInvalid);
}
void OpDispatchBuilder::AESEncOp(OpcodeArgs) {
Ref Dest = LoadSource(FPRClass, Op, Op->Dest, Op->Flags);
Ref Src = LoadSource(FPRClass, Op, Op->Src[0], Op->Flags);
Ref Result = _VAESEnc(16, Dest, Src, LoadZeroVector(16));
StoreResult(FPRClass, Op, Result, -1);
Ref Result = _VAESEnc(OpSize::i128Bit, Dest, Src, LoadZeroVector(OpSize::i128Bit));
StoreResult(FPRClass, Op, Result, OpSize::iInvalid);
}
void OpDispatchBuilder::VAESEncOp(OpcodeArgs) {
const auto DstSize = GetDstSize(Op);
[[maybe_unused]] const auto Is128Bit = DstSize == Core::CPUState::XMM_SSE_REG_SIZE;
const auto DstSize = OpSizeFromDst(Op);
[[maybe_unused]] const auto Is128Bit = DstSize == OpSize::i128Bit;
// TODO: Handle 256-bit VAESENC.
LOGMAN_THROW_A_FMT(Is128Bit, "256-bit VAESENC unimplemented");
@@ -325,19 +331,19 @@ void OpDispatchBuilder::VAESEncOp(OpcodeArgs) {
Ref Key = LoadSource(FPRClass, Op, Op->Src[1], Op->Flags);
Ref Result = _VAESEnc(DstSize, State, Key, LoadZeroVector(DstSize));
StoreResult(FPRClass, Op, Result, -1);
StoreResult(FPRClass, Op, Result, OpSize::iInvalid);
}
void OpDispatchBuilder::AESEncLastOp(OpcodeArgs) {
Ref Dest = LoadSource(FPRClass, Op, Op->Dest, Op->Flags);
Ref Src = LoadSource(FPRClass, Op, Op->Src[0], Op->Flags);
Ref Result = _VAESEncLast(16, Dest, Src, LoadZeroVector(16));
StoreResult(FPRClass, Op, Result, -1);
Ref Result = _VAESEncLast(OpSize::i128Bit, Dest, Src, LoadZeroVector(OpSize::i128Bit));
StoreResult(FPRClass, Op, Result, OpSize::iInvalid);
}
void OpDispatchBuilder::VAESEncLastOp(OpcodeArgs) {
const auto DstSize = GetDstSize(Op);
[[maybe_unused]] const auto Is128Bit = DstSize == Core::CPUState::XMM_SSE_REG_SIZE;
const auto DstSize = OpSizeFromDst(Op);
[[maybe_unused]] const auto Is128Bit = DstSize == OpSize::i128Bit;
// TODO: Handle 256-bit VAESENCLAST.
LOGMAN_THROW_A_FMT(Is128Bit, "256-bit VAESENCLAST unimplemented");
@@ -346,19 +352,19 @@ void OpDispatchBuilder::VAESEncLastOp(OpcodeArgs) {
Ref Key = LoadSource(FPRClass, Op, Op->Src[1], Op->Flags);
Ref Result = _VAESEncLast(DstSize, State, Key, LoadZeroVector(DstSize));
StoreResult(FPRClass, Op, Result, -1);
StoreResult(FPRClass, Op, Result, OpSize::iInvalid);
}
void OpDispatchBuilder::AESDecOp(OpcodeArgs) {
Ref Dest = LoadSource(FPRClass, Op, Op->Dest, Op->Flags);
Ref Src = LoadSource(FPRClass, Op, Op->Src[0], Op->Flags);
Ref Result = _VAESDec(16, Dest, Src, LoadZeroVector(16));
StoreResult(FPRClass, Op, Result, -1);
Ref Result = _VAESDec(OpSize::i128Bit, Dest, Src, LoadZeroVector(OpSize::i128Bit));
StoreResult(FPRClass, Op, Result, OpSize::iInvalid);
}
void OpDispatchBuilder::VAESDecOp(OpcodeArgs) {
const auto DstSize = GetDstSize(Op);
[[maybe_unused]] const auto Is128Bit = DstSize == Core::CPUState::XMM_SSE_REG_SIZE;
const auto DstSize = OpSizeFromDst(Op);
[[maybe_unused]] const auto Is128Bit = DstSize == OpSize::i128Bit;
// TODO: Handle 256-bit VAESDEC.
LOGMAN_THROW_A_FMT(Is128Bit, "256-bit VAESDEC unimplemented");
@@ -367,19 +373,19 @@ void OpDispatchBuilder::VAESDecOp(OpcodeArgs) {
Ref Key = LoadSource(FPRClass, Op, Op->Src[1], Op->Flags);
Ref Result = _VAESDec(DstSize, State, Key, LoadZeroVector(DstSize));
StoreResult(FPRClass, Op, Result, -1);
StoreResult(FPRClass, Op, Result, OpSize::iInvalid);
}
void OpDispatchBuilder::AESDecLastOp(OpcodeArgs) {
Ref Dest = LoadSource(FPRClass, Op, Op->Dest, Op->Flags);
Ref Src = LoadSource(FPRClass, Op, Op->Src[0], Op->Flags);
Ref Result = _VAESDecLast(16, Dest, Src, LoadZeroVector(16));
StoreResult(FPRClass, Op, Result, -1);
Ref Result = _VAESDecLast(OpSize::i128Bit, Dest, Src, LoadZeroVector(OpSize::i128Bit));
StoreResult(FPRClass, Op, Result, OpSize::iInvalid);
}
void OpDispatchBuilder::VAESDecLastOp(OpcodeArgs) {
const auto DstSize = GetDstSize(Op);
[[maybe_unused]] const auto Is128Bit = DstSize == Core::CPUState::XMM_SSE_REG_SIZE;
const auto DstSize = OpSizeFromDst(Op);
[[maybe_unused]] const auto Is128Bit = DstSize == OpSize::i128Bit;
// TODO: Handle 256-bit VAESDECLAST.
LOGMAN_THROW_A_FMT(Is128Bit, "256-bit VAESDECLAST unimplemented");
@@ -388,20 +394,20 @@ void OpDispatchBuilder::VAESDecLastOp(OpcodeArgs) {
Ref Key = LoadSource(FPRClass, Op, Op->Src[1], Op->Flags);
Ref Result = _VAESDecLast(DstSize, State, Key, LoadZeroVector(DstSize));
StoreResult(FPRClass, Op, Result, -1);
StoreResult(FPRClass, Op, Result, OpSize::iInvalid);
}
Ref OpDispatchBuilder::AESKeyGenAssistImpl(OpcodeArgs) {
Ref Src = LoadSource(FPRClass, Op, Op->Src[0], Op->Flags);
const uint64_t RCON = Op->Src[1].Literal();
auto KeyGenSwizzle = LoadAndCacheNamedVectorConstant(16, NAMED_VECTOR_AESKEYGENASSIST_SWIZZLE);
return _VAESKeyGenAssist(Src, KeyGenSwizzle, LoadZeroVector(16), RCON);
auto KeyGenSwizzle = LoadAndCacheNamedVectorConstant(OpSize::i128Bit, NAMED_VECTOR_AESKEYGENASSIST_SWIZZLE);
return _VAESKeyGenAssist(Src, KeyGenSwizzle, LoadZeroVector(OpSize::i128Bit), RCON);
}
void OpDispatchBuilder::AESKeyGenAssist(OpcodeArgs) {
Ref Result = AESKeyGenAssistImpl(Op);
StoreResult(FPRClass, Op, Result, -1);
StoreResult(FPRClass, Op, Result, OpSize::iInvalid);
}
void OpDispatchBuilder::PCLMULQDQOp(OpcodeArgs) {
@@ -409,19 +415,19 @@ void OpDispatchBuilder::PCLMULQDQOp(OpcodeArgs) {
Ref Src = LoadSource(FPRClass, Op, Op->Src[0], Op->Flags);
const auto Selector = static_cast<uint8_t>(Op->Src[1].Literal());
auto Res = _PCLMUL(16, Dest, Src, Selector & 0b1'0001);
StoreResult(FPRClass, Op, Res, -1);
auto Res = _PCLMUL(OpSize::i128Bit, Dest, Src, Selector & 0b1'0001);
StoreResult(FPRClass, Op, Res, OpSize::iInvalid);
}
void OpDispatchBuilder::VPCLMULQDQOp(OpcodeArgs) {
const auto DstSize = GetDstSize(Op);
const auto DstSize = OpSizeFromDst(Op);
Ref Src1 = LoadSource(FPRClass, Op, Op->Src[0], Op->Flags);
Ref Src2 = LoadSource(FPRClass, Op, Op->Src[1], Op->Flags);
const auto Selector = static_cast<uint8_t>(Op->Src[2].Literal());
Ref Res = _PCLMUL(DstSize, Src1, Src2, Selector & 0b1'0001);
StoreResult(FPRClass, Op, Res, -1);
StoreResult(FPRClass, Op, Res, OpSize::iInvalid);
}
} // namespace FEXCore::IR
@@ -0,0 +1,45 @@
// SPDX-License-Identifier: MIT
#pragma once
#include "Interface/Core/OpcodeDispatcher.h"
namespace FEXCore::IR {
constexpr std::tuple<uint8_t, uint8_t, FEXCore::X86Tables::OpDispatchPtr> OpDispatch_DDDTable[] = {
{0x0C, 1, &OpDispatchBuilder::PI2FWOp},
{0x0D, 1, &OpDispatchBuilder::Vector_CVT_Int_To_Float<OpSize::i32Bit, false>},
{0x1C, 1, &OpDispatchBuilder::PF2IWOp},
{0x1D, 1, &OpDispatchBuilder::Vector_CVT_Float_To_Int<OpSize::i32Bit, false>},
{0x86, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorUnaryOp, IR::OP_VFRECP, OpSize::i32Bit>},
{0x87, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorUnaryOp, IR::OP_VFRSQRT, OpSize::i32Bit>},
{0x8A, 1, &OpDispatchBuilder::PFNACCOp},
{0x8E, 1, &OpDispatchBuilder::PFPNACCOp},
{0x90, 1, &OpDispatchBuilder::VPFCMPOp<1>},
{0x94, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorALUOp, IR::OP_VFMIN, OpSize::i32Bit>},
{0x96, 1, &OpDispatchBuilder::VectorUnaryDuplicateOp<IR::OP_VFRECP, OpSize::i32Bit>},
{0x97, 1, &OpDispatchBuilder::VectorUnaryDuplicateOp<IR::OP_VFRSQRT, OpSize::i32Bit>},
{0x9A, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorALUOp, IR::OP_VFSUB, OpSize::i32Bit>},
{0x9E, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorALUOp, IR::OP_VFADD, OpSize::i32Bit>},
{0xA0, 1, &OpDispatchBuilder::VPFCMPOp<2>},
{0xA4, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorALUOp, IR::OP_VFMAX, OpSize::i32Bit>},
// Can be treated as a move
{0xA6, 1, &OpDispatchBuilder::MOVVectorUnalignedOp},
{0xA7, 1, &OpDispatchBuilder::MOVVectorUnalignedOp},
{0xAA, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorALUROp, IR::OP_VFSUB, OpSize::i32Bit>},
{0xAE, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorALUOp, IR::OP_VFADDP, OpSize::i32Bit>},
{0xB0, 1, &OpDispatchBuilder::VPFCMPOp<0>},
{0xB4, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorALUOp, IR::OP_VFMUL, OpSize::i32Bit>},
// Can be treated as a move
{0xB6, 1, &OpDispatchBuilder::MOVVectorUnalignedOp},
{0xB7, 1, &OpDispatchBuilder::PMULHRWOp},
{0xBB, 1, &OpDispatchBuilder::PSWAPDOp},
{0xBF, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorALUOp, IR::OP_VURAVG, OpSize::i8Bit>},
};
} // namespace FEXCore::IR
@@ -6,7 +6,6 @@ desc: Handles x86/64 flag generation
$end_info$
*/
#include "Interface/Context/Context.h"
#include "Interface/Core/OpcodeDispatcher.h"
#include "Interface/Core/X86Tables/X86Tables.h"
@@ -20,7 +19,7 @@ $end_info$
namespace FEXCore::IR {
constexpr std::array<uint32_t, 17> FlagOffsets = {
FEXCore::X86State::RFLAG_CF_RAW_LOC, FEXCore::X86State::RFLAG_PF_RAW_LOC, FEXCore::X86State::RFLAG_AF_RAW_LOC,
FEXCore::X86State::RFLAG_ZF_RAW_LOC, FEXCore::X86State::RFLAG_SF_RAW_LOC, FEXCore::X86State::RFLAG_TF_LOC,
FEXCore::X86State::RFLAG_ZF_RAW_LOC, FEXCore::X86State::RFLAG_SF_RAW_LOC, FEXCore::X86State::RFLAG_TF_RAW_LOC,
FEXCore::X86State::RFLAG_IF_LOC, FEXCore::X86State::RFLAG_DF_RAW_LOC, FEXCore::X86State::RFLAG_OF_RAW_LOC,
FEXCore::X86State::RFLAG_IOPL_LOC, FEXCore::X86State::RFLAG_NT_LOC, FEXCore::X86State::RFLAG_RF_LOC,
FEXCore::X86State::RFLAG_VM_LOC, FEXCore::X86State::RFLAG_AC_LOC, FEXCore::X86State::RFLAG_VIF_LOC,
@@ -37,13 +36,9 @@ void OpDispatchBuilder::SetPackedRFLAG(bool Lower8, Ref Src) {
size_t NumFlags = FlagOffsets.size();
if (Lower8) {
// Calculate flags early.
// Could use InvalidateDeferredFlags() if we had masked invalidation.
// This is only a partial overwrite of flags since OF isn't stored here.
CalculateDeferredFlags();
NumFlags = 5;
} else {
// We are overwriting all RFLAGS. Invalidate the deferred flag state.
InvalidateDeferredFlags();
}
// PF and CF are both stored inverted, so hoist the invert.
@@ -77,16 +72,13 @@ Ref OpDispatchBuilder::GetPackedRFLAG(uint32_t FlagsMask) {
// Calculate flags early.
CalculateDeferredFlags();
Ref Original = _Constant(0);
// SF/ZF and N/Z are together on both arm64 and x86_64, so we special case that.
bool GetNZ = (FlagsMask & (1 << FEXCore::X86State::RFLAG_SF_RAW_LOC)) && (FlagsMask & (1 << FEXCore::X86State::RFLAG_ZF_RAW_LOC));
// Handle CF first, since it's at bit 0 and hence doesn't need shift or OR.
if (FlagsMask & (1 << FEXCore::X86State::RFLAG_CF_RAW_LOC)) {
static_assert(FEXCore::X86State::RFLAG_CF_RAW_LOC == 0);
Original = GetRFLAG(FEXCore::X86State::RFLAG_CF_RAW_LOC);
}
LOGMAN_THROW_A_FMT(FlagsMask & (1 << FEXCore::X86State::RFLAG_CF_RAW_LOC), "CF always handled");
static_assert(FEXCore::X86State::RFLAG_CF_RAW_LOC == 0);
Ref Original = GetRFLAG(FEXCore::X86State::RFLAG_CF_RAW_LOC);
for (size_t i = 0; i < FlagOffsets.size(); ++i) {
const auto FlagOffset = FlagOffsets[i];
@@ -117,7 +109,7 @@ Ref OpDispatchBuilder::GetPackedRFLAG(uint32_t FlagsMask) {
// instead.
if (FlagsMask & (1 << FEXCore::X86State::RFLAG_PF_RAW_LOC)) {
// Set every bit except the bottommost.
auto OnesInvPF = _Or(OpSize::i64Bit, LoadPFRaw(false), _Constant(~1ull));
auto OnesInvPF = _Or(OpSize::i64Bit, LoadPFRaw(false, false), _InlineConstant(~1ull));
// Rotate the bottom bit to the appropriate location for PF, so we get
// something like 111P1111. Then invert that to get 000p0000. Then OR that
@@ -130,21 +122,21 @@ Ref OpDispatchBuilder::GetPackedRFLAG(uint32_t FlagsMask) {
if (GetNZ) {
static_assert(FEXCore::X86State::RFLAG_SF_RAW_LOC == (FEXCore::X86State::RFLAG_ZF_RAW_LOC + 1));
auto NZCV = GetNZCV();
auto NZ = _And(OpSize::i64Bit, NZCV, _Constant(0b11u << 30));
auto NZ = _And(OpSize::i64Bit, NZCV, _InlineConstant(0b11u << 30));
Original = _Orlshr(OpSize::i64Bit, Original, NZ, 31 - FEXCore::X86State::RFLAG_SF_RAW_LOC);
}
// The constant is OR'ed in at the end, to avoid a pointless or xzr, #2.
if ((1U << X86State::RFLAG_RESERVED_LOC) & FlagsMask) {
Original = _Or(OpSize::i64Bit, Original, _Constant(2));
Original = _Or(OpSize::i64Bit, Original, _InlineConstant(2));
}
return Original;
}
void OpDispatchBuilder::CalculateOF(uint8_t SrcSize, Ref Res, Ref Src1, Ref Src2, bool Sub) {
auto OpSize = SrcSize == 8 ? OpSize::i64Bit : OpSize::i32Bit;
uint64_t SignBit = (SrcSize * 8) - 1;
void OpDispatchBuilder::CalculateOF(IR::OpSize SrcSize, Ref Res, Ref Src1, Ref Src2, bool Sub) {
const auto OpSize = SrcSize == OpSize::i64Bit ? OpSize::i64Bit : OpSize::i32Bit;
const uint64_t SignBit = IR::OpSizeAsBits(SrcSize) - 1;
Ref Anded = nullptr;
// For add, OF is set iff the sources have the same sign but the destination
@@ -175,25 +167,15 @@ void OpDispatchBuilder::CalculateOF(uint8_t SrcSize, Ref Res, Ref Src1, Ref Src2
}
}
SetRFLAG<FEXCore::X86State::RFLAG_OF_RAW_LOC>(Anded, SrcSize * 8 - 1, true);
SetRFLAG<FEXCore::X86State::RFLAG_OF_RAW_LOC>(Anded, SignBit, true);
}
Ref OpDispatchBuilder::LoadPFRaw(bool Invert) {
// Read the stored byte. This is the original result (up to 64-bits), it needs
// parity calculated.
auto Result = GetRFLAG(FEXCore::X86State::RFLAG_PF_RAW_LOC);
Ref OpDispatchBuilder::LoadPFRaw(bool Mask, bool Invert) {
// Most blocks do not read parity, so PF optimization is gated on this flag.
CurrentHeader->ReadsParity = true;
// Cascade to calculate parity of bottom 8-bits to bottom bit.
Result = _XorShift(OpSize::i32Bit, Result, Result, ShiftType::LSR, 4);
Result = _XorShift(OpSize::i32Bit, Result, Result, ShiftType::LSR, 2);
if (Invert) {
Result = _XornShift(OpSize::i32Bit, Result, Result, ShiftType::LSR, 1);
} else {
Result = _XorShift(OpSize::i32Bit, Result, Result, ShiftType::LSR, 1);
}
return Result;
// Evaluate parity on the deferred raw value.
return _Parity(GetRFLAG(FEXCore::X86State::RFLAG_PF_RAW_LOC), Mask, Invert);
}
Ref OpDispatchBuilder::LoadAF() {
@@ -203,8 +185,9 @@ Ref OpDispatchBuilder::LoadAF() {
// Read the result, stored for PF.
auto Result = GetRFLAG(FEXCore::X86State::RFLAG_PF_RAW_LOC);
// What's left is to XOR and extract. This is the deferred part.
return _Bfe(OpSize::i32Bit, 1, 4, _Xor(OpSize::i32Bit, AFWord, Result));
// What's left is to XOR and extract. This is the deferred part. We
// specifically use a 64-bit Xor here as we don't need masking.
return _Bfe(OpSize::i32Bit, 1, 4, _Xor(OpSize::i64Bit, AFWord, Result));
}
void OpDispatchBuilder::FixupAF() {
@@ -217,7 +200,8 @@ void OpDispatchBuilder::FixupAF() {
auto PFRaw = GetRFLAG(FEXCore::X86State::RFLAG_PF_RAW_LOC);
auto AFRaw = GetRFLAG(FEXCore::X86State::RFLAG_AF_RAW_LOC);
Ref XorRes = _Xor(OpSize::i32Bit, AFRaw, PFRaw);
// Again 64-bit as masking is more expensive given our ConstProp design.
Ref XorRes = _Xor(OpSize::i64Bit, AFRaw, PFRaw);
SetRFLAG<FEXCore::X86State::RFLAG_AF_RAW_LOC>(XorRes);
}
@@ -256,8 +240,8 @@ void OpDispatchBuilder::CalculateAF(Ref Src1, Ref Src2) {
// We store the XOR of the arguments. At read time, we XOR with the
// appropriate bit of the result (available as the PF flag) and extract the
// appropriate bit.
Ref XorRes = _Xor(OpSize::i32Bit, Src1, Src2);
// appropriate bit. Again 64-bit to avoid masking.
Ref XorRes = _Xor(OpSize::i64Bit, Src1, Src2);
SetRFLAG<FEXCore::X86State::RFLAG_AF_RAW_LOC>(XorRes);
}
@@ -276,22 +260,22 @@ Ref OpDispatchBuilder::IncrementByCarry(OpSize OpSize, Ref Src) {
return _NZCVSelectIncrement(OpSize, {CFInverted ? COND_UGE : COND_ULT}, Src, Src);
}
Ref OpDispatchBuilder::CalculateFlags_ADC(uint8_t SrcSize, Ref Src1, Ref Src2) {
auto Zero = _Constant(0);
auto One = _Constant(1);
auto OpSize = SrcSize == 8 ? OpSize::i64Bit : OpSize::i32Bit;
Ref OpDispatchBuilder::CalculateFlags_ADC(IR::OpSize SrcSize, Ref Src1, Ref Src2) {
auto Zero = _InlineConstant(0);
auto One = _InlineConstant(1);
auto OpSize = SrcSize == OpSize::i64Bit ? OpSize::i64Bit : OpSize::i32Bit;
Ref Res;
CalculateAF(Src1, Src2);
if (SrcSize >= 4) {
if (SrcSize >= OpSize::i32Bit) {
RectifyCarryInvert(false);
HandleNZCV_RMW();
Res = _AdcWithFlags(OpSize, Src1, Src2);
CFInverted = false;
} else {
// Need to zero-extend for correct comparisons below
Src2 = _Bfe(OpSize, SrcSize * 8, 0, Src2);
Src2 = _Bfe(OpSize, IR::OpSizeAsBits(SrcSize), 0, Src2);
// Note that we do not extend Src2PlusCF, since we depend on proper
// 32-bit arithmetic to correctly handle the Src2 = 0xffff case.
@@ -299,7 +283,7 @@ Ref OpDispatchBuilder::CalculateFlags_ADC(uint8_t SrcSize, Ref Src1, Ref Src2) {
// Need to zero-extend for the comparison.
Res = _Add(OpSize, Src1, Src2PlusCF);
Res = _Bfe(OpSize, SrcSize * 8, 0, Res);
Res = _Bfe(OpSize, IR::OpSizeAsBits(SrcSize), 0, Res);
// TODO: We can fold that second Bfe in (cmp uxth).
auto SelectCFInv = _Select(FEXCore::IR::COND_UGE, Res, Src2PlusCF, One, Zero);
@@ -313,15 +297,15 @@ Ref OpDispatchBuilder::CalculateFlags_ADC(uint8_t SrcSize, Ref Src1, Ref Src2) {
return Res;
}
Ref OpDispatchBuilder::CalculateFlags_SBB(uint8_t SrcSize, Ref Src1, Ref Src2) {
auto Zero = _Constant(0);
auto One = _Constant(1);
auto OpSize = SrcSize == 8 ? OpSize::i64Bit : OpSize::i32Bit;
Ref OpDispatchBuilder::CalculateFlags_SBB(IR::OpSize SrcSize, Ref Src1, Ref Src2) {
auto Zero = _InlineConstant(0);
auto One = _InlineConstant(1);
auto OpSize = SrcSize == OpSize::i64Bit ? OpSize::i64Bit : OpSize::i32Bit;
CalculateAF(Src1, Src2);
Ref Res;
if (SrcSize >= 4) {
if (SrcSize >= OpSize::i32Bit) {
// Arm's subtraction has inverted CF from x86, so rectify the input and
// invert the output.
RectifyCarryInvert(true);
@@ -330,13 +314,13 @@ Ref OpDispatchBuilder::CalculateFlags_SBB(uint8_t SrcSize, Ref Src1, Ref Src2) {
CFInverted = true;
} else {
// Zero extend for correct comparison behaviour with Src1 = 0xffff.
Src1 = _Bfe(OpSize, SrcSize * 8, 0, Src1);
Src2 = _Bfe(OpSize, SrcSize * 8, 0, Src2);
Src1 = _Bfe(OpSize, IR::OpSizeAsBits(SrcSize), 0, Src1);
Src2 = _Bfe(OpSize, IR::OpSizeAsBits(SrcSize), 0, Src2);
auto Src2PlusCF = IncrementByCarry(OpSize, Src2);
Res = _Sub(OpSize, Src1, Src2PlusCF);
Res = _Bfe(OpSize, SrcSize * 8, 0, Res);
Res = _Bfe(OpSize, IR::OpSizeAsBits(SrcSize), 0, Res);
auto SelectCFInv = _Select(FEXCore::IR::COND_UGE, Src1, Src2PlusCF, One, Zero);
@@ -349,7 +333,7 @@ Ref OpDispatchBuilder::CalculateFlags_SBB(uint8_t SrcSize, Ref Src1, Ref Src2) {
return Res;
}
Ref OpDispatchBuilder::CalculateFlags_SUB(uint8_t SrcSize, Ref Src1, Ref Src2, bool UpdateCF) {
Ref OpDispatchBuilder::CalculateFlags_SUB(IR::OpSize SrcSize, Ref Src1, Ref Src2, bool UpdateCF) {
// Stash CF before stomping over it
auto OldCFInv = UpdateCF ? nullptr : GetRFLAG(FEXCore::X86State::RFLAG_CF_RAW_LOC, true);
@@ -358,10 +342,10 @@ Ref OpDispatchBuilder::CalculateFlags_SUB(uint8_t SrcSize, Ref Src1, Ref Src2, b
CalculateAF(Src1, Src2);
Ref Res;
if (SrcSize >= 4) {
Res = _SubWithFlags(IR::SizeToOpSize(SrcSize), Src1, Src2);
if (SrcSize >= OpSize::i32Bit) {
Res = _SubWithFlags(SrcSize, Src1, Src2);
} else {
_SubNZCV(IR::SizeToOpSize(SrcSize), Src1, Src2);
_SubNZCV(SrcSize, Src1, Src2);
Res = _Sub(OpSize::i32Bit, Src1, Src2);
}
@@ -379,7 +363,7 @@ Ref OpDispatchBuilder::CalculateFlags_SUB(uint8_t SrcSize, Ref Src1, Ref Src2, b
return Res;
}
Ref OpDispatchBuilder::CalculateFlags_ADD(uint8_t SrcSize, Ref Src1, Ref Src2, bool UpdateCF) {
Ref OpDispatchBuilder::CalculateFlags_ADD(IR::OpSize SrcSize, Ref Src1, Ref Src2, bool UpdateCF) {
// Stash CF before stomping over it
auto OldCFInv = UpdateCF ? nullptr : GetRFLAG(FEXCore::X86State::RFLAG_CF_RAW_LOC, true);
@@ -388,10 +372,10 @@ Ref OpDispatchBuilder::CalculateFlags_ADD(uint8_t SrcSize, Ref Src1, Ref Src2, b
CalculateAF(Src1, Src2);
Ref Res;
if (SrcSize >= 4) {
Res = _AddWithFlags(IR::SizeToOpSize(SrcSize), Src1, Src2);
if (SrcSize >= OpSize::i32Bit) {
Res = _AddWithFlags(SrcSize, Src1, Src2);
} else {
_AddNZCV(IR::SizeToOpSize(SrcSize), Src1, Src2);
_AddNZCV(SrcSize, Src1, Src2);
Res = _Add(OpSize::i32Bit, Src1, Src2);
}
@@ -408,18 +392,18 @@ Ref OpDispatchBuilder::CalculateFlags_ADD(uint8_t SrcSize, Ref Src1, Ref Src2, b
return Res;
}
void OpDispatchBuilder::CalculateFlags_MUL(uint8_t SrcSize, Ref Res, Ref High) {
void OpDispatchBuilder::CalculateFlags_MUL(IR::OpSize SrcSize, Ref Res, Ref High) {
HandleNZCVWrite();
InvalidatePF_AF();
// CF and OF are set if the result of the operation can't be fit in to the destination register
// If the value can fit then the top bits will be zero
auto SignBit = _Sbfe(OpSize::i64Bit, 1, SrcSize * 8 - 1, Res);
auto SignBit = _Sbfe(OpSize::i64Bit, 1, IR::OpSizeAsBits(SrcSize) - 1, Res);
_SubNZCV(OpSize::i64Bit, High, SignBit);
// If High = SignBit, then sets to nZCv. Else sets to nzcV. Since SF/ZF
// undefined, this does what we need after inverting carry.
auto Zero = _Constant(0);
auto Zero = _InlineConstant(0);
_CondSubNZCV(OpSize::i64Bit, Zero, Zero, CondClassType {COND_EQ}, 0x1 /* nzcV */);
CFInverted = true;
}
@@ -428,8 +412,8 @@ void OpDispatchBuilder::CalculateFlags_UMUL(Ref High) {
HandleNZCVWrite();
InvalidatePF_AF();
auto Zero = _Constant(0);
OpSize Size = IR::SizeToOpSize(GetOpSize(High));
auto Zero = _InlineConstant(0);
const auto Size = GetOpSize(High);
// CF and OF are set if the result of the operation can't be fit in to the destination register
// The result register will be all zero if it can't fit due to how multiplication behaves
@@ -441,7 +425,7 @@ void OpDispatchBuilder::CalculateFlags_UMUL(Ref High) {
CFInverted = true;
}
void OpDispatchBuilder::CalculateFlags_Logical(uint8_t SrcSize, Ref Res, Ref Src1, Ref Src2) {
void OpDispatchBuilder::CalculateFlags_Logical(IR::OpSize SrcSize, Ref Res, Ref Src1, Ref Src2) {
InvalidateAF();
CalculatePF(Res);
@@ -450,13 +434,13 @@ void OpDispatchBuilder::CalculateFlags_Logical(uint8_t SrcSize, Ref Res, Ref Src
SetNZ_ZeroCV(SrcSize, Res);
}
void OpDispatchBuilder::CalculateFlags_ShiftLeftImmediate(uint8_t SrcSize, Ref UnmaskedRes, Ref Src1, uint64_t Shift) {
void OpDispatchBuilder::CalculateFlags_ShiftLeftImmediate(IR::OpSize SrcSize, Ref UnmaskedRes, Ref Src1, uint64_t Shift) {
// No flags changed if shift is zero
if (Shift == 0) {
return;
}
auto OpSize = SrcSize == 8 ? OpSize::i64Bit : OpSize::i32Bit;
auto OpSize = SrcSize == OpSize::i64Bit ? OpSize::i64Bit : OpSize::i32Bit;
SetNZ_ZeroCV(SrcSize, UnmaskedRes);
@@ -465,7 +449,7 @@ void OpDispatchBuilder::CalculateFlags_ShiftLeftImmediate(uint8_t SrcSize, Ref U
// Extract the last bit shifted in to CF. Shift is already masked, but for
// 8/16-bit it might be >= SrcSizeBits, in which case CF is cleared. There's
// nothing to do in that case since we already cleared CF above.
auto SrcSizeBits = SrcSize * 8;
const auto SrcSizeBits = IR::OpSizeAsBits(SrcSize);
if (Shift < SrcSizeBits) {
SetCFDirect(Src1, SrcSizeBits - Shift, true);
}
@@ -478,13 +462,13 @@ void OpDispatchBuilder::CalculateFlags_ShiftLeftImmediate(uint8_t SrcSize, Ref U
// In the case of left shift. OF is only set from the result of <Top Source Bit> XOR <Top Result Bit>
if (Shift == 1) {
auto Xor = _Xor(OpSize, UnmaskedRes, Src1);
SetRFLAG<FEXCore::X86State::RFLAG_OF_RAW_LOC>(Xor, SrcSize * 8 - 1, true);
SetRFLAG<FEXCore::X86State::RFLAG_OF_RAW_LOC>(Xor, IR::OpSizeAsBits(SrcSize) - 1, true);
} else {
// Undefined, we choose to zero as part of SetNZ_ZeroCV
}
}
void OpDispatchBuilder::CalculateFlags_SignShiftRightImmediate(uint8_t SrcSize, Ref Res, Ref Src1, uint64_t Shift) {
void OpDispatchBuilder::CalculateFlags_SignShiftRightImmediate(IR::OpSize SrcSize, Ref Res, Ref Src1, uint64_t Shift) {
// No flags changed if shift is zero
if (Shift == 0) {
return;
@@ -504,7 +488,7 @@ void OpDispatchBuilder::CalculateFlags_SignShiftRightImmediate(uint8_t SrcSize,
// already zeroed there's nothing to do here.
}
void OpDispatchBuilder::CalculateFlags_ShiftRightImmediateCommon(uint8_t SrcSize, Ref Res, Ref Src1, uint64_t Shift) {
void OpDispatchBuilder::CalculateFlags_ShiftRightImmediateCommon(IR::OpSize SrcSize, Ref Res, Ref Src1, uint64_t Shift) {
// Set SF and PF. Clobbers OF, but OF only defined for Shift = 1 where it is
// set below.
SetNZ_ZeroCV(SrcSize, Res);
@@ -516,7 +500,7 @@ void OpDispatchBuilder::CalculateFlags_ShiftRightImmediateCommon(uint8_t SrcSize
InvalidateAF();
}
void OpDispatchBuilder::CalculateFlags_ShiftRightImmediate(uint8_t SrcSize, Ref Res, Ref Src1, uint64_t Shift) {
void OpDispatchBuilder::CalculateFlags_ShiftRightImmediate(IR::OpSize SrcSize, Ref Res, Ref Src1, uint64_t Shift) {
// No flags changed if shift is zero
if (Shift == 0) {
return;
@@ -529,18 +513,18 @@ void OpDispatchBuilder::CalculateFlags_ShiftRightImmediate(uint8_t SrcSize, Ref
// Only defined when Shift is 1 else undefined
// Is set to the MSB of the original value
if (Shift == 1) {
SetRFLAG<FEXCore::X86State::RFLAG_OF_RAW_LOC>(Src1, SrcSize * 8 - 1, true);
SetRFLAG<FEXCore::X86State::RFLAG_OF_RAW_LOC>(Src1, IR::OpSizeAsBits(SrcSize) - 1, true);
}
}
}
void OpDispatchBuilder::CalculateFlags_ShiftRightDoubleImmediate(uint8_t SrcSize, Ref Res, Ref Src1, uint64_t Shift) {
void OpDispatchBuilder::CalculateFlags_ShiftRightDoubleImmediate(IR::OpSize SrcSize, Ref Res, Ref Src1, uint64_t Shift) {
// No flags changed if shift is zero
if (Shift == 0) {
return;
}
const auto OpSize = SrcSize == 8 ? OpSize::i64Bit : OpSize::i32Bit;
const auto OpSize = SrcSize == OpSize::i64Bit ? OpSize::i64Bit : OpSize::i32Bit;
CalculateFlags_ShiftRightImmediateCommon(SrcSize, Res, Src1, Shift);
// OF
@@ -550,12 +534,12 @@ void OpDispatchBuilder::CalculateFlags_ShiftRightDoubleImmediate(uint8_t SrcSize
// XOR of Result and Src1
if (Shift == 1) {
auto val = _Xor(OpSize, Src1, Res);
SetRFLAG<FEXCore::X86State::RFLAG_OF_RAW_LOC>(val, SrcSize * 8 - 1, true);
SetRFLAG<FEXCore::X86State::RFLAG_OF_RAW_LOC>(val, IR::OpSizeAsBits(SrcSize) - 1, true);
}
}
}
void OpDispatchBuilder::CalculateFlags_ZCNT(uint8_t SrcSize, Ref Result) {
void OpDispatchBuilder::CalculateFlags_ZCNT(IR::OpSize SrcSize, Ref Result) {
// OF, SF, AF, PF all undefined
// Test ZF of result, SF is undefined so this is ok.
SetNZ_ZeroCV(SrcSize, Result);
@@ -563,7 +547,7 @@ void OpDispatchBuilder::CalculateFlags_ZCNT(uint8_t SrcSize, Ref Result) {
// Now set CF if the Result = SrcSize * 8. Since SrcSize is a power-of-two and
// Result is <= SrcSize * 8, we equivalently check if the log2(SrcSize * 8)
// bit is set. No masking is needed because no higher bits could be set.
unsigned CarryBit = FEXCore::ilog2(SrcSize * 8u);
unsigned CarryBit = FEXCore::ilog2(IR::OpSizeAsBits(SrcSize));
SetCFDirect(Result, CarryBit);
}
@@ -0,0 +1,82 @@
// SPDX-License-Identifier: MIT
#pragma once
#include "Interface/Core/OpcodeDispatcher.h"
namespace FEXCore::IR {
#define OPD(prefix, opcode) (((prefix) << 8) | opcode)
constexpr uint16_t PF_38_NONE = 0;
constexpr uint16_t PF_38_66 = (1U << 0);
constexpr uint16_t PF_38_F3 = (1U << 2);
constexpr std::tuple<uint16_t, uint8_t, FEXCore::X86Tables::OpDispatchPtr> OpDispatch_H0F38Table[] = {
{OPD(PF_38_NONE, 0x00), 1, &OpDispatchBuilder::PSHUFBOp},
{OPD(PF_38_66, 0x00), 1, &OpDispatchBuilder::PSHUFBOp},
{OPD(PF_38_NONE, 0x01), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorALUOp, IR::OP_VADDP, OpSize::i16Bit>},
{OPD(PF_38_66, 0x01), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorALUOp, IR::OP_VADDP, OpSize::i16Bit>},
{OPD(PF_38_NONE, 0x02), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorALUOp, IR::OP_VADDP, OpSize::i32Bit>},
{OPD(PF_38_66, 0x02), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorALUOp, IR::OP_VADDP, OpSize::i32Bit>},
{OPD(PF_38_NONE, 0x03), 1, &OpDispatchBuilder::PHADDS},
{OPD(PF_38_66, 0x03), 1, &OpDispatchBuilder::PHADDS},
{OPD(PF_38_NONE, 0x04), 1, &OpDispatchBuilder::PMADDUBSW},
{OPD(PF_38_66, 0x04), 1, &OpDispatchBuilder::PMADDUBSW},
{OPD(PF_38_NONE, 0x05), 1, &OpDispatchBuilder::PHSUB<OpSize::i16Bit>},
{OPD(PF_38_66, 0x05), 1, &OpDispatchBuilder::PHSUB<OpSize::i16Bit>},
{OPD(PF_38_NONE, 0x06), 1, &OpDispatchBuilder::PHSUB<OpSize::i32Bit>},
{OPD(PF_38_66, 0x06), 1, &OpDispatchBuilder::PHSUB<OpSize::i32Bit>},
{OPD(PF_38_NONE, 0x07), 1, &OpDispatchBuilder::PHSUBS},
{OPD(PF_38_66, 0x07), 1, &OpDispatchBuilder::PHSUBS},
{OPD(PF_38_NONE, 0x08), 1, &OpDispatchBuilder::PSIGN<OpSize::i8Bit>},
{OPD(PF_38_66, 0x08), 1, &OpDispatchBuilder::PSIGN<OpSize::i8Bit>},
{OPD(PF_38_NONE, 0x09), 1, &OpDispatchBuilder::PSIGN<OpSize::i16Bit>},
{OPD(PF_38_66, 0x09), 1, &OpDispatchBuilder::PSIGN<OpSize::i16Bit>},
{OPD(PF_38_NONE, 0x0A), 1, &OpDispatchBuilder::PSIGN<OpSize::i32Bit>},
{OPD(PF_38_66, 0x0A), 1, &OpDispatchBuilder::PSIGN<OpSize::i32Bit>},
{OPD(PF_38_NONE, 0x0B), 1, &OpDispatchBuilder::PMULHRSW},
{OPD(PF_38_66, 0x0B), 1, &OpDispatchBuilder::PMULHRSW},
{OPD(PF_38_66, 0x10), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorVariableBlend, OpSize::i8Bit>},
{OPD(PF_38_66, 0x14), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorVariableBlend, OpSize::i32Bit>},
{OPD(PF_38_66, 0x15), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorVariableBlend, OpSize::i64Bit>},
{OPD(PF_38_66, 0x17), 1, &OpDispatchBuilder::PTestOp},
{OPD(PF_38_NONE, 0x1C), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorUnaryOp, IR::OP_VABS, OpSize::i8Bit>},
{OPD(PF_38_66, 0x1C), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorUnaryOp, IR::OP_VABS, OpSize::i8Bit>},
{OPD(PF_38_NONE, 0x1D), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorUnaryOp, IR::OP_VABS, OpSize::i16Bit>},
{OPD(PF_38_66, 0x1D), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorUnaryOp, IR::OP_VABS, OpSize::i16Bit>},
{OPD(PF_38_NONE, 0x1E), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorUnaryOp, IR::OP_VABS, OpSize::i32Bit>},
{OPD(PF_38_66, 0x1E), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorUnaryOp, IR::OP_VABS, OpSize::i32Bit>},
{OPD(PF_38_66, 0x20), 1, &OpDispatchBuilder::ExtendVectorElements<OpSize::i8Bit, OpSize::i16Bit, true>},
{OPD(PF_38_66, 0x21), 1, &OpDispatchBuilder::ExtendVectorElements<OpSize::i8Bit, OpSize::i32Bit, true>},
{OPD(PF_38_66, 0x22), 1, &OpDispatchBuilder::ExtendVectorElements<OpSize::i8Bit, OpSize::i64Bit, true>},
{OPD(PF_38_66, 0x23), 1, &OpDispatchBuilder::ExtendVectorElements<OpSize::i16Bit, OpSize::i32Bit, true>},
{OPD(PF_38_66, 0x24), 1, &OpDispatchBuilder::ExtendVectorElements<OpSize::i16Bit, OpSize::i64Bit, true>},
{OPD(PF_38_66, 0x25), 1, &OpDispatchBuilder::ExtendVectorElements<OpSize::i32Bit, OpSize::i64Bit, true>},
{OPD(PF_38_66, 0x28), 1, &OpDispatchBuilder::PMULLOp<OpSize::i32Bit, true>},
{OPD(PF_38_66, 0x29), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorALUOp, IR::OP_VCMPEQ, OpSize::i64Bit>},
{OPD(PF_38_66, 0x2A), 1, &OpDispatchBuilder::MOVVectorNTOp},
{OPD(PF_38_66, 0x2B), 1, &OpDispatchBuilder::PACKUSOp<OpSize::i32Bit>},
{OPD(PF_38_66, 0x30), 1, &OpDispatchBuilder::ExtendVectorElements<OpSize::i8Bit, OpSize::i16Bit, false>},
{OPD(PF_38_66, 0x31), 1, &OpDispatchBuilder::ExtendVectorElements<OpSize::i8Bit, OpSize::i32Bit, false>},
{OPD(PF_38_66, 0x32), 1, &OpDispatchBuilder::ExtendVectorElements<OpSize::i8Bit, OpSize::i64Bit, false>},
{OPD(PF_38_66, 0x33), 1, &OpDispatchBuilder::ExtendVectorElements<OpSize::i16Bit, OpSize::i32Bit, false>},
{OPD(PF_38_66, 0x34), 1, &OpDispatchBuilder::ExtendVectorElements<OpSize::i16Bit, OpSize::i64Bit, false>},
{OPD(PF_38_66, 0x35), 1, &OpDispatchBuilder::ExtendVectorElements<OpSize::i32Bit, OpSize::i64Bit, false>},
{OPD(PF_38_66, 0x37), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorALUOp, IR::OP_VCMPGT, OpSize::i64Bit>},
{OPD(PF_38_66, 0x38), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorALUOp, IR::OP_VSMIN, OpSize::i8Bit>},
{OPD(PF_38_66, 0x39), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorALUOp, IR::OP_VSMIN, OpSize::i32Bit>},
{OPD(PF_38_66, 0x3A), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorALUOp, IR::OP_VUMIN, OpSize::i16Bit>},
{OPD(PF_38_66, 0x3B), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorALUOp, IR::OP_VUMIN, OpSize::i32Bit>},
{OPD(PF_38_66, 0x3C), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorALUOp, IR::OP_VSMAX, OpSize::i8Bit>},
{OPD(PF_38_66, 0x3D), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorALUOp, IR::OP_VSMAX, OpSize::i32Bit>},
{OPD(PF_38_66, 0x3E), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorALUOp, IR::OP_VUMAX, OpSize::i16Bit>},
{OPD(PF_38_66, 0x3F), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorALUOp, IR::OP_VUMAX, OpSize::i32Bit>},
{OPD(PF_38_66, 0x40), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorALUOp, IR::OP_VMUL, OpSize::i32Bit>},
{OPD(PF_38_66, 0x41), 1, &OpDispatchBuilder::PHMINPOSUWOp},
{OPD(PF_38_NONE, 0xF0), 2, &OpDispatchBuilder::MOVBEOp},
{OPD(PF_38_66, 0xF0), 2, &OpDispatchBuilder::MOVBEOp},
{OPD(PF_38_66, 0xF6), 1, &OpDispatchBuilder::ADXOp},
{OPD(PF_38_F3, 0xF6), 1, &OpDispatchBuilder::ADXOp},
};
#undef OPD
} // namespace FEXCore::IR
@@ -0,0 +1,77 @@
// SPDX-License-Identifier: MIT
#pragma once
#include "Interface/Core/OpcodeDispatcher.h"
namespace FEXCore::IR {
#define OPD(REX, prefix, opcode) ((REX << 9) | (prefix << 8) | opcode)
#define PF_3A_NONE 0
#define PF_3A_66 1
constexpr auto OpDispatchTableGenH0F3A = []() consteval {
constexpr auto OpDispatchTableGenH0F3AREX = []<uint16_t REX>() consteval {
constexpr std::tuple<uint16_t, uint8_t, FEXCore::X86Tables::OpDispatchPtr> Table[] = {
{OPD(REX, PF_3A_66, 0x08), 1, &OpDispatchBuilder::VectorRound<OpSize::i32Bit>},
{OPD(REX, PF_3A_66, 0x09), 1, &OpDispatchBuilder::VectorRound<OpSize::i64Bit>},
{OPD(REX, PF_3A_66, 0x0A), 1, &OpDispatchBuilder::InsertScalarRound<OpSize::i32Bit>},
{OPD(REX, PF_3A_66, 0x0B), 1, &OpDispatchBuilder::InsertScalarRound<OpSize::i64Bit>},
{OPD(REX, PF_3A_66, 0x0C), 1, &OpDispatchBuilder::VectorBlend<OpSize::i32Bit>},
{OPD(REX, PF_3A_66, 0x0D), 1, &OpDispatchBuilder::VectorBlend<OpSize::i64Bit>},
{OPD(REX, PF_3A_66, 0x0E), 1, &OpDispatchBuilder::VectorBlend<OpSize::i16Bit>},
{OPD(REX, PF_3A_NONE, 0x0F), 1, &OpDispatchBuilder::PAlignrOp},
{OPD(REX, PF_3A_66, 0x0F), 1, &OpDispatchBuilder::PAlignrOp},
{OPD(REX, PF_3A_66, 0x14), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::PExtrOp, OpSize::i8Bit>},
{OPD(REX, PF_3A_66, 0x15), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::PExtrOp, OpSize::i16Bit>},
{OPD(REX, PF_3A_66, 0x17), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::PExtrOp, OpSize::i32Bit>},
{OPD(REX, PF_3A_66, 0x20), 1, &OpDispatchBuilder::PINSROp<OpSize::i8Bit>},
{OPD(REX, PF_3A_66, 0x21), 1, &OpDispatchBuilder::InsertPSOp},
{OPD(REX, PF_3A_66, 0x40), 1, &OpDispatchBuilder::DPPOp<OpSize::i32Bit>},
{OPD(REX, PF_3A_66, 0x41), 1, &OpDispatchBuilder::DPPOp<OpSize::i64Bit>},
{OPD(REX, PF_3A_66, 0x42), 1, &OpDispatchBuilder::MPSADBWOp},
{OPD(REX, PF_3A_66, 0x60), 1, &OpDispatchBuilder::VPCMPESTRMOp},
{OPD(REX, PF_3A_66, 0x61), 1, &OpDispatchBuilder::VPCMPESTRIOp},
{OPD(REX, PF_3A_66, 0x62), 1, &OpDispatchBuilder::VPCMPISTRMOp},
{OPD(REX, PF_3A_66, 0x63), 1, &OpDispatchBuilder::VPCMPISTRIOp},
{OPD(REX, PF_3A_NONE, 0xCC), 1, &OpDispatchBuilder::SHA1RNDS4Op},
};
return std::to_array(Table);
};
auto REX0 = OpDispatchTableGenH0F3AREX.template operator()<0>();
auto REX1 = OpDispatchTableGenH0F3AREX.template operator()<1>();
auto concat = []<typename T, size_t N1, size_t N2>(std::array<T, N1> const& lhs,
std::array<T, N2> const& rhs) consteval -> std::array<T, N1 + N2> {
std::array<T, N1 + N2> Table {};
for (size_t i = 0; i < N1; ++i) {
Table[i] = lhs[i];
}
for (size_t i = 0; i < N2; ++i) {
Table[N1 + i] = rhs[i];
}
return Table;
};
return concat(REX0, REX1);
};
constexpr auto OpDispatch_H0F3ATableIgnoreREX = OpDispatchTableGenH0F3A();
constexpr std::tuple<uint16_t, uint8_t, FEXCore::X86Tables::OpDispatchPtr> OpDispatch_H0F3ATableNeedsREX0[] = {
{OPD(0, PF_3A_66, 0x16), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::PExtrOp, OpSize::i32Bit>},
{OPD(0, PF_3A_66, 0x22), 1, &OpDispatchBuilder::PINSROp<OpSize::i32Bit>},
};
constexpr std::tuple<uint16_t, uint8_t, FEXCore::X86Tables::OpDispatchPtr> OpDispatch_H0F3ATable_64[] = {
{OPD(1, PF_3A_66, 0x16), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::PExtrOp, OpSize::i64Bit>},
{OPD(1, PF_3A_66, 0x22), 1, &OpDispatchBuilder::PINSROp<OpSize::i64Bit>},
};
#undef PF_3A_NONE
#undef PF_3A_66
#undef OPD
} // namespace FEXCore::IR
@@ -0,0 +1,129 @@
// SPDX-License-Identifier: MIT
#pragma once
#include "Interface/Core/OpcodeDispatcher.h"
namespace FEXCore::IR {
using X86Tables::OpToIndex;
#define OPD(group, prefix, Reg) (((group - FEXCore::X86Tables::TYPE_GROUP_1) << 6) | (prefix) << 3 | (Reg))
constexpr std::tuple<uint16_t, uint8_t, FEXCore::X86Tables::OpDispatchPtr> OpDispatch_PrimaryGroupTables[] = {
// GROUP 1
{OPD(FEXCore::X86Tables::TYPE_GROUP_1, OpToIndex(0x80), 0), 1, &OpDispatchBuilder::SecondaryALUOp},
{OPD(FEXCore::X86Tables::TYPE_GROUP_1, OpToIndex(0x80), 1), 1, &OpDispatchBuilder::SecondaryALUOp},
{OPD(FEXCore::X86Tables::TYPE_GROUP_1, OpToIndex(0x80), 2), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::ADCOp, 1>},
{OPD(FEXCore::X86Tables::TYPE_GROUP_1, OpToIndex(0x80), 3), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::SBBOp, 1>},
{OPD(FEXCore::X86Tables::TYPE_GROUP_1, OpToIndex(0x80), 4), 1, &OpDispatchBuilder::SecondaryALUOp},
{OPD(FEXCore::X86Tables::TYPE_GROUP_1, OpToIndex(0x80), 5), 1, &OpDispatchBuilder::SecondaryALUOp},
{OPD(FEXCore::X86Tables::TYPE_GROUP_1, OpToIndex(0x80), 6), 1, &OpDispatchBuilder::SecondaryALUOp},
{OPD(FEXCore::X86Tables::TYPE_GROUP_1, OpToIndex(0x80), 7), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::CMPOp, 1>}, // CMP
{OPD(FEXCore::X86Tables::TYPE_GROUP_1, OpToIndex(0x81), 0), 1, &OpDispatchBuilder::SecondaryALUOp},
{OPD(FEXCore::X86Tables::TYPE_GROUP_1, OpToIndex(0x81), 1), 1, &OpDispatchBuilder::SecondaryALUOp},
{OPD(FEXCore::X86Tables::TYPE_GROUP_1, OpToIndex(0x81), 2), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::ADCOp, 1>},
{OPD(FEXCore::X86Tables::TYPE_GROUP_1, OpToIndex(0x81), 3), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::SBBOp, 1>},
{OPD(FEXCore::X86Tables::TYPE_GROUP_1, OpToIndex(0x81), 4), 1, &OpDispatchBuilder::SecondaryALUOp},
{OPD(FEXCore::X86Tables::TYPE_GROUP_1, OpToIndex(0x81), 5), 1, &OpDispatchBuilder::SecondaryALUOp},
{OPD(FEXCore::X86Tables::TYPE_GROUP_1, OpToIndex(0x81), 6), 1, &OpDispatchBuilder::SecondaryALUOp},
{OPD(FEXCore::X86Tables::TYPE_GROUP_1, OpToIndex(0x81), 7), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::CMPOp, 1>},
{OPD(FEXCore::X86Tables::TYPE_GROUP_1, OpToIndex(0x83), 0), 1, &OpDispatchBuilder::SecondaryALUOp},
{OPD(FEXCore::X86Tables::TYPE_GROUP_1, OpToIndex(0x83), 1), 1, &OpDispatchBuilder::SecondaryALUOp},
{OPD(FEXCore::X86Tables::TYPE_GROUP_1, OpToIndex(0x83), 2), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::ADCOp, 1>},
{OPD(FEXCore::X86Tables::TYPE_GROUP_1, OpToIndex(0x83), 3), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::SBBOp, 1>},
{OPD(FEXCore::X86Tables::TYPE_GROUP_1, OpToIndex(0x83), 4), 1, &OpDispatchBuilder::SecondaryALUOp},
{OPD(FEXCore::X86Tables::TYPE_GROUP_1, OpToIndex(0x83), 5), 1, &OpDispatchBuilder::SecondaryALUOp},
{OPD(FEXCore::X86Tables::TYPE_GROUP_1, OpToIndex(0x83), 6), 1, &OpDispatchBuilder::SecondaryALUOp},
{OPD(FEXCore::X86Tables::TYPE_GROUP_1, OpToIndex(0x83), 7), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::CMPOp, 1>},
// GROUP 2
{OPD(FEXCore::X86Tables::TYPE_GROUP_2, OpToIndex(0xC0), 0), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::RotateOp, true, true, false>},
{OPD(FEXCore::X86Tables::TYPE_GROUP_2, OpToIndex(0xC0), 1), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::RotateOp, false, true, false>},
{OPD(FEXCore::X86Tables::TYPE_GROUP_2, OpToIndex(0xC0), 2), 1, &OpDispatchBuilder::RCLOp},
{OPD(FEXCore::X86Tables::TYPE_GROUP_2, OpToIndex(0xC0), 3), 1, &OpDispatchBuilder::RCROp},
{OPD(FEXCore::X86Tables::TYPE_GROUP_2, OpToIndex(0xC0), 4), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::SHLImmediateOp, false>},
{OPD(FEXCore::X86Tables::TYPE_GROUP_2, OpToIndex(0xC0), 5), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::SHRImmediateOp, false>},
{OPD(FEXCore::X86Tables::TYPE_GROUP_2, OpToIndex(0xC0), 6), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::SHLImmediateOp, false>}, // SAL
{OPD(FEXCore::X86Tables::TYPE_GROUP_2, OpToIndex(0xC0), 7), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::ASHROp, true, false>}, // SAR
{OPD(FEXCore::X86Tables::TYPE_GROUP_2, OpToIndex(0xC1), 0), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::RotateOp, true, true, false>},
{OPD(FEXCore::X86Tables::TYPE_GROUP_2, OpToIndex(0xC1), 1), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::RotateOp, false, true, false>},
{OPD(FEXCore::X86Tables::TYPE_GROUP_2, OpToIndex(0xC1), 2), 1, &OpDispatchBuilder::RCLOp},
{OPD(FEXCore::X86Tables::TYPE_GROUP_2, OpToIndex(0xC1), 3), 1, &OpDispatchBuilder::RCROp},
{OPD(FEXCore::X86Tables::TYPE_GROUP_2, OpToIndex(0xC1), 4), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::SHLImmediateOp, false>},
{OPD(FEXCore::X86Tables::TYPE_GROUP_2, OpToIndex(0xC1), 5), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::SHRImmediateOp, false>},
{OPD(FEXCore::X86Tables::TYPE_GROUP_2, OpToIndex(0xC1), 6), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::SHLImmediateOp, false>}, // SAL
{OPD(FEXCore::X86Tables::TYPE_GROUP_2, OpToIndex(0xC1), 7), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::ASHROp, true, false>}, // SAR
{OPD(FEXCore::X86Tables::TYPE_GROUP_2, OpToIndex(0xD0), 0), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::RotateOp, true, true, true>},
{OPD(FEXCore::X86Tables::TYPE_GROUP_2, OpToIndex(0xD0), 1), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::RotateOp, false, true, true>},
{OPD(FEXCore::X86Tables::TYPE_GROUP_2, OpToIndex(0xD0), 2), 1, &OpDispatchBuilder::RCLOp1Bit},
{OPD(FEXCore::X86Tables::TYPE_GROUP_2, OpToIndex(0xD0), 3), 1, &OpDispatchBuilder::RCROp8x1Bit},
{OPD(FEXCore::X86Tables::TYPE_GROUP_2, OpToIndex(0xD0), 4), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::SHLImmediateOp, true>},
{OPD(FEXCore::X86Tables::TYPE_GROUP_2, OpToIndex(0xD0), 5), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::SHRImmediateOp, true>}, // 1Bit SHR
{OPD(FEXCore::X86Tables::TYPE_GROUP_2, OpToIndex(0xD0), 6), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::SHLImmediateOp, true>}, // SAL
{OPD(FEXCore::X86Tables::TYPE_GROUP_2, OpToIndex(0xD0), 7), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::ASHROp, true, true>}, // SAR
{OPD(FEXCore::X86Tables::TYPE_GROUP_2, OpToIndex(0xD1), 0), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::RotateOp, true, true, true>},
{OPD(FEXCore::X86Tables::TYPE_GROUP_2, OpToIndex(0xD1), 1), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::RotateOp, false, true, true>},
{OPD(FEXCore::X86Tables::TYPE_GROUP_2, OpToIndex(0xD1), 2), 1, &OpDispatchBuilder::RCLOp1Bit},
{OPD(FEXCore::X86Tables::TYPE_GROUP_2, OpToIndex(0xD1), 3), 1, &OpDispatchBuilder::RCROp1Bit},
{OPD(FEXCore::X86Tables::TYPE_GROUP_2, OpToIndex(0xD1), 4), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::SHLImmediateOp, true>},
{OPD(FEXCore::X86Tables::TYPE_GROUP_2, OpToIndex(0xD1), 5), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::SHRImmediateOp, true>}, // 1Bit SHR
{OPD(FEXCore::X86Tables::TYPE_GROUP_2, OpToIndex(0xD1), 6), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::SHLImmediateOp, true>}, // SAL
{OPD(FEXCore::X86Tables::TYPE_GROUP_2, OpToIndex(0xD1), 7), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::ASHROp, true, true>}, // SAR
{OPD(FEXCore::X86Tables::TYPE_GROUP_2, OpToIndex(0xD2), 0), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::RotateOp, true, false, false>},
{OPD(FEXCore::X86Tables::TYPE_GROUP_2, OpToIndex(0xD2), 1), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::RotateOp, false, false, false>},
{OPD(FEXCore::X86Tables::TYPE_GROUP_2, OpToIndex(0xD2), 2), 1, &OpDispatchBuilder::RCLSmallerOp},
{OPD(FEXCore::X86Tables::TYPE_GROUP_2, OpToIndex(0xD2), 3), 1, &OpDispatchBuilder::RCRSmallerOp},
{OPD(FEXCore::X86Tables::TYPE_GROUP_2, OpToIndex(0xD2), 4), 1, &OpDispatchBuilder::SHLOp},
{OPD(FEXCore::X86Tables::TYPE_GROUP_2, OpToIndex(0xD2), 5), 1, &OpDispatchBuilder::SHROp}, // SHR by CL
{OPD(FEXCore::X86Tables::TYPE_GROUP_2, OpToIndex(0xD2), 6), 1, &OpDispatchBuilder::SHLOp}, // SAL
{OPD(FEXCore::X86Tables::TYPE_GROUP_2, OpToIndex(0xD2), 7), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::ASHROp, false, false>}, // SAR
{OPD(FEXCore::X86Tables::TYPE_GROUP_2, OpToIndex(0xD3), 0), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::RotateOp, true, false, false>},
{OPD(FEXCore::X86Tables::TYPE_GROUP_2, OpToIndex(0xD3), 1), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::RotateOp, false, false, false>},
{OPD(FEXCore::X86Tables::TYPE_GROUP_2, OpToIndex(0xD3), 2), 1, &OpDispatchBuilder::RCLOp},
{OPD(FEXCore::X86Tables::TYPE_GROUP_2, OpToIndex(0xD3), 3), 1, &OpDispatchBuilder::RCROp},
{OPD(FEXCore::X86Tables::TYPE_GROUP_2, OpToIndex(0xD3), 4), 1, &OpDispatchBuilder::SHLOp},
{OPD(FEXCore::X86Tables::TYPE_GROUP_2, OpToIndex(0xD3), 5), 1, &OpDispatchBuilder::SHROp}, // SHR by CL
{OPD(FEXCore::X86Tables::TYPE_GROUP_2, OpToIndex(0xD3), 6), 1, &OpDispatchBuilder::SHLOp}, // SAL
{OPD(FEXCore::X86Tables::TYPE_GROUP_2, OpToIndex(0xD3), 7), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::ASHROp, false, false>}, // SAR
// GROUP 3
{OPD(FEXCore::X86Tables::TYPE_GROUP_3, OpToIndex(0xF6), 0), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::TESTOp, 1>},
{OPD(FEXCore::X86Tables::TYPE_GROUP_3, OpToIndex(0xF6), 1), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::TESTOp, 1>},
{OPD(FEXCore::X86Tables::TYPE_GROUP_3, OpToIndex(0xF6), 2), 1, &OpDispatchBuilder::NOTOp},
{OPD(FEXCore::X86Tables::TYPE_GROUP_3, OpToIndex(0xF6), 3), 1, &OpDispatchBuilder::NEGOp}, // NEG
{OPD(FEXCore::X86Tables::TYPE_GROUP_3, OpToIndex(0xF6), 4), 1, &OpDispatchBuilder::MULOp},
{OPD(FEXCore::X86Tables::TYPE_GROUP_3, OpToIndex(0xF6), 5), 1, &OpDispatchBuilder::IMULOp},
{OPD(FEXCore::X86Tables::TYPE_GROUP_3, OpToIndex(0xF6), 6), 1, &OpDispatchBuilder::DIVOp}, // DIV
{OPD(FEXCore::X86Tables::TYPE_GROUP_3, OpToIndex(0xF6), 7), 1, &OpDispatchBuilder::IDIVOp}, // IDIV
{OPD(FEXCore::X86Tables::TYPE_GROUP_3, OpToIndex(0xF7), 0), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::TESTOp, 1>},
{OPD(FEXCore::X86Tables::TYPE_GROUP_3, OpToIndex(0xF7), 1), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::TESTOp, 1>},
{OPD(FEXCore::X86Tables::TYPE_GROUP_3, OpToIndex(0xF7), 2), 1, &OpDispatchBuilder::NOTOp},
{OPD(FEXCore::X86Tables::TYPE_GROUP_3, OpToIndex(0xF7), 3), 1, &OpDispatchBuilder::NEGOp}, // NEG
{OPD(FEXCore::X86Tables::TYPE_GROUP_3, OpToIndex(0xF7), 4), 1, &OpDispatchBuilder::MULOp}, // MUL
{OPD(FEXCore::X86Tables::TYPE_GROUP_3, OpToIndex(0xF7), 5), 1, &OpDispatchBuilder::IMULOp},
{OPD(FEXCore::X86Tables::TYPE_GROUP_3, OpToIndex(0xF7), 6), 1, &OpDispatchBuilder::DIVOp}, // DIV
{OPD(FEXCore::X86Tables::TYPE_GROUP_3, OpToIndex(0xF7), 7), 1, &OpDispatchBuilder::IDIVOp}, // IDIV
// GROUP 4
{OPD(FEXCore::X86Tables::TYPE_GROUP_4, OpToIndex(0xFE), 0), 1, &OpDispatchBuilder::INCOp}, // INC
{OPD(FEXCore::X86Tables::TYPE_GROUP_4, OpToIndex(0xFE), 1), 1, &OpDispatchBuilder::DECOp}, // DEC
// GROUP 5
{OPD(FEXCore::X86Tables::TYPE_GROUP_5, OpToIndex(0xFF), 0), 1, &OpDispatchBuilder::INCOp}, // INC
{OPD(FEXCore::X86Tables::TYPE_GROUP_5, OpToIndex(0xFF), 1), 1, &OpDispatchBuilder::DECOp}, // DEC
{OPD(FEXCore::X86Tables::TYPE_GROUP_5, OpToIndex(0xFF), 2), 1, &OpDispatchBuilder::CALLAbsoluteOp},
{OPD(FEXCore::X86Tables::TYPE_GROUP_5, OpToIndex(0xFF), 4), 1, &OpDispatchBuilder::JUMPAbsoluteOp},
{OPD(FEXCore::X86Tables::TYPE_GROUP_5, OpToIndex(0xFF), 6), 1, &OpDispatchBuilder::PUSHOp},
// GROUP 11
{OPD(FEXCore::X86Tables::TYPE_GROUP_11, OpToIndex(0xC6), 0), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::MOVGPROp, 1>},
{OPD(FEXCore::X86Tables::TYPE_GROUP_11, OpToIndex(0xC7), 0), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::MOVGPROp, 1>},
};
#undef OPD
} // namespace FEXCore::IR
@@ -0,0 +1,161 @@
// SPDX-License-Identifier: MIT
#pragma once
#include "Interface/Core/OpcodeDispatcher.h"
namespace FEXCore::IR {
#define OPD(group, prefix, Reg) (((group - FEXCore::X86Tables::TYPE_GROUP_6) << 5) | (prefix) << 3 | (Reg))
constexpr uint16_t PF_NONE = 0;
constexpr uint16_t PF_F3 = 1;
constexpr uint16_t PF_66 = 2;
constexpr uint16_t PF_F2 = 3;
constexpr std::tuple<uint16_t, uint8_t, FEXCore::X86Tables::OpDispatchPtr> OpDispatch_SecondaryGroupTables[] = {
// GROUP 6
{OPD(FEXCore::X86Tables::TYPE_GROUP_6, PF_NONE, 3), 1, &OpDispatchBuilder::PermissionRestrictedOp},
{OPD(FEXCore::X86Tables::TYPE_GROUP_6, PF_F3, 3), 1, &OpDispatchBuilder::PermissionRestrictedOp},
{OPD(FEXCore::X86Tables::TYPE_GROUP_6, PF_66, 3), 1, &OpDispatchBuilder::PermissionRestrictedOp},
{OPD(FEXCore::X86Tables::TYPE_GROUP_6, PF_F2, 3), 1, &OpDispatchBuilder::PermissionRestrictedOp},
// GROUP 7
{OPD(FEXCore::X86Tables::TYPE_GROUP_7, PF_NONE, 0), 1, &OpDispatchBuilder::SGDTOp},
{OPD(FEXCore::X86Tables::TYPE_GROUP_7, PF_F3, 0), 1, &OpDispatchBuilder::SGDTOp},
{OPD(FEXCore::X86Tables::TYPE_GROUP_7, PF_66, 0), 1, &OpDispatchBuilder::SGDTOp},
{OPD(FEXCore::X86Tables::TYPE_GROUP_7, PF_F2, 0), 1, &OpDispatchBuilder::SGDTOp},
{OPD(FEXCore::X86Tables::TYPE_GROUP_7, PF_NONE, 3), 1, &OpDispatchBuilder::PermissionRestrictedOp},
{OPD(FEXCore::X86Tables::TYPE_GROUP_7, PF_F3, 3), 1, &OpDispatchBuilder::PermissionRestrictedOp},
{OPD(FEXCore::X86Tables::TYPE_GROUP_7, PF_66, 3), 1, &OpDispatchBuilder::PermissionRestrictedOp},
{OPD(FEXCore::X86Tables::TYPE_GROUP_7, PF_F2, 3), 1, &OpDispatchBuilder::PermissionRestrictedOp},
{OPD(FEXCore::X86Tables::TYPE_GROUP_7, PF_NONE, 4), 1, &OpDispatchBuilder::SMSWOp},
{OPD(FEXCore::X86Tables::TYPE_GROUP_7, PF_F3, 4), 1, &OpDispatchBuilder::SMSWOp},
{OPD(FEXCore::X86Tables::TYPE_GROUP_7, PF_66, 4), 1, &OpDispatchBuilder::SMSWOp},
{OPD(FEXCore::X86Tables::TYPE_GROUP_7, PF_F2, 4), 1, &OpDispatchBuilder::SMSWOp},
{OPD(FEXCore::X86Tables::TYPE_GROUP_7, PF_NONE, 6), 1, &OpDispatchBuilder::PermissionRestrictedOp},
{OPD(FEXCore::X86Tables::TYPE_GROUP_7, PF_F3, 6), 1, &OpDispatchBuilder::PermissionRestrictedOp},
{OPD(FEXCore::X86Tables::TYPE_GROUP_7, PF_66, 6), 1, &OpDispatchBuilder::PermissionRestrictedOp},
{OPD(FEXCore::X86Tables::TYPE_GROUP_7, PF_F2, 6), 1, &OpDispatchBuilder::PermissionRestrictedOp},
// GROUP 8
{OPD(FEXCore::X86Tables::TYPE_GROUP_8, PF_NONE, 4), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::BTOp, 1, BTAction::BTNone>},
{OPD(FEXCore::X86Tables::TYPE_GROUP_8, PF_F3, 4), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::BTOp, 1, BTAction::BTNone>},
{OPD(FEXCore::X86Tables::TYPE_GROUP_8, PF_66, 4), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::BTOp, 1, BTAction::BTNone>},
{OPD(FEXCore::X86Tables::TYPE_GROUP_8, PF_F2, 4), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::BTOp, 1, BTAction::BTNone>},
{OPD(FEXCore::X86Tables::TYPE_GROUP_8, PF_NONE, 5), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::BTOp, 1, BTAction::BTSet>},
{OPD(FEXCore::X86Tables::TYPE_GROUP_8, PF_F3, 5), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::BTOp, 1, BTAction::BTSet>},
{OPD(FEXCore::X86Tables::TYPE_GROUP_8, PF_66, 5), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::BTOp, 1, BTAction::BTSet>},
{OPD(FEXCore::X86Tables::TYPE_GROUP_8, PF_F2, 5), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::BTOp, 1, BTAction::BTSet>},
{OPD(FEXCore::X86Tables::TYPE_GROUP_8, PF_NONE, 6), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::BTOp, 1, BTAction::BTClear>},
{OPD(FEXCore::X86Tables::TYPE_GROUP_8, PF_F3, 6), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::BTOp, 1, BTAction::BTClear>},
{OPD(FEXCore::X86Tables::TYPE_GROUP_8, PF_66, 6), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::BTOp, 1, BTAction::BTClear>},
{OPD(FEXCore::X86Tables::TYPE_GROUP_8, PF_F2, 6), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::BTOp, 1, BTAction::BTClear>},
{OPD(FEXCore::X86Tables::TYPE_GROUP_8, PF_NONE, 7), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::BTOp, 1, BTAction::BTComplement>},
{OPD(FEXCore::X86Tables::TYPE_GROUP_8, PF_F3, 7), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::BTOp, 1, BTAction::BTComplement>},
{OPD(FEXCore::X86Tables::TYPE_GROUP_8, PF_66, 7), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::BTOp, 1, BTAction::BTComplement>},
{OPD(FEXCore::X86Tables::TYPE_GROUP_8, PF_F2, 7), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::BTOp, 1, BTAction::BTComplement>},
// GROUP 9
{OPD(FEXCore::X86Tables::TYPE_GROUP_9, PF_NONE, 1), 1, &OpDispatchBuilder::CMPXCHGPairOp},
{OPD(FEXCore::X86Tables::TYPE_GROUP_9, PF_F3, 1), 1, &OpDispatchBuilder::CMPXCHGPairOp},
{OPD(FEXCore::X86Tables::TYPE_GROUP_9, PF_66, 1), 1, &OpDispatchBuilder::CMPXCHGPairOp},
{OPD(FEXCore::X86Tables::TYPE_GROUP_9, PF_F2, 1), 1, &OpDispatchBuilder::CMPXCHGPairOp},
{OPD(FEXCore::X86Tables::TYPE_GROUP_9, PF_F3, 7), 1, &OpDispatchBuilder::RDPIDOp},
// GROUP 12
{OPD(FEXCore::X86Tables::TYPE_GROUP_12, PF_NONE, 2), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::PSRLI, OpSize::i16Bit>},
{OPD(FEXCore::X86Tables::TYPE_GROUP_12, PF_NONE, 4), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::PSRAIOp, OpSize::i16Bit>},
{OPD(FEXCore::X86Tables::TYPE_GROUP_12, PF_NONE, 6), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::PSLLI, OpSize::i16Bit>},
{OPD(FEXCore::X86Tables::TYPE_GROUP_12, PF_66, 2), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::PSRLI, OpSize::i16Bit>},
{OPD(FEXCore::X86Tables::TYPE_GROUP_12, PF_66, 4), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::PSRAIOp, OpSize::i16Bit>},
{OPD(FEXCore::X86Tables::TYPE_GROUP_12, PF_66, 6), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::PSLLI, OpSize::i16Bit>},
// GROUP 13
{OPD(FEXCore::X86Tables::TYPE_GROUP_13, PF_NONE, 2), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::PSRLI, OpSize::i32Bit>},
{OPD(FEXCore::X86Tables::TYPE_GROUP_13, PF_NONE, 4), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::PSRAIOp, OpSize::i32Bit>},
{OPD(FEXCore::X86Tables::TYPE_GROUP_13, PF_NONE, 6), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::PSLLI, OpSize::i32Bit>},
{OPD(FEXCore::X86Tables::TYPE_GROUP_13, PF_66, 2), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::PSRLI, OpSize::i32Bit>},
{OPD(FEXCore::X86Tables::TYPE_GROUP_13, PF_66, 4), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::PSRAIOp, OpSize::i32Bit>},
{OPD(FEXCore::X86Tables::TYPE_GROUP_13, PF_66, 6), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::PSLLI, OpSize::i32Bit>},
// GROUP 14
{OPD(FEXCore::X86Tables::TYPE_GROUP_14, PF_NONE, 2), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::PSRLI, OpSize::i64Bit>},
{OPD(FEXCore::X86Tables::TYPE_GROUP_14, PF_NONE, 6), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::PSLLI, OpSize::i64Bit>},
{OPD(FEXCore::X86Tables::TYPE_GROUP_14, PF_66, 2), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::PSRLI, OpSize::i64Bit>},
{OPD(FEXCore::X86Tables::TYPE_GROUP_14, PF_66, 3), 1, &OpDispatchBuilder::PSRLDQ},
{OPD(FEXCore::X86Tables::TYPE_GROUP_14, PF_66, 6), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::PSLLI, OpSize::i64Bit>},
{OPD(FEXCore::X86Tables::TYPE_GROUP_14, PF_66, 7), 1, &OpDispatchBuilder::PSLLDQ},
// GROUP 15
{OPD(FEXCore::X86Tables::TYPE_GROUP_15, PF_NONE, 0), 1, &OpDispatchBuilder::FXSaveOp},
{OPD(FEXCore::X86Tables::TYPE_GROUP_15, PF_NONE, 1), 1, &OpDispatchBuilder::FXRStoreOp},
{OPD(FEXCore::X86Tables::TYPE_GROUP_15, PF_NONE, 2), 1, &OpDispatchBuilder::LDMXCSR},
{OPD(FEXCore::X86Tables::TYPE_GROUP_15, PF_NONE, 3), 1, &OpDispatchBuilder::STMXCSR},
{OPD(FEXCore::X86Tables::TYPE_GROUP_15, PF_NONE, 4), 1, &OpDispatchBuilder::XSaveOp},
{OPD(FEXCore::X86Tables::TYPE_GROUP_15, PF_NONE, 5), 1, &OpDispatchBuilder::LoadFenceOrXRSTOR}, // LFENCE (or XRSTOR)
{OPD(FEXCore::X86Tables::TYPE_GROUP_15, PF_NONE, 6), 1, &OpDispatchBuilder::MemFenceOrXSAVEOPT}, // MFENCE (or XSAVEOPT)
{OPD(FEXCore::X86Tables::TYPE_GROUP_15, PF_NONE, 7), 1, &OpDispatchBuilder::StoreFenceOrCLFlush}, // SFENCE (or CLFLUSH)
{OPD(FEXCore::X86Tables::TYPE_GROUP_15, PF_F3, 5), 1, &OpDispatchBuilder::UnimplementedOp},
{OPD(FEXCore::X86Tables::TYPE_GROUP_15, PF_F3, 6), 1, &OpDispatchBuilder::UnimplementedOp},
{OPD(FEXCore::X86Tables::TYPE_GROUP_15, PF_66, 6), 1, &OpDispatchBuilder::CLWB},
{OPD(FEXCore::X86Tables::TYPE_GROUP_15, PF_66, 7), 1, &OpDispatchBuilder::CLFLUSHOPT},
// GROUP 16
{OPD(FEXCore::X86Tables::TYPE_GROUP_16, PF_NONE, 0), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::Prefetch, false, true, 1>},
{OPD(FEXCore::X86Tables::TYPE_GROUP_16, PF_NONE, 1), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::Prefetch, false, false, 1>},
{OPD(FEXCore::X86Tables::TYPE_GROUP_16, PF_NONE, 2), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::Prefetch, false, false, 2>},
{OPD(FEXCore::X86Tables::TYPE_GROUP_16, PF_NONE, 3), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::Prefetch, false, false, 3>},
{OPD(FEXCore::X86Tables::TYPE_GROUP_16, PF_NONE, 4), 4, &OpDispatchBuilder::NOPOp},
{OPD(FEXCore::X86Tables::TYPE_GROUP_16, PF_F3, 0), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::Prefetch, false, true, 1>},
{OPD(FEXCore::X86Tables::TYPE_GROUP_16, PF_F3, 1), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::Prefetch, false, false, 1>},
{OPD(FEXCore::X86Tables::TYPE_GROUP_16, PF_F3, 2), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::Prefetch, false, false, 2>},
{OPD(FEXCore::X86Tables::TYPE_GROUP_16, PF_F3, 3), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::Prefetch, false, false, 3>},
{OPD(FEXCore::X86Tables::TYPE_GROUP_16, PF_F3, 4), 4, &OpDispatchBuilder::NOPOp},
{OPD(FEXCore::X86Tables::TYPE_GROUP_16, PF_66, 0), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::Prefetch, false, true, 1>},
{OPD(FEXCore::X86Tables::TYPE_GROUP_16, PF_66, 1), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::Prefetch, false, false, 1>},
{OPD(FEXCore::X86Tables::TYPE_GROUP_16, PF_66, 2), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::Prefetch, false, false, 2>},
{OPD(FEXCore::X86Tables::TYPE_GROUP_16, PF_66, 3), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::Prefetch, false, false, 3>},
{OPD(FEXCore::X86Tables::TYPE_GROUP_16, PF_66, 4), 4, &OpDispatchBuilder::NOPOp},
{OPD(FEXCore::X86Tables::TYPE_GROUP_16, PF_F2, 0), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::Prefetch, false, true, 1>},
{OPD(FEXCore::X86Tables::TYPE_GROUP_16, PF_F2, 1), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::Prefetch, false, false, 1>},
{OPD(FEXCore::X86Tables::TYPE_GROUP_16, PF_F2, 2), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::Prefetch, false, false, 2>},
{OPD(FEXCore::X86Tables::TYPE_GROUP_16, PF_F2, 3), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::Prefetch, false, false, 3>},
{OPD(FEXCore::X86Tables::TYPE_GROUP_16, PF_F2, 4), 4, &OpDispatchBuilder::NOPOp},
// GROUP P
{OPD(FEXCore::X86Tables::TYPE_GROUP_P, PF_NONE, 0), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::Prefetch, false, false, 1>},
{OPD(FEXCore::X86Tables::TYPE_GROUP_P, PF_NONE, 1), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::Prefetch, true, false, 1>},
{OPD(FEXCore::X86Tables::TYPE_GROUP_P, PF_NONE, 2), 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::Prefetch, true, false, 1>},
{OPD(FEXCore::X86Tables::TYPE_GROUP_P, PF_NONE, 3), 5, &OpDispatchBuilder::NOPOp},
{OPD(FEXCore::X86Tables::TYPE_GROUP_P, PF_F3, 0), 8, &OpDispatchBuilder::NOPOp},
{OPD(FEXCore::X86Tables::TYPE_GROUP_P, PF_66, 0), 8, &OpDispatchBuilder::NOPOp},
{OPD(FEXCore::X86Tables::TYPE_GROUP_P, PF_F2, 0), 8, &OpDispatchBuilder::NOPOp},
};
constexpr std::tuple<uint16_t, uint8_t, FEXCore::X86Tables::OpDispatchPtr> OpDispatch_SecondaryGroupTables_64[] = {
// GROUP 15
{OPD(FEXCore::X86Tables::TYPE_GROUP_15, PF_F3, 0), 1,
&OpDispatchBuilder::Bind<&OpDispatchBuilder::ReadSegmentReg, OpDispatchBuilder::Segment::FS>},
{OPD(FEXCore::X86Tables::TYPE_GROUP_15, PF_F3, 1), 1,
&OpDispatchBuilder::Bind<&OpDispatchBuilder::ReadSegmentReg, OpDispatchBuilder::Segment::GS>},
{OPD(FEXCore::X86Tables::TYPE_GROUP_15, PF_F3, 2), 1,
&OpDispatchBuilder::Bind<&OpDispatchBuilder::WriteSegmentReg, OpDispatchBuilder::Segment::FS>},
{OPD(FEXCore::X86Tables::TYPE_GROUP_15, PF_F3, 3), 1,
&OpDispatchBuilder::Bind<&OpDispatchBuilder::WriteSegmentReg, OpDispatchBuilder::Segment::GS>},
};
#undef OPD
} // namespace FEXCore::IR
@@ -0,0 +1,22 @@
// SPDX-License-Identifier: MIT
#pragma once
#include "Interface/Core/OpcodeDispatcher.h"
namespace FEXCore::IR {
constexpr std::tuple<uint8_t, uint8_t, FEXCore::X86Tables::OpDispatchPtr> OpDispatch_SecondaryModRMTables[] = {
// REG /1
{((0 << 3) | 0), 1, &OpDispatchBuilder::UnimplementedOp},
{((0 << 3) | 1), 1, &OpDispatchBuilder::UnimplementedOp},
// REG /2
{((1 << 3) | 0), 1, &OpDispatchBuilder::XGetBVOp},
// REG /3
{((2 << 3) | 7), 1, &OpDispatchBuilder::PermissionRestrictedOp},
// REG /7
{((3 << 3) | 0), 1, &OpDispatchBuilder::PermissionRestrictedOp},
{((3 << 3) | 1), 1, &OpDispatchBuilder::RDTSCPOp},
};
} // namespace FEXCore::IR
@@ -0,0 +1,329 @@
// SPDX-License-Identifier: MIT
#pragma once
#include "Interface/Core/OpcodeDispatcher.h"
namespace FEXCore::IR {
constexpr std::tuple<uint8_t, uint8_t, FEXCore::X86Tables::OpDispatchPtr> OpDispatch_TwoByteOpTable[] = {
// Instructions
{0x06, 1, &OpDispatchBuilder::PermissionRestrictedOp},
{0x07, 1, &OpDispatchBuilder::PermissionRestrictedOp},
{0x0B, 1, &OpDispatchBuilder::INTOp},
{0x0E, 1, &OpDispatchBuilder::X87EMMS},
{0x19, 7, &OpDispatchBuilder::NOPOp}, // NOP with ModRM
{0x20, 4, &OpDispatchBuilder::PermissionRestrictedOp},
{0x30, 1, &OpDispatchBuilder::PermissionRestrictedOp},
{0x31, 1, &OpDispatchBuilder::RDTSCOp},
{0x32, 2, &OpDispatchBuilder::PermissionRestrictedOp},
{0x34, 3, &OpDispatchBuilder::UnimplementedOp},
{0x3F, 1, &OpDispatchBuilder::ThunkOp},
{0x40, 16, &OpDispatchBuilder::CMOVOp},
{0x6E, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::MOVBetweenGPR_FPR, OpDispatchBuilder::VectorOpType::MMX>},
{0x6F, 1, &OpDispatchBuilder::MOVQMMXOp},
{0x7E, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::MOVBetweenGPR_FPR, OpDispatchBuilder::VectorOpType::MMX>},
{0x7F, 1, &OpDispatchBuilder::MOVQMMXOp},
{0x80, 16, &OpDispatchBuilder::CondJUMPOp},
{0x90, 16, &OpDispatchBuilder::SETccOp},
{0xA2, 1, &OpDispatchBuilder::CPUIDOp},
{0xA3, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::BTOp, 0, BTAction::BTNone>}, // BT
{0xA4, 1, &OpDispatchBuilder::SHLDImmediateOp},
{0xA5, 1, &OpDispatchBuilder::SHLDOp},
{0xAB, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::BTOp, 0, BTAction::BTSet>}, // BTS
{0xAC, 1, &OpDispatchBuilder::SHRDImmediateOp},
{0xAD, 1, &OpDispatchBuilder::SHRDOp},
{0xAF, 1, &OpDispatchBuilder::IMUL1SrcOp},
{0xB0, 2, &OpDispatchBuilder::CMPXCHGOp}, // CMPXCHG
{0xB3, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::BTOp, 0, BTAction::BTClear>}, // BTR
{0xB6, 2, &OpDispatchBuilder::MOVZXOp},
{0xBB, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::BTOp, 0, BTAction::BTComplement>}, // BTC
{0xBC, 1, &OpDispatchBuilder::BSFOp}, // BSF
{0xBD, 1, &OpDispatchBuilder::BSROp}, // BSF
{0xBE, 2, &OpDispatchBuilder::MOVSXOp},
{0xC0, 2, &OpDispatchBuilder::XADDOp},
{0xC3, 1, &OpDispatchBuilder::MOVGPRNTOp},
{0xC4, 1, &OpDispatchBuilder::PINSROp<OpSize::i16Bit>},
{0xC5, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::PExtrOp, OpSize::i16Bit>},
{0xC8, 8, &OpDispatchBuilder::BSWAPOp},
// SSE
{0x10, 2, &OpDispatchBuilder::MOVVectorUnalignedOp},
{0x12, 2, &OpDispatchBuilder::MOVLPOp},
{0x14, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::PUNPCKLOp, OpSize::i32Bit>},
{0x15, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::PUNPCKHOp, OpSize::i32Bit>},
{0x16, 2, &OpDispatchBuilder::MOVHPDOp},
{0x28, 2, &OpDispatchBuilder::MOVVectorAlignedOp},
{0x2A, 1, &OpDispatchBuilder::InsertMMX_To_XMM_Vector_CVT_Int_To_Float},
{0x2B, 1, &OpDispatchBuilder::MOVVectorNTOp},
{0x2C, 1, &OpDispatchBuilder::XMM_To_MMX_Vector_CVT_Float_To_Int<OpSize::i32Bit, false>},
{0x2D, 1, &OpDispatchBuilder::XMM_To_MMX_Vector_CVT_Float_To_Int<OpSize::i32Bit, true>},
{0x2E, 2, &OpDispatchBuilder::UCOMISxOp<OpSize::i32Bit>},
{0x50, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::MOVMSKOp, OpSize::i32Bit>},
{0x51, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorUnaryOp, IR::OP_VFSQRT, OpSize::i32Bit>},
{0x52, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorUnaryOp, IR::OP_VFRSQRT, OpSize::i32Bit>},
{0x53, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorUnaryOp, IR::OP_VFRECP, OpSize::i32Bit>},
{0x54, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorALUOp, IR::OP_VAND, OpSize::i128Bit>},
{0x55, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorALUROp, IR::OP_VANDN, OpSize::i64Bit>},
{0x56, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorALUOp, IR::OP_VOR, OpSize::i128Bit>},
{0x57, 1, &OpDispatchBuilder::VectorXOROp},
{0x58, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorALUOp, IR::OP_VFADD, OpSize::i32Bit>},
{0x59, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorALUOp, IR::OP_VFMUL, OpSize::i32Bit>},
{0x5A, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::Vector_CVT_Float_To_Float, OpSize::i64Bit, OpSize::i32Bit, false>},
{0x5B, 1, &OpDispatchBuilder::Vector_CVT_Int_To_Float<OpSize::i32Bit, false>},
{0x5C, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorALUOp, IR::OP_VFSUB, OpSize::i32Bit>},
{0x5D, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorALUOp, IR::OP_VFMIN, OpSize::i32Bit>},
{0x5E, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorALUOp, IR::OP_VFDIV, OpSize::i32Bit>},
{0x5F, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorALUOp, IR::OP_VFMAX, OpSize::i32Bit>},
{0x60, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::PUNPCKLOp, OpSize::i8Bit>},
{0x61, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::PUNPCKLOp, OpSize::i16Bit>},
{0x62, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::PUNPCKLOp, OpSize::i32Bit>},
{0x63, 1, &OpDispatchBuilder::PACKSSOp<OpSize::i16Bit>},
{0x64, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorALUOp, IR::OP_VCMPGT, OpSize::i8Bit>},
{0x65, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorALUOp, IR::OP_VCMPGT, OpSize::i16Bit>},
{0x66, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorALUOp, IR::OP_VCMPGT, OpSize::i32Bit>},
{0x67, 1, &OpDispatchBuilder::PACKUSOp<OpSize::i16Bit>},
{0x68, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::PUNPCKHOp, OpSize::i8Bit>},
{0x69, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::PUNPCKHOp, OpSize::i16Bit>},
{0x6A, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::PUNPCKHOp, OpSize::i32Bit>},
{0x6B, 1, &OpDispatchBuilder::PACKSSOp<OpSize::i32Bit>},
{0x70, 1, &OpDispatchBuilder::PSHUFW8ByteOp},
{0x74, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorALUOp, IR::OP_VCMPEQ, OpSize::i8Bit>},
{0x75, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorALUOp, IR::OP_VCMPEQ, OpSize::i16Bit>},
{0x76, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorALUOp, IR::OP_VCMPEQ, OpSize::i32Bit>},
{0x77, 1, &OpDispatchBuilder::X87EMMS},
{0xC2, 1, &OpDispatchBuilder::VFCMPOp<OpSize::i32Bit>},
{0xC6, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::SHUFOp, OpSize::i32Bit>},
{0xD1, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::PSRLDOp, OpSize::i16Bit>},
{0xD2, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::PSRLDOp, OpSize::i32Bit>},
{0xD3, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::PSRLDOp, OpSize::i64Bit>},
{0xD4, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorALUOp, IR::OP_VADD, OpSize::i64Bit>},
{0xD5, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorALUOp, IR::OP_VMUL, OpSize::i16Bit>},
{0xD7, 1, &OpDispatchBuilder::MOVMSKOpOne}, // PMOVMSKB
{0xD8, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorALUOp, IR::OP_VUQSUB, OpSize::i8Bit>},
{0xD9, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorALUOp, IR::OP_VUQSUB, OpSize::i16Bit>},
{0xDA, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorALUOp, IR::OP_VUMIN, OpSize::i8Bit>},
{0xDB, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorALUOp, IR::OP_VAND, OpSize::i64Bit>},
{0xDC, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorALUOp, IR::OP_VUQADD, OpSize::i8Bit>},
{0xDD, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorALUOp, IR::OP_VUQADD, OpSize::i16Bit>},
{0xDE, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorALUOp, IR::OP_VUMAX, OpSize::i8Bit>},
{0xDF, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorALUROp, IR::OP_VANDN, OpSize::i64Bit>},
{0xE0, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorALUOp, IR::OP_VURAVG, OpSize::i8Bit>},
{0xE1, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::PSRAOp, OpSize::i16Bit>},
{0xE2, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::PSRAOp, OpSize::i32Bit>},
{0xE3, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorALUOp, IR::OP_VURAVG, OpSize::i16Bit>},
{0xE4, 1, &OpDispatchBuilder::PMULHW<false>},
{0xE5, 1, &OpDispatchBuilder::PMULHW<true>},
{0xE7, 1, &OpDispatchBuilder::MOVVectorNTOp},
{0xE8, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorALUOp, IR::OP_VSQSUB, OpSize::i8Bit>},
{0xE9, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorALUOp, IR::OP_VSQSUB, OpSize::i16Bit>},
{0xEA, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorALUOp, IR::OP_VSMIN, OpSize::i16Bit>},
{0xEB, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorALUOp, IR::OP_VOR, OpSize::i64Bit>},
{0xEC, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorALUOp, IR::OP_VSQADD, OpSize::i8Bit>},
{0xED, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorALUOp, IR::OP_VSQADD, OpSize::i16Bit>},
{0xEE, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorALUOp, IR::OP_VSMAX, OpSize::i16Bit>},
{0xEF, 1, &OpDispatchBuilder::VectorXOROp},
{0xF1, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::PSLL, OpSize::i16Bit>},
{0xF2, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::PSLL, OpSize::i32Bit>},
{0xF3, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::PSLL, OpSize::i64Bit>},
{0xF4, 1, &OpDispatchBuilder::PMULLOp<OpSize::i32Bit, false>},
{0xF5, 1, &OpDispatchBuilder::PMADDWD},
{0xF6, 1, &OpDispatchBuilder::PSADBW},
{0xF7, 1, &OpDispatchBuilder::MASKMOVOp},
{0xF8, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorALUOp, IR::OP_VSUB, OpSize::i8Bit>},
{0xF9, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorALUOp, IR::OP_VSUB, OpSize::i16Bit>},
{0xFA, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorALUOp, IR::OP_VSUB, OpSize::i32Bit>},
{0xFB, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorALUOp, IR::OP_VSUB, OpSize::i64Bit>},
{0xFC, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorALUOp, IR::OP_VADD, OpSize::i8Bit>},
{0xFD, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorALUOp, IR::OP_VADD, OpSize::i16Bit>},
{0xFE, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorALUOp, IR::OP_VADD, OpSize::i32Bit>},
// FEX reserved instructions
{0x37, 1, &OpDispatchBuilder::CallbackReturnOp},
};
constexpr std::tuple<uint8_t, uint8_t, FEXCore::X86Tables::OpDispatchPtr> OpDispatch_SecondaryRepModTables[] = {
{0x10, 2, &OpDispatchBuilder::MOVSSOp},
{0x12, 1, &OpDispatchBuilder::VMOVSLDUPOp},
{0x16, 1, &OpDispatchBuilder::VMOVSHDUPOp},
{0x2A, 1, &OpDispatchBuilder::InsertCVTGPR_To_FPR<OpSize::i32Bit>},
{0x2B, 1, &OpDispatchBuilder::MOVVectorNTOp},
{0x2C, 1, &OpDispatchBuilder::CVTFPR_To_GPR<OpSize::i32Bit, false>},
{0x2D, 1, &OpDispatchBuilder::CVTFPR_To_GPR<OpSize::i32Bit, true>},
{0x51, 1, &OpDispatchBuilder::VectorScalarUnaryInsertALUOp<IR::OP_VFSQRTSCALARINSERT, OpSize::i32Bit>},
{0x52, 1, &OpDispatchBuilder::VectorScalarUnaryInsertALUOp<IR::OP_VFRSQRTSCALARINSERT, OpSize::i32Bit>},
{0x53, 1, &OpDispatchBuilder::VectorScalarUnaryInsertALUOp<IR::OP_VFRECPSCALARINSERT, OpSize::i32Bit>},
{0x58, 1, &OpDispatchBuilder::VectorScalarInsertALUOp<IR::OP_VFADDSCALARINSERT, OpSize::i32Bit>},
{0x59, 1, &OpDispatchBuilder::VectorScalarInsertALUOp<IR::OP_VFMULSCALARINSERT, OpSize::i32Bit>},
{0x5A, 1, &OpDispatchBuilder::InsertScalar_CVT_Float_To_Float<OpSize::i64Bit, OpSize::i32Bit>},
{0x5B, 1, &OpDispatchBuilder::Vector_CVT_Float_To_Int<OpSize::i32Bit, false>},
{0x5C, 1, &OpDispatchBuilder::VectorScalarInsertALUOp<IR::OP_VFSUBSCALARINSERT, OpSize::i32Bit>},
{0x5D, 1, &OpDispatchBuilder::VectorScalarInsertALUOp<IR::OP_VFMINSCALARINSERT, OpSize::i32Bit>},
{0x5E, 1, &OpDispatchBuilder::VectorScalarInsertALUOp<IR::OP_VFDIVSCALARINSERT, OpSize::i32Bit>},
{0x5F, 1, &OpDispatchBuilder::VectorScalarInsertALUOp<IR::OP_VFMAXSCALARINSERT, OpSize::i32Bit>},
{0x6F, 1, &OpDispatchBuilder::MOVVectorUnalignedOp},
{0x70, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::PSHUFWOp, false>},
{0x7E, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::MOVQOp, OpDispatchBuilder::VectorOpType::SSE>},
{0x7F, 1, &OpDispatchBuilder::MOVVectorUnalignedOp},
{0xB8, 1, &OpDispatchBuilder::PopcountOp},
{0xBC, 1, &OpDispatchBuilder::TZCNT},
{0xBD, 1, &OpDispatchBuilder::LZCNT},
{0xC2, 1, &OpDispatchBuilder::InsertScalarFCMPOp<OpSize::i32Bit>},
{0xD6, 1, &OpDispatchBuilder::MOVQ2DQ<true>},
{0xE6, 1, &OpDispatchBuilder::Vector_CVT_Int_To_Float<OpSize::i32Bit, true>},
};
constexpr std::tuple<uint8_t, uint8_t, FEXCore::X86Tables::OpDispatchPtr> OpDispatch_SecondaryRepNEModTables[] = {
{0x10, 2, &OpDispatchBuilder::MOVSDOp},
{0x12, 1, &OpDispatchBuilder::MOVDDUPOp},
{0x2A, 1, &OpDispatchBuilder::InsertCVTGPR_To_FPR<OpSize::i64Bit>},
{0x2B, 1, &OpDispatchBuilder::MOVVectorNTOp},
{0x2C, 1, &OpDispatchBuilder::CVTFPR_To_GPR<OpSize::i64Bit, false>},
{0x2D, 1, &OpDispatchBuilder::CVTFPR_To_GPR<OpSize::i64Bit, true>},
{0x51, 1, &OpDispatchBuilder::VectorScalarUnaryInsertALUOp<IR::OP_VFSQRTSCALARINSERT, OpSize::i64Bit>},
// x52 = Invalid
{0x58, 1, &OpDispatchBuilder::VectorScalarInsertALUOp<IR::OP_VFADDSCALARINSERT, OpSize::i64Bit>},
{0x59, 1, &OpDispatchBuilder::VectorScalarInsertALUOp<IR::OP_VFMULSCALARINSERT, OpSize::i64Bit>},
{0x5A, 1, &OpDispatchBuilder::InsertScalar_CVT_Float_To_Float<OpSize::i32Bit, OpSize::i64Bit>},
{0x5C, 1, &OpDispatchBuilder::VectorScalarInsertALUOp<IR::OP_VFSUBSCALARINSERT, OpSize::i64Bit>},
{0x5D, 1, &OpDispatchBuilder::VectorScalarInsertALUOp<IR::OP_VFMINSCALARINSERT, OpSize::i64Bit>},
{0x5E, 1, &OpDispatchBuilder::VectorScalarInsertALUOp<IR::OP_VFDIVSCALARINSERT, OpSize::i64Bit>},
{0x5F, 1, &OpDispatchBuilder::VectorScalarInsertALUOp<IR::OP_VFMAXSCALARINSERT, OpSize::i64Bit>},
{0x70, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::PSHUFWOp, true>},
{0x7C, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorALUOp, IR::OP_VFADDP, OpSize::i32Bit>},
{0x7D, 1, &OpDispatchBuilder::HSUBP<OpSize::i32Bit>},
{0xD0, 1, &OpDispatchBuilder::ADDSUBPOp<OpSize::i32Bit>},
{0xD6, 1, &OpDispatchBuilder::MOVQ2DQ<false>},
{0xC2, 1, &OpDispatchBuilder::InsertScalarFCMPOp<OpSize::i64Bit>},
{0xE6, 1, &OpDispatchBuilder::Vector_CVT_Float_To_Int<OpSize::i64Bit, true>},
{0xF0, 1, &OpDispatchBuilder::MOVVectorUnalignedOp},
};
constexpr std::tuple<uint8_t, uint8_t, FEXCore::X86Tables::OpDispatchPtr> OpDispatch_SecondaryOpSizeModTables[] = {
{0x10, 2, &OpDispatchBuilder::MOVVectorUnalignedOp},
{0x12, 2, &OpDispatchBuilder::MOVLPOp},
{0x14, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::PUNPCKLOp, OpSize::i64Bit>},
{0x15, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::PUNPCKHOp, OpSize::i64Bit>},
{0x16, 2, &OpDispatchBuilder::MOVHPDOp},
{0x28, 2, &OpDispatchBuilder::MOVVectorAlignedOp},
{0x2A, 1, &OpDispatchBuilder::MMX_To_XMM_Vector_CVT_Int_To_Float},
{0x2B, 1, &OpDispatchBuilder::MOVVectorNTOp},
{0x2C, 1, &OpDispatchBuilder::XMM_To_MMX_Vector_CVT_Float_To_Int<OpSize::i64Bit, false>},
{0x2D, 1, &OpDispatchBuilder::XMM_To_MMX_Vector_CVT_Float_To_Int<OpSize::i64Bit, true>},
{0x2E, 2, &OpDispatchBuilder::UCOMISxOp<OpSize::i64Bit>},
{0x50, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::MOVMSKOp, OpSize::i64Bit>},
{0x51, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorUnaryOp, IR::OP_VFSQRT, OpSize::i64Bit>},
{0x54, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorALUOp, IR::OP_VAND, OpSize::i128Bit>},
{0x55, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorALUROp, IR::OP_VANDN, OpSize::i64Bit>},
{0x56, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorALUOp, IR::OP_VOR, OpSize::i128Bit>},
{0x57, 1, &OpDispatchBuilder::VectorXOROp},
{0x58, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorALUOp, IR::OP_VFADD, OpSize::i64Bit>},
{0x59, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorALUOp, IR::OP_VFMUL, OpSize::i64Bit>},
{0x5A, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::Vector_CVT_Float_To_Float, OpSize::i32Bit, OpSize::i64Bit, false>},
{0x5B, 1, &OpDispatchBuilder::Vector_CVT_Float_To_Int<OpSize::i32Bit, true>},
{0x5C, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorALUOp, IR::OP_VFSUB, OpSize::i64Bit>},
{0x5D, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorALUOp, IR::OP_VFMIN, OpSize::i64Bit>},
{0x5E, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorALUOp, IR::OP_VFDIV, OpSize::i64Bit>},
{0x5F, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorALUOp, IR::OP_VFMAX, OpSize::i64Bit>},
{0x60, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::PUNPCKLOp, OpSize::i8Bit>},
{0x61, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::PUNPCKLOp, OpSize::i16Bit>},
{0x62, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::PUNPCKLOp, OpSize::i32Bit>},
{0x63, 1, &OpDispatchBuilder::PACKSSOp<OpSize::i16Bit>},
{0x64, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorALUOp, IR::OP_VCMPGT, OpSize::i8Bit>},
{0x65, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorALUOp, IR::OP_VCMPGT, OpSize::i16Bit>},
{0x66, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorALUOp, IR::OP_VCMPGT, OpSize::i32Bit>},
{0x67, 1, &OpDispatchBuilder::PACKUSOp<OpSize::i16Bit>},
{0x68, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::PUNPCKHOp, OpSize::i8Bit>},
{0x69, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::PUNPCKHOp, OpSize::i16Bit>},
{0x6A, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::PUNPCKHOp, OpSize::i32Bit>},
{0x6B, 1, &OpDispatchBuilder::PACKSSOp<OpSize::i32Bit>},
{0x6C, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::PUNPCKLOp, OpSize::i64Bit>},
{0x6D, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::PUNPCKHOp, OpSize::i64Bit>},
{0x6E, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::MOVBetweenGPR_FPR, OpDispatchBuilder::VectorOpType::SSE>},
{0x6F, 1, &OpDispatchBuilder::MOVVectorAlignedOp},
{0x70, 1, &OpDispatchBuilder::PSHUFDOp},
{0x74, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorALUOp, IR::OP_VCMPEQ, OpSize::i8Bit>},
{0x75, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorALUOp, IR::OP_VCMPEQ, OpSize::i16Bit>},
{0x76, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorALUOp, IR::OP_VCMPEQ, OpSize::i32Bit>},
{0x78, 1, nullptr}, // GROUP 17
{0x7C, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorALUOp, IR::OP_VFADDP, OpSize::i64Bit>},
{0x7D, 1, &OpDispatchBuilder::HSUBP<OpSize::i64Bit>},
{0x7E, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::MOVBetweenGPR_FPR, OpDispatchBuilder::VectorOpType::SSE>},
{0x7F, 1, &OpDispatchBuilder::MOVVectorAlignedOp},
{0xC2, 1, &OpDispatchBuilder::VFCMPOp<OpSize::i64Bit>},
{0xC4, 1, &OpDispatchBuilder::PINSROp<OpSize::i16Bit>},
{0xC5, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::PExtrOp, OpSize::i16Bit>},
{0xC6, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::SHUFOp, OpSize::i64Bit>},
{0xD0, 1, &OpDispatchBuilder::ADDSUBPOp<OpSize::i64Bit>},
{0xD1, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::PSRLDOp, OpSize::i16Bit>},
{0xD2, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::PSRLDOp, OpSize::i32Bit>},
{0xD3, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::PSRLDOp, OpSize::i64Bit>},
{0xD4, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorALUOp, IR::OP_VADD, OpSize::i64Bit>},
{0xD5, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorALUOp, IR::OP_VMUL, OpSize::i16Bit>},
{0xD6, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::MOVQOp, OpDispatchBuilder::VectorOpType::SSE>},
{0xD7, 1, &OpDispatchBuilder::MOVMSKOpOne}, // PMOVMSKB
{0xD8, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorALUOp, IR::OP_VUQSUB, OpSize::i8Bit>},
{0xD9, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorALUOp, IR::OP_VUQSUB, OpSize::i16Bit>},
{0xDA, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorALUOp, IR::OP_VUMIN, OpSize::i8Bit>},
{0xDB, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorALUOp, IR::OP_VAND, OpSize::i128Bit>},
{0xDC, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorALUOp, IR::OP_VUQADD, OpSize::i8Bit>},
{0xDD, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorALUOp, IR::OP_VUQADD, OpSize::i16Bit>},
{0xDE, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorALUOp, IR::OP_VUMAX, OpSize::i8Bit>},
{0xDF, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorALUROp, IR::OP_VANDN, OpSize::i64Bit>},
{0xE0, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorALUOp, IR::OP_VURAVG, OpSize::i8Bit>},
{0xE1, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::PSRAOp, OpSize::i16Bit>},
{0xE2, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::PSRAOp, OpSize::i32Bit>},
{0xE3, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorALUOp, IR::OP_VURAVG, OpSize::i16Bit>},
{0xE4, 1, &OpDispatchBuilder::PMULHW<false>},
{0xE5, 1, &OpDispatchBuilder::PMULHW<true>},
{0xE6, 1, &OpDispatchBuilder::Vector_CVT_Float_To_Int<OpSize::i64Bit, false>},
{0xE7, 1, &OpDispatchBuilder::MOVVectorNTOp},
{0xE8, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorALUOp, IR::OP_VSQSUB, OpSize::i8Bit>},
{0xE9, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorALUOp, IR::OP_VSQSUB, OpSize::i16Bit>},
{0xEA, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorALUOp, IR::OP_VSMIN, OpSize::i16Bit>},
{0xEB, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorALUOp, IR::OP_VOR, OpSize::i128Bit>},
{0xEC, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorALUOp, IR::OP_VSQADD, OpSize::i8Bit>},
{0xED, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorALUOp, IR::OP_VSQADD, OpSize::i16Bit>},
{0xEE, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorALUOp, IR::OP_VSMAX, OpSize::i16Bit>},
{0xEF, 1, &OpDispatchBuilder::VectorXOROp},
{0xF1, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::PSLL, OpSize::i16Bit>},
{0xF2, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::PSLL, OpSize::i32Bit>},
{0xF3, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::PSLL, OpSize::i64Bit>},
{0xF4, 1, &OpDispatchBuilder::PMULLOp<OpSize::i32Bit, false>},
{0xF5, 1, &OpDispatchBuilder::PMADDWD},
{0xF6, 1, &OpDispatchBuilder::PSADBW},
{0xF7, 1, &OpDispatchBuilder::MASKMOVOp},
{0xF8, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorALUOp, IR::OP_VSUB, OpSize::i8Bit>},
{0xF9, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorALUOp, IR::OP_VSUB, OpSize::i16Bit>},
{0xFA, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorALUOp, IR::OP_VSUB, OpSize::i32Bit>},
{0xFB, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorALUOp, IR::OP_VSUB, OpSize::i64Bit>},
{0xFC, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorALUOp, IR::OP_VADD, OpSize::i8Bit>},
{0xFD, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorALUOp, IR::OP_VADD, OpSize::i16Bit>},
{0xFE, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::VectorALUOp, IR::OP_VADD, OpSize::i32Bit>},
};
constexpr std::tuple<uint8_t, uint8_t, FEXCore::X86Tables::OpDispatchPtr> OpDispatch_TwoByteOpTable_64[] = {
{0x05, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::SyscallOp, true>},
{0xA0, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::PUSHSegmentOp, FEXCore::X86Tables::DecodeFlags::FLAG_FS_PREFIX>},
{0xA1, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::POPSegmentOp, FEXCore::X86Tables::DecodeFlags::FLAG_FS_PREFIX>},
{0xA8, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::PUSHSegmentOp, FEXCore::X86Tables::DecodeFlags::FLAG_GS_PREFIX>},
{0xA9, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::POPSegmentOp, FEXCore::X86Tables::DecodeFlags::FLAG_GS_PREFIX>},
};
constexpr std::tuple<uint8_t, uint8_t, FEXCore::X86Tables::OpDispatchPtr> OpDispatch_TwoByteOpTable_32[] = {
{0x05, 1, &OpDispatchBuilder::NOPOp},
{0xA0, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::PUSHSegmentOp, FEXCore::X86Tables::DecodeFlags::FLAG_FS_PREFIX>},
{0xA1, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::POPSegmentOp, FEXCore::X86Tables::DecodeFlags::FLAG_FS_PREFIX>},
{0xA8, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::PUSHSegmentOp, FEXCore::X86Tables::DecodeFlags::FLAG_GS_PREFIX>},
{0xA9, 1, &OpDispatchBuilder::Bind<&OpDispatchBuilder::POPSegmentOp, FEXCore::X86Tables::DecodeFlags::FLAG_GS_PREFIX>},
};
} // namespace FEXCore::IR
@@ -0,0 +1,26 @@
// SPDX-License-Identifier: MIT
#pragma once
#include "Interface/Core/OpcodeDispatcher.h"
namespace FEXCore::IR {
#define OPD(map_select, pp, opcode) (((map_select - 1) << 10) | (pp << 8) | (opcode))
constexpr std::tuple<uint16_t, uint8_t, FEXCore::X86Tables::OpDispatchPtr> OpDispatch_VEXTable[] = {
{OPD(2, 0b00, 0xF2), 1, &OpDispatchBuilder::ANDNBMIOp}, {OPD(2, 0b00, 0xF5), 1, &OpDispatchBuilder::BZHI},
{OPD(2, 0b10, 0xF5), 1, &OpDispatchBuilder::PEXT}, {OPD(2, 0b11, 0xF5), 1, &OpDispatchBuilder::PDEP},
{OPD(2, 0b11, 0xF6), 1, &OpDispatchBuilder::MULX}, {OPD(2, 0b00, 0xF7), 1, &OpDispatchBuilder::BEXTRBMIOp},
{OPD(2, 0b01, 0xF7), 1, &OpDispatchBuilder::BMI2Shift}, {OPD(2, 0b10, 0xF7), 1, &OpDispatchBuilder::BMI2Shift},
{OPD(2, 0b11, 0xF7), 1, &OpDispatchBuilder::BMI2Shift},
{OPD(3, 0b11, 0xF0), 1, &OpDispatchBuilder::RORX},
};
#undef OPD
#define OPD(group, pp, opcode) (((group - X86Tables::InstType::TYPE_VEX_GROUP_12) << 4) | (pp << 3) | (opcode))
constexpr std::tuple<uint8_t, uint8_t, X86Tables::OpDispatchPtr> OpDispatch_VEXGroupTable[] = {
{OPD(X86Tables::InstType::TYPE_VEX_GROUP_17, 0, 0b001), 1, &OpDispatchBuilder::BLSRBMIOp},
{OPD(X86Tables::InstType::TYPE_VEX_GROUP_17, 0, 0b010), 1, &OpDispatchBuilder::BLSMSKBMIOp},
{OPD(X86Tables::InstType::TYPE_VEX_GROUP_17, 0, 0b011), 1, &OpDispatchBuilder::BLSIBMIOp},
};
#undef OPD
} // namespace FEXCore::IR
File diff suppressed because it is too large. Load diff
@@ -16,6 +16,7 @@ $end_info$
#include <FEXCore/Utils/LogManager.h>
#include <FEXCore/Utils/FPState.h>
#include <cmath>
#include <stddef.h>
#include <stdint.h>
@@ -26,7 +27,7 @@ class OrderedNode;
Ref OpDispatchBuilder::GetX87Top() {
// Yes, we are storing 3 bits in a single flag register.
// Deal with it
return _LoadContext(1, GPRClass, offsetof(FEXCore::Core::CPUState, flags) + FEXCore::X86State::X87FLAG_TOP_LOC);
return _LoadContext(OpSize::i8Bit, GPRClass, offsetof(FEXCore::Core::CPUState, flags) + FEXCore::X86State::X87FLAG_TOP_LOC);
}
Ref OpDispatchBuilder::GetX87Tag(Ref Value, Ref AbridgedFTW) {
@@ -56,17 +57,17 @@ void OpDispatchBuilder::SetX87FTW(Ref FTW) {
}
void OpDispatchBuilder::SetX87Top(Ref Value) {
_StoreContext(1, GPRClass, Value, offsetof(FEXCore::Core::CPUState, flags) + FEXCore::X86State::X87FLAG_TOP_LOC);
_StoreContext(OpSize::i8Bit, GPRClass, Value, offsetof(FEXCore::Core::CPUState, flags) + FEXCore::X86State::X87FLAG_TOP_LOC);
}
// Float LoaD operation with memory operand
void OpDispatchBuilder::FLD(OpcodeArgs, size_t Width) {
size_t ReadWidth = (Width == 80) ? 16 : Width / 8;
void OpDispatchBuilder::FLD(OpcodeArgs, IR::OpSize Width) {
const auto ReadWidth = (Width == OpSize::f80Bit) ? OpSize::i128Bit : Width;
Ref Data = LoadSource_WithOpSize(FPRClass, Op, Op->Src[0], ReadWidth, Op->Flags);
Ref Data = LoadSource_WithOpSize(FPRClass, Op, Op->Src[0], Width, Op->Flags);
Ref ConvertedData = Data;
// Convert to 80bit float
if (Width == 32 || Width == 64) {
if (Width == OpSize::i32Bit || Width == OpSize::i64Bit) {
ConvertedData = _F80CVTTo(Data, ReadWidth);
}
_PushStack(ConvertedData, Data, ReadWidth, true);
@@ -79,31 +80,31 @@ void OpDispatchBuilder::FLDFromStack(OpcodeArgs) {
void OpDispatchBuilder::FBLD(OpcodeArgs) {
// Read from memory
Ref Data = LoadSource_WithOpSize(FPRClass, Op, Op->Src[0], 16, Op->Flags);
Ref Data = LoadSource_WithOpSize(FPRClass, Op, Op->Src[0], OpSize::f80Bit, Op->Flags);
Ref ConvertedData = _F80BCDLoad(Data);
_PushStack(ConvertedData, Data, 16, true);
_PushStack(ConvertedData, Data, OpSize::i128Bit, true);
}
void OpDispatchBuilder::FBSTP(OpcodeArgs) {
Ref converted = _F80BCDStore(_ReadStackValue(0));
StoreResult_WithOpSize(FPRClass, Op, Op->Dest, converted, 10, 1);
StoreResult_WithOpSize(FPRClass, Op, Op->Dest, converted, OpSize::f80Bit, OpSize::i8Bit);
_PopStackDestroy();
}
void OpDispatchBuilder::FLD_Const(OpcodeArgs, NamedVectorConstant Constant) {
// Update TOP
Ref Data = LoadAndCacheNamedVectorConstant(16, Constant);
_PushStack(Data, Data, 16, true);
Ref Data = LoadAndCacheNamedVectorConstant(OpSize::i128Bit, Constant);
_PushStack(Data, Data, OpSize::i128Bit, true);
}
void OpDispatchBuilder::FILD(OpcodeArgs) {
size_t ReadWidth = GetSrcSize(Op);
const auto ReadWidth = OpSizeFromSrc(Op);
// Read from memory
Ref Data = LoadSource_WithOpSize(GPRClass, Op, Op->Src[0], ReadWidth, Op->Flags);
// Sign extend to 64bits
if (ReadWidth != 8) {
Data = _Sbfe(OpSize::i64Bit, ReadWidth * 8, 0, Data);
if (ReadWidth != OpSize::i64Bit) {
Data = _Sbfe(OpSize::i64Bit, IR::OpSizeAsBits(ReadWidth), 0, Data);
}
// We're about to clobber flags to grab the sign, so save NZCV.
@@ -123,14 +124,29 @@ void OpDispatchBuilder::FILD(OpcodeArgs) {
auto zeroed_exponent = _Select(COND_EQ, absolute, zero, zero, adjusted_exponent);
auto upper = _Or(OpSize::i64Bit, sign, zeroed_exponent);
Ref ConvertedData = _VCastFromGPR(16, 8, shifted);
ConvertedData = _VInsElement(16, 8, 1, 0, ConvertedData, _VCastFromGPR(16, 8, upper));
Ref ConvertedData = _VCastFromGPR(OpSize::i64Bit, OpSize::i64Bit, shifted);
ConvertedData = _VInsElement(OpSize::i128Bit, OpSize::i64Bit, 1, 0, ConvertedData, _VCastFromGPR(OpSize::i128Bit, OpSize::i64Bit, upper));
_PushStack(ConvertedData, Data, ReadWidth, false);
}
void OpDispatchBuilder::FST(OpcodeArgs, size_t Width) {
Ref Mem = LoadSource(GPRClass, Op, Op->Dest, Op->Flags, {.LoadData = false});
_StoreStackMemory(Mem, OpSize::i128Bit, true, Width / 8);
void OpDispatchBuilder::FST(OpcodeArgs, IR::OpSize Width) {
// Ref Mem = LoadSource(GPRClass, Op, Op->Dest, Op->Flags, {.LoadData = false});
// FIXME: Is TSO relevant for x87?
AddressMode A = DecodeAddress(Op, Op->Dest, MemoryAccessType::DEFAULT, false);
// Index scale is a power of 2?
LOGMAN_THROW_A_FMT(A.IndexScale > 0 && (A.IndexScale & (A.IndexScale - 1)) == 0, "Invalid index scale");
Ref Addr = A.Base ? A.Base : _Constant(0);
if (A.Index) {
Ref ScaledIndex = A.Index;
if (A.IndexScale > 1) {
ScaledIndex = _Lshl(A.AddrSize, ScaledIndex, _Constant(std::log2(A.IndexScale)));
}
Addr = _Add(A.AddrSize, Addr, ScaledIndex);
}
_StoreStackMem(OpSize::i128Bit, Width, Addr, _Constant(A.Offset), /*Float=*/true);
if (Op->TableInfo->Flags & X86Tables::InstFlags::FLAGS_POP) {
_PopStackDestroy();
}
@@ -149,20 +165,18 @@ void OpDispatchBuilder::FSTToStack(OpcodeArgs) {
// Store integer to memory (possibly with truncation)
void OpDispatchBuilder::FIST(OpcodeArgs, bool Truncate) {
auto Size = GetSrcSize(Op);
// FIXME(pmatos): is there any advantage of using STORESTACKMEMORY here?
// Do we need STORESTACKMEMORY at all?
const auto Size = OpSizeFromSrc(Op);
Ref Data = _ReadStackValue(0);
Data = _F80CVTInt(Size, Data, Truncate);
StoreResult_WithOpSize(GPRClass, Op, Op->Dest, Data, Size, 1);
StoreResult_WithOpSize(GPRClass, Op, Op->Dest, Data, Size, OpSize::i8Bit);
if ((Op->TableInfo->Flags & X86Tables::InstFlags::FLAGS_POP) != 0) {
_PopStackDestroy();
}
}
void OpDispatchBuilder::FADD(OpcodeArgs, size_t Width, bool Integer, OpDispatchBuilder::OpResult ResInST0) {
void OpDispatchBuilder::FADD(OpcodeArgs, IR::OpSize Width, bool Integer, OpDispatchBuilder::OpResult ResInST0) {
if (Op->Src[0].IsNone()) { // Implicit argument case
auto Offset = Op->OP & 7;
auto St0 = 0;
@@ -177,23 +191,22 @@ void OpDispatchBuilder::FADD(OpcodeArgs, size_t Width, bool Integer, OpDispatchB
return;
}
LOGMAN_THROW_A_FMT(Width != OpSize::f80Bit, "No 80-bit floats from memory");
// We have one memory argument
Ref Arg {};
if (Width == 16 || Width == 32 || Width == 64) {
if (Integer) {
Arg = LoadSource(GPRClass, Op, Op->Src[0], Op->Flags);
Arg = _F80CVTToInt(Arg, Width / 8);
} else {
Arg = LoadSource(FPRClass, Op, Op->Src[0], Op->Flags);
Arg = _F80CVTTo(Arg, Width / 8);
}
if (Integer) {
Arg = LoadSource(GPRClass, Op, Op->Src[0], Op->Flags);
Arg = _F80CVTToInt(Arg, Width);
} else {
Arg = LoadSource(FPRClass, Op, Op->Src[0], Op->Flags);
Arg = _F80CVTTo(Arg, Width);
}
// top of stack is at offset zero
_F80AddValue(0, Arg);
}
void OpDispatchBuilder::FMUL(OpcodeArgs, size_t Width, bool Integer, OpDispatchBuilder::OpResult ResInST0) {
void OpDispatchBuilder::FMUL(OpcodeArgs, IR::OpSize Width, bool Integer, OpDispatchBuilder::OpResult ResInST0) {
if (Op->Src[0].IsNone()) { // Implicit argument case
auto offset = Op->OP & 7;
auto st0 = 0;
@@ -208,16 +221,15 @@ void OpDispatchBuilder::FMUL(OpcodeArgs, size_t Width, bool Integer, OpDispatchB
return;
}
LOGMAN_THROW_A_FMT(Width != OpSize::f80Bit, "No 80-bit floats from memory");
// We have one memory argument
Ref arg {};
if (Width == 16 || Width == 32 || Width == 64) {
if (Integer) {
arg = LoadSource(GPRClass, Op, Op->Src[0], Op->Flags);
arg = _F80CVTToInt(arg, Width / 8);
} else {
arg = LoadSource(FPRClass, Op, Op->Src[0], Op->Flags);
arg = _F80CVTTo(arg, Width / 8);
}
if (Integer) {
arg = LoadSource(GPRClass, Op, Op->Src[0], Op->Flags);
arg = _F80CVTToInt(arg, Width);
} else {
arg = LoadSource(FPRClass, Op, Op->Src[0], Op->Flags);
arg = _F80CVTTo(arg, Width);
}
// top of stack is at offset zero
@@ -228,11 +240,11 @@ void OpDispatchBuilder::FMUL(OpcodeArgs, size_t Width, bool Integer, OpDispatchB
}
}
void OpDispatchBuilder::FDIV(OpcodeArgs, size_t Width, bool Integer, bool Reverse, OpDispatchBuilder::OpResult ResInST0) {
void OpDispatchBuilder::FDIV(OpcodeArgs, IR::OpSize Width, bool Integer, bool Reverse, OpDispatchBuilder::OpResult ResInST0) {
if (Op->Src[0].IsNone()) {
const auto Offset = Op->OP & 7;
const auto St0 = 0;
const auto Result = (ResInST0 == OpResult::RES_STI) ? Offset : St0;
const uint8_t Offset = Op->OP & 7;
const uint8_t St0 = 0;
const uint8_t Result = (ResInST0 == OpResult::RES_STI) ? Offset : St0;
if (Reverse ^ (ResInST0 == OpResult::RES_STI)) {
_F80DivStack(Result, Offset, St0);
@@ -246,16 +258,15 @@ void OpDispatchBuilder::FDIV(OpcodeArgs, size_t Width, bool Integer, bool Revers
return;
}
LOGMAN_THROW_A_FMT(Width != OpSize::f80Bit, "No 80-bit floats from memory");
// We have one memory argument
Ref arg {};
if (Width == 16 || Width == 32 || Width == 64) {
if (Integer) {
arg = LoadSource(GPRClass, Op, Op->Src[0], Op->Flags);
arg = _F80CVTToInt(arg, Width / 8);
} else {
arg = LoadSource(FPRClass, Op, Op->Src[0], Op->Flags);
arg = _F80CVTTo(arg, Width / 8);
}
if (Integer) {
arg = LoadSource(GPRClass, Op, Op->Src[0], Op->Flags);
arg = _F80CVTToInt(arg, Width);
} else {
arg = LoadSource(FPRClass, Op, Op->Src[0], Op->Flags);
arg = _F80CVTTo(arg, Width);
}
// top of stack is at offset zero
@@ -270,7 +281,7 @@ void OpDispatchBuilder::FDIV(OpcodeArgs, size_t Width, bool Integer, bool Revers
}
}
void OpDispatchBuilder::FSUB(OpcodeArgs, size_t Width, bool Integer, bool Reverse, OpDispatchBuilder::OpResult ResInST0) {
void OpDispatchBuilder::FSUB(OpcodeArgs, IR::OpSize Width, bool Integer, bool Reverse, OpDispatchBuilder::OpResult ResInST0) {
if (Op->Src[0].IsNone()) {
const auto Offset = Op->OP & 7;
const auto St0 = 0;
@@ -288,16 +299,15 @@ void OpDispatchBuilder::FSUB(OpcodeArgs, size_t Width, bool Integer, bool Revers
return;
}
LOGMAN_THROW_A_FMT(Width != OpSize::f80Bit, "No 80-bit floats from memory");
// We have one memory argument
Ref Arg {};
if (Width == 16 || Width == 32 || Width == 64) {
if (Integer) {
Arg = LoadSource(GPRClass, Op, Op->Src[0], Op->Flags);
Arg = _F80CVTToInt(Arg, Width / 8);
} else {
Arg = LoadSource(FPRClass, Op, Op->Src[0], Op->Flags);
Arg = _F80CVTTo(Arg, Width / 8);
}
if (Integer) {
Arg = LoadSource(GPRClass, Op, Op->Src[0], Op->Flags);
Arg = _F80CVTToInt(Arg, Width);
} else {
Arg = LoadSource(FPRClass, Op, Op->Src[0], Op->Flags);
Arg = _F80CVTTo(Arg, Width);
}
// top of stack is at offset zero
@@ -348,42 +358,42 @@ void OpDispatchBuilder::X87FNSTENV(OpcodeArgs) {
// Before we store anything we need to sync our stack to the registers.
_SyncStackToSlow();
auto Size = GetDstSize(Op);
const auto Size = OpSizeFromSrc(Op);
Ref Mem = LoadSource(GPRClass, Op, Op->Dest, Op->Flags, {.LoadData = false});
Mem = AppendSegmentOffset(Mem, Op->Flags);
{
auto FCW = _LoadContext(2, GPRClass, offsetof(FEXCore::Core::CPUState, FCW));
auto FCW = _LoadContext(OpSize::i16Bit, GPRClass, offsetof(FEXCore::Core::CPUState, FCW));
_StoreMem(GPRClass, Size, Mem, FCW, Size);
}
{ _StoreMem(GPRClass, Size, ReconstructFSW_Helper(), Mem, _Constant(Size * 1), Size, MEM_OFFSET_SXTX, 1); }
{ _StoreMem(GPRClass, Size, ReconstructFSW_Helper(), Mem, _Constant(IR::OpSizeToSize(Size) * 1), Size, MEM_OFFSET_SXTX, 1); }
auto ZeroConst = _Constant(0);
{
// FTW
_StoreMem(GPRClass, Size, GetX87FTW_Helper(), Mem, _Constant(Size * 2), Size, MEM_OFFSET_SXTX, 1);
_StoreMem(GPRClass, Size, GetX87FTW_Helper(), Mem, _Constant(IR::OpSizeToSize(Size) * 2), Size, MEM_OFFSET_SXTX, 1);
}
{
// Instruction Offset
_StoreMem(GPRClass, Size, ZeroConst, Mem, _Constant(Size * 3), Size, MEM_OFFSET_SXTX, 1);
_StoreMem(GPRClass, Size, ZeroConst, Mem, _Constant(IR::OpSizeToSize(Size) * 3), Size, MEM_OFFSET_SXTX, 1);
}
{
// Instruction CS selector (+ Opcode)
_StoreMem(GPRClass, Size, ZeroConst, Mem, _Constant(Size * 4), Size, MEM_OFFSET_SXTX, 1);
_StoreMem(GPRClass, Size, ZeroConst, Mem, _Constant(IR::OpSizeToSize(Size) * 4), Size, MEM_OFFSET_SXTX, 1);
}
{
// Data pointer offset
_StoreMem(GPRClass, Size, ZeroConst, Mem, _Constant(Size * 5), Size, MEM_OFFSET_SXTX, 1);
_StoreMem(GPRClass, Size, ZeroConst, Mem, _Constant(IR::OpSizeToSize(Size) * 5), Size, MEM_OFFSET_SXTX, 1);
}
{
// Data pointer selector
_StoreMem(GPRClass, Size, ZeroConst, Mem, _Constant(Size * 6), Size, MEM_OFFSET_SXTX, 1);
_StoreMem(GPRClass, Size, ZeroConst, Mem, _Constant(IR::OpSizeToSize(Size) * 6), Size, MEM_OFFSET_SXTX, 1);
}
}
@@ -406,26 +416,27 @@ Ref OpDispatchBuilder::ReconstructX87StateFromFSW_Helper(Ref FSW) {
void OpDispatchBuilder::X87LDENV(OpcodeArgs) {
_StackForceSlow();
auto Size = GetSrcSize(Op);
const auto Size = OpSizeFromSrc(Op);
Ref Mem = LoadSource(GPRClass, Op, Op->Src[0], Op->Flags, {.LoadData = false});
Mem = AppendSegmentOffset(Mem, Op->Flags);
auto NewFCW = _LoadMem(GPRClass, 2, Mem, 2);
_StoreContext(2, GPRClass, NewFCW, offsetof(FEXCore::Core::CPUState, FCW));
auto NewFCW = _LoadMem(GPRClass, OpSize::i16Bit, Mem, OpSize::i16Bit);
_StoreContext(OpSize::i16Bit, GPRClass, NewFCW, offsetof(FEXCore::Core::CPUState, FCW));
Ref MemLocation = _Add(OpSize::i64Bit, Mem, _Constant(Size * 1));
Ref MemLocation = _Add(OpSize::i64Bit, Mem, _Constant(IR::OpSizeToSize(Size) * 1));
auto NewFSW = _LoadMem(GPRClass, Size, MemLocation, Size);
ReconstructX87StateFromFSW_Helper(NewFSW);
{
// FTW
Ref MemLocation = _Add(OpSize::i64Bit, Mem, _Constant(Size * 2));
Ref MemLocation = _Add(OpSize::i64Bit, Mem, _Constant(IR::OpSizeToSize(Size) * 2));
SetX87FTW(_LoadMem(GPRClass, Size, MemLocation, Size));
}
}
void OpDispatchBuilder::X87FNSAVE(OpcodeArgs) {
_SyncStackToSlow();
// 14 bytes for 16bit
// 2 Bytes : FCW
// 2 Bytes : FSW
@@ -444,60 +455,66 @@ void OpDispatchBuilder::X87FNSAVE(OpcodeArgs) {
// 2 bytes : Opcode
// 4 bytes : data pointer offset
// 4 bytes : data pointer selector
const auto Size = GetDstSize(Op);
const auto Size = OpSizeFromDst(Op);
Ref Mem = MakeSegmentAddress(Op, Op->Dest);
Ref Top = GetX87Top();
{
auto FCW = _LoadContext(2, GPRClass, offsetof(FEXCore::Core::CPUState, FCW));
auto FCW = _LoadContext(OpSize::i16Bit, GPRClass, offsetof(FEXCore::Core::CPUState, FCW));
_StoreMem(GPRClass, Size, Mem, FCW, Size);
}
{ _StoreMem(GPRClass, Size, ReconstructFSW_Helper(), Mem, _Constant(Size * 1), Size, MEM_OFFSET_SXTX, 1); }
{ _StoreMem(GPRClass, Size, ReconstructFSW_Helper(), Mem, _Constant(IR::OpSizeToSize(Size) * 1), Size, MEM_OFFSET_SXTX, 1); }
auto ZeroConst = _Constant(0);
{
// FTW
_StoreMem(GPRClass, Size, GetX87FTW_Helper(), Mem, _Constant(Size * 2), Size, MEM_OFFSET_SXTX, 1);
_StoreMem(GPRClass, Size, GetX87FTW_Helper(), Mem, _Constant(IR::OpSizeToSize(Size) * 2), Size, MEM_OFFSET_SXTX, 1);
}
{
// Instruction Offset
_StoreMem(GPRClass, Size, ZeroConst, Mem, _Constant(Size * 3), Size, MEM_OFFSET_SXTX, 1);
_StoreMem(GPRClass, Size, ZeroConst, Mem, _Constant(IR::OpSizeToSize(Size) * 3), Size, MEM_OFFSET_SXTX, 1);
}
{
// Instruction CS selector (+ Opcode)
_StoreMem(GPRClass, Size, ZeroConst, Mem, _Constant(Size * 4), Size, MEM_OFFSET_SXTX, 1);
_StoreMem(GPRClass, Size, ZeroConst, Mem, _Constant(IR::OpSizeToSize(Size) * 4), Size, MEM_OFFSET_SXTX, 1);
}
{
// Data pointer offset
_StoreMem(GPRClass, Size, ZeroConst, Mem, _Constant(Size * 5), Size, MEM_OFFSET_SXTX, 1);
_StoreMem(GPRClass, Size, ZeroConst, Mem, _Constant(IR::OpSizeToSize(Size) * 5), Size, MEM_OFFSET_SXTX, 1);
}
{
// Data pointer selector
_StoreMem(GPRClass, Size, ZeroConst, Mem, _Constant(Size * 6), Size, MEM_OFFSET_SXTX, 1);
_StoreMem(GPRClass, Size, ZeroConst, Mem, _Constant(IR::OpSizeToSize(Size) * 6), Size, MEM_OFFSET_SXTX, 1);
}
auto OneConst = _Constant(1);
auto SevenConst = _Constant(7);
const auto LoadSize = ReducedPrecisionMode ? OpSize::i64Bit : OpSize::i128Bit;
for (int i = 0; i < 7; ++i) {
auto data = _LoadContextIndexed(Top, 16, MMBaseOffset(), 16, FPRClass);
_StoreMem(FPRClass, 16, data, Mem, _Constant((Size * 7) + (10 * i)), 1, MEM_OFFSET_SXTX, 1);
Ref data = _LoadContextIndexed(Top, LoadSize, MMBaseOffset(), IR::OpSizeToSize(OpSize::i128Bit), FPRClass);
if (ReducedPrecisionMode) {
data = _F80CVTTo(data, OpSize::i64Bit);
}
_StoreMem(FPRClass, OpSize::i128Bit, data, Mem, _Constant((IR::OpSizeToSize(Size) * 7) + (10 * i)), OpSize::i8Bit, MEM_OFFSET_SXTX, 1);
Top = _And(OpSize::i32Bit, _Add(OpSize::i32Bit, Top, OneConst), SevenConst);
}
// The final st(7) needs a bit of special handling here
auto data = _LoadContextIndexed(Top, 16, MMBaseOffset(), 16, FPRClass);
Ref data = _LoadContextIndexed(Top, LoadSize, MMBaseOffset(), IR::OpSizeToSize(OpSize::i128Bit), FPRClass);
if (ReducedPrecisionMode) {
data = _F80CVTTo(data, OpSize::i64Bit);
}
// ST7 broken in to two parts
// Lower 64bits [63:0]
// upper 16 bits [79:64]
_StoreMem(FPRClass, 8, data, Mem, _Constant((Size * 7) + (7 * 10)), 1, MEM_OFFSET_SXTX, 1);
auto topBytes = _VDupElement(16, 2, data, 4);
_StoreMem(FPRClass, 2, topBytes, Mem, _Constant((Size * 7) + (7 * 10) + 8), 1, MEM_OFFSET_SXTX, 1);
_StoreMem(FPRClass, OpSize::i64Bit, data, Mem, _Constant((IR::OpSizeToSize(Size) * 7) + (7 * 10)), OpSize::i8Bit, MEM_OFFSET_SXTX, 1);
auto topBytes = _VDupElement(OpSize::i128Bit, OpSize::i16Bit, data, 4);
_StoreMem(FPRClass, OpSize::i16Bit, topBytes, Mem, _Constant((IR::OpSizeToSize(Size) * 7) + (7 * 10) + 8), OpSize::i8Bit, MEM_OFFSET_SXTX, 1);
// reset to default
FNINIT(Op);
@@ -505,17 +522,27 @@ void OpDispatchBuilder::X87FNSAVE(OpcodeArgs) {
void OpDispatchBuilder::X87FRSTOR(OpcodeArgs) {
_StackForceSlow();
const auto Size = GetSrcSize(Op);
const auto Size = OpSizeFromSrc(Op);
Ref Mem = MakeSegmentAddress(Op, Op->Src[0]);
auto NewFCW = _LoadMem(GPRClass, 2, Mem, 2);
_StoreContext(2, GPRClass, NewFCW, offsetof(FEXCore::Core::CPUState, FCW));
auto NewFCW = _LoadMem(GPRClass, OpSize::i16Bit, Mem, OpSize::i16Bit);
_StoreContext(OpSize::i16Bit, GPRClass, NewFCW, offsetof(FEXCore::Core::CPUState, FCW));
if (ReducedPrecisionMode) {
// ignore the rounding precision, we're always 64-bit in F64.
// extract rounding mode
Ref roundingMode = NewFCW;
auto roundShift = _Constant(10);
auto roundMask = _Constant(3);
roundingMode = _Lshr(OpSize::i32Bit, roundingMode, roundShift);
roundingMode = _And(OpSize::i32Bit, roundingMode, roundMask);
_SetRoundingMode(roundingMode, false, roundingMode);
}
auto NewFSW = _LoadMem(GPRClass, Size, Mem, _Constant(Size * 1), Size, MEM_OFFSET_SXTX, 1);
auto NewFSW = _LoadMem(GPRClass, Size, Mem, _Constant(IR::OpSizeToSize(Size) * 1), Size, MEM_OFFSET_SXTX, 1);
Ref Top = ReconstructX87StateFromFSW_Helper(NewFSW);
{
// FTW
SetX87FTW(_LoadMem(GPRClass, Size, Mem, _Constant(Size * 2), Size, MEM_OFFSET_SXTX, 1));
SetX87FTW(_LoadMem(GPRClass, Size, Mem, _Constant(IR::OpSizeToSize(Size) * 2), Size, MEM_OFFSET_SXTX, 1));
}
auto OneConst = _Constant(1);
@@ -523,15 +550,18 @@ void OpDispatchBuilder::X87FRSTOR(OpcodeArgs) {
auto low = _Constant(~0ULL);
auto high = _Constant(0xFFFF);
Ref Mask = _VCastFromGPR(16, 8, low);
Mask = _VInsGPR(16, 8, 1, Mask, high);
Ref Mask = _VCastFromGPR(OpSize::i128Bit, OpSize::i64Bit, low);
Mask = _VInsGPR(OpSize::i128Bit, OpSize::i64Bit, 1, Mask, high);
const auto StoreSize = ReducedPrecisionMode ? OpSize::i64Bit : OpSize::i128Bit;
for (int i = 0; i < 7; ++i) {
Ref Reg = _LoadMem(FPRClass, 16, Mem, _Constant((Size * 7) + (10 * i)), 1, MEM_OFFSET_SXTX, 1);
Ref Reg = _LoadMem(FPRClass, OpSize::i128Bit, Mem, _Constant((IR::OpSizeToSize(Size) * 7) + (10 * i)), OpSize::i8Bit, MEM_OFFSET_SXTX, 1);
// Mask off the top bits
Reg = _VAnd(16, 16, Reg, Mask);
_StoreContextIndexed(Reg, Top, 16, MMBaseOffset(), 16, FPRClass);
Reg = _VAnd(OpSize::i128Bit, OpSize::i128Bit, Reg, Mask);
if (ReducedPrecisionMode) {
// Convert to double precision
Reg = _F80CVT(OpSize::i64Bit, Reg);
}
_StoreContextIndexed(Reg, Top, StoreSize, MMBaseOffset(), IR::OpSizeToSize(OpSize::i128Bit), FPRClass);
Top = _And(OpSize::i32Bit, _Add(OpSize::i32Bit, Top, OneConst), SevenConst);
}
@@ -540,29 +570,31 @@ void OpDispatchBuilder::X87FRSTOR(OpcodeArgs) {
// ST7 broken in to two parts
// Lower 64bits [63:0]
// upper 16 bits [79:64]
Ref Reg = _LoadMem(FPRClass, 8, Mem, _Constant((Size * 7) + (10 * 7)), 1, MEM_OFFSET_SXTX, 1);
Ref RegHigh = _LoadMem(FPRClass, 2, Mem, _Constant((Size * 7) + (10 * 7) + 8), 1, MEM_OFFSET_SXTX, 1);
Reg = _VInsElement(16, 2, 4, 0, Reg, RegHigh);
_StoreContextIndexed(Reg, Top, 16, MMBaseOffset(), 16, FPRClass);
Ref Reg = _LoadMem(FPRClass, OpSize::i64Bit, Mem, _Constant((IR::OpSizeToSize(Size) * 7) + (10 * 7)), OpSize::i8Bit, MEM_OFFSET_SXTX, 1);
Ref RegHigh =
_LoadMem(FPRClass, OpSize::i16Bit, Mem, _Constant((IR::OpSizeToSize(Size) * 7) + (10 * 7) + 8), OpSize::i8Bit, MEM_OFFSET_SXTX, 1);
Reg = _VInsElement(OpSize::i128Bit, OpSize::i16Bit, 4, 0, Reg, RegHigh);
if (ReducedPrecisionMode) {
Reg = _F80CVT(OpSize::i64Bit, Reg); // Convert to double precision
}
_StoreContextIndexed(Reg, Top, StoreSize, MMBaseOffset(), IR::OpSizeToSize(OpSize::i128Bit), FPRClass);
}
// Load / Store Control Word
void OpDispatchBuilder::X87FSTCW(OpcodeArgs) {
auto FCW = _LoadContext(2, GPRClass, offsetof(FEXCore::Core::CPUState, FCW));
StoreResult(GPRClass, Op, FCW, -1);
auto FCW = _LoadContext(OpSize::i16Bit, GPRClass, offsetof(FEXCore::Core::CPUState, FCW));
StoreResult(GPRClass, Op, FCW, OpSize::iInvalid);
}
void OpDispatchBuilder::X87FLDCW(OpcodeArgs) {
// FIXME: Because loading control flags will affect several instructions in fast path, we might have
// to switch for now to slow mode whenever these are manually changed.
// Remove the next line and try DF_04.asm in fast path.
_StackForceSlow();
Ref NewFCW = LoadSource(GPRClass, Op, Op->Src[0], Op->Flags);
_StoreContext(2, GPRClass, NewFCW, offsetof(FEXCore::Core::CPUState, FCW));
_StoreContext(OpSize::i16Bit, GPRClass, NewFCW, offsetof(FEXCore::Core::CPUState, FCW));
}
void OpDispatchBuilder::FXCH(OpcodeArgs) {
uint8_t Offset = Op->OP & 7;
// fxch st0, st0 is for us essentially a nop
@@ -575,15 +607,15 @@ void OpDispatchBuilder::FXCH(OpcodeArgs) {
void OpDispatchBuilder::X87FYL2X(OpcodeArgs, bool IsFYL2XP1) {
if (IsFYL2XP1) {
// create an add between top of stack and 1.
Ref One = ReducedPrecisionMode ? _VCastFromGPR(8, 8, _Constant(0x3FF0000000000000)) :
LoadAndCacheNamedVectorConstant(16, NamedVectorConstant::NAMED_VECTOR_X87_ONE);
Ref One = ReducedPrecisionMode ? _VCastFromGPR(OpSize::i64Bit, OpSize::i64Bit, _Constant(0x3FF0000000000000)) :
LoadAndCacheNamedVectorConstant(OpSize::i128Bit, NamedVectorConstant::NAMED_VECTOR_X87_ONE);
_F80AddValue(0, One);
}
_F80FYL2XStack();
}
void OpDispatchBuilder::FCOMI(OpcodeArgs, size_t Width, bool Integer, OpDispatchBuilder::FCOMIFlags WhichFlags, bool PopTwice) {
void OpDispatchBuilder::FCOMI(OpcodeArgs, IR::OpSize Width, bool Integer, OpDispatchBuilder::FCOMIFlags WhichFlags, bool PopTwice) {
Ref arg {};
Ref b {};
@@ -593,15 +625,17 @@ void OpDispatchBuilder::FCOMI(OpcodeArgs, size_t Width, bool Integer, OpDispatch
uint8_t Offset = Op->OP & 7;
Res = _F80CmpStack(Offset);
} else {
// Memory arg
if (Width == 16 || Width == 32 || Width == 64) {
if (Width == OpSize::i16Bit || Width == OpSize::i32Bit || Width == OpSize::i64Bit) {
// Memory arg
if (Integer) {
arg = LoadSource(GPRClass, Op, Op->Src[0], Op->Flags);
b = _F80CVTToInt(arg, Width / 8);
b = _F80CVTToInt(arg, Width);
} else {
arg = LoadSource(FPRClass, Op, Op->Src[0], Op->Flags);
b = _F80CVTTo(arg, Width / 8);
b = _F80CVTTo(arg, Width);
}
} else {
FEX_UNREACHABLE;
}
Res = _F80CmpValue(b);
}
@@ -618,10 +652,7 @@ void OpDispatchBuilder::FCOMI(OpcodeArgs, size_t Width, bool Integer, OpDispatch
SetRFLAG<FEXCore::X86State::X87FLAG_C2_LOC>(HostFlag_Unordered);
SetRFLAG<FEXCore::X86State::X87FLAG_C3_LOC>(HostFlag_ZF);
} else {
// Invalidate deferred flags early
// OF, SF, AF, PF all undefined
InvalidateDeferredFlags();
SetCFDirect(HostFlag_CF);
SetRFLAG<FEXCore::X86State::RFLAG_ZF_RAW_LOC>(HostFlag_ZF);
@@ -681,7 +712,6 @@ void OpDispatchBuilder::X87ModifySTP(OpcodeArgs, bool Inc) {
// Optionally we can pass a pre calculated value for Top, otherwise we calculate it
// during the function runtime.
Ref OpDispatchBuilder::ReconstructFSW_Helper(Ref T) {
// Start with the top value
auto Top = T ? T : GetX87Top();
Ref FSW = _Lshl(OpSize::i64Bit, Top, _Constant(11));
@@ -706,18 +736,21 @@ Ref OpDispatchBuilder::ReconstructFSW_Helper(Ref T) {
// There's no load Status Word instruction but you can load it through frstor
// or fldenv.
void OpDispatchBuilder::X87FNSTSW(OpcodeArgs) {
Ref TopValue = _SyncStackToSlow();
Ref StatusWord = ReconstructFSW_Helper(TopValue);
StoreResult(GPRClass, Op, StatusWord, -1);
StoreResult(GPRClass, Op, StatusWord, OpSize::iInvalid);
}
void OpDispatchBuilder::FNINIT(OpcodeArgs) {
auto Zero = _Constant(0);
if (ReducedPrecisionMode) {
_SetRoundingMode(Zero, false, Zero);
}
// Init FCW to 0x037F
auto NewFCW = _Constant(16, 0x037F);
_StoreContext(2, GPRClass, NewFCW, offsetof(FEXCore::Core::CPUState, FCW));
auto NewFCW = _Constant(OpSize::i16Bit, 0x037F);
_StoreContext(OpSize::i16Bit, GPRClass, NewFCW, offsetof(FEXCore::Core::CPUState, FCW));
// Set top to zero
SetX87Top(Zero);
@@ -734,13 +767,11 @@ void OpDispatchBuilder::FNINIT(OpcodeArgs) {
}
void OpDispatchBuilder::X87FFREE(OpcodeArgs) {
_InvalidateStack(Op->OP & 7);
}
void OpDispatchBuilder::X87EMMS(OpcodeArgs) {
// Tags all get set to 0b11
_InvalidateStack(0xff);
}
@@ -782,13 +813,14 @@ void OpDispatchBuilder::X87FCMOV(OpcodeArgs) {
auto AllOneConst = _Constant(0xffff'ffff'ffff'ffffull);
Ref SrcCond = SelectCC(CC, OpSize::i64Bit, AllOneConst, ZeroConst);
Ref VecCond = _VDupFromGPR(16, 8, SrcCond);
_F80VBSLStack(16, VecCond, Op->OP & 7, 0);
Ref VecCond = _VDupFromGPR(OpSize::i128Bit, OpSize::i64Bit, SrcCond);
_F80VBSLStack(OpSize::i128Bit, VecCond, Op->OP & 7, 0);
}
void OpDispatchBuilder::X87FXAM(OpcodeArgs) {
auto a = _ReadStackValue(0);
Ref Result = ReducedPrecisionMode ? _VExtractToGPR(8, 8, a, 0) : _VExtractToGPR(16, 8, a, 1);
Ref Result =
ReducedPrecisionMode ? _VExtractToGPR(OpSize::i64Bit, OpSize::i64Bit, a, 0) : _VExtractToGPR(OpSize::i128Bit, OpSize::i64Bit, a, 1);
// Extract the sign bit
Result = ReducedPrecisionMode ? _Bfe(OpSize::i64Bit, 1, 63, Result) : _Bfe(OpSize::i64Bit, 1, 15, Result);
@@ -810,4 +842,14 @@ void OpDispatchBuilder::X87FXAM(OpcodeArgs) {
SetRFLAG<FEXCore::X86State::X87FLAG_C3_LOC>(C3);
}
void OpDispatchBuilder::X87FXTRACT(OpcodeArgs) {
auto Top = _ReadStackValue(0);
_PopStackDestroy();
auto Exp = _F80XTRACT_EXP(Top);
auto Sig = _F80XTRACT_SIG(Top);
_PushStack(Exp, Exp, OpSize::f80Bit, true);
_PushStack(Sig, Sig, OpSize::f80Bit, true);
}
} // namespace FEXCore::IR
@@ -8,6 +8,7 @@ $end_info$
#include "Interface/Core/OpcodeDispatcher.h"
#include "Interface/Core/X86Tables/X86Tables.h"
#include "Interface/IR/IR.h"
#include <FEXCore/Core/CoreState.h>
#include <FEXCore/Core/X86Enums.h>
@@ -22,58 +23,28 @@ class OrderedNode;
#define OpcodeArgs [[maybe_unused]] FEXCore::X86Tables::DecodedOp Op
// Functions in X87.cpp (no change required)
// GetX87Top
// SetX87ValidTag
// GetX87ValidTag
// GetX87Tag (will need changing once special tag handling is implemented)
// SetX87FTW
// GetX87FTW (will need changing once special tag handling is implemented)
// SetX87Top
// X87ModifySTP
// EMMS
// FFREE
// FNSTENV
// FSTCW
// LDSW
// FNSTSW
// FXCH
// FCMOV
// FST(register to register)
// FCHS
void OpDispatchBuilder::FNINITF64(OpcodeArgs) {
// Init host rounding mode to zero
auto Zero = _Constant(0);
_SetRoundingMode(Zero, false, Zero);
// Call generic version
FNINIT(Op);
}
void OpDispatchBuilder::X87LDENVF64(OpcodeArgs) {
_StackForceSlow();
const auto Size = GetSrcSize(Op);
const auto Size = OpSizeFromSrc(Op);
Ref Mem = MakeSegmentAddress(Op, Op->Src[0]);
auto NewFCW = _LoadMem(GPRClass, 2, Mem, 2);
auto NewFCW = _LoadMem(GPRClass, OpSize::i16Bit, Mem, OpSize::i16Bit);
// ignore the rounding precision, we're always 64-bit in F64.
// extract rounding mode
Ref roundingMode = _Bfe(OpSize::i32Bit, 3, 10, NewFCW);
_SetRoundingMode(roundingMode, false, roundingMode);
_StoreContext(2, GPRClass, NewFCW, offsetof(FEXCore::Core::CPUState, FCW));
_StoreContext(OpSize::i16Bit, GPRClass, NewFCW, offsetof(FEXCore::Core::CPUState, FCW));
auto NewFSW = _LoadMem(GPRClass, Size, Mem, _Constant(Size * 1), Size, MEM_OFFSET_SXTX, 1);
auto NewFSW = _LoadMem(GPRClass, Size, Mem, _Constant(IR::OpSizeToSize(Size)), Size, MEM_OFFSET_SXTX, 1);
ReconstructX87StateFromFSW_Helper(NewFSW);
{
// FTW
SetX87FTW(_LoadMem(GPRClass, Size, Mem, _Constant(Size * 2), Size, MEM_OFFSET_SXTX, 1));
SetX87FTW(_LoadMem(GPRClass, Size, Mem, _Constant(IR::OpSizeToSize(Size) * 2), Size, MEM_OFFSET_SXTX, 1));
}
}
void OpDispatchBuilder::X87FLDCWF64(OpcodeArgs) {
_StackForceSlow();
@@ -82,82 +53,94 @@ void OpDispatchBuilder::X87FLDCWF64(OpcodeArgs) {
// extract rounding mode
Ref roundingMode = _Bfe(OpSize::i32Bit, 3, 10, NewFCW);
_SetRoundingMode(roundingMode, false, roundingMode);
_StoreContext(2, GPRClass, NewFCW, offsetof(FEXCore::Core::CPUState, FCW));
_StoreContext(OpSize::i16Bit, GPRClass, NewFCW, offsetof(FEXCore::Core::CPUState, FCW));
}
// F64 ops
// Float load op with memory operand
void OpDispatchBuilder::FLDF64(OpcodeArgs, size_t Width) {
size_t ReadWidth = (Width == 80) ? 16 : Width / 8;
void OpDispatchBuilder::FLDF64(OpcodeArgs, IR::OpSize Width) {
const auto ReadWidth = (Width == OpSize::f80Bit) ? OpSize::i128Bit : Width;
Ref Data = LoadSource_WithOpSize(FPRClass, Op, Op->Src[0], ReadWidth, Op->Flags);
// Convert to 64bit float
Ref ConvertedData = Data;
if (Width == 32) {
ConvertedData = _Float_FToF(8, 4, Data);
} else if (Width == 80) {
ConvertedData = _F80CVT(8, Data);
if (Width == OpSize::i32Bit) {
ConvertedData = _Float_FToF(OpSize::i64Bit, OpSize::i32Bit, Data);
} else if (Width == OpSize::f80Bit) {
ConvertedData = _F80CVT(OpSize::i64Bit, Data);
}
_PushStack(ConvertedData, Data, ReadWidth, true);
}
void OpDispatchBuilder::FBLDF64(OpcodeArgs) {
// Read from memory
Ref Data = LoadSource_WithOpSize(FPRClass, Op, Op->Src[0], 16, Op->Flags);
Ref Data = LoadSource_WithOpSize(FPRClass, Op, Op->Src[0], OpSize::i128Bit, Op->Flags);
Ref ConvertedData = _F80BCDLoad(Data);
ConvertedData = _F80CVT(8, ConvertedData);
_PushStack(ConvertedData, Data, 8, true);
ConvertedData = _F80CVT(OpSize::i64Bit, ConvertedData);
_PushStack(ConvertedData, Data, OpSize::i64Bit, true);
}
void OpDispatchBuilder::FBSTPF64(OpcodeArgs) {
Ref converted = _F80CVTTo(_ReadStackValue(0), 8);
Ref converted = _F80CVTTo(_ReadStackValue(0), OpSize::i64Bit);
converted = _F80BCDStore(converted);
StoreResult_WithOpSize(FPRClass, Op, Op->Dest, converted, 10, 1);
StoreResult_WithOpSize(FPRClass, Op, Op->Dest, converted, OpSize::f80Bit, OpSize::i8Bit);
_PopStackDestroy();
}
void OpDispatchBuilder::FLDF64_Const(OpcodeArgs, uint64_t Num) {
auto Data = _VCastFromGPR(8, 8, _Constant(Num));
_PushStack(Data, Data, 8, true);
auto Data = _VCastFromGPR(OpSize::i64Bit, OpSize::i64Bit, _Constant(Num));
_PushStack(Data, Data, OpSize::i64Bit, true);
}
void OpDispatchBuilder::FILDF64(OpcodeArgs) {
size_t ReadWidth = GetSrcSize(Op);
const auto ReadWidth = OpSizeFromSrc(Op);
// Read from memory
Ref Data = LoadSource_WithOpSize(GPRClass, Op, Op->Src[0], ReadWidth, Op->Flags);
if (ReadWidth == 2) {
Data = _Sbfe(OpSize::i64Bit, ReadWidth * 8, 0, Data);
if (ReadWidth == OpSize::i16Bit) {
Data = _Sbfe(OpSize::i64Bit, IR::OpSizeAsBits(ReadWidth), 0, Data);
}
auto ConvertedData = _Float_FromGPR_S(8, ReadWidth == 4 ? 4 : 8, Data);
auto ConvertedData = _Float_FromGPR_S(OpSize::i64Bit, ReadWidth == OpSize::i32Bit ? OpSize::i32Bit : OpSize::i64Bit, Data);
_PushStack(ConvertedData, Data, ReadWidth, false);
}
void OpDispatchBuilder::FSTF64(OpcodeArgs, size_t Width) {
Ref Mem = LoadSource(GPRClass, Op, Op->Dest, Op->Flags, {.LoadData = false});
_StoreStackMemory(Mem, OpSize::i64Bit, true, Width / 8);
void OpDispatchBuilder::FSTF64(OpcodeArgs, IR::OpSize Width) {
AddressMode A = DecodeAddress(Op, Op->Dest, MemoryAccessType::DEFAULT, false);
// Index scale is a power of 2?
LOGMAN_THROW_A_FMT(A.IndexScale > 0 && (A.IndexScale & (A.IndexScale - 1)) == 0, "Invalid index scale");
Ref Addr = A.Base ? A.Base : _Constant(0);
if (A.Index) {
Ref ScaledIndex = A.Index;
if (A.IndexScale > 1) {
ScaledIndex = _Lshl(A.AddrSize, ScaledIndex, _Constant(std::log2(A.IndexScale)));
}
Addr = _Add(A.AddrSize, Addr, ScaledIndex);
}
_StoreStackMem(OpSize::i64Bit, Width, Addr, _Constant(A.Offset), /*Float=*/true);
if (Op->TableInfo->Flags & X86Tables::InstFlags::FLAGS_POP) {
_PopStackDestroy();
}
}
void OpDispatchBuilder::FISTF64(OpcodeArgs, bool Truncate) {
auto Size = GetSrcSize(Op);
const auto Size = OpSizeFromSrc(Op);
Ref data = _ReadStackValue(0);
if (Truncate) {
data = _Float_ToGPR_ZS(Size == 4 ? 4 : 8, 8, data);
data = _Float_ToGPR_ZS(Size == OpSize::i32Bit ? OpSize::i32Bit : OpSize::i64Bit, OpSize::i64Bit, data);
} else {
data = _Float_ToGPR_S(Size == 4 ? 4 : 8, 8, data);
data = _Float_ToGPR_S(Size == OpSize::i32Bit ? OpSize::i32Bit : OpSize::i64Bit, OpSize::i64Bit, data);
}
StoreResult_WithOpSize(GPRClass, Op, Op->Dest, data, Size, 1);
StoreResult_WithOpSize(GPRClass, Op, Op->Dest, data, Size, OpSize::i8Bit);
if ((Op->TableInfo->Flags & X86Tables::InstFlags::FLAGS_POP) != 0) {
_PopStackDestroy();
}
}
void OpDispatchBuilder::FADDF64(OpcodeArgs, size_t Width, bool Integer, OpDispatchBuilder::OpResult ResInST0) {
void OpDispatchBuilder::FADDF64(OpcodeArgs, IR::OpSize Width, bool Integer, OpDispatchBuilder::OpResult ResInST0) {
if (Op->Src[0].IsNone()) { // Implicit argument case
auto Offset = Op->OP & 7;
auto St0 = 0;
@@ -177,15 +160,17 @@ void OpDispatchBuilder::FADDF64(OpcodeArgs, size_t Width, bool Integer, OpDispat
if (Integer) {
arg = LoadSource(GPRClass, Op, Op->Src[0], Op->Flags);
if (Width == 16) {
if (Width == OpSize::i16Bit) {
arg = _Sbfe(OpSize::i64Bit, 16, 0, arg);
}
arg = _Float_FromGPR_S(8, Width == 64 ? 8 : 4, arg);
} else if (Width == 32) {
arg = _Float_FromGPR_S(OpSize::i64Bit, Width == OpSize::i64Bit ? OpSize::i64Bit : OpSize::i32Bit, arg);
} else if (Width == OpSize::i32Bit) {
arg = LoadSource(FPRClass, Op, Op->Src[0], Op->Flags);
arg = _Float_FToF(8, 4, arg);
} else if (Width == 64) {
arg = _Float_FToF(OpSize::i64Bit, OpSize::i32Bit, arg);
} else if (Width == OpSize::i64Bit) {
arg = LoadSource(FPRClass, Op, Op->Src[0], Op->Flags);
} else {
FEX_UNREACHABLE;
}
// top of stack is at offset zero
@@ -193,7 +178,7 @@ void OpDispatchBuilder::FADDF64(OpcodeArgs, size_t Width, bool Integer, OpDispat
}
// FIXME: following is very similar to FADDF64
void OpDispatchBuilder::FMULF64(OpcodeArgs, size_t Width, bool Integer, OpDispatchBuilder::OpResult ResInST0) {
void OpDispatchBuilder::FMULF64(OpcodeArgs, IR::OpSize Width, bool Integer, OpDispatchBuilder::OpResult ResInST0) {
if (Op->Src[0].IsNone()) { // Implicit argument case
auto offset = Op->OP & 7;
auto st0 = 0;
@@ -213,15 +198,17 @@ void OpDispatchBuilder::FMULF64(OpcodeArgs, size_t Width, bool Integer, OpDispat
if (Integer) {
arg = LoadSource(GPRClass, Op, Op->Src[0], Op->Flags);
if (Width == 16) {
if (Width == OpSize::i16Bit) {
arg = _Sbfe(OpSize::i64Bit, 16, 0, arg);
}
arg = _Float_FromGPR_S(8, Width == 64 ? 8 : 4, arg);
} else if (Width == 32) {
arg = _Float_FromGPR_S(OpSize::i64Bit, Width == OpSize::i64Bit ? OpSize::i64Bit : OpSize::i32Bit, arg);
} else if (Width == OpSize::i32Bit) {
arg = LoadSource(FPRClass, Op, Op->Src[0], Op->Flags);
arg = _Float_FToF(8, 4, arg);
} else if (Width == 64) {
arg = _Float_FToF(OpSize::i64Bit, OpSize::i32Bit, arg);
} else if (Width == OpSize::i64Bit) {
arg = LoadSource(FPRClass, Op, Op->Src[0], Op->Flags);
} else {
FEX_UNREACHABLE;
}
// top of stack is at offset zero
@@ -232,7 +219,7 @@ void OpDispatchBuilder::FMULF64(OpcodeArgs, size_t Width, bool Integer, OpDispat
}
}
void OpDispatchBuilder::FDIVF64(OpcodeArgs, size_t Width, bool Integer, bool Reverse, OpDispatchBuilder::OpResult ResInST0) {
void OpDispatchBuilder::FDIVF64(OpcodeArgs, IR::OpSize Width, bool Integer, bool Reverse, OpDispatchBuilder::OpResult ResInST0) {
if (Op->Src[0].IsNone()) {
const auto offset = Op->OP & 7;
const auto st0 = 0;
@@ -260,19 +247,21 @@ void OpDispatchBuilder::FDIVF64(OpcodeArgs, size_t Width, bool Integer, bool Rev
// We have one memory argument
Ref Arg {};
if (Width == 16 || Width == 32 || Width == 64) {
if (Width == OpSize::i16Bit || Width == OpSize::i32Bit || Width == OpSize::i64Bit) {
if (Integer) {
Arg = LoadSource(GPRClass, Op, Op->Src[0], Op->Flags);
if (Width == 16) {
if (Width == OpSize::i16Bit) {
Arg = _Sbfe(OpSize::i64Bit, 16, 0, Arg);
}
Arg = _Float_FromGPR_S(8, Width == 64 ? 8 : 4, Arg);
} else if (Width == 32) {
Arg = _Float_FromGPR_S(OpSize::i64Bit, Width == OpSize::i64Bit ? OpSize::i64Bit : OpSize::i32Bit, Arg);
} else if (Width == OpSize::i32Bit) {
Arg = LoadSource(FPRClass, Op, Op->Src[0], Op->Flags);
Arg = _Float_FToF(8, 4, Arg);
} else if (Width == 64) {
Arg = _Float_FToF(OpSize::i64Bit, OpSize::i32Bit, Arg);
} else if (Width == OpSize::i64Bit) {
Arg = LoadSource(FPRClass, Op, Op->Src[0], Op->Flags);
}
} else {
FEX_UNREACHABLE;
}
// top of stack is at offset zero
@@ -287,7 +276,7 @@ void OpDispatchBuilder::FDIVF64(OpcodeArgs, size_t Width, bool Integer, bool Rev
}
}
void OpDispatchBuilder::FSUBF64(OpcodeArgs, size_t Width, bool Integer, bool Reverse, OpDispatchBuilder::OpResult ResInST0) {
void OpDispatchBuilder::FSUBF64(OpcodeArgs, IR::OpSize Width, bool Integer, bool Reverse, OpDispatchBuilder::OpResult ResInST0) {
if (Op->Src[0].IsNone()) {
const auto Offset = Op->OP & 7;
const auto St0 = 0;
@@ -315,19 +304,21 @@ void OpDispatchBuilder::FSUBF64(OpcodeArgs, size_t Width, bool Integer, bool Rev
// We have one memory argument
Ref arg {};
if (Width == 16 || Width == 32 || Width == 64) {
if (Width == OpSize::i16Bit || Width == OpSize::i32Bit || Width == OpSize::i64Bit) {
if (Integer) {
arg = LoadSource(GPRClass, Op, Op->Src[0], Op->Flags);
if (Width == 16) {
if (Width == OpSize::i16Bit) {
arg = _Sbfe(OpSize::i64Bit, 16, 0, arg);
}
arg = _Float_FromGPR_S(8, Width == 64 ? 8 : 4, arg);
} else if (Width == 32) {
arg = _Float_FromGPR_S(OpSize::i64Bit, Width == OpSize::i64Bit ? OpSize::i64Bit : OpSize::i32Bit, arg);
} else if (Width == OpSize::i32Bit) {
arg = LoadSource(FPRClass, Op, Op->Src[0], Op->Flags);
arg = _Float_FToF(8, 4, arg);
} else if (Width == 64) {
arg = _Float_FToF(OpSize::i64Bit, OpSize::i32Bit, arg);
} else if (Width == OpSize::i64Bit) {
arg = LoadSource(FPRClass, Op, Op->Src[0], Op->Flags);
}
} else {
FEX_UNREACHABLE;
}
// top of stack is at offset zero
@@ -348,11 +339,10 @@ void OpDispatchBuilder::FTSTF64(OpcodeArgs) {
// Now we do our comparison.
_F80StackTest(0);
PossiblySetNZCVBits = ~0;
ConvertNZCVToX87();
}
void OpDispatchBuilder::FCOMIF64(OpcodeArgs, size_t Width, bool Integer, OpDispatchBuilder::FCOMIFlags WhichFlags, bool PopTwice) {
void OpDispatchBuilder::FCOMIF64(OpcodeArgs, IR::OpSize Width, bool Integer, OpDispatchBuilder::FCOMIFlags WhichFlags, bool PopTwice) {
Ref arg {};
Ref b {};
@@ -360,22 +350,22 @@ void OpDispatchBuilder::FCOMIF64(OpcodeArgs, size_t Width, bool Integer, OpDispa
// Implicit arg
uint8_t offset = Op->OP & 7;
b = _ReadStackValue(offset);
} else {
} else if (Width == OpSize::i16Bit || Width == OpSize::i32Bit || Width == OpSize::i64Bit) {
// Memory arg
if (Width == 16 || Width == 32 || Width == 64) {
if (Integer) {
arg = LoadSource(GPRClass, Op, Op->Src[0], Op->Flags);
if (Width == 16) {
arg = _Sbfe(OpSize::i64Bit, 16, 0, arg);
}
b = _Float_FromGPR_S(8, Width == 64 ? 8 : 4, arg);
} else if (Width == 32) {
arg = LoadSource(FPRClass, Op, Op->Src[0], Op->Flags);
b = _Float_FToF(8, 4, arg);
} else if (Width == 64) {
b = LoadSource(FPRClass, Op, Op->Src[0], Op->Flags);
if (Integer) {
arg = LoadSource(GPRClass, Op, Op->Src[0], Op->Flags);
if (Width == OpSize::i16Bit) {
arg = _Sbfe(OpSize::i64Bit, 16, 0, arg);
}
b = _Float_FromGPR_S(OpSize::i64Bit, Width == OpSize::i64Bit ? OpSize::i64Bit : OpSize::i32Bit, arg);
} else if (Width == OpSize::i32Bit) {
arg = LoadSource(FPRClass, Op, Op->Src[0], Op->Flags);
b = _Float_FToF(OpSize::i64Bit, OpSize::i32Bit, arg);
} else if (Width == OpSize::i64Bit) {
b = LoadSource(FPRClass, Op, Op->Src[0], Op->Flags);
}
} else {
FEX_UNREACHABLE;
}
if (WhichFlags == FCOMIFlags::FLAGS_X87) {
@@ -383,7 +373,6 @@ void OpDispatchBuilder::FCOMIF64(OpcodeArgs, size_t Width, bool Integer, OpDispa
GetNZCV();
_F80CmpValue(b);
PossiblySetNZCVBits = ~0;
ConvertNZCVToX87();
} else {
HandleNZCVWrite();
@@ -399,144 +388,37 @@ void OpDispatchBuilder::FCOMIF64(OpcodeArgs, size_t Width, bool Integer, OpDispa
}
}
// This function converts to F80 on save for compatibility
void OpDispatchBuilder::X87FNSAVEF64(OpcodeArgs) {
_SyncStackToSlow();
// 14 bytes for 16bit
// 2 Bytes : FCW
// 2 Bytes : FSW
// 2 bytes : FTW
// 2 bytes : Instruction offset
// 2 bytes : Instruction CS selector
// 2 bytes : Data offset
// 2 bytes : Data selector
void OpDispatchBuilder::X87FXTRACTF64(OpcodeArgs) {
// Split node into SIG and EXP while handling the special zero case.
// i.e. if val == 0.0, then sig = 0.0, exp = -inf
// if val == -0.0, then sig = -0.0, exp = -inf
// otherwise we just extract the 64-bit sig and exp as normal.
Ref Node = _ReadStackValue(0);
// 28 bytes for 32bit
// 4 bytes : FCW
// 4 bytes : FSW
// 4 bytes : FTW
// 4 bytes : Instruction pointer
// 2 bytes : instruction pointer selector
// 2 bytes : Opcode
// 4 bytes : data pointer offset
// 4 bytes : data pointer selector
Ref Gpr = _VExtractToGPR(OpSize::i64Bit, OpSize::i64Bit, Node, 0);
const auto Size = GetDstSize(Op);
Ref Mem = MakeSegmentAddress(Op, Op->Dest);
Ref Top = GetX87Top();
{
auto FCW = _LoadContext(2, GPRClass, offsetof(FEXCore::Core::CPUState, FCW));
_StoreMem(GPRClass, Size, Mem, FCW, Size);
}
// zero case
Ref ExpZV = _VCastFromGPR(OpSize::i64Bit, OpSize::i64Bit, _Constant(0xfff0'0000'0000'0000UL));
Ref SigZV = Node;
{ _StoreMem(GPRClass, Size, ReconstructFSW_Helper(), Mem, _Constant(Size * 1), Size, MEM_OFFSET_SXTX, 1); }
// non zero case
Ref ExpNZ = _Bfe(OpSize::i64Bit, 11, 52, Gpr);
ExpNZ = _Sub(OpSize::i64Bit, ExpNZ, _Constant(1023));
Ref ExpNZV = _Float_FromGPR_S(OpSize::i64Bit, OpSize::i64Bit, ExpNZ);
auto ZeroConst = _Constant(0);
Ref SigNZ = _And(OpSize::i64Bit, Gpr, _Constant(0x800f'ffff'ffff'ffffLL));
SigNZ = _Or(OpSize::i64Bit, SigNZ, _Constant(0x3ff0'0000'0000'0000LL));
Ref SigNZV = _VCastFromGPR(OpSize::i64Bit, OpSize::i64Bit, SigNZ);
{
// FTW
_StoreMem(GPRClass, Size, GetX87FTW_Helper(), Mem, _Constant(Size * 2), Size, MEM_OFFSET_SXTX, 1);
}
// Comparison and select to push onto stack
SaveNZCV();
_TestNZ(OpSize::i64Bit, Gpr, _Constant(0x7fff'ffff'ffff'ffffUL));
{
// Instruction Offset
_StoreMem(GPRClass, Size, ZeroConst, Mem, _Constant(Size * 3), Size, MEM_OFFSET_SXTX, 1);
}
Ref Sig = _NZCVSelectV(OpSize::i64Bit, {COND_EQ}, SigZV, SigNZV);
Ref Exp = _NZCVSelectV(OpSize::i64Bit, {COND_EQ}, ExpZV, ExpNZV);
{
// Instruction CS selector (+ Opcode)
_StoreMem(GPRClass, Size, ZeroConst, Mem, _Constant(Size * 4), Size, MEM_OFFSET_SXTX, 1);
}
{
// Data pointer offset
_StoreMem(GPRClass, Size, ZeroConst, Mem, _Constant(Size * 5), Size, MEM_OFFSET_SXTX, 1);
}
{
// Data pointer selector
_StoreMem(GPRClass, Size, ZeroConst, Mem, _Constant(Size * 6), Size, MEM_OFFSET_SXTX, 1);
}
auto OneConst = _Constant(1);
auto SevenConst = _Constant(7);
for (int i = 0; i < 7; ++i) {
Ref data = _LoadContextIndexed(Top, 8, MMBaseOffset(), 16, FPRClass);
data = _F80CVTTo(data, 8);
_StoreMem(FPRClass, 16, data, Mem, _Constant((Size * 7) + (i * 10)), 1, MEM_OFFSET_SXTX, 1);
Top = _And(OpSize::i32Bit, _Add(OpSize::i32Bit, Top, OneConst), SevenConst);
}
// The final st(7) needs a bit of special handling here
Ref data = _LoadContextIndexed(Top, 8, MMBaseOffset(), 16, FPRClass);
data = _F80CVTTo(data, 8);
// ST7 broken in to two parts
// Lower 64bits [63:0]
// upper 16 bits [79:64]
_StoreMem(FPRClass, 8, data, Mem, _Constant((Size * 7) + (7 * 10)), 1, MEM_OFFSET_SXTX, 1);
auto topBytes = _VDupElement(16, 2, data, 4);
_StoreMem(FPRClass, 2, topBytes, Mem, _Constant((Size * 7) + (7 * 10) + 8), 1, MEM_OFFSET_SXTX, 1);
// reset to default
FNINITF64(Op);
_PopStackDestroy();
_PushStack(Exp, Exp, OpSize::i64Bit, true);
_PushStack(Sig, Sig, OpSize::i64Bit, true);
}
// This function converts from F80 on load for compatibility
void OpDispatchBuilder::X87FRSTORF64(OpcodeArgs) {
_StackForceSlow();
const auto Size = GetSrcSize(Op);
Ref Mem = MakeSegmentAddress(Op, Op->Src[0]);
auto NewFCW = _LoadMem(GPRClass, 2, Mem, 2);
// ignore the rounding precision, we're always 64-bit in F64.
// extract rounding mode
Ref roundingMode = NewFCW;
auto roundShift = _Constant(10);
auto roundMask = _Constant(3);
roundingMode = _Lshr(OpSize::i32Bit, roundingMode, roundShift);
roundingMode = _And(OpSize::i32Bit, roundingMode, roundMask);
_SetRoundingMode(roundingMode, false, roundingMode);
_StoreContext(2, GPRClass, NewFCW, offsetof(FEXCore::Core::CPUState, FCW));
_StoreContext(2, GPRClass, NewFCW, offsetof(FEXCore::Core::CPUState, FCW));
auto NewFSW = _LoadMem(GPRClass, Size, Mem, _Constant(Size * 1), Size, MEM_OFFSET_SXTX, 1);
Ref Top = ReconstructX87StateFromFSW_Helper(NewFSW);
{
// FTW
SetX87FTW(_LoadMem(GPRClass, Size, Mem, _Constant(Size * 2), Size, MEM_OFFSET_SXTX, 1));
}
auto OneConst = _Constant(1);
auto SevenConst = _Constant(7);
auto low = _Constant(~0ULL);
auto high = _Constant(0xFFFF);
Ref Mask = _VCastFromGPR(16, 8, low);
Mask = _VInsGPR(16, 8, 1, Mask, high);
for (int i = 0; i < 7; ++i) {
Ref Reg = _LoadMem(FPRClass, 16, Mem, _Constant((Size * 7) + (i * 10)), 1, MEM_OFFSET_SXTX, 1);
// Mask off the top bits
Reg = _VAnd(16, 16, Reg, Mask);
// Convert to double precision
Reg = _F80CVT(8, Reg);
_StoreContextIndexed(Reg, Top, 8, MMBaseOffset(), 16, FPRClass);
Top = _And(OpSize::i32Bit, _Add(OpSize::i32Bit, Top, OneConst), SevenConst);
}
// The final st(7) needs a bit of special handling here
// ST7 broken in to two parts
// Lower 64bits [63:0]
// upper 16 bits [79:64]
Ref Reg = _LoadMem(FPRClass, 8, Mem, _Constant((Size * 7) + (7 * 10)), 1, MEM_OFFSET_SXTX, 1);
Ref RegHigh = _LoadMem(FPRClass, 2, Mem, _Constant((Size * 7) + (7 * 10) + 8), 1, MEM_OFFSET_SXTX, 1);
Reg = _VInsElement(16, 2, 4, 0, Reg, RegHigh);
Reg = _F80CVT(8, Reg); // Convert to double precision
_StoreContextIndexed(Reg, Top, 8, MMBaseOffset(), 16, FPRClass);
}
} // namespace FEXCore::IR
@@ -14,12 +14,14 @@ namespace FEXCore::X86Tables {
void InitializeBaseTables(Context::OperatingMode Mode);
void InitializeSecondaryTables(Context::OperatingMode Mode);
void InitializeSecondaryGroupTables(Context::OperatingMode Mode);
void InitializePrimaryGroupTables(Context::OperatingMode Mode);
void InitializeH0F3ATables(Context::OperatingMode Mode);
void InitializeInfoTables(Context::OperatingMode Mode) {
InitializeBaseTables(Mode);
InitializeSecondaryTables(Mode);
InitializeSecondaryGroupTables(Mode);
InitializePrimaryGroupTables(Mode);
InitializeH0F3ATables(Mode);
}
@@ -6,6 +6,7 @@ $end_info$
*/
#include "Interface/Core/X86Tables/X86Tables.h"
#include "Interface/Core/OpcodeDispatcher/BaseTables.h"
#include <FEXCore/Core/Context.h>
@@ -144,7 +145,7 @@ std::array<X86InstInfo, MAX_PRIMARY_TABLE_SIZE> BaseOps = []() consteval {
// These three are all X87 instructions
{0x9B, 1, X86InstInfo{"FWAIT", TYPE_INST, FLAGS_NONE, 0, nullptr}},
{0x9C, 1, X86InstInfo{"PUSHF", TYPE_INST, GenFlagsSameSize(SIZE_64BITDEF), 0, nullptr}},
{0x9D, 1, X86InstInfo{"POPF", TYPE_INST, GenFlagsSameSize(SIZE_64BITDEF), 0, nullptr}},
{0x9D, 1, X86InstInfo{"POPF", TYPE_INST, GenFlagsSameSize(SIZE_64BITDEF) | FLAGS_BLOCK_END, 0, nullptr}},
{0x9E, 1, X86InstInfo{"SAHF", TYPE_INST, FLAGS_NONE, 0, nullptr}},
{0x9F, 1, X86InstInfo{"LAHF", TYPE_INST, FLAGS_NONE, 0, nullptr}},
@@ -236,6 +237,7 @@ std::array<X86InstInfo, MAX_PRIMARY_TABLE_SIZE> BaseOps = []() consteval {
};
GenerateTable(&Table.at(0), BaseOpTable, std::size(BaseOpTable));
IR::InstallToTable(Table, IR::OpDispatch_BaseOpTable);
return Table;
}();
@@ -301,9 +303,11 @@ void InitializeBaseTables(Context::OperatingMode Mode) {
if (Mode == Context::MODE_64BIT) {
GenerateTable(&BaseOps.at(0), BaseOpTable_64, std::size(BaseOpTable_64));
IR::InstallToTable(BaseOps, IR::OpDispatch_BaseOpTable_64);
}
else {
GenerateTable(&BaseOps.at(0), BaseOpTable_32, std::size(BaseOpTable_32));
IR::InstallToTable(BaseOps, IR::OpDispatch_BaseOpTable_32);
}
}
}
@@ -6,6 +6,7 @@ $end_info$
*/
#include "Interface/Core/X86Tables/X86Tables.h"
#include "Interface/Core/OpcodeDispatcher/DDDTables.h"
#include <iterator>
@@ -54,6 +55,8 @@ std::array<X86InstInfo, MAX_3DNOW_TABLE_SIZE> DDDNowOps = []() consteval {
};
GenerateTable(&Table.at(0), DDDNowOpTable, std::size(DDDNowOpTable));
IR::InstallToTable(Table, IR::OpDispatch_DDDTable);
return Table;
}();
@@ -1,37 +0,0 @@
// SPDX-License-Identifier: MIT
/*
$info$
tags: frontend|x86-tables
$end_info$
*/
#include "Interface/Core/X86Tables/X86Tables.h"
#include <iterator>
namespace FEXCore::X86Tables {
using namespace InstFlags;
std::array<X86InstInfo, MAX_EVEX_TABLE_SIZE> EVEXTableOps = []() consteval {
std::array<X86InstInfo, MAX_EVEX_TABLE_SIZE> Table{};
constexpr U16U8InfoStruct EVEXTable[] = {
{0x10, 1, X86InstInfo{"VMOVUPS", TYPE_INST, FLAGS_MODRM | FLAGS_XMM_FLAGS, 0, nullptr}},
{0x11, 1, X86InstInfo{"VMOVUPS", TYPE_INST, FLAGS_MODRM | FLAGS_SF_MOD_DST | FLAGS_XMM_FLAGS, 0, nullptr}},
{0x18, 1, X86InstInfo{"VBROADCASTSS", TYPE_INST, FLAGS_MODRM | FLAGS_XMM_FLAGS, 0, nullptr}},
{0x19, 1, X86InstInfo{"VBROADCASTD", TYPE_INST, FLAGS_MODRM | FLAGS_XMM_FLAGS, 0, nullptr}},
{0x1A, 1, X86InstInfo{"VBROADCASTSD", TYPE_INST, FLAGS_MODRM | FLAGS_XMM_FLAGS, 0, nullptr}},
{0x1B, 1, X86InstInfo{"VBROADCASTF64X4", TYPE_INST, FLAGS_MODRM | FLAGS_XMM_FLAGS, 0, nullptr}},
{0x28, 1, X86InstInfo{"VMOVAPS", TYPE_INST, FLAGS_MODRM | FLAGS_XMM_FLAGS, 0, nullptr}},
{0x29, 1, X86InstInfo{"VMOVAPS", TYPE_INST, FLAGS_MODRM | FLAGS_SF_MOD_DST | FLAGS_XMM_FLAGS, 0, nullptr}},
{0x59, 1, X86InstInfo{"VBROADCASTQ", TYPE_INST, FLAGS_MODRM | FLAGS_SF_MOD_DST | FLAGS_XMM_FLAGS, 0, nullptr}},
{0x6F, 1, X86InstInfo{"VMOVDQU64", TYPE_INST, FLAGS_MODRM | FLAGS_XMM_FLAGS, 0, nullptr}},
{0x73, 1, X86InstInfo{"VPSLLDQ", TYPE_INST, FLAGS_MODRM | FLAGS_SF_MOD_DST | FLAGS_XMM_FLAGS, 0, nullptr}},
{0x7F, 1, X86InstInfo{"VMOVDQU64", TYPE_INST, FLAGS_MODRM | FLAGS_SF_MOD_DST | FLAGS_XMM_FLAGS, 0, nullptr}},
{0xE7, 1, X86InstInfo{"VMOVNTDQ", TYPE_INST, FLAGS_MODRM | FLAGS_SF_MOD_DST | FLAGS_XMM_FLAGS, 0, nullptr}},
};
GenerateTable(&Table.at(0), EVEXTable, std::size(EVEXTable));
return Table;
}();
}
@@ -6,6 +6,7 @@ $end_info$
*/
#include "Interface/Core/X86Tables/X86Tables.h"
#include "Interface/Core/OpcodeDispatcher/H0F38Tables.h"
#include <iterator>
#include <stdint.h>
@@ -119,6 +120,8 @@ std::array<X86InstInfo, MAX_0F_38_TABLE_SIZE> H0F38TableOps = []() consteval {
#undef OPD
GenerateTable(&Table.at(0), H0F38Table, std::size(H0F38Table));
IR::InstallToTable(Table, IR::OpDispatch_H0F38Table);
return Table;
}();
@@ -6,6 +6,7 @@ $end_info$
*/
#include "Interface/Core/X86Tables/X86Tables.h"
#include "Interface/Core/OpcodeDispatcher/H0F3ATables.h"
#include <FEXCore/Core/Context.h>
@@ -20,47 +21,60 @@ constexpr uint16_t PF_3A_66 = 1;
std::array<X86InstInfo, MAX_0F_3A_TABLE_SIZE> H0F3ATableOps = []() consteval {
std::array<X86InstInfo, MAX_0F_3A_TABLE_SIZE> Table{};
constexpr U16U8InfoStruct H0F3ATable[] = {
{OPD(0, PF_3A_NONE, 0x0F), 1, X86InstInfo{"PALIGNR", TYPE_INST, GenFlagsSameSize(SIZE_64BIT) | FLAGS_MODRM | FLAGS_XMM_FLAGS | FLAGS_SF_MMX, 1, nullptr}},
{OPD(0, PF_3A_66, 0x08), 1, X86InstInfo{"ROUNDPS", TYPE_INST, GenFlagsSameSize(SIZE_128BIT) | FLAGS_MODRM | FLAGS_XMM_FLAGS, 1, nullptr}},
{OPD(0, PF_3A_66, 0x09), 1, X86InstInfo{"ROUNDPD", TYPE_INST, GenFlagsSameSize(SIZE_128BIT) | FLAGS_MODRM | FLAGS_XMM_FLAGS, 1, nullptr}},
{OPD(0, PF_3A_66, 0x0A), 1, X86InstInfo{"ROUNDSS", TYPE_INST, GenFlagsSizes(SIZE_128BIT, SIZE_32BIT) | FLAGS_MODRM | FLAGS_XMM_FLAGS, 1, nullptr}},
{OPD(0, PF_3A_66, 0x0B), 1, X86InstInfo{"ROUNDSD", TYPE_INST, GenFlagsSizes(SIZE_128BIT, SIZE_64BIT) | FLAGS_MODRM | FLAGS_XMM_FLAGS, 1, nullptr}},
{OPD(0, PF_3A_66, 0x0C), 1, X86InstInfo{"BLENDPS", TYPE_INST, GenFlagsSameSize(SIZE_128BIT) | FLAGS_MODRM | FLAGS_XMM_FLAGS, 1, nullptr}},
{OPD(0, PF_3A_66, 0x0D), 1, X86InstInfo{"BLENDPD", TYPE_INST, GenFlagsSameSize(SIZE_128BIT) | FLAGS_MODRM | FLAGS_XMM_FLAGS, 1, nullptr}},
{OPD(0, PF_3A_66, 0x0E), 1, X86InstInfo{"PBLENDW", TYPE_INST, GenFlagsSameSize(SIZE_128BIT) | FLAGS_MODRM | FLAGS_XMM_FLAGS, 1, nullptr}},
{OPD(0, PF_3A_66, 0x0F), 1, X86InstInfo{"PALIGNR", TYPE_INST, GenFlagsSameSize(SIZE_128BIT) | FLAGS_MODRM | FLAGS_XMM_FLAGS, 1, nullptr}},
auto TableGen = []<uint16_t REX>() consteval {
constexpr U16U8InfoStruct Table[] = {
{OPD(REX, PF_3A_NONE, 0x0F), 1, X86InstInfo{"PALIGNR", TYPE_INST, GenFlagsSameSize(SIZE_64BIT) | FLAGS_MODRM | FLAGS_XMM_FLAGS | FLAGS_SF_MMX, 1, nullptr}},
{OPD(REX, PF_3A_66, 0x08), 1, X86InstInfo{"ROUNDPS", TYPE_INST, GenFlagsSameSize(SIZE_128BIT) | FLAGS_MODRM | FLAGS_XMM_FLAGS, 1, nullptr}},
{OPD(REX, PF_3A_66, 0x09), 1, X86InstInfo{"ROUNDPD", TYPE_INST, GenFlagsSameSize(SIZE_128BIT) | FLAGS_MODRM | FLAGS_XMM_FLAGS, 1, nullptr}},
{OPD(REX, PF_3A_66, 0x0A), 1, X86InstInfo{"ROUNDSS", TYPE_INST, GenFlagsSizes(SIZE_128BIT, SIZE_32BIT) | FLAGS_MODRM | FLAGS_XMM_FLAGS, 1, nullptr}},
{OPD(REX, PF_3A_66, 0x0B), 1, X86InstInfo{"ROUNDSD", TYPE_INST, GenFlagsSizes(SIZE_128BIT, SIZE_64BIT) | FLAGS_MODRM | FLAGS_XMM_FLAGS, 1, nullptr}},
{OPD(REX, PF_3A_66, 0x0C), 1, X86InstInfo{"BLENDPS", TYPE_INST, GenFlagsSameSize(SIZE_128BIT) | FLAGS_MODRM | FLAGS_XMM_FLAGS, 1, nullptr}},
{OPD(REX, PF_3A_66, 0x0D), 1, X86InstInfo{"BLENDPD", TYPE_INST, GenFlagsSameSize(SIZE_128BIT) | FLAGS_MODRM | FLAGS_XMM_FLAGS, 1, nullptr}},
{OPD(REX, PF_3A_66, 0x0E), 1, X86InstInfo{"PBLENDW", TYPE_INST, GenFlagsSameSize(SIZE_128BIT) | FLAGS_MODRM | FLAGS_XMM_FLAGS, 1, nullptr}},
{OPD(REX, PF_3A_66, 0x0F), 1, X86InstInfo{"PALIGNR", TYPE_INST, GenFlagsSameSize(SIZE_128BIT) | FLAGS_MODRM | FLAGS_XMM_FLAGS, 1, nullptr}},
{OPD(0, PF_3A_66, 0x14), 1, X86InstInfo{"PEXTRB", TYPE_INST, GenFlagsSizes(SIZE_32BIT, SIZE_128BIT) | FLAGS_MODRM | FLAGS_SF_MOD_DST | FLAGS_SF_DST_GPR | FLAGS_XMM_FLAGS, 1, nullptr}},
{OPD(0, PF_3A_66, 0x15), 1, X86InstInfo{"PEXTRW", TYPE_INST, GenFlagsSizes(SIZE_16BIT, SIZE_128BIT) | FLAGS_MODRM | FLAGS_SF_MOD_DST | FLAGS_SF_DST_GPR | FLAGS_XMM_FLAGS, 1, nullptr}},
{OPD(0, PF_3A_66, 0x16), 1, X86InstInfo{"PEXTRD", TYPE_INST, GenFlagsSizes(SIZE_32BIT, SIZE_128BIT) | FLAGS_MODRM | FLAGS_SF_MOD_DST | FLAGS_SF_DST_GPR | FLAGS_XMM_FLAGS, 1, nullptr}},
{OPD(0, PF_3A_66, 0x17), 1, X86InstInfo{"EXTRACTPS", TYPE_INST, GenFlagsSizes(SIZE_32BIT, SIZE_128BIT) | FLAGS_MODRM | FLAGS_SF_MOD_DST | FLAGS_SF_DST_GPR | FLAGS_XMM_FLAGS, 1, nullptr}},
{OPD(REX, PF_3A_66, 0x14), 1, X86InstInfo{"PEXTRB", TYPE_INST, GenFlagsSizes(SIZE_32BIT, SIZE_128BIT) | FLAGS_MODRM | FLAGS_SF_MOD_DST | FLAGS_SF_DST_GPR | FLAGS_XMM_FLAGS, 1, nullptr}},
{OPD(REX, PF_3A_66, 0x15), 1, X86InstInfo{"PEXTRW", TYPE_INST, GenFlagsSizes(SIZE_16BIT, SIZE_128BIT) | FLAGS_MODRM | FLAGS_SF_MOD_DST | FLAGS_SF_DST_GPR | FLAGS_XMM_FLAGS, 1, nullptr}},
{OPD(REX, PF_3A_66, 0x17), 1, X86InstInfo{"EXTRACTPS", TYPE_INST, GenFlagsSizes(SIZE_32BIT, SIZE_128BIT) | FLAGS_MODRM | FLAGS_SF_MOD_DST | FLAGS_SF_DST_GPR | FLAGS_XMM_FLAGS, 1, nullptr}},
{OPD(0, PF_3A_66, 0x20), 1, X86InstInfo{"PINSRB", TYPE_INST, GenFlagsDstSize(SIZE_128BIT) | FLAGS_MODRM | FLAGS_XMM_FLAGS | FLAGS_SF_SRC_GPR, 1, nullptr}},
{OPD(0, PF_3A_66, 0x21), 1, X86InstInfo{"INSERTPS", TYPE_INST, GenFlagsSameSize(SIZE_128BIT) | FLAGS_MODRM | FLAGS_XMM_FLAGS, 1, nullptr}},
{OPD(0, PF_3A_66, 0x22), 1, X86InstInfo{"PINSRD", TYPE_INST, GenFlagsSizes(SIZE_128BIT, SIZE_32BIT) | FLAGS_MODRM | FLAGS_XMM_FLAGS | FLAGS_SF_SRC_GPR, 1, nullptr}},
{OPD(0, PF_3A_66, 0x40), 1, X86InstInfo{"DPPS", TYPE_INST, GenFlagsSameSize(SIZE_128BIT) | FLAGS_MODRM | FLAGS_XMM_FLAGS, 1, nullptr}},
{OPD(0, PF_3A_66, 0x41), 1, X86InstInfo{"DPPD", TYPE_INST, GenFlagsSameSize(SIZE_128BIT) | FLAGS_MODRM | FLAGS_XMM_FLAGS, 1, nullptr}},
{OPD(0, PF_3A_66, 0x42), 1, X86InstInfo{"MPSADBW", TYPE_INST, GenFlagsSameSize(SIZE_128BIT) | FLAGS_MODRM | FLAGS_XMM_FLAGS, 1, nullptr}},
{OPD(0, PF_3A_66, 0x44), 1, X86InstInfo{"PCLMULQDQ", TYPE_INST, GenFlagsSameSize(SIZE_128BIT) | FLAGS_MODRM | FLAGS_XMM_FLAGS, 1, nullptr}},
{OPD(REX, PF_3A_66, 0x20), 1, X86InstInfo{"PINSRB", TYPE_INST, GenFlagsDstSize(SIZE_128BIT) | FLAGS_MODRM | FLAGS_XMM_FLAGS | FLAGS_SF_SRC_GPR, 1, nullptr}},
{OPD(REX, PF_3A_66, 0x21), 1, X86InstInfo{"INSERTPS", TYPE_INST, GenFlagsSameSize(SIZE_128BIT) | FLAGS_MODRM | FLAGS_XMM_FLAGS, 1, nullptr}},
{OPD(REX, PF_3A_66, 0x40), 1, X86InstInfo{"DPPS", TYPE_INST, GenFlagsSameSize(SIZE_128BIT) | FLAGS_MODRM | FLAGS_XMM_FLAGS, 1, nullptr}},
{OPD(REX, PF_3A_66, 0x41), 1, X86InstInfo{"DPPD", TYPE_INST, GenFlagsSameSize(SIZE_128BIT) | FLAGS_MODRM | FLAGS_XMM_FLAGS, 1, nullptr}},
{OPD(REX, PF_3A_66, 0x42), 1, X86InstInfo{"MPSADBW", TYPE_INST, GenFlagsSameSize(SIZE_128BIT) | FLAGS_MODRM | FLAGS_XMM_FLAGS, 1, nullptr}},
{OPD(REX, PF_3A_66, 0x44), 1, X86InstInfo{"PCLMULQDQ", TYPE_INST, GenFlagsSameSize(SIZE_128BIT) | FLAGS_MODRM | FLAGS_XMM_FLAGS, 1, nullptr}},
{OPD(0, PF_3A_66, 0x60), 1, X86InstInfo{"PCMPESTRM", TYPE_INST, GenFlagsSizes(SIZE_128BIT, SIZE_32BIT) | FLAGS_MODRM | FLAGS_XMM_FLAGS, 1, nullptr}},
{OPD(0, PF_3A_66, 0x61), 1, X86InstInfo{"PCMPESTRI", TYPE_INST, GenFlagsSizes(SIZE_128BIT, SIZE_32BIT) | FLAGS_MODRM | FLAGS_XMM_FLAGS, 1, nullptr}},
{OPD(0, PF_3A_66, 0x62), 1, X86InstInfo{"PCMPISTRM", TYPE_INST, GenFlagsSameSize(SIZE_128BIT) | FLAGS_MODRM | FLAGS_XMM_FLAGS, 1, nullptr}},
{OPD(0, PF_3A_66, 0x63), 1, X86InstInfo{"PCMPISTRI", TYPE_INST, GenFlagsSameSize(SIZE_128BIT) | FLAGS_MODRM | FLAGS_XMM_FLAGS, 1, nullptr}},
{OPD(REX, PF_3A_66, 0x60), 1, X86InstInfo{"PCMPESTRM", TYPE_INST, GenFlagsSizes(SIZE_128BIT, SIZE_32BIT) | FLAGS_MODRM | FLAGS_XMM_FLAGS, 1, nullptr}},
{OPD(REX, PF_3A_66, 0x61), 1, X86InstInfo{"PCMPESTRI", TYPE_INST, GenFlagsSizes(SIZE_128BIT, SIZE_32BIT) | FLAGS_MODRM | FLAGS_XMM_FLAGS, 1, nullptr}},
{OPD(REX, PF_3A_66, 0x62), 1, X86InstInfo{"PCMPISTRM", TYPE_INST, GenFlagsSameSize(SIZE_128BIT) | FLAGS_MODRM | FLAGS_XMM_FLAGS, 1, nullptr}},
{OPD(REX, PF_3A_66, 0x63), 1, X86InstInfo{"PCMPISTRI", TYPE_INST, GenFlagsSameSize(SIZE_128BIT) | FLAGS_MODRM | FLAGS_XMM_FLAGS, 1, nullptr}},
{OPD(0, PF_3A_NONE, 0xCC), 1, X86InstInfo{"SHA1RNDS4", TYPE_INST, GenFlagsSameSize(SIZE_128BIT) | FLAGS_MODRM | FLAGS_XMM_FLAGS, 1, nullptr}},
{OPD(REX, PF_3A_NONE, 0xCC), 1, X86InstInfo{"SHA1RNDS4", TYPE_INST, GenFlagsSameSize(SIZE_128BIT) | FLAGS_MODRM | FLAGS_XMM_FLAGS, 1, nullptr}},
{OPD(0, PF_3A_66, 0xDF), 1, X86InstInfo{"AESKEYGENASSIST", TYPE_INST, GenFlagsSameSize(SIZE_128BIT) | FLAGS_MODRM | FLAGS_XMM_FLAGS, 1, nullptr}},
{OPD(REX, PF_3A_66, 0xDF), 1, X86InstInfo{"AESKEYGENASSIST", TYPE_INST, GenFlagsSameSize(SIZE_128BIT) | FLAGS_MODRM | FLAGS_XMM_FLAGS, 1, nullptr}},
};
return std::to_array(Table);
};
constexpr auto H0F3ATable_IgnoresREX0 = TableGen.template operator()<0>();
constexpr auto H0F3ATable_IgnoresREX1 = TableGen.template operator()<1>();
GenerateTable(&Table.at(0), &H0F3ATable_IgnoresREX0.at(0), H0F3ATable_IgnoresREX0.size());
GenerateTable(&Table.at(0), &H0F3ATable_IgnoresREX1.at(0), H0F3ATable_IgnoresREX1.size());
constexpr U16U8InfoStruct TableNeedsREX[] = {
{OPD(0, PF_3A_66, 0x16), 1, X86InstInfo{"PEXTRD", TYPE_INST, GenFlagsSizes(SIZE_32BIT, SIZE_128BIT) | FLAGS_MODRM | FLAGS_SF_MOD_DST | FLAGS_SF_DST_GPR | FLAGS_XMM_FLAGS, 1, nullptr}},
{OPD(0, PF_3A_66, 0x22), 1, X86InstInfo{"PINSRD", TYPE_INST, GenFlagsSizes(SIZE_128BIT, SIZE_32BIT) | FLAGS_MODRM | FLAGS_XMM_FLAGS | FLAGS_SF_SRC_GPR, 1, nullptr}},
};
GenerateTable(&Table.at(0), TableNeedsREX, std::size(TableNeedsREX));
IR::InstallToTable(Table, IR::OpDispatch_H0F3ATableIgnoreREX);
IR::InstallToTable(Table, IR::OpDispatch_H0F3ATableNeedsREX0);
GenerateTable(&Table.at(0), H0F3ATable, std::size(H0F3ATable));
return Table;
}();
void InitializeH0F3ATables(Context::OperatingMode Mode) {
static constexpr U16U8InfoStruct H0F3ATable_64[] = {
{OPD(1, PF_3A_66, 0x0F), 1, X86InstInfo{"PALIGNR", TYPE_INST, GenFlagsSameSize(SIZE_128BIT) | FLAGS_MODRM | FLAGS_XMM_FLAGS, 1, nullptr}},
{OPD(1, PF_3A_66, 0x16), 1, X86InstInfo{"PEXTRQ", TYPE_INST, GenFlagsSizes(SIZE_64BIT, SIZE_128BIT) | FLAGS_MODRM | FLAGS_SF_MOD_DST | FLAGS_SF_DST_GPR | FLAGS_XMM_FLAGS, 1, nullptr}},
{OPD(1, PF_3A_66, 0x22), 1, X86InstInfo{"PINSRQ", TYPE_INST, GenFlagsSizes(SIZE_128BIT, SIZE_64BIT) | FLAGS_MODRM | FLAGS_XMM_FLAGS | FLAGS_SF_SRC_GPR, 1, nullptr}},
};
@@ -69,6 +83,7 @@ void InitializeH0F3ATables(Context::OperatingMode Mode) {
if (Mode == Context::MODE_64BIT) {
GenerateTable(&H0F3ATableOps.at(0), H0F3ATable_64, std::size(H0F3ATable_64));
IR::InstallToTable(H0F3ATableOps, IR::OpDispatch_H0F3ATable_64);
}
}
}
@@ -6,6 +6,7 @@ $end_info$
*/
#include "Interface/Core/X86Tables/X86Tables.h"
#include "Interface/Core/OpcodeDispatcher/PrimaryGroupTables.h"
#include <FEXCore/Core/Context.h>
@@ -144,6 +145,8 @@ std::array<X86InstInfo, MAX_INST_GROUP_TABLE_SIZE> PrimaryInstGroupOps = []() co
};
GenerateTable(&Table.at(0), PrimaryGroupOpTable, std::size(PrimaryGroupOpTable));
IR::InstallToTable(Table, IR::OpDispatch_PrimaryGroupTables);
return Table;
}();
@@ -6,6 +6,7 @@ $end_info$
*/
#include "Interface/Core/X86Tables/X86Tables.h"
#include "Interface/Core/OpcodeDispatcher/SecondaryGroupTables.h"
#include <iterator>
#include <stdint.h>
@@ -488,7 +489,15 @@ std::array<X86InstInfo, MAX_INST_SECOND_GROUP_TABLE_SIZE> SecondInstGroupOps = [
#undef OPD
GenerateTable(&Table.at(0), SecondaryExtensionOpTable, std::size(SecondaryExtensionOpTable));
IR::InstallToTable(Table, IR::OpDispatch_SecondaryGroupTables);
return Table;
}();
void InitializeSecondaryGroupTables(Context::OperatingMode Mode) {
if (Mode == Context::MODE_64BIT) {
IR::InstallToTable(SecondInstGroupOps, IR::OpDispatch_SecondaryGroupTables_64);
}
}
}
@@ -6,6 +6,7 @@ $end_info$
*/
#include "Interface/Core/X86Tables/X86Tables.h"
#include "Interface/Core/OpcodeDispatcher/SecondaryModRMTables.h"
#include <iterator>
@@ -56,6 +57,8 @@ std::array<X86InstInfo, MAX_SECOND_MODRM_TABLE_SIZE> SecondModRMTableOps = []()
};
GenerateTable(&Table.at(0), SecondaryModRMExtensionOpTable, std::size(SecondaryModRMExtensionOpTable));
IR::InstallToTable(Table, IR::OpDispatch_SecondaryModRMTables);
return Table;
}();
@@ -6,6 +6,7 @@ $end_info$
*/
#include "Interface/Core/X86Tables/X86Tables.h"
#include "Interface/Core/OpcodeDispatcher/SecondaryTables.h"
#include <FEXCore/Core/Context.h>
@@ -270,6 +271,8 @@ auto BaseOpsLambda = []() consteval {
GenerateTable(&Table.at(0), TwoByteOpTable, std::size(TwoByteOpTable));
IR::InstallToTable(Table, IR::OpDispatch_TwoByteOpTable);
return Table;
};
@@ -297,7 +300,7 @@ std::array<X86InstInfo, MAX_REP_MOD_TABLE_SIZE> RepModOps = []() consteval {
{0x2E, 2, X86InstInfo{"", TYPE_INVALID, FLAGS_NONE, 0, nullptr}},
{0x30, 16, X86InstInfo{"", TYPE_COPY_OTHER, FLAGS_NONE, 0, nullptr}},
{0x40, 16, X86InstInfo{"", TYPE_COPY_OTHER, FLAGS_NONE, 0, nullptr}},
{0x40, 16, X86InstInfo{"", TYPE_INVALID, FLAGS_NONE, 0, nullptr}},
{0x50, 1, X86InstInfo{"", TYPE_INVALID, FLAGS_NONE, 0, nullptr}},
{0x51, 1, X86InstInfo{"SQRTSS", TYPE_INST, GenFlagsSizes(SIZE_128BIT, SIZE_32BIT) | FLAGS_MODRM | FLAGS_XMM_FLAGS, 0, nullptr}},
@@ -359,6 +362,7 @@ std::array<X86InstInfo, MAX_REP_MOD_TABLE_SIZE> RepModOps = []() consteval {
GenerateTableWithCopy(&Table.at(0), RepModOpTable, std::size(RepModOpTable), &BaseOpsLambda().at(0));
IR::InstallToTable(Table, IR::OpDispatch_SecondaryRepModTables);
return Table;
}();
@@ -383,7 +387,7 @@ std::array<X86InstInfo, MAX_REPNE_MOD_TABLE_SIZE> RepNEModOps = []() consteval {
{0x2E, 2, X86InstInfo{"", TYPE_INVALID, FLAGS_NONE, 0, nullptr}},
{0x30, 16, X86InstInfo{"", TYPE_COPY_OTHER, FLAGS_NONE, 0, nullptr}},
{0x40, 16, X86InstInfo{"", TYPE_COPY_OTHER, FLAGS_NONE, 0, nullptr}},
{0x40, 16, X86InstInfo{"", TYPE_INVALID, FLAGS_NONE, 0, nullptr}},
{0x50, 1, X86InstInfo{"", TYPE_INVALID, FLAGS_NONE, 0, nullptr}},
{0x51, 1, X86InstInfo{"SQRTSD", TYPE_INST, GenFlagsSizes(SIZE_128BIT, SIZE_64BIT) | FLAGS_MODRM | FLAGS_XMM_FLAGS, 0, nullptr}},
@@ -440,6 +444,7 @@ std::array<X86InstInfo, MAX_REPNE_MOD_TABLE_SIZE> RepNEModOps = []() consteval {
GenerateTableWithCopy(&Table.at(0), RepNEModOpTable, std::size(RepNEModOpTable), &BaseOpsLambda().at(0));
IR::InstallToTable(Table, IR::OpDispatch_SecondaryRepNEModTables);
return Table;
}();
@@ -595,6 +600,7 @@ std::array<X86InstInfo, MAX_OPSIZE_MOD_TABLE_SIZE> OpSizeModOps = []() consteval
GenerateTableWithCopy(&Table.at(0), OpSizeModOpTable, std::size(OpSizeModOpTable), &BaseOpsLambda().at(0));
IR::InstallToTable(Table, IR::OpDispatch_SecondaryOpSizeModTables);
return Table;
}();
@@ -620,12 +626,16 @@ void InitializeSecondaryTables(Context::OperatingMode Mode) {
LateInitCopyTable(&RepModOps.at(0), TwoByteOpTable_64, std::size(TwoByteOpTable_64));
LateInitCopyTable(&RepNEModOps.at(0), TwoByteOpTable_64, std::size(TwoByteOpTable_64));
LateInitCopyTable(&OpSizeModOps.at(0), TwoByteOpTable_64, std::size(TwoByteOpTable_64));
IR::InstallToTable(SecondBaseOps, IR::OpDispatch_TwoByteOpTable_64);
}
else {
LateInitCopyTable(&SecondBaseOps.at(0), TwoByteOpTable_32, std::size(TwoByteOpTable_32));
LateInitCopyTable(&RepModOps.at(0), TwoByteOpTable_32, std::size(TwoByteOpTable_32));
LateInitCopyTable(&RepNEModOps.at(0), TwoByteOpTable_32, std::size(TwoByteOpTable_32));
LateInitCopyTable(&OpSizeModOps.at(0), TwoByteOpTable_32, std::size(TwoByteOpTable_32));
IR::InstallToTable(SecondBaseOps, IR::OpDispatch_TwoByteOpTable_32);
}
}
Loaded 100 of 1381 files, more files were not shown because too many files have changed in this diff. Show more