Compare commits

...
420 Commits
Author SHA1 Message Date
Ryan Houdek a408749eef Docs: Update for release FEX-2203 2022-03-06 04:49:31 -08:00
Ryan Houdek d8a3687ac3 Merge pull request #1604 from Sonicadvance1/fix_cmpxchg_66h
OpcodeDispatcher: Fixes CMPXCHG8B/16B with 66h/72h/73h prefix
2022-03-06 04:10:58 -08:00
Ryan Houdek a32c7f6ce2 unittests: Adds cmpxchg unit tests for prefixes 2022-03-06 03:54:45 -08:00
Ryan Houdek 540feb857b OpcodeDispatcher: Fixes CMPXCHG8B/16B with 66h/72h/73h prefix
The documentation is incorrect about this instruction. It claims that
you use 66h prefix to choose between operating at 8B or 16B.
This is incorrect, real hardware only responds to REX.W for choosing the
operating size. These other prefixes are ignored but is still accepted as
an instruction decoding.
2022-03-06 03:54:45 -08:00
Ryan Houdek a3902a0d2d FEXCore: Fixes usage of GPRPair in operations
These were working around the previous quirks by accident
2022-03-06 03:54:45 -08:00
Ryan Houdek 5fbd01536f IR: Fixes some GPRPair IR op definitions
These were always wrong but how it the operations were handled meant
that it happened to work even though the IR representation was broken
2022-03-06 03:54:45 -08:00
Ryan Houdek bebcab0277 Merge pull request #1603 from Sonicadvance1/rng_support
FEXCore: Adds support for RDRAND/RDSEED
2022-03-06 03:54:20 -08:00
Ryan Houdek d0f17d400e unittests: Adds RDRAND/RDSEED unit tests 2022-03-06 03:40:25 -08:00
Ryan Houdek 11a07eb3f2 FEXCore: Adds support for RDRAND/RDSEED
This matches the AArch64 implementation fairly well.
Bundles RDRAND and RDSEED together for simplification, both instructions
are a single flag on AArch64.
2022-03-06 03:40:25 -08:00
Ryan Houdek cb13e1bdb9 X86Tables: Fixes secondary group decoding
If we're hitting these group tables then it needs the ignore overlay extension
since the prefixes are used to select ops inside this table.
2022-03-06 03:40:25 -08:00
Ryan Houdek 3e8c6d0be0 Merge pull request #1601 from Sonicadvance1/new_ir_json
IR: New IR JSON format
2022-03-06 03:40:05 -08:00
Ryan Houdek 847b6e5026 LinuxSyscalls: Fixes struct verifier on Ubuntu 20.04
The `linux/types.h` header needs to be included before the rest
otherwise we are missing types.
2022-03-04 17:44:26 -08:00
Ryan Houdek 7f6e9d3dae unittests/IR: Resolves fallout from recent JSON changes 2022-03-04 16:39:11 -08:00
Ryan Houdek e244e8f66b x86Jit: Resolves fallout from recent JSON changes 2022-03-04 16:39:11 -08:00
Ryan Houdek 0adaa85d97 ArmJit: Resolves fallout from recent JSON changes 2022-03-04 16:39:11 -08:00
Ryan Houdek f1d34c7407 Interpreter: Resolves fallout from recent JSON changes 2022-03-04 16:39:11 -08:00
Ryan Houdek 7a9492ceca IRPasses: Resolves fallout from recent JSON changes 2022-03-04 16:39:11 -08:00
Ryan Houdek 39192cd092 OpcodeDispatcher: Resolves fallout from recent JSON changes
Also fixes a bug in FCVTIntTo where it was getting passed an FPR when it
wants a GPR. Would cause RA problems on the JITs
2022-03-04 05:10:45 -08:00
Ryan Houdek 73abf9b9dd IRParser: Resolves fallout from recent JSON changes 2022-03-04 04:07:25 -08:00
Ryan Houdek 0387c24e21 IREmitter: Resolves the fallout from recent JSON changes 2022-03-04 04:07:25 -08:00
Ryan Houdek 9d14bc7846 IR: Updates generators and json to new format
This greatly simplifies the IR format by using string parsing for
gathering the information.
Tons of redundant information removed.
Significantly more difficult to mess up adding a new IR op.

Significantly improves the generator functions in IREmitter
2022-03-04 03:51:44 -08:00
Ryan Houdek ac32ecadbe Merge pull request #1597 from Sonicadvance1/3dnow_and_back_again
OpcodeDispatcher: Implements all the 3DNow! instructions
2022-03-04 03:36:40 -08:00
Mai M 08938ecb11 Merge pull request #1596 from Sonicadvance1/fix_old_kernel_bug
LinuxAllocator: Fixes bug with old kernels and hint allocation
2022-03-02 00:26:51 -05:00
Ryan Houdek 4a3cbf1b1f Merge pull request #1598 from Sonicadvance1/hypervisor_cpuid
CPUID: Implements leaf 4000_0000
2022-03-01 21:02:32 -08:00
Ryan Houdek 0a0bc21c88 CPUID: Implements leaf 4000_0000
This region is reserved for hypervisor uses. Let's follow other examples
and return a hypervisor vendor id signature as another way for software
to find if it is running under FEX-Emu.
2022-03-01 20:47:03 -08:00
Mai M a7ad7f4456 Merge pull request #1593 from Sonicadvance1/add_robin_map
Adds tsl::robin_map
2022-03-01 23:24:34 -05:00
Mai M 46ac05e3cf Merge pull request #1599 from Sonicadvance1/deprecated_distutils
Scripts: Stop using deprecated Distutils
2022-03-01 23:23:55 -05:00
Ryan Houdek 9911fe68d4 Scripts: Stop using deprecated Distutils
According to PEP 386: https://www.python.org/dev/peps/pep-0386/

distutils is deprecated and will be removed in an upcoming python
version.

Switch over to pkg_resources for version parsing and comparison
2022-03-01 07:10:41 -08:00
Ryan Houdek 57a56545b2 Merge pull request #1589 from Sonicadvance1/fix_vixl_assert
Update vixl to fix assert
2022-03-01 05:35:31 -08:00
Ryan Houdek b4e0565907 Merge pull request #1590 from Sonicadvance1/termux_fixes
Termux fixes
2022-03-01 04:08:48 -08:00
Ryan Houdek 5137af5bae Fixes epoxy include in FEXConfig and FEXLogServer 2022-03-01 03:55:19 -08:00
Ryan Houdek 385ed2c2ec Update imgui to fix autodetect 2022-03-01 03:55:16 -08:00
Ryan Houdek 3418cc8054 LinuxSyscalls: More type fixes 2022-03-01 03:55:15 -08:00
Ryan Houdek 1f835ea1f4 LinuxSyscalls: Fixes semid and ipc types
Newer headers redefine semid_ds and ipc_perm as semid64_ds and
ipc64_perm silently.
Use the new types directly since we are a 64-bit only application.
2022-03-01 03:55:12 -08:00
Ryan Houdek 0a40753624 More missing include fixes 2022-03-01 03:55:10 -08:00
Ryan Houdek 609538e758 Utils/Allocator: Use a namespace alias for pmr
This still lives under experimental in Termux environment
2022-03-01 03:55:08 -08:00
Ryan Houdek 09010e1292 LinuxSyscalls: Remove unused headers now
These don't even exist on termux.
2022-03-01 03:54:01 -08:00
Ryan Houdek bddf3871ba LinuxSyscalls/x32/Types: Fixes stat type definitions
The time argument definitions are defines in termux.
Rename our definition name of these so we don't get caught by define.
2022-03-01 03:50:18 -08:00
Ryan Houdek 1ca9e56502 LinuxSyscalls/x64/Types: Fix guest_stat definition
kernel types don't exist in termux. Use uint64_t and int64_t directly.

Also using reserved `__` causes compile failure.
2022-03-01 03:50:17 -08:00
Ryan Houdek fe58a9ae2c LinuxSyscalls/Thread: Don't use set_robust_list on Termux
Would get caught by seccomp and crash FEX
2022-03-01 03:50:15 -08:00
Ryan Houdek bc7c4dfa74 LinuxSyscalls/x32/Types: Don't redefine SIGEV defines 2022-03-01 03:50:12 -08:00
Ryan Houdek 7af7b1309d Msg: Switch msqd_t to FEX defined type 2022-03-01 03:50:10 -08:00
Ryan Houdek dd4630750f LinuxSyscalls/Types: Adds missing types for Termux 2022-03-01 03:50:07 -08:00
Ryan Houdek 302da029eb Work around Termux not supporting hardlinks
The Android filesystem they are on just doesn't support them
Instead of hardlinking FEXLoader to FEXInterpter, just build the
executable twice and eat the filesystem cost.
2022-03-01 03:50:05 -08:00
Ryan Houdek 3cc59bc68e LinuxSyscalls: Switches to a bunch of raw syscalls
For older and Termux build environments these helper libc functions
don't exist.
2022-03-01 03:49:31 -08:00
Ryan Houdek 30c27851ef FEXRootFSFetcher: Termux build environments 2022-03-01 03:30:47 -08:00
Ryan Houdek 9b29ff61e9 Adds some missing headers 2022-03-01 03:30:45 -08:00
Ryan Houdek 91f780223b Stop self-defining PAGE_SIZE
We only work on targets with 4096 byte page sizes.
Adds a cmake compile test to ensure this is adhered to.
2022-03-01 03:30:43 -08:00
Ryan Houdek 2933a00b12 Merge pull request #1579 from Sonicadvance1/optimize_syscalls_with_flags
Allow classifying syscalls with flags
2022-03-01 03:23:43 -08:00
Ryan Houdek a0efd2b01f Resolve comments. 2022-02-28 21:04:03 -08:00
Ryan Houdek 3e9dbda146 Classify syscalls 2022-02-28 21:04:03 -08:00
Ryan Houdek 6501715a3c Allow classifying syscalls with flags
In some cases we can generate more optimal code if we have more
information about a syscall which number gets const-propagated.

In particular optimizing through syscalls, not synchronizing state, and
never returning.

- Noreturn is used by a syscall that never returns, like exit.

This means that it never needs to try and synchronize state coming back

- Not synchronizing state and optimizing through syscalls

Useful for syscalls that don't read the state past arguments and only
returns a value.
2022-02-28 21:03:54 -08:00
Ryan Houdek 2af23d9bec Merge pull request #1591 from Sonicadvance1/new_cpus_in_native_fit
Scripts: Updates CPU fitting script for latest CPUs
2022-02-28 07:05:45 -08:00
Ryan Houdek 3ba2d6cfb4 Merge pull request #1586 from Sonicadvance1/testharness_env
TestHarnessRunner: Wire up environment variable option setting
2022-02-28 07:05:28 -08:00
Ryan Houdek abd266441c unittests: Implements 3DNow! unit tests
Covers the full space, of which there aren't many.

3DNow! unit tests are disabled on the CI runner since the x86 CPU in CI
doesn't support it.
2022-02-28 04:06:14 -08:00
Ryan Houdek b1b6078518 CPUID: Enables 3DNow! + Extensions
Now that we support these
2022-02-28 04:05:18 -08:00
Ryan Houdek a9a89bea9f OpcodeDispatcher: Implements all the 3DNow! instructions
This picks up all the instruction implementations, including 3DNow!
Extended and the Geode specific instructions that were added.

Most of these match preexisting SSE instructions except that they
operate at 64-bit and in the MMX registers.
2022-02-28 04:03:55 -08:00
Ryan Houdek 9b1b2e6496 X86Tables: Fills out 3DNow tables
Fully decoded the same way and adds the Geode specific instruction
decodings as well.
2022-02-28 04:02:57 -08:00
Ryan Houdek 638da92f45 Frontend: Fixes minor bug decoding 3DNow!
We already decoded the modrm `rm` bits, check while decoding modrm to
ensure we don't try decoding it again.
Was causing double decoding of SIB and displacement bytes, breaking
things
2022-02-28 04:01:32 -08:00
Ryan Houdek 0f3a169cc4 Opdispatcher: Minor bug fix with unimplemented op
If multiblock isn't enabled then on Unimplemented op we shouldn't create
a new block.

Was causing IR validation to get angry
2022-02-28 04:00:41 -08:00
Ryan Houdek c7dd176799 IR: Adds a VRev64 op
This directly matches the AArch64 instruction and will be used shortly
2022-02-28 04:00:13 -08:00
Ryan Houdek 23daf4cb72 LinuxAllocator: Fixes bug with old kernels and hint allocation
In the face of an application using MAP_FIXED_NOREPLACE AND the host
linux kernel doesn't understand this flag. Then we were falling down the
hint allocation path which would allocate a pointer in 64-bit space,
returning this pointer to a 32-bit userspace and breaking things.

Now when the hint fails with this flag, we know that it intersecting a
range and can early exit.
2022-02-26 21:12:44 -08:00
Ryan Houdek bb7fa84fb8 Scripts: Updates CPU fitting script for latest CPUs
Clang-13 doesn't yet understand the latest ARM CPUs so just document them
and set to the closest thing.
2022-02-26 02:14:52 -08:00
Ryan Houdek 3325ba52b9 Adds tsl::robin_map
This will be used with the code serialization service soon
2022-02-26 00:43:43 -08:00
Ryan Houdek 7f47fe6d73 Update vixl to fix assert
Any hardware using MTE will assert without this
2022-02-24 13:46:56 -08:00
Ryan Houdek ee165379c5 TestHarnessRunner: Wire up environment variable option setting
Wire up the environment variable option setting so asm files can set
these and it works
2022-02-24 13:37:39 -08:00
Ryan Houdek fa554d3096 Merge pull request #1588 from Azkali/main
Improve compatibility with older uapi kernel headers
2022-02-24 01:34:24 -08:00
The Great Wizard Azkali d60710d3f1 Define proper statx syscall depending on CPU architecture 2022-02-24 10:20:28 +01:00
Azkali 75988b2ae5 Improve compatibility with older uapi kernel headers
Following up the work previously done in 2079f6b3c7.
Adding more defines for older Linux uapi headers missing some defines.
2022-02-24 09:49:56 +01:00
Mai M 30803c66f7 Merge pull request #1587 from Sonicadvance1/fix_missing_telemetry_names
Telemetry: Fix missing telemetry names
2022-02-22 21:46:00 -05:00
Ryan Houdek a9d838fa27 Telemetry: Fix missing telemetry names
Didn't have names for tearing
2022-02-22 18:30:40 -08:00
Ryan Houdek 2b8f60c108 TestHarness: Support for asm files having the option to set config options
Allows some something like the following:
"Env": {
  "FEX_MAXINST": "500"
}

Not that I would recommend overriding MAXINST in the asm tests, as
command line overrides that
2022-02-21 14:53:27 -08:00
Mai M 5ec6ee5b69 Merge pull request #1578 from Sonicadvance1/update_vixl
Updates vixl for new cursor updating methods
2022-02-17 16:18:26 -05:00
Mai M 0db7205f61 Merge pull request #1582 from Sonicadvance1/ccache_option
Adds option to disable ccache
2022-02-16 22:49:48 -05:00
Ryan Houdek 6027494d69 Adds option to disable ccache
Can be useful when running static analysis tools
2022-02-16 19:24:24 -08:00
Mai M 252dcfe26f Merge pull request #1581 from Sonicadvance1/add_required_growsdown
FEXLoader: Adds back required MAP_GROWSDOWN
2022-02-16 19:07:40 -05:00
Ryan Houdek 944c93c10c FEXLoader: Adds back required MAP_GROWSDOWN
I was overzealous with my removal of MAP_GROWSDOWN.
We still require the primary thread to have this flag set.
2022-02-16 15:41:44 -08:00
Ryan Houdek a5fb7e7313 Merge pull request #1580 from Sonicadvance1/fix_hostthunks_install
Fixes Host and guest thunks install path
2022-02-15 15:46:10 -08:00
Ryan Houdek 364b3380fc Fixes Host and guest thunks install path
Hosts were using the cmake install path with $DESTDIR which duplicates
paths.

GuestThunks were doing some magic that wasn't actually necessary
2022-02-15 15:36:31 -08:00
Ryan Houdek a0edab8040 Updates vixl for new cursor updating methods
These will be required for code cache
2022-02-14 16:33:51 -08:00
Ryan Houdek b65194f433 Merge pull request #1576 from Sonicadvance1/move_x87_constant_helpers
JIT: Implements x87 fallback helpers as lookups in to state
2022-02-14 14:13:33 -08:00
Ryan Houdek 0d1c9cd7df JIT: Implements x87 fallback helpers as lookups in to state
This allows x87 fallbacks to be loaded from the upcoming code cache
without relocations.

Only 40 pointers necessary to store and means x87 code won't hit
relocations heavily.

Probably improves performance slightly on the x86 host side but should
be neglible.

Needs #1574 and #1575 merged first.
2022-02-14 14:00:10 -08:00
Ryan Houdek 99dcda7f8c Merge pull request #1575 from Sonicadvance1/move_constant_functions_x86
JITx86: Switches over to loading pointers from state
2022-02-14 13:59:07 -08:00
Ryan Houdek 2aa4d33de5 JITx86: Switches over to loading pointers from state
Just like the previous AArch64 JIT.
These pointers are process or thread specific depending on the pointer
and should be loaded from the State object.

Performance here might slightly increase.

This is required for code cache on x86

Needs #1574 merged first.
2022-02-14 13:49:54 -08:00
Ryan Houdek 1ef78d0a00 Merge pull request #1574 from Sonicadvance1/move_constant_functions
ARMJIT: Switches over to loading pointers from state
2022-02-14 13:39:28 -08:00
Stefanos Kornilios Mitsis Poiitidis 64d1840f1e Merge pull request #1571 from Sonicadvance1/fix_syscall_strace
Linux: Fix missing types for syscall strace
2022-02-14 19:05:48 +02:00
Stefanos Kornilios Mitsis Poiitidis 34530236f9 Merge pull request #1573 from Sonicadvance1/disable_int_tests_with_no_int
unittests: Disables Interpreter tests when its disabled
2022-02-14 19:00:13 +02:00
Ryan Houdek f5a9da082f ARMJIT: Switches over to loading pointers from state
These pointers are process or thread specific depending on which pointer
it is.
All of these pointers end up getting used inside of the JIT blocks
themselves and with code caching would result in a ton of relocations
occuring inside the code.

The pointers used within the dispatcher don't currently matter since I'm
not expecting to cache the dispatcher itself. It's only a page per
thread after all. This does move us significantly closer towards using a
single dispatcher for all threads though.

The performance impact of this change is unlikely to be felt at all,
some locations have less code generation which could improve perf
slightly. Some locations move from a 1-3 cycle constant calculation to a
4 cycle load, hard to be felt since it gets hidden by other
instructions.

x86-64 JIT will be added soon after this
2022-02-13 23:06:26 -08:00
Ryan Houdek 9b49e8cb59 FEXCore: Adds utility class for class member function casting
Adds validation for our class member casting to ensure we don't try
casting a virtual member.
2022-02-13 23:06:25 -08:00
Ryan Houdek 1b6d20b731 unittests: Disables Interpreter tests when its disabled
Would result in failures if you weren't expecting it.
2022-02-11 12:55:10 -08:00
Mai M e9c7c76174 Merge pull request #1572 from Sonicadvance1/LoadConstant_no_opt
Arm64Emitter: Allow non-optimizing LoadConstant
2022-02-11 00:50:56 -05:00
Ryan Houdek cbce06d012 Arm64Emitter: Allow non-optimizing LoadConstant
This is pulled from the code cache PR. Will be necessary for supporting
relocations.

Not currently being used but will be once we have code caching in place.
2022-02-10 20:04:02 -08:00
Ryan Houdek d95326b23b Linux: Fix missing types for syscall strace 2022-02-10 19:46:43 -08:00
Mai M 43fada7555 Merge pull request #1568 from Sonicadvance1/fix_musl_load
ELFCodeLoader: Fixes typo in AT_BASE calculation
2022-02-10 21:25:48 -05:00
Mai M afa7172cb1 Merge pull request #1570 from Sonicadvance1/remove_growsdown
Removes MAP_GROWSDOWN usage
2022-02-10 21:25:23 -05:00
Ryan Houdek b88d8a7cc4 Removes MAP_GROWSDOWN usage
This is just a memory leak waiting to happen.
Only the primary thread in an application really should have this set
since the kernel cleans it up.

We only ever allocate the primary thread of the guest application then
every host thread's stack on top of that. It's up to the guest when it
is cloning to set up new stack pointers, we don't manage that.

We are already allocating the first thread's size at the soft stack
limit with RLIMIT_STACK anyway.

Fixes #1556
2022-02-10 18:03:18 -08:00
Ryan Houdek 6add09b78b ELFCodeLoader: Fixes typo in AT_BASE calculation
Fixes executing musl applications with the dynamic linker.

It was using the main executable's p_offset instead of the
interpreter's.
Wasn't a problem with glibc since it uses a different symbol to find the
base (Don't ask me why it does this).

musl dynamic linker on the other hand just uses AT_BASE directly and
since it was calculated incorrectly it was crashing.

Testing application was `ls` which had a p_offset of 0x40, so it would
try and read some values from AT_BASE, starting at an offset below where
it was mapped.
2022-02-10 17:43:58 -08:00
Mai M 4bb3a54ccf Merge pull request #1566 from neobrain/refactor_thunk_misc
Miscellaneous thunk cleanups
2022-02-10 15:33:34 -05:00
Mai M 76f86e51a6 Merge pull request #1567 from Sonicadvance1/fix_fexgetconfig_rootfs
FEXGetConfig: Fix --current-rootfs option
2022-02-10 15:15:26 -05:00
Ryan Houdek 2a1b27df58 FEXGetConfig: Fix --current-rootfs option
If the configured rootfs wasn't a squashfs then it was failing to return
the directory.

Now it works for both squashfs and directory rootfs again.
2022-02-10 11:45:29 -08:00
Tony Wasserka 68426735a5 Thunks: Clean up ASTMatcher-based testing helpers
The run_thunkgen* helpers now parse generated source code and return its AST
representation, so HasASTMatching helper calls don't each need to redundantly
compile it themselves. This also ensures the generator output actually compiles
in tests where we didn't explicitly check that before.

This also allows printing the full AST of the generator output on test
failures. This must be enabled manually by changing a variable in the ostream
output operator for SourceWithAST.
2022-02-10 12:11:52 +01:00
Tony Wasserka ee6b558000 Thunks: Fix warning about unused field 2022-02-10 12:11:52 +01:00
Tony Wasserka 3c7872c6d5 Thunks: Rename FrontendAction to GenerateThunkLibsAction 2022-02-10 12:11:52 +01:00
Tony Wasserka 4aed6fc56c Thunks/gen: Remove now unneeded code 2022-02-10 12:11:52 +01:00
Tony Wasserka 5d555a10a2 Thunks: Explicitly put thunks into the text library section
Previously, defining zero-initialized variables right before LOAD_LIB
could cause the compiler to put thunk definitions into bss, hence triggering
errors during assembly ("attempt to store non-zero value in section `.bss'").
2022-02-10 12:11:52 +01:00
Ryan Houdek 5854d4ad1c Merge pull request #1565 from neobrain/refactor_thunk_ide_integration
Enable proper IDE integration of thunk libraries
2022-02-10 02:44:29 -08:00
Tony Wasserka bd6999eb87 CMake: Clean up build architecture for ThunkLibs
Host thunk libraries are always built as part of the main project now.
Guest thunk libraries are still cross-compiled in a CMake ExternalProject,
but *additionally* there are CMake targets in the main project to make
sure IDE engines can properly handle guest source files.
2022-02-10 11:23:43 +01:00
Tony Wasserka 3ebb2eaf0d Thunks: Fix guest libs build on clang 2022-02-10 11:23:41 +01:00
Mai M defd3be30c Merge pull request #1563 from Sonicadvance1/fix_auto_script
Updates Readme to fix install script
2022-02-09 18:18:39 -05:00
Mai M a8e5a0a68e Merge pull request #1562 from Sonicadvance1/remove_debug_memory_mapping
FEXLoader: Removes memory mapping check on startup
2022-02-09 18:18:18 -05:00
Ryan Houdek f27c43ff8a Updates Readme to fix install script
Fixes an issue where the FEXRootFSFetcher wouldn't get a any user input
and just fail out.
Save it to the tmp folder and execute from there instead.

Fixes #1557
2022-02-09 14:01:48 -08:00
Ryan Houdek 8840fc818c FEXLoader: Removes memory mapping check on startup
FEX always builds with PIE and we don't hit this issue anymore anyway.
If some application wants to inject a page in to the lower 32-bits then
we have no reason to complain about it anymore. Just let it go and
hopefully they know what they are doing.

Fixes #1559
2022-02-09 13:52:05 -08:00
Mai M ffcaf294e3 Merge pull request #1555 from Sonicadvance1/weirdo_edge_case
OpcodeDispatcher: Fixes weirdo edge case in segment moving
2022-02-08 00:26:55 -05:00
Ryan Houdek 6b7a84bef2 OpcodeDispatcher: Fixes weirdo edge case in segment moving
Just noticed this while casually reading the x86 architecture manuals.
The move segment registers instructions ignore the REX.R prefix on the
segment register.

Previously this was expected to create an invalid register selection.
A little bit silly but sure, support it.
2022-02-07 21:12:53 -08:00
Mai M 287c65dc64 Merge pull request #1554 from Sonicadvance1/fix_tricky_stat
Linux: x32: Fixes tricky stat64 defines
2022-02-06 19:52:31 -05:00
Mai M 62397bb19c Merge pull request #1553 from Sonicadvance1/fix_sigevent
Linux: Make sure to use correct accessors for sigevent
2022-02-06 19:52:14 -05:00
Ryan Houdek 0216bcf27f Linux: x32: Fixes tricky stat64 defines
Some build environments use a define to change stat64 and statfs64 to be
the same definition as stat and statfs.

Check if the define exists and if it does then remove the 64bit
constructors.
2022-02-06 16:05:27 -08:00
Ryan Houdek b981fcfe42 Linux: Make sure to use correct accessors for sigevent
Some of these are defined differently depending on environment
2022-02-06 15:33:07 -08:00
Mai M cb491a8acb Merge pull request #1552 from Sonicadvance1/fix_ucontext_copy
UContext: Fixes 32-bit siginfo_t copying definition
2022-02-06 18:23:04 -05:00
Mai M 1b99495b4b Merge pull request #1551 from Sonicadvance1/fix_older_env
Some fixes for older environments
2022-02-06 18:22:26 -05:00
Ryan Houdek 3c5a2cec90 UContext: Fixes 32-bit siginfo_t copying definition
The host provided siginfo_t definition can vary depending on the build
environment.
What doesn't change however is how the data is laid out.
It's always 128bytes, The first three 32-bit words are always known.
The 64-bit host side always has an additional 32-bit pad member.
Then the remaining bytes is the sifields.
2022-02-06 15:03:56 -08:00
Ryan Houdek 9a64e7f100 Linux: Renamed some 64-bit syscall names
Some build environments use defines to rename these. Which breaks our
naming
2022-02-06 14:19:30 -08:00
Ryan Houdek 8e8baec47a Linux: Use raw syscalls for pkey syscalls
For older libc environments
2022-02-06 14:19:30 -08:00
Ryan Houdek 59ca60e39f Fixes a bunch of header includes
Necessary for older build environments
2022-02-06 14:19:30 -08:00
Ryan Houdek 8c956e6ce1 Docs: Update for release FEX-2202 2022-02-05 22:48:35 -08:00
Ryan Houdek 832d013c92 Merge pull request #1513 from Sonicadvance1/reduce_flags_memory_usage
FEXCore: Defer a significant number of ALU flag calculation
2022-02-04 16:22:24 -08:00
Mai M 1c24206117 Merge pull request #1550 from Sonicadvance1/fix_weirdo_crc32
OpcodeDispatcher: Fixes CRC32 decoding in 0F_38 table
2022-02-04 01:01:50 -05:00
Ryan Houdek f1979c15a2 unittests: Adds new CRC32 unittests
The instruction decode tables for crc32 introduced some dumb.
`F2h` and `F2h && 66h` prefixes both work for crc32.
This is a failure on Intel's part for sticking crc32 in to the vector
table.

MOVBE without any prefixes also does the same garbage where prefix `66h`
acts as an operand prefix size ONLY.
2022-02-03 21:11:50 -08:00
Ryan Houdek 556a1dab24 OpcodeDispatcher: Fixes CRC32 decoding in 0F_38 table
This table is particularly terrible. CRC32 is the first instruction in
this table that needs either prefix `72h` OR `66h && F2h`

For 8bit CRC32, this ignores the 66h operand size override prefix.
  - But our table decoding didn't handle this
For 16bit/32bit/64bit CRC32 this behaviour changes depending on 66h
prefix AND REX.W
  - 66h prefix is ignored when REX.W is set, always 64bit but it falls
    down the other table path

This is an absolutely weird edge case that nobody should hit, but here
we are.
2022-02-03 21:11:50 -08:00
Mai M caffad8562 Merge pull request #1549 from Sonicadvance1/implement_pcmpgtq
OpcodeDispatcher: Implements PCMPGTQ
2022-02-03 21:46:13 -05:00
Mai M 5978143141 Merge pull request #1547 from Sonicadvance1/remove_system_xxhash
CMake: Always use local xxhash to statically link
2022-02-03 21:45:58 -05:00
Ryan Houdek 594c70b5e0 OpcodeDispatcher: Implements PCMPGTQ
I thought we already had this implemented but I guess it was missed.

Required for SSE 4.2
2022-02-03 18:36:29 -08:00
Ryan Houdek 655e6989ca FEXCore: Defer a significant number of ALU flag calculation
This was mainly an optimization around memory usage. ALU ops tend to
bloat the IR quite heavily, but I also noticed a 2-4% uplift in
performance of some applications. So a nice side effect.

Should let us more aggressively target reducing our IR intrusive
allocator size since this is quite reduced.

In a pedantic heavy ALU op code block this reduces the number of IR ops
from 14,756 IR ops to 2,016 prior to optimization.
After optimization both had reduced down to 50 IR ops, proving the
output IR was the same.
2022-02-03 01:37:59 -08:00
Ryan Houdek afeb228a89 CMake: Always use local xxhash to statically link
Dynamically linking xxhash is causing problems with pressure-vessel.

With this in place we only have the typical C++ dependencies
```
$ ldd ./Bin/FEXLoader
        linux-vdso.so.1 (0x00007fff44d9d000)
        libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f4c4d884000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f4c4d7a0000)
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f4c4d786000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f4c4d55e000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f4c4e0fa000)
```
2022-02-03 01:31:43 -08:00
Mai M d308a438ea Merge pull request #1546 from Sonicadvance1/fix_fexconfig
Fixes FEXConfig build
2022-02-01 21:13:08 -05:00
Ryan Houdek 260fc8ba52 Fixes FEXConfig build
Oops. This was added late and didn't test it.
2022-02-01 15:44:06 -08:00
Mai M ade0d0f241 Merge pull request #1543 from Sonicadvance1/fixes_for_1423
Linux: Fixes for older build environments
2022-02-01 16:52:40 -05:00
Mai M 11a5105547 Merge pull request #1544 from Sonicadvance1/allow_disable_interpreter
Adds an option to disable the IR interpreter
2022-02-01 16:52:22 -05:00
Ryan Houdek 10ad5db686 Adds an option to disable the IR interpreter
By default we won't build with the interpeter to reduce user confusion.
The interpreter isn't really useful to end users so remove it.

Completely removes it from building except for the fallback operations.

This also removes the selection from FEXConfig to remove selection
confusion there.

File Stats:
FEXLoader Size with Interpreter:    3422768 bytes
FEXLoader Size without Interpreter: 3301944 bytes
Size difference:                    96.4699915%
Bytes removed:                      120824 bytes
4k pages removed:                   29.498046875 -> 30 rounded up

VM Stats (Reported from bloaty):
Memory Size with Interpreter:    6.50Mi
Memory Size without Interpreter: 6.38Mi
Size difference:                 98.1538462%
2022-02-01 13:00:29 -08:00
Ryan Houdek 68c441575d Linux: Fixes for older build environments
Should resolve the new building issues from #1423
2022-02-01 12:17:09 -08:00
Ryan Houdek 334a8ef87c Merge pull request #1542 from Sonicadvance1/fix_pressure_vessel_hangs
Fix pressure vessel hangs
2022-01-31 08:57:20 -08:00
Ryan Houdek b7a76af72f Merge pull request #1541 from Sonicadvance1/implement_crc
OpcodeDispatcher: Implements CRC32 instruction
2022-01-31 08:57:01 -08:00
Ryan Houdek 4c92b562b8 Merge pull request #1540 from Sonicadvance1/remove_extract
OpcodeDispatcher: Removes extraneous extract in VFCMP
2022-01-31 08:56:47 -08:00
Stefanos Kornilios Mitsis Poiitidis 9d08451903 Merge pull request #1536 from Sonicadvance1/fix_orbitals
Softfloat: Stop doing special handling for FREM
2022-01-31 16:51:18 +02:00
Stefanos Kornilios Mitsis Poiitidis c252f8bfc5 Merge pull request #1539 from Sonicadvance1/fix_wrong_offsets
IR: Fixes some wrong offsets in passes
2022-01-31 15:40:33 +02:00
Ryan Houdek dc7ec6377b Linux: Safely handle Filemanagement mutex on fork
If an application is forking heavily with threaded file accesses
happening then the mutex can end up in an unknown state.

On fork make sure to lock the mutex then immediately unlock after fork
occurs.

This final step resolves hanging that pressure-vessel hits on startup.
Since it is doing a ton of file opening and forking during
initialization.
2022-01-30 18:15:57 -08:00
Ryan Houdek ce6f4edaaa FileManagement: Use ScopedSignalMaskWithMutex
When using mutexes in syscall helpers we need to be extra careful around
signals.
2022-01-30 18:15:57 -08:00
Ryan Houdek 983c35ea3b Allocator: Use ScopedSignalMaskWithMutex
Instead of just a basic mutex, also mask the signals.
This fixes the problem where we can end up receiving a signal in the
middle of memory allocation. Thus leaving the locked mutex in a broken
state.

This more closely matches the Linux kernel behaviour.
Since if you're in the middle of a memory allocating syscall, you won't
get signaled.
2022-01-30 18:15:57 -08:00
Ryan Houdek 70aaa1117a FEXHeaderUtils: Adds ScopedSignalMaskWithMutex
This class allows a scoped region lock a mutex and mask signals.

This is necessary for thread and signal safety coming up
2022-01-30 18:15:57 -08:00
Ryan Houdek 59e9859087 unittests: Implements CRC32 unit tests 2022-01-30 15:38:26 -08:00
Ryan Houdek d9453ff639 OpcodeDispatcher: Implements CRC32 instruction
Now that the rest of the code matches behaviour, we just need to pass
this through.

Easy enough and get Horizon Zero Dawn running.
2022-01-30 15:38:26 -08:00
Ryan Houdek 70754991d1 CPUID: Fill out CPUID for SSE4.2 feature
Currently force disabled until the rest of SSE 4.2 is enabled
This is to remind us in the future that SSE4.2 can only be enabled in
CPUID with CRC32 instruction support.
2022-01-30 15:38:26 -08:00
Ryan Houdek 57ebfceb48 HostFeatures: Check for CRC32 op support
Available with CRC32 bit on Arm64 or SSE4.2 on x86-64
2022-01-30 15:38:26 -08:00
Ryan Houdek 9e224d2bb0 x86 JIT: Implements CRC32 op 2022-01-30 15:38:26 -08:00
Ryan Houdek e43bd04901 JITArm64: Implements CRC32 op 2022-01-30 15:38:26 -08:00
Ryan Houdek 48762e03a6 Interpreter: Implements CRC32 op 2022-01-30 15:38:26 -08:00
Ryan Houdek 858924309e IR: Implements CRC32 op 2022-01-29 23:33:53 -08:00
Ryan Houdek cab02d1e65 OpcodeDispatcher: Removes extraneous extract in VFCMP
We don't need to extract the element to compare it.
2022-01-28 22:22:34 -08:00
Ryan Houdek 2a64f80567 IR: Fixes some wrong offsets in passes
GPR and FPR ending offsets were off by one here. Just a quick fix.
2022-01-28 22:19:51 -08:00
Ryan Houdek 174ddea99d Softfloat: Stop doing special handling for FREM
This isn't correct and breaks games.
This makes the FREM and REM1 implementation the same.
While not 100% correct, it is still better than before.
New issues will be created to handle the differences in the future.

Fixes #1374.
Also fixes most of the HL2 issues, just not the seam issue.
2022-01-28 19:42:57 -08:00
Ryan Houdek 6fb0e3c0cf Softfloat: Allow x87 fallback for all ops 2022-01-28 19:42:25 -08:00
Ryan Houdek 2b044bbdf4 Disable fprem unittests
These are about to be broken
2022-01-28 19:39:03 -08:00
Mai M ea76de0fd2 Merge pull request #1533 from Sonicadvance1/revise_posix_tests
unittests: Revise POSIX tests known failures and disabled
2022-01-25 15:19:24 -05:00
Ryan Houdek 9c8642e0dc unittests: Revise POSIX tests known failures and disabled
Some of these behaviours have changed now, particularly around signal
handling.

Some things still fail now of course. But most everything is now
documented as to why it is failing or disabled.

Fixes #955
2022-01-25 11:41:45 -08:00
Ryan Houdek 13f35f7b79 Merge pull request #1530 from Sonicadvance1/rootfs_fetcher_fixes
FEXRootFSFetcher: Fixes some edge case behaviours
2022-01-25 10:29:55 -08:00
Ryan Houdek e2798e370e Merge pull request #1518 from Sonicadvance1/fix_signed_branch
JIT: Fixes signed displacement wraparound on 32-bit
2022-01-25 10:29:46 -08:00
Ryan Houdek d9149548b5 Merge pull request #1531 from Sonicadvance1/fix_sockopt
Linux: Fixes 32-bit getsockopt and setsockopt
2022-01-25 09:09:16 -08:00
Ryan Houdek 4823933f79 Linux: Fixes 32-bit getsockopt and setsockopt
On Set, we have four options that need to be converted.
On Get, we have two options that need to be converted.

This fixes a crash that Tomb Raider 2013 was having on launch.
2022-01-24 17:14:50 -08:00
Ryan Houdek ee04067424 FEXRootFSFetcher: Fixes some edge case behaviours
Makes curl do its continue feature to give the users the best chance of
downloading a rootfs. We don't need to restart the full file transfer on
failure. Helps people with slower connections.

On failure to download, asks the user if they want to retry the download
rather than just exiting with a weird error about hash failure.

Once the image is downloaded, now changes options depending on if
squashfuse or unsquashfs works.

Prevents the user from selecting a bad option and getting unexpected
behaviour. Ideally we would do a squashfs mount test as well for
platforms that don't have working FUSE, like termux. This is harder to
get right and its for an unsupported platform, so I'm not going to
invest more time with it.

Fixes #1525
Fixes #1526
Fixes #1527
2022-01-23 22:54:05 -08:00
Ryan Houdek e4aef26ef5 FEXRootFSFetcher: Adds helper namespace for tool checking
Location to check if curl, squashfuse, and unsquashfs are working.

unsquashfs is a bit more complex where it needs to parse the help output
to see if zstd is supported
2022-01-23 22:44:31 -08:00
Ryan Houdek a41dc8eafa FEXRootFSFetcher: Fix pipe redirecting
In the case of launching without stdout/stderr then redirection could
have these constants be a redirected FD that sits in the same fd number.

Use -2 to indicate no redirection.
Use -1 to indicate closing traditional stderr/stdout
The rest will indicate if stdout and stderr should be replaced as
normal.
Making sure not to close the incoming fds if they matched the
stdout/stderr FD numbers.
2022-01-23 22:41:39 -08:00
Ryan Houdek f41cd8deff OpcodeDispatcher: Renamed GetDynamicPC to GetRelocatedPC
For clarity.
2022-01-23 18:52:53 -08:00
Ryan Houdek 2a0c3cce30 Core: Have GetDynamicPC mask based on operating size
This ensures on 32-bit we overflow correctly under relocation.
2022-01-23 18:50:16 -08:00
Ryan Houdek e817f5d98c unittests: Adds 32-bit tests for signed displacement wraparound
A bit meta since it needs to JIT some minor code but easy enough.
Ensures something like #1517 won't happen again.
2022-01-23 18:38:44 -08:00
Ryan Houdek 8b8cda9b80 JIT: Fixes signed displacement wraparound on 32-bit
This cropped up mostly with multiblock and `jmp <signed displacement>`
This also happened with non multiblock `jcc <signed displacement>`

Due to how IR relocations occur, this needs to happen fairly late but
isn't a big deal.

Fixes #1517
2022-01-23 18:38:43 -08:00
Ryan Houdek 8e3893df07 Merge pull request #1523 from lioncash/vixl-update
Externals: Update vixl
2022-01-20 15:13:36 -08:00
lioncash 51b335914c github: Synchronize submodules before checking them out
Ensures that we don't get stale remotes.
2022-01-20 17:57:45 -05:00
lioncash 8835d57ae3 Arm64Emitter: Adjust XRegister to Register
With the updated API, we need to make use of Register as opposed to
XRegister in our arrays.
2022-01-20 16:42:04 -05:00
lioncash eba1b65fb0 Externals: Update vixl to updated branch
Now we have access to some SVE goodies.
2022-01-20 16:42:02 -05:00
Ryan Houdek a3a138ef7e Merge pull request #1520 from lioncash/vixl
External: Point vixl submodule towards FEX's fork
2022-01-14 13:40:18 -08:00
lioncash 62d9a494cd External: Point vixl submodule towards FEX's fork
This allows it to be managed by all organization members
2022-01-14 12:21:19 -05:00
Stefanos Kornilios Mitsis Poiitidis 6744a06a53 Merge pull request #1519 from Sonicadvance1/aarch64_single_instruction_opt
AArch64: Single instruction optimization for AESKeyGenAssist
2022-01-14 15:41:57 +02:00
Ryan Houdek 140e9824b7 Merge pull request #1516 from lioncash/fmt
externals: Update fmt to 8.1.1
2022-01-14 01:55:50 -08:00
Ryan Houdek bad84f61fa AArch64: Single instruction optimization for AESKeyGenAssist
No need to do adr when loads can do a 1MB offset loadstore
2022-01-14 01:49:21 -08:00
lioncash 2296126af3 externals: Update fmt to 8.1.1
Brings along a bunch of enhancements and ensures we always build against
the latest version.

Also fixes up a few issues that arose due to changes in fmt
2022-01-13 14:48:35 -05:00
Stefanos Kornilios Mitsis Poiitidis 0a8717d9a8 Merge pull request #1515 from Sonicadvance1/fix_ptest
OpcodeDispatcher: Fixes ptest flags calculation.
2022-01-13 10:38:38 +02:00
Ryan Houdek 4e2220c27f unittests: Adds ptest unit test to ensure correct flag setting
ptest wasn't correctly setting OF, SF, AF, and PF to zero until now.
Do a unit test to ensure correct behaviour here
2022-01-11 16:45:56 -08:00
Ryan Houdek c87e11cee9 OpcodeDispatcher: Fixes ptest flags calculation.
We were missing four flags that require setting zero.
2022-01-11 16:45:11 -08:00
Ryan Houdek 7768f6965a Merge pull request #1501 from Sonicadvance1/finish_siginfo_32bit
Linux: Handles the remaining 32-bit siginfo_t usage
2022-01-11 00:02:55 -08:00
Ryan Houdek 023aaaae0c Merge pull request #1499 from Sonicadvance1/resolve_rootfs_path_in_interpreter
FEXLoader: Resolve the absolute path to rootfs if possible
2022-01-11 00:02:27 -08:00
Stefanos Kornilios Mitsis Poiitidis a2aa9f3fc1 Merge pull request #1512 from Sonicadvance1/fix_ssa_id_print
IR: Fixes SSA ID printing
2022-01-11 09:09:06 +02:00
Ryan Houdek e46ec9a0ce IR: Fixes SSA ID printing
These should print as decimal. They were ending up as hex
2022-01-10 18:09:03 -08:00
Ryan Houdek 784cbdd973 Merge pull request #1500 from Sonicadvance1/rootfsfetch_check_curl
FEXRootFSFetcher: Check if curl is installed and fail before running
2022-01-10 16:23:05 -08:00
Ryan Houdek 6022715a9b Merge pull request #1510 from Sonicadvance1/fix_asan_cpuid
CPUID: Fixes ASAN problem with reading midr
2022-01-10 02:12:10 -08:00
Ryan Houdek 9eb5ba5ad1 Merge pull request #1509 from Sonicadvance1/fix_logserver_sync
SocketLogging: Fixes MsgHandler not syncing with Assert level
2022-01-10 02:12:01 -08:00
Ryan Houdek 82e5977709 Merge pull request #1506 from Sonicadvance1/fix_apitest_syscalls
APITests: Fixes InterruptableConditionVariable test to use the syscal…
2022-01-10 02:11:44 -08:00
Ryan Houdek 7c08b67dff Merge pull request #1504 from Sonicadvance1/fix_unittest_rootfs_define
unittests: Fixes ROOTFS needing to be defined prior to cmake
2022-01-10 02:11:35 -08:00
Ryan Houdek 609587f9ee Merge pull request #1503 from Sonicadvance1/implement_bcd_tests
unittests: Adds a BCD unit test
2022-01-10 02:11:07 -08:00
Ryan Houdek 5dda3a1599 Merge pull request #1497 from Sonicadvance1/fix_alternative_links
Linux: Fixes emulatedpath with symlink following
2022-01-10 02:10:54 -08:00
Ryan Houdek 73aaa4c3a6 CPUID: Fixes ASAN problem with reading midr
Needs to be a string_view for the MIDR for the StrConv helper to work in
this instance.
There is no null terminator character when reading from the file is why.
2022-01-10 01:21:56 -08:00
Ryan Houdek 3ba5371d36 SocketLogging: Fixes MsgHandler not syncing with Assert level
AssertHandler by default synchronizes but MsgHandler with Assert level
should also synchronize.

Fixes an issue where LogMan::Msg::AFmt wasn't syncing so the
FEXLogServer would never see the messages.
2022-01-10 01:16:53 -08:00
Ryan Houdek bf581decde Merge pull request #1507 from Sonicadvance1/fix_warnings
Fixes some of the warnings that cropped up
2022-01-10 01:12:39 -08:00
Ryan Houdek 250504502a Fixes some of the warnings that cropped up 2022-01-10 00:46:10 -08:00
Ryan Houdek a0e826feaf APITests: Fixes InterruptableConditionVariable test to use the syscall wrappers.
Fixes a build error on old Ubuntu
2022-01-09 22:00:57 -08:00
Ryan Houdek eb17edec05 Merge pull request #1502 from Sonicadvance1/fix_fexlog_server_message
FEXLogServer: Stop duplicating and dropping messages
2022-01-09 03:22:24 -08:00
Ryan Houdek 228aed98c7 unittests: Fixes ROOTFS needing to be defined prior to cmake
cmake will bake in the environment variable in to the build scripts.
Instead have the guest_test_runner fetch it at runtime.

This means if you forget to set ROOTFS prior to running cmake, you can
now set it afterwards and rerun with just ctest instead of a cmake
dance.

Fixes #315
2022-01-09 01:56:13 -08:00
Ryan Houdek 6ba4aec88e unittests: Adds a BCD unit test
Nothing really fantastical found here. Just that sub-precision results
weren't rounded correctly on store

Fixes #770
2022-01-09 01:36:26 -08:00
Ryan Houdek b8d6b2cd4a F80: Ensures BCDStore rounds to the current rounding mode
BCD storing will round any subprecision results depending on the current
rounding mode.
2022-01-09 01:35:28 -08:00
Ryan Houdek d4e2f42f90 FEXLogServer: Stop duplicating and dropping messages
In the case that multiple messages appearing in a single packet then we
were repeating the first message and dropping any subsequent messages.

Fixes #1496
2022-01-09 00:22:12 -08:00
Ryan Houdek 3055c23365 Linux: Handles the remaining 32-bit siginfo_t usage
Just need to translate them between 32-bit and 64-bit versions.

Fixes #1254
2022-01-09 00:03:17 -08:00
Ryan Houdek cb7feaefbb Types: Allows passing 64-bit host siginfo_t to 32-bit siginfo_t
Needed for waitid
2022-01-09 00:00:13 -08:00
Ryan Houdek 5a5a498ed6 FEXRootFSFetcher: Check if curl is installed and fail before running
Before doing anything that requires curl, actually check if it is
installed.
Then instruct the user to install curl before using.

Doesn't try installing curl itself since we don't have a clean way to
execute sudo from potentially GUI.

Fixes #1498
2022-01-08 21:35:29 -08:00
Ryan Houdek 285ed8f1e0 FEXRootFSFetcher: Adds new Exec function with stdout,stderr redirection
Just so we can test for applications without spamming terminal
2022-01-08 21:34:59 -08:00
Ryan Houdek a68da468a5 FEXRootFSFetcher: ExecAndWaitForResponse sign extend program result
Only the lower 8bits of the execve result is the program result.
Makes sure to sign extend it so -1 is a true -1 instead of 255
2022-01-08 21:33:24 -08:00
Ryan Houdek d59aa6874e FEXLoader: Resolve the absolute path to rootfs if possible
If the user passes in an absolute path then check to see if it exists in
the rootfs before executing.

Useful for launching applications directly out of the rootfs with
FEXInterpreter.

In the case that the absolute path doesn't exist in the rootfs then
fallback to the host system as usual
2022-01-07 03:42:41 -08:00
Ryan Houdek 19fd89d2bf Linux: Fixes emulatedpath with symlink following
Some syscalls support `AT_SYMLINK_NOFOLLOW` In these instances we need
to follow the symlink on a couple of syscalls.

Fixes executing wine using the basic wine path
eg:
FEXBash "wine dxcapsviewer.exe"
2022-01-07 02:26:26 -08:00
Mai M e36beb8dbe Merge pull request #1495 from Sonicadvance1/add_tune_arch
CMake: Adds TUNE_ARCH option
2022-01-05 18:09:26 -05:00
Ryan Houdek 70f447b265 CMake: Adds TUNE_ARCH option
I forgot about this option working for tuning arch on AArch64. This will
be used in PPA releases in the future. Will leave the previous option
since it can be used in testing.
2022-01-05 13:48:35 -08:00
Ryan Houdek a0026c92a8 Merge pull request #1494 from Seas0/main
ThunkLibs: Add meta data to libvulkan_device
2022-01-04 23:37:54 -08:00
Seas0 8fc5f66a5b ThunkLibs: Add meta data to libvulkan_device 2022-01-05 14:28:26 +08:00
Ryan Houdek affbb40cc1 Docs: Update for release FEX-2201 2022-01-03 19:08:53 -08:00
Ryan Houdek b276750c8a Merge pull request #1488 from Sonicadvance1/python_auto_install
Scripts: Adds a python script that can hand hold a user through FEX install
2022-01-03 19:07:41 -08:00
Ryan Houdek 6d5322b349 Readme: Updates with links to the wiki's quick start guide and python command
Quick start guide on the wiki is currently not populated but will be
soon.
2022-01-03 16:05:05 -08:00
Ryan Houdek f836ff0897 Scripts: Adds a python script that can hand hold a user through FEX install
This is definitely a bit divisive but overall this is a win.
This is a user pattern that is emerging in a bunch of projects.
Allow an officially sourced script that lets you pipe a script directly
in to python/bash and setup the environment entirely.

This only supports Ubuntu {20.04, 21.04, 21.10, 22.04} which matches
exactly what we expose in the PPA.

Once this is in the repo, and our PPA is updated to the latest release
tag you can run this script like:
`curl --silent <Direct Github raw link> | python3`

Once the PPA is updated, the README and Wiki will be updated with Quick
Start guides to use this path.

The script steps
1) Checks if ARMv8, or fail
2) Checks if supported Ubuntu, or fail
3) Checks if PPA installed
3a) Install PPA if not, or fail
4) Check if packages are installed
4a) Install non-installed packages, or fail
5) Check if RootFS is configured/exists
5a) Run through FEXRootFSFetcher to get/setup rootfs, or fail
6) Attempt to run emulated uname -a through FEX, or fail
7) Provide some examples for how to use FEX
8) Exit with success!
2022-01-03 14:20:09 -08:00
Mai M 0bf577654d Merge pull request #1491 from Sonicadvance1/fix_signal_pause
FEXCore: Don't use Run() inside RunUntilExit()
2022-01-03 06:56:43 -05:00
Mai M ac63a2bac4 Merge pull request #1493 from Sonicadvance1/fix_pidfd_send_signal32
Linux: Fixes pidfd_send_signal for 32-bit
2022-01-03 06:37:13 -05:00
Ryan Houdek 6b8e6c18ee SignalDelegator: Increase signal stack size
Just to help in the case that a program will dequeue a bucketload of
signals in one go. Basic signal stack size wasn't enough.
2022-01-03 03:29:35 -08:00
Ryan Houdek 93dccffc81 unittests: Updates posix tests with new passing tests 2022-01-03 03:29:35 -08:00
Ryan Houdek 22810208b4 FEXCore: Don't use Run() inside RunUntilExit()
Run() is expected to be used after a pause at this point.
Make RunUntilExit() notify using the StartRunning variable entirely.

This is because if we used Run() then the initial thread hasn't yet set
up its core or signal handlers. So it'll ignore SIG63. Then its reason
for pause is still set to Resume.
Once an application tries sending SIG63, we try restoring the context
from a paused state that does exist and crash
2022-01-03 03:29:35 -08:00
Mai M 154468c172 Merge pull request #1492 from Sonicadvance1/disable_unsync_log_message
Dispatcher: Disables unsync context message
2022-01-03 06:26:08 -05:00
Mai M 0e45b98fb2 Merge pull request #1490 from Sonicadvance1/fix_cpuid_memcpy
CPUID: Fixes another memory overflow issue
2022-01-03 06:24:47 -05:00
Ryan Houdek 2d573f7bf9 Linux: Fixes pidfd_send_signal for 32-bit
Due to the way this syscall handles siginfo_t, we only need to shift
data through the padding.

This is due to the syscall not allowing you to send siginfo_t with
si_code being positive. Which is what the kernel uses to interpret the
data if it is reinterpreting it.
2022-01-03 03:19:12 -08:00
Ryan Houdek 01587270d9 Dispatcher: Disables unsync context message
This message can actually cause a perfectly running application to crash
due to signaling at the wrong time.

Instead constexpr disable it so if you want to see it. You can still
enable it
2022-01-03 03:02:33 -08:00
Ryan Houdek 0a63a15287 CPUID: Fixes another memory overflow issue
Just like the previous one, but this time when we are NOT on a release
2022-01-03 02:56:48 -08:00
Mai M a65e7a09e7 Merge pull request #1489 from Sonicadvance1/release_process_documentation
Docs: Adds documentation about the FEX monthly release process
2022-01-03 01:01:38 -05:00
Ryan Houdek 7e5c561849 Docs: Adds documentation about the FEX monthly release process
Walks through all the steps required for a monthly release.
2022-01-02 21:13:54 -08:00
Mai M a45bc7598c Merge pull request #1487 from Sonicadvance1/rootfs_stop_trying_so_hard
RootFS: Stop trying to retry rootfs after five times
2022-01-01 03:40:40 -05:00
Ryan Houdek 9f8071ce8f RootFS: Stop trying to retry rootfs after five times
In most cases this will be fixed on the first retry, but give it a
chance at five.
Otherwise you have a chance to infinite loop.
2021-12-31 20:04:57 -08:00
Mai M 0808599813 Merge pull request #1486 from Sonicadvance1/add_version_override
CMake: Adds an OVERRIDE_VERSION option
2021-12-31 02:51:15 -05:00
Mai M 340699ca33 Merge pull request #1485 from Sonicadvance1/fix_fortify
FEXCore: Fixes a memcpy overflow in processor brand
2021-12-31 02:51:06 -05:00
Ryan Houdek 00a572c00c CMake: Adds an OVERRIDE_VERSION option
This is intended to be used by a package maintainer to override the
version when the git repo or git executable isn't available.

Not expected to be used by normal users. Instead by our automated ppa
tooling.
2021-12-30 22:55:46 -08:00
Ryan Houdek d300181ab9 FEXCore: Fixes a memcpy overflow in processor brand
When our FEX version string has less than 16 characters this would
result in size_t underflow, resulting in a crash.

This only ever occurs on a release tag so it went uncaught at first.
The PPA builder managed to uncover this problem since it only deals with
release tags.
2021-12-30 22:54:44 -08:00
Ryan Houdek 5146651908 Merge pull request #1482 from Sonicadvance1/override_tune_cpu
CMake: Adds a TUNE_CPU option
2021-12-29 20:36:04 -08:00
Ryan Houdek 875bae41a3 CMake: Adds a TUNE_CPU option
By default we tune to the native CPU, using some heuristics to determine
the true native CPU since Apple doesn't expose an MIDR.

Adds an option that allows the user to pass in the CPU to tune for.
This will be useful for debugging and also for package building.
2021-12-29 17:27:19 -08:00
Mai M 95bd309db9 Merge pull request #1481 from Sonicadvance1/fix_config_dependency
FEXCore: Fixes FEXCore_Base config dependency
2021-12-29 20:07:22 -05:00
Ryan Houdek 6f1e7e7a4c FEXCore: Fixes FEXCore_Base config dependency
This depends on Config. Then everything else links to this, pulling in
the dependency.
2021-12-29 16:52:26 -08:00
Stefanos Kornilios Mitsis Poiitidis b078f51199 Merge pull request #1480 from Sonicadvance1/fix_asmtest_dependencies
unittests: Fixes a missing dependency on ASM tests
2021-12-29 15:37:32 +02:00
Ryan Houdek d03856f0b7 unittests: Fixes a missing dependency on ASM tests
The output asm folder needs to be created before the config file can be
generated. Otherwise the python script will fail.
2021-12-29 00:27:13 -08:00
Ryan Houdek 98210fdf47 Merge pull request #1479 from Sonicadvance1/fix_msqid_ds
Linux: Finish 32-bit msqid_ds usage
2021-12-28 23:37:20 -08:00
Ryan Houdek 3d1cef4afe Linux: Documents msgctl syscall only supporting IPC_64
I thought this was incorrect but since it only supports IPC_64 it is
actually correct.
If a 32-bit application wants to use the old encoding then it needs to
use the ipc syscall instead.
Fixes #1253
2021-12-28 23:04:53 -08:00
Ryan Houdek 060ab98378 Linux: Implements 32-bit msgctl IPC_SET
Needed struct version conversion support.
Removes the remaining UNHANDLED defines since we support them all now.
2021-12-28 23:02:18 -08:00
Ryan Houdek 1d5bd1520a Merge pull request #1478 from Sonicadvance1/stop_hardcode
Stop hardcoding /usr/bin paths
2021-12-28 22:35:15 -08:00
Ryan Houdek 9eca823e84 Stop hardcoding /usr/bin paths
Instead of using execve, use execvpe.
This glibc helper will search PATH or `confstr(_CS_PATH)`

Fixes #1475
2021-12-28 21:40:22 -08:00
Ryan Houdek cd4269f4e8 Merge pull request #1477 from Sonicadvance1/implement_into
FEXCore: Implements recoverable INTO instruction
2021-12-28 20:38:32 -08:00
Ryan Houdek 2f1d44f838 unittests: Basic INTO test
This only tests for the non-faulting INTO instruction.

This is because our ASM tests can not test for signals. Nor can it
recover.
2021-12-28 19:58:35 -08:00
Ryan Houdek 95eb456065 FEXCore: Implements recoverable INTO instruction
This instruction is only available on 32-bit x86. On overflow flag set,
this instruction will raise a SIGSEGV with overflow exception set in
siginfo_t's si_code.

This implements the everything required to raise the signal and modify
our signal results that we are giving to the guest.

This is effectively the first step in generating signal frames with
custom data in it but only enough for INTO today.
2021-12-28 19:58:35 -08:00
Ryan Houdek d4b31cd4c6 Merge pull request #1476 from Sonicadvance1/squashfs_robust_lock
RootFS: Be more robust against stale lock files
2021-12-28 17:26:54 -08:00
Ryan Houdek c611eb228f RootFS: Be more robust against stale lock files
In the case of having a stale lock file on the system, usually due to an
unclean shutdown, have FEX retry if the socket for that lock file is
also not available.

This resolves the exact case of:
```
FEXBash xeyes &
kill -9 `pidof FEXMountDaemon`
sudo shutdown -r now
FEXBash xeyes & <--- This command would now fail
```

The first command would do:
 - Check lock file and socket
 - Spin up FEXMountDaemon
 - Create Lock file and Socket file
 - Start executing XEyes in FEXInterpreter

The second command would do:
 - Kills the FEXMountDaemon process
 - This leaves squashfuse running
 - The lock file and socket file are now unmanaged and just files on the filesystem

The third command would do:
 - unmounts the squashfuse mount
 - Leaving a dangling folder as a mount point

The fourth command would do:
 - Check for lock file and socket
 - Attempt to use lock file, socket, and mount that is dangling
 - Fail with error

Now instead fourth command will do:
 - Check for lock file and socket
 - Check if filesystem is valid
 - Knows that lock file exists but socket isn't active at all
 - Deletes lock file
 - Spin up FEXMountDaemon
 - Create Lock File and Socket File
 - Start executing XEyes in FEXInterpreter
2021-12-28 16:48:46 -08:00
Ryan Houdek bc120b9f97 Merge pull request #1471 from Sonicadvance1/remaining_syscalls
Linux: Implements the remaining 32-bit syscalls
2021-12-27 19:25:35 -08:00
Ryan Houdek 995672921f Linux: Implements the remaining 32-bit syscalls
The main one that we can't implement is readdir. Falls in to the same
problem space as getdents.

With this, we have the full entry tables filled out for both 64-bit and
32-bit. With some holes in the implementation, we have almost all
coverage now.
2021-12-27 02:19:30 -08:00
Ryan Houdek 514b6a822f Linux: Oops, forgot to pass usig to io_pggetevents
Easy enough to fix
2021-12-27 02:17:52 -08:00
Ryan Houdek 0945c728e9 Linux: Oops, missed fsmount syscall implementation
Simple fix
2021-12-27 02:17:20 -08:00
Ryan Houdek 3a3e2776ba Adds Game bug issue template 2021-12-25 12:29:59 -08:00
Ryan Houdek b72245a2d5 Merge pull request #1469 from Sonicadvance1/remove_0x
FEXRootFSFetcher: Remove 0x prefix on hash
2021-12-24 20:46:43 -08:00
Ryan Houdek 9c49bd3c8e FEXRootFSFetcher: Remove 0x prefix on hash
So as to not confuse users who just copy and paste the hash without
thinking
2021-12-24 20:33:23 -08:00
Ryan Houdek c691d70919 Merge pull request #1467 from Sonicadvance1/fix_really_old_ubuntu
Improves compile ability for older libraries
2021-12-24 18:50:39 -08:00
Ryan Houdek f3a27a57f1 Merge pull request #1468 from Sonicadvance1/experimental_libcxx
CMake: Adds experimental libc++ option
2021-12-24 18:50:31 -08:00
Ryan Houdek edce981824 Merge pull request #1463 from Sonicadvance1/enable_threads_default
Config: Enables all host threads by default
2021-12-24 18:50:20 -08:00
Ryan Houdek 9bffaeea40 Merge pull request #1462 from Sonicadvance1/sanitize_core_option
Config: Sanitize Core option
2021-12-24 18:50:11 -08:00
Ryan Houdek 88ce9b5cd2 Merge pull request #1461 from Sonicadvance1/fix_gdb_map
GDBServer: Fixes long string packet encodings
2021-12-24 18:50:01 -08:00
Ryan Houdek 72e8a997f6 Merge pull request #1460 from Sonicadvance1/FEXRootFSFetcher
FEXRootFSFetcher: Adds a new tool to help set up a new RootFS
2021-12-24 18:49:50 -08:00
Ryan Houdek d708cbad5e Config: Always fixup Threads option
In the case of nothing being set then with the default being zero now we
will not have fixed it up to calculate the number of threads based on
host core count.
2021-12-24 16:32:58 -08:00
Ryan Houdek 152eaff00b CMake: Adds experimental libc++ option
FEX currently doesn't compile with it mainly because libc++ doesn't
support c++ pmr at all.
2021-12-24 15:46:58 -08:00
Ryan Houdek 2079f6b3c7 Improves compile ability for older libraries
Adds a header only include utility folder that can be included from
everywhere.

Contains syscall helpers for older glibc and defines for older Linux
uapi headers missing some defines.
2021-12-24 15:43:25 -08:00
Ryan Houdek 2c31080fd4 FEXRootFSFetcher: Adds a new tool to help set up a new RootFS
This tool supports both a zenity and tty interface.
TTY will be presented if available while Zenity will be used otherwise.

This tool pulls a rootfs list from https://rootfs.fex-emu.org/
It then allows you to select a rootfs from the list, download it, place
it in to the correct working folder, extract it if desired, and set it
as the current default RootFS.

This requires curl and potentially zenity installed to use.
Maybe also unsquashfs if the user chooses to extract the image.

This is an all in one tool to quickly get a new user up and running.

Additionally if you pass in a file path in to the tool, it will generate
an xxhash of the file and exit out. This is the hash used to ensure the
files are valid.
2021-12-24 13:38:20 -08:00
Ryan Houdek ef7b77dff7 Config: Enables all host threads by default
We've solved the few threading problems that we had before. So now allow
FEX to query the host for the number of threads.
2021-12-24 13:22:01 -08:00
Ryan Houdek 777aadb73e Config: Sanitize Core option
If set to an invalid Core option from the json file then sanitize it
back to JIT.

Otherwise FEX has a chance of just crashing.
2021-12-24 13:21:47 -08:00
Ryan Houdek ccd06e2097 GDBServer: Fixes long string packet encodings
Fixes thread, memory-map, and OS data packet types.
These were attempting to substr when the encode function already handles
that.
Was making it so gdb was only ever receiving the first 1000 bytes of the
data and then decoding incorrectly.
2021-12-24 13:21:35 -08:00
Ryan Houdek 09a5f8c6b5 FEX: Move common config loading to a helper 2021-12-24 13:21:24 -08:00
Ryan Houdek 8f835678f5 Merge pull request #1466 from Sonicadvance1/fix_old_ubuntu3
Linux: Fixes build on Ubuntu 20.04 take 3
2021-12-24 13:20:45 -08:00
Ryan Houdek 8f2cd39802 Linux: Fixes build on Ubuntu 20.04 take 3
Fixes #1457
Header is necessary for x64 types as well for struct verifier
2021-12-24 13:05:32 -08:00
Ryan Houdek 059e8a9099 Merge pull request #1465 from Sonicadvance1/fix_old_ubuntu2
Linux: Fixes build on Ubuntu 20.04 take 2
2021-12-24 12:40:34 -08:00
Ryan Houdek 5fae07f6ee Linux: Fixes build on Ubuntu 20.04 take 2
Fixes #1457

Header include order matters here. Rude.
2021-12-24 12:23:52 -08:00
Ryan Houdek 02005e71ca Merge pull request #1464 from Sonicadvance1/fix_old_ubuntu
Linux: Fixes build on Ubuntu 20.04
2021-12-24 12:11:49 -08:00
Ryan Houdek 59cc5228b8 Linux: Fixes build on Ubuntu 20.04
Fixes #1457
2021-12-24 02:46:29 -08:00
Ryan Houdek 117cbde226 Merge pull request #1455 from Sonicadvance1/detect_fpcr_exceptions
HostFeatures: Detect if the host CPU suports float exceptions
2021-12-23 13:42:08 -08:00
Ryan Houdek 565d1e27d7 Merge pull request #1456 from Sonicadvance1/fix_sized_constant
IR: Fixes sized constant mask
2021-12-18 23:31:33 -08:00
Ryan Houdek 22ecff05c9 IR: Fixes sized constant mask
This mask was subtracting backwards, which made anything smaller than 8
byte constants not mask correctly
2021-12-18 23:14:55 -08:00
Ryan Houdek 17480c0e2d HostFeatures: Detect if the host CPU suports float exceptions
On x86 this is always supported.
On ARM this is only supported if FPCR writes actually enable the things.
Also detects the AFP feature for flushing input denormals to zero.

These are all part of the x86 MXCSR.
No Cortex supports FPCR exceptions, while Apple M1 CPUs support
Exceptions but not the true "AFP" extension
Apple instead supports some additional flags in their
`SYS_APL_AFPCR_EL0` register for enabling this.
2021-12-17 16:12:31 -08:00
Ryan Houdek 0608d95322 Updates externals vixl 2021-12-17 16:11:40 -08:00
Ryan Houdek 47bd47950b Merge pull request #1454 from Sonicadvance1/fix_aarch64_ubuntu_20.04
Linux: Fixes Ubuntu 20.04 compilation on AArch64
2021-12-16 22:41:05 -08:00
Ryan Houdek 9aab7e07f4 Linux: Fixes Ubuntu 20.04 compilation on AArch64
This header was missing
2021-12-16 22:23:32 -08:00
Mai M f2fd9d9e3f Merge pull request #1451 from Sonicadvance1/fix_semid_ds
Linux: Fixes semid_ds_64 definition
2021-12-17 01:19:41 -05:00
Mai M 45430e9569 Merge pull request #1452 from Sonicadvance1/fix_32bit_shmctl
Linux: Fixes 32-bit shmctl
2021-12-17 01:19:30 -05:00
Mai M be3e3a351a Merge pull request #1453 from Sonicadvance1/fix_cpuid_crash
CPUID: Fixes crash on unknown CPU
2021-12-17 01:19:19 -05:00
Ryan Houdek 2cee9e5d4b CPUID: Fixes crash on unknown CPU
If the CPU is unknown inside of the ARM CPU detection then the
MIDROption selected could have fallen down a path where it is set to
nullptr.

Resolve this crash by doing a nullptr check.
2021-12-16 21:50:42 -08:00
Ryan Houdek 03cc35f341 Linux: Fixes 32-bit shmctl
We were dereferencing the shmun ptr instead of using it directly.
Resulting in an almost immediate crash

Additionally IPC_SET doesn't write back to the shmid_ds provided.

Additional still SHM_STATE/SHM_STAT_ANY/IPC_STAT wasn't writing its
result back to the guest. Resulting in invalid stat information for the
guest.

Additionally SHM_INFO doesn't follow IPC64 behaviour since there is no
shm_info 64-bit type for a 32-bit OS.
The kernel just truncates the results in this case. Could be an
oversight on the kernel dev's side?

Fixes #1252
2021-12-15 23:34:15 -08:00
Ryan Houdek 3aead5ef65 Linux: Move some 32-bit ipc types to Types.h
This way we can run the struct verifier over it
2021-12-15 23:29:08 -08:00
Ryan Houdek b9abd091d5 Linux: Fixes semid_ds_64 definition
Wasn't quite the expected definition.

Fixes #1450
2021-12-15 22:02:37 -08:00
Ryan Houdek 43dc232e5a Merge pull request #1442 from Sonicadvance1/more_aot_changes
More AOT code movement
2021-12-15 03:27:01 -08:00
Ryan Houdek ea73c9d7ea Merge pull request #1443 from Sonicadvance1/squashfs_fixes
FEXMountDaemon squashfs fixes.
2021-12-15 03:04:15 -08:00
Ryan Houdek fa6f1b1d90 Merge pull request #1446 from Sonicadvance1/more_fault_reconstruction
Dispatcher: Adds more state reconstruction to state restore
2021-12-15 03:04:09 -08:00
Ryan Houdek e24eb7a72d Merge pull request #1447 from Sonicadvance1/consolidate_host_features
HostFeatures: Consolidates HostFeatures flags
2021-12-15 02:55:42 -08:00
Ryan Houdek e05d116b05 Merge pull request #1448 from Sonicadvance1/fix_proton_experimental
Linux: Adds the safe syscall unimplemented gap for x86-64
2021-12-15 02:55:35 -08:00
Ryan Houdek ee821b9cbf Merge pull request #1449 from Sonicadvance1/implement_rdtscp
Implements RDTSCP
2021-12-15 02:52:47 -08:00
Ryan Houdek c29e563836 unittests: Adds tests for RDTSCP 2021-12-15 01:21:16 -08:00
Ryan Houdek fd59fb1a7a OpcodeDispatcher: Implements support for RDTSCP
Have fun
2021-12-15 01:18:34 -08:00
Ryan Houdek 04deeb3911 IR: Adds Processor ID IR op
For the x86-64 JIT this is implemented with pulling rdtscp's result for
this value.
For Interpreter and AArch64 JIT this is implemented with the getcpu
syscall.

Theoretically AArch64 could implement this with MPIDR_EL1 but because
SoC vendors hecked this up, we can't. Thanks.
Kernel just returns zero + reserved bits if you try reading it.
2021-12-15 01:17:24 -08:00
Ryan Houdek 4a7c65b20d Linux: Adds the safe syscall unimplemented gap for x86-64
Syscalls on x86-64 have a gap in the range of [335, 424) where these are
defined as returning ENOSYS.
This was a decision that the Linux developers decided on so x86-64 can
realign its syscall numbers to a common infrastructure.

Turns out that Proton Experimental was using a 32-bit only syscall to
check if the feature existed and was listening for ENOSYS.
They did this unconditionally on both x86 and x86-64 and it hits this
gap section.

This was introduced in Proton Experimental with commit
563fb0fbe2a49734aede87bedccb20daefc553b0
Message: "ntdll: Use clock_gettime64 if supported."

This is technically valid since it falls within this gap section but
leaves a bad taste.

We can safely ignore this section so set it up with the
UnimplementedSyscallSafe handler and also assign it to Syscall MAX so it
can go down our fast handler.

Not that this is likely to be a performance critical path.

Fixes Proton Experimental crashing when run under FEX.
2021-12-14 03:45:55 -08:00
Ryan Houdek 23cb0dea00 HostFeatures: Consolidates HostFeatures flags
Some of these were in the Emitter class and some were in the
HostFeatures.

Merge these together since in the future I'm going to be using all of
this data as a key for our AOT code cache.
2021-12-14 02:41:42 -08:00
Ryan Houdek bcad7a9eea Dispatcher: Adds more state reconstruction to state restore
Still only setting the new state if RIP is affected for now.
Noticed a bug where we weren't setting our frame RIP to the new RIP on
32-bit.
Decided to walk through more of the state setting while fixing that.

This gets #1214 further but then it eventually crashes with a read to
0x11.
2021-12-14 02:36:24 -08:00
Ryan Houdek 0a3a270a63 Merge pull request #1445 from lioncash/pdep
OpcodeDispatcher: Handle PDEP
2021-12-13 16:43:35 -08:00
lioncash 06e4a5a5b7 CPUID: Signify full support for BMI2
With PDEP support dropped in, we now support all of BMI2, so we can
signify that we support it in our emulated CPUID.
2021-12-13 14:07:41 -05:00
lioncash 6ff80670b3 OpcodeDispatcher: Handle PDEP
Now all of BMI2 is handled.
2021-12-13 14:06:55 -05:00
lioncash 38eea80b8d IR: Add PDep IR opcode 2021-12-13 13:54:15 -05:00
Ryan Houdek c43af0e10f Merge pull request #1444 from neobrain/fix_jemalloc_overrides
Update jemalloc for fixed libc overrides
2021-12-13 04:48:05 -08:00
Tony Wasserka 2fa8cd7e97 Update jemalloc for fixed libc overrides 2021-12-13 13:38:39 +01:00
Ryan Houdek 5d0734a7f2 FEXMountDaemon: Attempt to increase FD max when close to limit
On FEXMountDaemon startup, fetch the FD limit and current number of open
files. This allows us to track the number of open FDs we have.

Once we get close the the safe threshold we then try increasing the soft
limit towards the hard limit. If we bump up to the max (Say someone
setting the hard limit to 256) then print some fairly big warning
messages.

With the previously fixed FD leak, we could very quickly hit the pipe
limit because Steam is constantly opening new processes all the time.
With the leak fixed, running Steam and idling, FEXMountDaemon only has
around 16 pipes being watched.
2021-12-12 22:51:53 -08:00
Ryan Houdek 3e0e922fef FEXMountDaemon: Fixes EPoll thread not shutting down on SIGTERM
This is another edge case where if the process was sent a SIGTERM then
it can get in to a weird state where pipe tracking is broken.

We already check for zero pipes being tracked at the end of this loop,
just check for the shutdown variable to be set instead
2021-12-12 22:51:15 -08:00
Ryan Houdek e2004f4999 FEXMountDaemon: Fix leaked pipe FDs
Even though we were removing the pipe from the epoll interest list,
we were failing to close the pipe fd. This was leaking the FD and
running in to the Linux process FD limitation over time.

This fixes a class of issues where if you were running FEXMountDaemon
that hit the max, then there were FEX processes not being watched. Which
means that FEXMountDaemon could exit while FEX was still trying to run.

Usually a non-issue since squashfuse would keep running in the
background since open files in the mount were still in use.

Could hit fun race conditions because of it though.
2021-12-12 22:45:40 -08:00
Ryan Houdek 13f37efe12 SquashFS: Work around a race condition on FEXMountDaemon ack
On an overburdened system then FEXMountDaemon may notrespond within the two
seconds of FEX waiting for an ack.
In this case then we hit a bad edge case where we overwrite the lock
file with a new FEXMountDaemon instance then spin up a new
FEXMountDaemon.
This new FEXMountDaemon will recognize that one is still running and
early exit, but now that we've overwritten the lock file with a new
location, all future FEX instances won't be able to find the squashfs
rootfs.

Instead, ignore that the ack wasn't received and attempt running
instead.

FEXMountDaemon should still pick up the pipe message and monitor FEX
regardless.

This fixes the `/usr not found` messages and crashes resulting from it.
2021-12-12 22:44:48 -08:00
Ryan Houdek 80e66a36eb FEXConfig: Adds AOT options to the UI 2021-12-12 18:22:02 -08:00
Ryan Houdek 4998d35ec5 FEXLoader: Moves AOT handling to its own independent files
This is only moving it outside of FEXLoader to make it easier to work
on.

I'll be hammering on this more soon.
2021-12-12 18:19:12 -08:00
Ryan Houdek f0655874e3 ELFCodeLoader: Fixes bug in ELF section tracking
Each section map invocation was clearing the sections pointer, meaning
we only ever had one section in the vector.

We want to keep all of the sections around from the ELFCodeLoader.
Fixes a bug where we were effectively never catching any code segments
from the executable passed in.
2021-12-12 18:15:06 -08:00
Ryan Houdek 9394e49c95 Merge pull request #1431 from Sonicadvance1/expose_arm_names
CPUID Expose Hybrid flag and CPU names
2021-12-12 18:07:53 -08:00
Ryan Houdek 91984003b6 Merge pull request #1440 from lioncash/pext
OpcodeDispatcher: Handle PEXT
2021-12-10 16:27:49 -08:00
lioncash dbf571fdfb OpcodeDispatcher: Handle PEXT 2021-12-10 19:03:58 -05:00
lioncash b6abcc5e3c IR: Add PExt opcode 2021-12-10 19:03:54 -05:00
Ryan Houdek e99a23cfc6 Merge pull request #1439 from neobrain/opt_x86tables_init
Speed up initialization of X86Tables
2021-12-10 09:11:47 -08:00
Ryan Houdek 425d9323d5 Merge pull request #1295 from neobrain/feature_new_thunk_gen
Implement thunk library generation using libclang
2021-12-10 09:10:00 -08:00
Tony Wasserka a7c0997daf OpcodeDispatcher: Mark const-initialized instruction tables as constexpr
This allows the compiler back these tables into the executable, which
reduces the amount of work the function has to do at runtime.

Reduces the runtime of this function by 30% relative to the previous commit.
2021-12-10 13:25:24 +01:00
Tony Wasserka 6eba3f331e OpcodeDispatcher: Store instruction tables as arrays instead of std::vector
Reduces the runtime of this function by 90%.
2021-12-10 13:25:23 +01:00
Tony Wasserka bcc75e3312 X86Tables: Use stack-allocated arrays instead of std::vector 2021-12-10 13:25:23 +01:00
Tony Wasserka 30672e1517 X86Tables: Remove unneeded initialization code from the Debug mode path
The tables have recently been changed to be zeroed out as a whole on startup.

Speeds up InstallDebugInfo by about two orders of magnitude and reduces Debug
executable size by 1.8%.
2021-12-10 13:25:23 +01:00
Tony Wasserka b6e46fd44d Thunks: Update documentation to reflect generator changes 2021-12-10 11:25:01 +01:00
Tony Wasserka 32a0e37569 Thunks: Remove now unneeded generator files 2021-12-10 11:25:01 +01:00
Tony Wasserka 5b6175b702 Remove now unused Vulkan-Docs submodule 2021-12-10 11:25:01 +01:00
Tony Wasserka 0285e35c87 Thunks: Use libclang-based code generation for libvulkan 2021-12-10 11:25:01 +01:00
Tony Wasserka bbfb8713a7 Thunks: Drop struct verifier tests for Vulkan
This is currently not supported with the new generator.
2021-12-10 11:25:00 +01:00
Tony Wasserka c1d6967cb6 Thunks/gen: Add support for expanding symbol lists using a macro 2021-12-10 11:25:00 +01:00
Tony Wasserka 20e24eae9c Thunks: Use libclang-based code generation for libdrm 2021-12-10 11:25:00 +01:00
Tony Wasserka b8d9027680 Thunks: Use libclang-based code generation for libxshmfence 2021-12-10 11:25:00 +01:00
Tony Wasserka 688ef9f5af Thunks: Use libclang-based code generation for libxcb-xfixes 2021-12-10 11:25:00 +01:00
Tony Wasserka 181c9074ab Thunks: Use libclang-based code generation for libxcb-sync 2021-12-10 11:25:00 +01:00
Tony Wasserka 6f85f64bfd Thunks: Use libclang-based code generation for libxcb-shm 2021-12-10 11:25:00 +01:00
Tony Wasserka ba6ee61db2 Thunks: Use libclang-based code generation for libxcb-randr 2021-12-10 11:25:00 +01:00
Tony Wasserka e735281b2c Thunks: Use libclang-based code generation for libxcb-present 2021-12-10 11:24:59 +01:00
Tony Wasserka c8b2d714a8 Thunks: Use libclang-based code generation for libxcb-glx 2021-12-10 11:24:59 +01:00
Tony Wasserka bba0625a57 Thunks: Use libclang-based code generation for libxcb-dri3 2021-12-10 11:24:59 +01:00
Tony Wasserka c9a7b36210 Thunks: Use libclang-based code generation for libxcb-dri2 2021-12-10 11:24:59 +01:00
Tony Wasserka 99bf02f27f Thunks/gen: Add support for libraries with "-" in the filename 2021-12-10 11:24:59 +01:00
Tony Wasserka a2c02ae51e Thunks: Use libclang-based code generation for libxcb
Notably, xcb_take_socket uses the callback_guest annotation, since the
given callback is never called on the host (instead it's manually forwarded
back to a helper guest thread for calling).
2021-12-10 11:24:59 +01:00
Tony Wasserka c7b59143c6 Thunks/gen: Add custom guest entrypoint annotation 2021-12-10 11:24:59 +01:00
Tony Wasserka 1e23e61572 Thunks: Use libclang-based code generation for libEGL 2021-12-10 11:24:59 +01:00
Tony Wasserka f4dd1a895e Thunks: Use libclang-based code generation for libGL 2021-12-10 11:24:58 +01:00
Tony Wasserka 3a2f9a4d46 Thunks/gen: Support guest-side symbol tables and custom host symbol loaders 2021-12-10 11:24:58 +01:00
Tony Wasserka 4a848d7202 Thunks: Use libclang-based code generation for libX11 2021-12-10 11:24:58 +01:00
Tony Wasserka 1b58ed9f57 Thunks/X11: Fix incorrect library version 2021-12-10 11:24:58 +01:00
Tony Wasserka 1c86f7ed36 Thunks/gen: Support annotating guest-exclusive function pointers
Function pointer arguments given to the guest thunk library aren't
callable on the host, so they need special handling on a case-by-case
basis. Similar problems arise when the guest-side tries to consume a
pointer returned from a native host library. To automate some common
scenarios, this change adds two new annotations:

"callback_guest" indicates the callback parameter is never called on the
host and hence can be marshalled like any other argument. Accidental host
calls to the function pointer are prevented by wrapping it in an opaque
type alias.

"returns_guest_pointer" indicates that the host function returns a pointer
usable in the guest context. This applies e.g. to functions that derive
the returned pointer from input guest pointer arguments.
2021-12-10 11:24:58 +01:00
Tony Wasserka 2733b2ee1e Thunks/gen: Add annotation for custom host implementations 2021-12-10 11:24:58 +01:00
Tony Wasserka 4ceb2dfdf2 Thunks: Use libclang-based code generation for libXext 2021-12-10 11:24:58 +01:00
Tony Wasserka bd380e0f15 Thunks: Use libclang-based code generation for libXrender 2021-12-10 11:24:58 +01:00
Tony Wasserka 73ec786c60 Thunks: Use libclang-based code generation for libasound 2021-12-10 11:24:57 +01:00
Tony Wasserka e3a2c8dc80 Thunks: Use libclang-based code generation for libXfixes 2021-12-10 11:24:57 +01:00
Tony Wasserka d3b14df840 Thunks/gen: Add versioning support 2021-12-10 11:24:57 +01:00
Tony Wasserka a12ab8f98a Thunks/gen: Add support for "callback_stub" annotations
Some applications set callbacks that never get called in practice (such as
error handlers). It's sensible to just not implement these instead of
cluttering the code with effectively unused callback wrappers.
2021-12-10 11:24:57 +01:00
Tony Wasserka 966b9a69d8 Thunks/gen: Add support for function pointer parameters ("callbacks") 2021-12-10 11:24:57 +01:00
Tony Wasserka 927d3d00e2 Thunks/gen: Add support for thunking variadic functions 2021-12-10 11:24:57 +01:00
Tony Wasserka bfb9cabeb8 Thunks/gen: Add support for generation of host library functions 2021-12-10 11:24:57 +01:00
Tony Wasserka 22466a973c Thunks/gen: Add unit tests 2021-12-10 11:24:57 +01:00
Tony Wasserka c05e1c9797 Thunks: Add a new code generator based on libclang 2021-12-10 11:24:56 +01:00
Ryan Houdek c65be9f55d Merge pull request #1438 from Sonicadvance1/remove_jemalloc_from_fexconfig
FEXConfig: Removes jemalloc usage
2021-12-09 18:10:32 -08:00
Ryan Houdek edc31fe0a3 FEXConfig: Removes jemalloc usage
Splits out the few required dependencies to a FEXCore_Base static
library.

FEXCore then links to this directly.

Then make it so the FEX Common code links to FEXCore_Base so the
jemalloc dependency doesn't get pulled in.
2021-12-09 17:54:53 -08:00
Ryan Houdek d4655fbb17 Merge pull request #1424 from Sonicadvance1/aot_code_movement
FEXCore: Reorganizes some AOT related code
2021-12-09 13:55:50 -08:00
Ryan Houdek 5f0dfcd715 Merge pull request #1422 from Sonicadvance1/implement_clzero
Implements CLZero instruction
2021-12-09 13:55:31 -08:00
Ryan Houdek e62cb417c2 Merge pull request #1437 from Sonicadvance1/disable_flake_gvisor
gvisor: Disables flaky test
2021-12-09 12:58:29 -08:00
Ryan Houdek df486c0786 gvisor: Disables flaky test 2021-12-09 12:37:58 -08:00
Ryan Houdek 9d43904792 Merge pull request #1436 from lioncash/context-const
Context: Take some arguments as pointer-to-const
2021-12-09 12:23:34 -08:00
Ryan Houdek 5654f9a030 Merge pull request #1435 from Sonicadvance1/fix_vixl_assertions
Arm64: Fixes vixl assertions around ubfm usage
2021-12-09 11:54:41 -08:00
Ryan Houdek d74cf6d8d8 CPUID Expose Hybrid flag and CPU names
Had some idle time so I implemented this logic.

We do some tricky logic to have a big.little configuration even with
unknown CPU core types. Promoting or demoting a single MIDR depending on
if we have a mixed configuration or not.

In a non-hybrid design we only claim product names inside the CPUID
product string.

This will appear if you `/proc/cpuinfo` or read the CPUID registers
directly

eg on Snapdragon 888:
processor       : 0
model name      : FEX-2112-1-g13b14b85            Cortex-A55
processor       : 1
model name      : FEX-2112-1-g13b14b85            Cortex-A55
processor       : 2
model name      : FEX-2112-1-g13b14b85            Cortex-A55
processor       : 3
model name      : FEX-2112-1-g13b14b85            Cortex-A55
processor       : 4
model name      : FEX-2112-1-g13b14b85            Cortex-A78
processor       : 5
model name      : FEX-2112-1-g13b14b85            Cortex-A78
processor       : 6
model name      : FEX-2112-1-g13b14b85            Cortex-A78
processor       : 7
model name      : FEX-2112-1-g13b14b85            Cortex-X1

eg on Macbook Pro VM which can't see the CPU type:
processor       : 0
model name      : FEX-2112-1-g13b14b85            Unknown ARM CPU
processor       : 1
model name      : FEX-2112-1-g13b14b85            Unknown ARM CPU
processor       : 2
model name      : FEX-2112-1-g13b14b85            Unknown ARM CPU
processor       : 3
model name      : FEX-2112-1-g13b14b85            Unknown ARM CPU
processor       : 4
model name      : FEX-2112-1-g13b14b85            Unknown ARM CPU
processor       : 5
model name      : FEX-2112-1-g13b14b85            Unknown ARM CPU
processor       : 6
model name      : FEX-2112-1-g13b14b85            Unknown ARM CPU
processor       : 7
model name      : FEX-2112-1-g13b14b85            Unknown ARM CPU
2021-12-09 11:39:52 -08:00
Ryan Houdek 84905d2856 unittests: Adds 32-bit inline syscall test
Triggers the ubfm bug on AArch64.
Uses a fairly benign syscall that gets inlined
2021-12-09 11:34:35 -08:00
Ryan Houdek 92cb9d477e Arm64: Fixes vixl assertions around ubfm usage
vixl has an assert check to ensure the register sizes are the same for
ubfm.

32-bit Inline syscalls hit this for both arguments and return value.
VCastFromGPR would have hit this but there aren't any x86 instructions
that move 8-bit and 16-bit values in to a vector register.
2021-12-09 11:34:20 -08:00
lioncash 7ed6007252 Context: std::move functions in signal registration functions
Prevents potential reallocations, given we're using std::function here.
2021-12-09 14:31:47 -05:00
Ryan Houdek b6499ac724 Merge pull request #1434 from lioncash/syscallthread
{x32, x64}/Thread: Make use of .data() instead of .at(0)
2021-12-09 11:28:39 -08:00
lioncash c73991f467 Context: Take some arguments as pointer-to-const
Several API functions act as state querying functions. These can take
some parameters by const to communicate that we don't intend to modify
the respective passed in instance.
2021-12-09 14:26:35 -05:00
lioncash 0bf0fe4779 {x32, x64}/Thread: Make use of .data() instead of .at(0)
Follows the previous PRs of using .data() instead of .at(0) to retrieve
the buffer pointers.
2021-12-09 14:14:40 -05:00
Ryan Houdek 097f48f3ff Merge pull request #1433 from lioncash/syscallmsg
x32/Socket: Resolve sign comparison mismatch cases
2021-12-09 11:09:07 -08:00
Ryan Houdek 75e5df545a Merge pull request #1432 from lioncash/syscall-warn
LinuxSyscalls: Enable warnings
2021-12-09 10:51:21 -08:00
lioncash 171d5f7263 x32/Socket: Use .data() instead of .at(0) for pointer retrieval
.at(0) can blow up in the case we receive a message from user code with msg_iovlen=0
.data() will simply return a pointer to the empty buffer and we can
proceed with forwarding the call to the syscall.
2021-12-09 13:49:15 -05:00
lioncash 970067d19b x32/Socket: Resolve sign comparison mismatch cases 2021-12-09 13:43:26 -05:00
lioncash c26ff60949 LinuxSyscalls: Enable warnings
Enables some basic warnings that we have enabled for FEXCore in order to
prevent some small things from slipping through.
2021-12-09 12:58:54 -05:00
lioncash 04d830ed39 LinuxSyscalls: Minor tidy up of CMakeLists 2021-12-09 12:46:39 -05:00
Ryan Houdek b9c49027c7 Merge pull request #1409 from Sonicadvance1/reentrantmutex
Adds a new ReentrantMutex to use for FEXCore
2021-12-08 14:00:12 -08:00
Ryan Houdek 9860e8b71b Merge pull request #1430 from lioncash/net
NetStream: Move NetBuf definition into cpp file
2021-12-08 13:10:06 -08:00
lioncash a1e94a9863 NetStream: Move NetBuf definition into cpp file
Keeps the NetBuf class completely internal and also lessens the header
dependencies for NetStream.
2021-12-08 10:41:55 -05:00
lioncash 2945c13dcb NetStream: Mark virtual functions as override
Makes it explicit that we're overriding the interface of the class being
derived from.
2021-12-08 10:33:18 -05:00
lioncash 4f68821aef NetStream: Mark constructors as explicit
Prevents implicit conversion of ints to NetStreams.
2021-12-08 10:30:38 -05:00
Ryan Houdek 13087f8425 Merge pull request #1426 from Sonicadvance1/fcmp_unittests
Arm64: Fixes MapSelectCC FGT flag
2021-12-08 05:26:15 -08:00
Ryan Houdek fd0424768b Merge pull request #1427 from Sonicadvance1/enable_unaligned_atomic_tests
unittests: Enables remaining unaligned atomic tests on ARMv8.0
2021-12-08 05:26:03 -08:00
Ryan Houdek 5c4112f103 Merge pull request #1428 from Sonicadvance1/fix_known_failures
unittests: Fixes known failures
2021-12-08 05:25:50 -08:00
Ryan Houdek e8670bbab2 Merge pull request #1425 from Sonicadvance1/x86tables_unknown
X86Tables: Build Unknown op definition tables at compile time
2021-12-08 00:22:57 -08:00
Ryan Houdek 13b14b857b unittests: Fixes known failures
These were just incorrect results and I had failed to fix them.
2021-12-07 22:06:36 -08:00
Ryan Houdek 48c7ff2a23 unittests: Enables remaining unaligned atomic tests on ARMv8.0 2021-12-07 21:54:03 -08:00
Ryan Houdek 77a032a286 unittests: Adds tests for handling cmp merging
As a continuation to #1404 these unit tests were needing to be added.
This tests both select and conditional branch merging for integer ops
and float ops.

This is what discovered the issue fixed in f2de640395
2021-12-07 18:45:54 -08:00
Ryan Houdek f2de640395 Arm64: Fixes MapSelectCC FGT flag
As a continuation to #1404, this was mapped to the incorrect Arm64 flags
2021-12-07 18:43:50 -08:00
Ryan Houdek 9a642158e0 Workaround fmt not handling nullptr strings
Simple ternary check each time a name is used
2021-12-07 12:23:56 -08:00
Ryan Houdek dc44caa178 Github: Adds the new APITests to the github workflow 2021-12-07 11:55:26 -08:00
Ryan Houdek 081a003c6c unittests: Adds a new catch2 test for testing the InterruptableConditionVariable 2021-12-07 11:55:26 -08:00
Ryan Houdek 4c5fc6e813 FEXCore: Uses the InterruptableConditionVariable for StartRunning event
This can be used with the gdbstub and on pause could cause longjumps
out of the event.
Resolves the hang on InternalThreadState destruction.
2021-12-07 11:55:26 -08:00
Ryan Houdek 253cdb552d FEXCore/Utils: Adds InterruptableConditionVariable class
Mutex destruction is not safe in the face of longjump and signals.
Adds a new class that is very simple and will still work in this
instance.
2021-12-07 11:55:26 -08:00
Ryan Houdek 6404aba6e2 X86Tables: Build Unknown op definition tables at compile time
It doesn't make any sense anymore to have specific instruction names
set for UND versus a nullptr string anymore.

Was useful when we could use it to determine the difference between
undefined from the start versus set in the tables but with unknown
decoding. Which is an edge case.

Now instead just zero initialize the data, which means it is an unknown
type and nullptr name. Which works for use.

Improves initialization time of the InitializeInfoTables function from
423 microseconds to 37 microseconds.
2021-12-06 18:57:02 -08:00
Ryan Houdek 45f919683a FEXCore: Reorganizes some AOT related code
Specifically this tries to avoid changing much behaviour and keeping the
code the same. So most of it is a direct transplant without any
modifications. This is step one of the process so I can start logically
separating the code and making sense of it.

This mostly moves the AOT IR handling to its own independent file for
separation. Cleaning up the Core.cpp file quite heavily.

Two minor behaviour changes that got mixed up with this change.

The first one is an ASAN fix.
This is the FEX_PACKED on the RegisterAllocationData class.
I didn't want to change too heavily how this serialization works but I
wanted to resolve the ASAN error. This may change in the coming work.
Problem was the padding betwene the uint32_t and the PhysicalRegister
wasn't initialized but was being read.
Since it is all uint8_t types afterwards there isn't a perf issue here.

Second fix was a crash that occurs if you're attempting to both capture
and load IR on the same run. This is a quirk where we mmap the original
IR file. Then on shutdown the IR file is getting saved.
At which point we open the IR file again, truncate it, and start
serializing all of the IR data.
The truncation makes it so our mmap of the file is no longer resident,
resulting in a crash when reading our IR cache from the mmap region.
Now open a temporary file and rename it after storing.
Resolves the crash but still doesn't really solve the issue of multiple
processes overwriting the same IR files.
2021-12-06 18:42:21 -08:00
Ryan Houdek c3a6890a40 CMake: Adds Catch build when tests are enabled 2021-12-06 18:27:12 -08:00
Tony Wasserka 07be5a0bae Add Catch2 submodule 2021-12-06 18:27:11 -08:00
Ryan Houdek d6e4da7e77 CPUID: Exposes support for CLZero
Only exposed if the host if the HostFeatures for ARM claim to support
it.
2021-12-03 21:20:40 -08:00
Ryan Houdek ed5d7f6a62 unittests: Adds CLZero unit tests
First test just ensures that an aligned cacheline clear will clear out
the data.
Second test ensures an unaligned address will still only clear the
cacheline the address lies in without touching the other surrounding
cachelines.

Disabled on the host runner since the CI host runner doesn't support the
instruction
2021-12-03 21:19:14 -08:00
Ryan Houdek 18b223811a OpcodeDispatcher: Implements CLZero instruction
This instruction zeroes a cacheline in memory that is weakly ordered and
non-temporal.

It uses the RAX register for where in memory to clear and aligns the
address on cacheline regardless of actual alignment.
2021-12-03 21:15:51 -08:00
Ryan Houdek e1d21f0bff IR: Implements new CacheLineZero op
This zeroes out an emulated 64byte cacheline. Writing zeros to memory.
This very specifically is only 64bytes to match x86 behaviour.
Also specifically non-temporal and weakly ordered. Which matches x86
CLZero behaviour.
2021-12-03 21:14:02 -08:00
Ryan Houdek d2783f2edd HostFeatures: Adds new host feature flag for CLZero
If the DCZID block size matches the CLZero cacheline size. Then claim we
support CLZero here.
Otherwise we don't want to expose support for it.
2021-12-03 21:12:52 -08:00
Ryan Houdek 3377e5a50e HarnessHelpers: Fixes accidental fmt::format
Was meant to be fmt::print. Happened with 62dfccc989

oops
2021-12-03 21:01:59 -08:00
440 changed files with 32601 additions and 24152 deletions

No files matched your search

@@ -0,0 +1,45 @@
---
name: Potential Game Bug
about: A bug in FEX-Emu that causes a problem in a game
title: "[Game]: [Short Problem Description]"
labels: Game related
assignees: ''
---
**What Game**
The game name.
A link to the storefront where to get the game. GOG, Steam, Itch.io, etc
**Describe the bug**
A clear and concise description of what the bug is.
**To Reproduce**
Steps to reproduce the behavior:
1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error
**Expected behavior**
A clear and concise description of what you expected to happen.
**Screenshots and Video**
If applicable, add screenshots and video to help explain your problem.
**System information:**
- OS: [eg: Ubuntu 21.10]
- CPU/SoC: [eg: Snapdragon 888, Intel Core i8-12900k]
- Video driver version: [eg: OpenGL ES 3.2 Mesa 22.0.0-devel (git-9ff086052a)]
- RootFS used: [eg: Ubuntu 21.10 Official Rootfs]
- FEX version: (FEXGetConfig --version) [eg: FEX-2112-155-gc691d709]
- Thunks Enabled: [Yes/No]
**Additional context**
- Is this an x86 or x86-64 game: [x86/x86-64/Both]
- Does this reproduce on x86-64 host with FEX: [Yes/No/Untested]
- Does this reproduce on AArch64 with Radeon/Intel/Nvidia: [Yes/No/Untested]
- Is this a Vulkan game: [Yes/No/Unknown]
- If Yes, What is your Vulkan driver:
Add any other context about the problem here.
+15 -2
View File
@@ -31,7 +31,9 @@ jobs:
- name : submodule checkout
# Need to update submodules
run: git submodule update --init --depth 1
run: |
git submodule sync --recursive
git submodule update --init --depth 1
- name: Clean Build Environment
run: rm -Rf ${{runner.workspace}}/build
@@ -49,7 +51,7 @@ jobs:
# Note the current convention is to use the -S and -B options here to specify source
# and build directories, but this is only available with CMake 3.13 and higher.
# The CMake binaries on the Github Actions machines are (as of this writing) 3.12
run: cmake $GITHUB_WORKSPACE -DCMAKE_BUILD_TYPE=$BUILD_TYPE -G Ninja -DENABLE_LTO=False -DENABLE_ASSERTIONS=True -DENABLE_X86_HOST_DEBUG=True
run: cmake $GITHUB_WORKSPACE -DCMAKE_BUILD_TYPE=$BUILD_TYPE -G Ninja -DENABLE_LTO=False -DENABLE_ASSERTIONS=True -DENABLE_X86_HOST_DEBUG=True -DENABLE_INTERPRETER=True
- name: Build
working-directory: ${{runner.workspace}}/build
@@ -140,6 +142,17 @@ jobs:
working-directory: ${{runner.workspace}}/build
run: mv ${{runner.workspace}}/build/Testing/Temporary/LastTest.log ${{runner.workspace}}/build/Testing/Temporary/LastTest_StructVerifier.log || true
- name: APITest tests
working-directory: ${{runner.workspace}}/build
shell: bash
run: cmake --build . --config $BUILD_TYPE --target api_tests
- name: APITest Test Results move
if: ${{ always() }}
shell: bash
working-directory: ${{runner.workspace}}/build
run: mv ${{runner.workspace}}/build/Testing/Temporary/LastTest.log ${{runner.workspace}}/build/Testing/Temporary/LastTest_APITests.log || true
- name: Truncate test results
if: ${{ always() }}
shell: bash
+8 -5
View File
@@ -1,7 +1,7 @@
[submodule "External/vixl"]
shallow = true
path = External/vixl
url = https://github.com/Sonicadvance1/vixl.git
url = https://github.com/FEX-Emu/vixl.git
[submodule "External/cpp-optparse"]
path = External/cpp-optparse
url = https://github.com/Sonicadvance1/cpp-optparse
@@ -42,7 +42,10 @@
[submodule "External/xxhash"]
path = External/xxhash
url = https://github.com/FEX-Emu/xxHash.git
[submodule "External/Vulkan-Docs"]
shallow = true
path = External/Vulkan-Docs
url = https://github.com/KhronosGroup/Vulkan-Docs.git
[submodule "External/Catch2"]
path = External/Catch2
url = https://github.com/catchorg/Catch2.git
[submodule "External/robin-map"]
shallow = true
path = External/robin-map
url = https://github.com/Tessil/robin-map.git
+188 -85
View File
@@ -18,11 +18,19 @@ option(ENABLE_STATIC_PIE "Enables static-pie build" FALSE)
option(ENABLE_JEMALLOC "Enables jemalloc allocator" TRUE)
option(ENABLE_OFFLINE_TELEMETRY "Enables FEX offline telemetry" TRUE)
option(ENABLE_COMPILE_TIME_TRACE "Enables time trace compile option" FALSE)
option(ENABLE_LIBCXX "Enables LLVM libc++" FALSE)
option(ENABLE_INTERPRETER "Enables FEX's Interpreter" FALSE)
option(ENABLE_CCACHE "Enables ccache for compile caching" TRUE)
set (X86_C_COMPILER "x86_64-linux-gnu-gcc" CACHE STRING "c compiler for compiling x86 guest libs")
set (X86_CXX_COMPILER "x86_64-linux-gnu-g++" CACHE STRING "c++ compiler for compiling x86 guest libs")
set (DATA_DIRECTORY "${CMAKE_INSTALL_PREFIX}/share/fex-emu" CACHE PATH "global data directory")
# These options are meant for package management
set (TUNE_CPU "native" CACHE STRING "Override the CPU the build is tuned for")
set (TUNE_ARCH "generic" CACHE STRING "Override the Arch the build is tuned for")
set (OVERRIDE_VERSION "detect" CACHE STRING "Override the FEX version in the format of <MMYY>{.<REV>}")
string(TOUPPER "${CMAKE_BUILD_TYPE}" CMAKE_BUILD_TYPE)
if (CMAKE_BUILD_TYPE MATCHES "DEBUG")
set(ENABLE_ASSERTIONS TRUE)
@@ -33,6 +41,11 @@ if (ENABLE_ASSERTIONS)
add_definitions(-DASSERTIONS_ENABLED=1)
endif()
if (ENABLE_INTERPRETER)
message(STATUS "Interpreter enabled")
add_definitions(-DINTERPRETER_ENABLED=1)
endif()
set(CMAKE_CXX_STANDARD 20)
set(CMAKE_EXPORT_COMPILE_COMMANDS ON)
set(CMAKE_RUNTIME_OUTPUT_DIRECTORY ${CMAKE_BINARY_DIR}/Bin)
@@ -67,10 +80,12 @@ if (CMAKE_SYSTEM_PROCESSOR MATCHES "aarch64")
add_definitions(-D_M_ARM_64=1)
endif()
find_program(CCACHE_PROGRAM ccache)
if(CCACHE_PROGRAM)
message(STATUS "CCache enabled")
set_property(GLOBAL PROPERTY RULE_LAUNCH_COMPILE "${CCACHE_PROGRAM}")
if (ENABLE_CCACHE)
find_program(CCACHE_PROGRAM ccache)
if(CCACHE_PROGRAM)
message(STATUS "CCache enabled")
set_property(GLOBAL PROPERTY RULE_LAUNCH_COMPILE "${CCACHE_PROGRAM}")
endif()
endif()
if (ENABLE_XRAY)
@@ -89,11 +104,76 @@ if (ENABLE_LLD)
link_libraries(${LD_OVERRIDE})
endif()
if (ENABLE_LIBCXX)
message(WARNING "This is an unsupported configuration and should only be used for testing")
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -std=c++11 -stdlib=libc++")
set(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} -stdlib=libc++ -lc++abi")
endif()
if (NOT ENABLE_OFFLINE_TELEMETRY)
# Disable FEX offline telemetry entirely if asked
add_definitions(-DFEX_DISABLE_TELEMETRY=1)
endif()
# Check if the build target page size is 4096
include(CheckCSourceRuns)
check_c_source_runs(
"#include <unistd.h>
int main(int argc, char* argv[])
{
return getpagesize() == 4096 ? 0 : 1;
}"
PAGEFILE_RESULT
)
if (NOT ${PAGEFILE_RESULT})
message(FATAL_ERROR "Host PAGE_SIZE is not 4096. Can't build on this target")
endif()
include(CheckCXXSourceCompiles)
check_cxx_source_compiles(
"#include <sys/user.h>
int main() {
return PAGE_SIZE;
}
"
HAS_PAGESIZE)
check_cxx_source_compiles(
"#include <sys/user.h>
int main() {
return PAGE_SHIFT;
}
"
HAS_PAGESHIFT)
check_cxx_source_compiles(
"#include <sys/user.h>
int main() {
return PAGE_MASK;
}
"
HAS_PAGEMASK)
if (NOT HAS_PAGESIZE)
add_definitions(-DPAGE_SIZE=4096)
endif()
if (NOT HAS_PAGESHIFT)
add_definitions(-DPAGE_SHIFT=12)
endif()
if (NOT HAS_PAGEMASK)
add_definitions("-DPAGE_MASK=(~(PAGE_SIZE-1))")
endif()
if(DEFINED ENV{TERMUX_VERSION})
add_definitions(-DTERMUX_BUILD=1)
set(TERMUX_BUILD 1)
# Termux doesn't support Jemalloc due to bad interactions between emutls, jemalloc, and scudo
set(ENABLE_JEMALLOC FALSE)
endif()
if (ENABLE_STATIC_PIE)
if (_M_ARM_64 AND ENABLE_LLD)
message (FATAL_ERROR "Static linking does not currently work with AArch64+LLD. Use GNU ld for now.")
@@ -247,6 +327,8 @@ set (CMAKE_LINKER_FLAGS_RELWITHDEBINFO "${CMAKE_LINKER_FLAGS_RELWITHDEBINFO} -fn
set (CMAKE_CXX_FLAGS_RELEASE "${CMAKE_CXX_FLAGS_RELEASE} -fomit-frame-pointer")
set (CMAKE_LINKER_FLAGS_RELEASE "${CMAKE_LINKER_FLAGS_RELEASE} -fomit-frame-pointer")
include_directories(External/robin-map/include/)
add_subdirectory(External/vixl/)
include_directories(External/vixl/src/)
@@ -257,17 +339,23 @@ endif()
find_package(PkgConfig REQUIRED)
find_package(Python 3.0 REQUIRED COMPONENTS Interpreter)
pkg_check_modules(XXHASH libxxhash>=0.8.0 QUIET)
if (NOT XXHASH_FOUND)
message(STATUS "xxHash not found. Using Externals")
add_subdirectory(External/xxhash/)
include_directories(External/xxhash/)
endif()
add_subdirectory(External/xxhash/)
include_directories(External/xxhash/)
add_definitions(-Wno-trigraphs)
add_definitions(-DGLOBAL_DATA_DIRECTORY="${DATA_DIRECTORY}/")
if (BUILD_TESTS)
option(CATCH_BUILD_STATIC_LIBRARY "" ON)
set(CATCH_BUILD_STATIC_LIBRARY ON)
add_subdirectory(External/Catch2/)
# Pull in catch_discover_tests definition
list(APPEND CMAKE_MODULE_PATH "${CMAKE_CURRENT_SOURCE_DIR}/External/Catch2/contrib/")
include(Catch)
endif()
add_subdirectory(External/cpp-optparse/)
include_directories(External/cpp-optparse/)
@@ -314,33 +402,51 @@ if(ENABLE_WERROR OR ENABLE_STRICT_WERROR)
endif()
endif()
if(_M_ARM_64)
if (CMAKE_CXX_COMPILER_VERSION VERSION_GREATER_EQUAL 999999.0)
# Clang 12.0 fixed the -mcpu=native bug with mixed big.little implementers
# Clang can not currently check for native Apple M1 type in hypervisor. Currently disabled
check_cxx_compiler_flag("-mcpu=native" COMPILER_SUPPORTS_CPU_TYPE)
if(COMPILER_SUPPORTS_CPU_TYPE)
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -mcpu=native")
if (NOT TUNE_ARCH STREQUAL "generic")
check_cxx_compiler_flag("-march=${TUNE_ARCH}" COMPILER_SUPPORTS_ARCH_TYPE)
if(COMPILER_SUPPORTS_ARCH_TYPE)
add_compile_options("-march=${TUNE_ARCH}")
else()
message(FATAL_ERROR "Trying to compile arch type '${TUNE_ARCH}' but the compiler doesn't support this")
endif()
endif()
if (TUNE_CPU STREQUAL "native")
if(_M_ARM_64)
if (CMAKE_CXX_COMPILER_VERSION VERSION_GREATER_EQUAL 999999.0)
# Clang 12.0 fixed the -mcpu=native bug with mixed big.little implementers
# Clang can not currently check for native Apple M1 type in hypervisor. Currently disabled
check_cxx_compiler_flag("-mcpu=native" COMPILER_SUPPORTS_CPU_TYPE)
if(COMPILER_SUPPORTS_CPU_TYPE)
add_compile_options("-mcpu=native")
endif()
else()
# Due to an oversight in llvm, it declares any reasonably new Kryo CPU to only be ARMv8.0
# Manually detect newer CPU revisions until clang and llvm fixes their bug
# This script will either provide a supported CPU or 'native'
# Additionally -march doesn't work under AArch64+Clang, so you have to use -mcpu or -mtune
execute_process(COMMAND python3 "${PROJECT_SOURCE_DIR}/Scripts/aarch64_fit_native.py" "/proc/cpuinfo" "${CMAKE_CXX_COMPILER_VERSION}"
OUTPUT_VARIABLE AARCH64_CPU)
string(STRIP ${AARCH64_CPU} AARCH64_CPU)
check_cxx_compiler_flag("-mcpu=${AARCH64_CPU}" COMPILER_SUPPORTS_CPU_TYPE)
if(COMPILER_SUPPORTS_CPU_TYPE)
add_compile_options("-mcpu=${AARCH64_CPU}")
endif()
endif()
else()
# Due to an oversight in llvm, it declares any reasonably new Kryo CPU to only be ARMv8.0
# Manually detect newer CPU revisions until clang and llvm fixes their bug
# This script will either provide a supported CPU or 'native'
# Additionally -march doesn't work under AArch64+Clang, so you have to use -mcpu or -mtune
execute_process(COMMAND python3 "${PROJECT_SOURCE_DIR}/Scripts/aarch64_fit_native.py" "/proc/cpuinfo" "${CMAKE_CXX_COMPILER_VERSION}"
OUTPUT_VARIABLE AARCH64_CPU)
string(STRIP ${AARCH64_CPU} AARCH64_CPU)
check_cxx_compiler_flag("-mcpu=${AARCH64_CPU}" COMPILER_SUPPORTS_CPU_TYPE)
if(COMPILER_SUPPORTS_CPU_TYPE)
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -mcpu=${AARCH64_CPU}")
check_cxx_compiler_flag("-march=native" COMPILER_SUPPORTS_MARCH_NATIVE)
if(COMPILER_SUPPORTS_MARCH_NATIVE)
add_compile_options("-march=native")
endif()
endif()
else()
check_cxx_compiler_flag("-march=native" COMPILER_SUPPORTS_MARCH_NATIVE)
if(COMPILER_SUPPORTS_MARCH_NATIVE)
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -march=native")
check_cxx_compiler_flag("-mcpu=${TUNE_CPU}" COMPILER_SUPPORTS_CPU_TYPE)
if(COMPILER_SUPPORTS_CPU_TYPE)
add_compile_options("-mcpu=${TUNE_CPU}")
else()
message(FATAL_ERROR "Trying to compile cpu type '${TUNE_CPU}' but the compiler doesn't support this")
endif()
endif()
@@ -414,6 +520,10 @@ if (BUILD_TESTS)
enable_testing()
message(STATUS "Unit tests are enabled")
endif()
add_subdirectory(FEXHeaderUtils/)
include_directories(FEXHeaderUtils/)
add_subdirectory(External/FEXCore)
# Binfmt_misc files must be installed prior to Source/ installs
@@ -432,29 +542,16 @@ if (BUILD_TESTS)
endif()
if (BUILD_THUNKS)
add_subdirectory(ThunkLibs/Generator)
# Thunk targets for both host libraries and IDE integration
add_subdirectory(ThunkLibs/HostLibs)
# Thunk targets for IDE integration of guest code, only
add_subdirectory(ThunkLibs/GuestLibs)
# Thunk targets for guest libraries
include(ExternalProject)
ExternalProject_Add(host-libs
PREFIX host-libs
SOURCE_DIR "${CMAKE_CURRENT_SOURCE_DIR}/ThunkLibs/HostLibs"
BINARY_DIR "Host"
CMAKE_ARGS
"-DCMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE}"
"-DCMAKE_INSTALL_PREFIX=${CMAKE_INSTALL_PREFIX}"
"-DVULKAN_XML=${CMAKE_SOURCE_DIR}/External/Vulkan-Docs/xml/vk.xml"
INSTALL_COMMAND ""
BUILD_ALWAYS ON
)
install(
CODE "MESSAGE(\"-- Installing: host-libs\")"
CODE "
EXECUTE_PROCESS(COMMAND ${CMAKE_COMMAND} --build . --target ThunkHostsInstall
WORKING_DIRECTORY ${CMAKE_BINARY_DIR}/Host
)"
DEPENDS host-libs
)
ExternalProject_Add(guest-libs
PREFIX guest-libs
SOURCE_DIR "${CMAKE_CURRENT_SOURCE_DIR}/ThunkLibs/GuestLibs"
@@ -465,15 +562,16 @@ if (BUILD_THUNKS)
"-DX86_CXX_COMPILER:STRING=${X86_CXX_COMPILER}"
"-DCMAKE_INSTALL_PREFIX=${CMAKE_INSTALL_PREFIX}"
"-DSTRUCT_VERIFIER=${CMAKE_SOURCE_DIR}/Scripts/StructPackVerifier.py"
"-DVULKAN_XML=${CMAKE_SOURCE_DIR}/External/Vulkan-Docs/xml/vk.xml"
"-DGENERATOR_EXE=$<TARGET_FILE:thunkgen>"
INSTALL_COMMAND ""
BUILD_ALWAYS ON
DEPENDS thunkgen
)
install(
CODE "MESSAGE(\"-- Installing: guest-libs\")"
CODE "
EXECUTE_PROCESS(COMMAND ${CMAKE_COMMAND} --build . --target ThunkGuestsInstall
EXECUTE_PROCESS(COMMAND ${CMAKE_COMMAND} --build . --target install
WORKING_DIRECTORY ${CMAKE_BINARY_DIR}/Guest
)"
DEPENDS guest-libs
@@ -484,43 +582,48 @@ set(FEX_VERSION_MAJOR "0")
set(FEX_VERSION_MINOR "0")
set(FEX_VERSION_PATCH "0")
find_package(Git)
if (GIT_FOUND)
execute_process(
COMMAND ${GIT_EXECUTABLE} describe --abbrev=0
WORKING_DIRECTORY "${CMAKE_SOURCE_DIR}"
OUTPUT_VARIABLE GIT_DESCRIBE_STRING
RESULT_VARIABLE GIT_ERROR
ERROR_QUIET
OUTPUT_STRIP_TRAILING_WHITESPACE
)
if (OVERRIDE_VERSION STREQUAL "detect")
find_package(Git)
if (GIT_FOUND)
execute_process(
COMMAND ${GIT_EXECUTABLE} describe --abbrev=0
WORKING_DIRECTORY "${CMAKE_SOURCE_DIR}"
OUTPUT_VARIABLE GIT_DESCRIBE_STRING
RESULT_VARIABLE GIT_ERROR
ERROR_QUIET
OUTPUT_STRIP_TRAILING_WHITESPACE
)
if (NOT ${GIT_ERROR} EQUAL 0)
# Likely built in a way that doesn't have tags
# Setup a version tag that is unknown
set(GIT_DESCRIBE_STRING "FEX-0000")
if (NOT ${GIT_ERROR} EQUAL 0)
# Likely built in a way that doesn't have tags
# Setup a version tag that is unknown
set(GIT_DESCRIBE_STRING "FEX-0000")
endif()
endif()
else()
set(GIT_DESCRIBE_STRING "FEX-${OVERRIDE_VERSION}")
endif()
# Change something like `FEX-2106.1-76-<hash>` in to a list
string(REPLACE "-" ";" DESCRIBE_LIST ${GIT_DESCRIBE_STRING})
# Parse the version here
# Change something like `FEX-2106.1-76-<hash>` in to a list
string(REPLACE "-" ";" DESCRIBE_LIST ${GIT_DESCRIBE_STRING})
# Extract the `2106.1` element
list(GET DESCRIBE_LIST 1 DESCRIBE_LIST)
# Extract the `2106.1` element
list(GET DESCRIBE_LIST 1 DESCRIBE_LIST)
# Change `2106.1` in to a list
string(REPLACE "." ";" DESCRIBE_LIST ${DESCRIBE_LIST})
# Change `2106.1` in to a list
string(REPLACE "." ";" DESCRIBE_LIST ${DESCRIBE_LIST})
# Calculate list size
list(LENGTH DESCRIBE_LIST LIST_SIZE)
# Calculate list size
list(LENGTH DESCRIBE_LIST LIST_SIZE)
# Pull out the major version
list(GET DESCRIBE_LIST 0 FEX_VERSION_MAJOR)
# Pull out the major version
list(GET DESCRIBE_LIST 0 FEX_VERSION_MAJOR)
# Minor version only exists if there is a .1 at the end
# eg: 2106 versus 2106.1
if (LIST_SIZE GREATER 1)
list(GET DESCRIBE_LIST 1 FEX_VERSION_MINOR)
endif()
# Minor version only exists if there is a .1 at the end
# eg: 2106 versus 2106.1
if (LIST_SIZE GREATER 1)
list(GET DESCRIBE_LIST 1 FEX_VERSION_MINOR)
endif()
# Package creation
Vendored Submodule
+1
Submodule External/Catch2 added at c4e3767e26.
+23 -18
View File
@@ -37,27 +37,32 @@ endif()
set(CMAKE_CXX_STANDARD 20)
set(CMAKE_EXPORT_COMPILE_COMMANDS ON)
# Find our git hash
find_package(Git)
set(GIT_SHORT_HASH "Unknown")
set(GIT_DESCRIBE_STRING "FEX-Unknown")
if (GIT_FOUND)
execute_process(
COMMAND ${GIT_EXECUTABLE} rev-parse --short HEAD
WORKING_DIRECTORY "${CMAKE_SOURCE_DIR}"
OUTPUT_VARIABLE GIT_SHORT_HASH
ERROR_QUIET
OUTPUT_STRIP_TRAILING_WHITESPACE
)
execute_process(
COMMAND ${GIT_EXECUTABLE} describe
WORKING_DIRECTORY "${CMAKE_SOURCE_DIR}"
OUTPUT_VARIABLE GIT_DESCRIBE_STRING
ERROR_QUIET
OUTPUT_STRIP_TRAILING_WHITESPACE
)
if (OVERRIDE_VERSION STREQUAL "detect")
# Find our git hash
find_package(Git)
if (GIT_FOUND)
execute_process(
COMMAND ${GIT_EXECUTABLE} rev-parse --short HEAD
WORKING_DIRECTORY "${CMAKE_SOURCE_DIR}"
OUTPUT_VARIABLE GIT_SHORT_HASH
ERROR_QUIET
OUTPUT_STRIP_TRAILING_WHITESPACE
)
execute_process(
COMMAND ${GIT_EXECUTABLE} describe
WORKING_DIRECTORY "${CMAKE_SOURCE_DIR}"
OUTPUT_VARIABLE GIT_DESCRIBE_STRING
ERROR_QUIET
OUTPUT_STRIP_TRAILING_WHITESPACE
)
endif()
else()
set(GIT_SHORT_HASH "${OVERRIDE_VERSION}")
set(GIT_DESCRIBE_STRING "FEX-${OVERRIDE_VERSION}")
endif()
configure_file(
+6 -43
View File
@@ -7,17 +7,12 @@ OpClasses = collections.OrderedDict()
def get_ir_classes(ops, defines):
global OpClasses
for op_key, op_vals in ops.items():
if not ("Last" in op_vals):
OpClass = "#Unknown"
for op_class, opslist in ops.items():
if not (op_class in OpClasses):
OpClasses[op_class] = []
if ("OpClass" in op_vals):
OpClass = op_vals["OpClass"]
if not (OpClass in OpClasses):
OpClasses[OpClass] = []
OpClasses[OpClass].append([op_key, op_vals])
for op, op_val in opslist.items():
OpClasses[op_class].append([op, op_val])
# Sort the dictionary after we are done parsing it
OpClasses = collections.OrderedDict(sorted(OpClasses.items()))
@@ -38,41 +33,9 @@ def print_ir_ops():
op_key = op[0]
op_vals = op[1]
output_file.write("## %s\n" % (op_key))
HasDest = ("HasDest" in op_vals and op_vals["HasDest"] == True)
HasSSAArgs = ("SSAArgs" in op_vals and len(op_vals["SSAArgs"]) > 0)
HasSSAArgNames = "SSANames" in op_vals
HasArgs = "Args" in op_vals
SSAArgsCount = 0
ArgCount = 0
if (HasSSAArgs):
SSAArgsCount = int(op_vals["SSAArgs"])
if (HasArgs):
ArgCount = len(op_vals["Args"])
TotalArgsCount = SSAArgsCount + (ArgCount / 2)
output_file.write(">")
if (HasDest):
output_file.write("%dest = ")
output_file.write("%s " % op_key)
ArgComma = (", ", "")
if (HasSSAArgs):
for i in range(0, SSAArgsCount):
FinalArg = (i + 1) == TotalArgsCount
if (HasSSAArgNames):
output_file.write("%%%s%s" % (op_vals["SSANames"][i], ArgComma[FinalArg]))
else:
output_file.write("%%ssa%d%s" % (i, ArgComma[FinalArg]))
if (HasArgs):
Args = op_vals["Args"]
for i in range(0, ArgCount, 2):
FinalArg = ((i / 2) + SSAArgsCount + 1) == TotalArgsCount
data_type = Args[i]
data_name = Args[i + 1]
output_file.write("\<%s %s\>%s" % (data_type, data_name, ArgComma[FinalArg]))
output_file.write(op_key)
output_file.write("\n\n")
Vendored Regular → Executable
+469 -382
View File
File diff suppressed because it is too large. Load diff
+53 -41
View File
@@ -1,7 +1,14 @@
set (MAN_DIR ${CMAKE_INSTALL_PREFIX}/share/man CACHE PATH "MAN_DIR")
set (SRCS
set (FEXCORE_BASE_SRCS
Common/Paths.cpp
Interface/Config/Config.cpp
Utils/FileLoading.cpp
Utils/ForcedAssert.cpp
Utils/LogManager.cpp
)
set (SRCS
Common/JitSymbols.cpp
Common/SoftFloat-3e/extF80_add.c
Common/SoftFloat-3e/extF80_div.c
@@ -70,7 +77,6 @@ set (SRCS
Common/SoftFloat-3e/f32_to_extF80.c
Common/SoftFloat-3e/s_normSubnormalF32Sig.c
Common/SoftFloat-3e/s_f32UIToCommonNaN.c
Interface/Config/Config.cpp
Interface/Context/Context.cpp
Interface/Core/LookupCache.cpp
Interface/Core/BlockSamplingData.cpp
@@ -94,19 +100,7 @@ set (SRCS
Interface/Core/Dispatcher/Dispatcher.cpp
Interface/Core/Dispatcher/X86Dispatcher.cpp
Interface/Core/Dispatcher/Arm64Dispatcher.cpp
Interface/Core/Interpreter/InterpreterCore.cpp
Interface/Core/Interpreter/InterpreterOps.cpp
Interface/Core/Interpreter/ALUOps.cpp
Interface/Core/Interpreter/AtomicOps.cpp
Interface/Core/Interpreter/BranchOps.cpp
Interface/Core/Interpreter/ConversionOps.cpp
Interface/Core/Interpreter/EncryptionOps.cpp
Interface/Core/Interpreter/F80Ops.cpp
Interface/Core/Interpreter/FlagOps.cpp
Interface/Core/Interpreter/MemoryOps.cpp
Interface/Core/Interpreter/MiscOps.cpp
Interface/Core/Interpreter/MoveOps.cpp
Interface/Core/Interpreter/VectorOps.cpp
Interface/Core/Interpreter/InterpreterFallbacks.cpp
Interface/Core/X86Tables/BaseTables.cpp
Interface/Core/X86Tables/DDDTables.cpp
Interface/Core/X86Tables/EVEXTables.cpp
@@ -120,6 +114,7 @@ set (SRCS
Interface/Core/X86Tables/X87Tables.cpp
Interface/Core/X86Tables/XOPTables.cpp
Interface/HLE/Thunks/Thunks.cpp
Interface/IR/AOTIR.cpp
Interface/IR/IRDumper.cpp
Interface/IR/IRParser.cpp
Interface/IR/IREmitter.cpp
@@ -140,14 +135,28 @@ set (SRCS
Interface/IR/Passes/SyscallOptimization.cpp
Utils/Allocator.cpp
Utils/Allocator/64BitAllocator.cpp
Utils/FileLoading.cpp
Utils/ForcedAssert.cpp
Utils/LogManager.cpp
Utils/NetStream.cpp
Utils/Telemetry.cpp
Utils/Threads.cpp
)
if (ENABLE_INTERPRETER)
list(APPEND SRCS
Interface/Core/Interpreter/InterpreterCore.cpp
Interface/Core/Interpreter/InterpreterOps.cpp
Interface/Core/Interpreter/ALUOps.cpp
Interface/Core/Interpreter/AtomicOps.cpp
Interface/Core/Interpreter/BranchOps.cpp
Interface/Core/Interpreter/ConversionOps.cpp
Interface/Core/Interpreter/EncryptionOps.cpp
Interface/Core/Interpreter/F80Ops.cpp
Interface/Core/Interpreter/FlagOps.cpp
Interface/Core/Interpreter/MemoryOps.cpp
Interface/Core/Interpreter/MiscOps.cpp
Interface/Core/Interpreter/MoveOps.cpp
Interface/Core/Interpreter/VectorOps.cpp)
endif()
if(_M_ARM_64)
list(APPEND SRCS
Interface/Core/ArchHelpers/Arm64.cpp)
@@ -195,7 +204,7 @@ if (ENABLE_JIT_ARM64)
Interface/Core/JIT/Arm64/VectorOps.cpp)
endif()
set (LIBS vixl dl fmt::fmt xxhash tiny-json)
set (LIBS vixl dl xxhash tiny-json)
if (ENABLE_JEMALLOC)
list (APPEND LIBS FEX_jemalloc)
endif()
@@ -296,17 +305,10 @@ install(FILES ${OUTPUT_MAN_NAME_COMPRESS} DESTINATION ${MAN_DIR}/man1)
check_cxx_compiler_flag(-fdiagnostics-color=always GCC_COLOR)
check_cxx_compiler_flag(-fcolor-diagnostics CLANG_COLOR)
function(AddObject Name Type)
add_library(${Name} ${Type} ${SRCS})
add_dependencies(${Name} IR_INC)
add_dependencies(${Name} CONFIG_INC)
target_link_libraries(${Name} ${LIBS})
set_target_properties(${Name} PROPERTIES OUTPUT_NAME FEXCore)
function(AddDefaultOptionsToTarget Name)
set_target_properties(${Name} PROPERTIES C_VISIBILITY_PRESET hidden)
set_target_properties(${Name} PROPERTIES CXX_VISIBILITY_PRESET hidden)
set_target_properties(${Name} PROPERTIES VISIBILITY_INLINES_HIDDEN TRUE)
target_include_directories(${Name} PUBLIC "${CMAKE_CURRENT_BINARY_DIR}")
target_include_directories(${Name} PRIVATE IncludePrivate/)
@@ -316,6 +318,7 @@ function(AddObject Name Type)
target_include_directories(${Name} PUBLIC "${CMAKE_BINARY_DIR}/include/")
target_compile_definitions(${Name} PRIVATE ${DEFINES})
add_dependencies(${Name} CONFIG_INC)
target_compile_options(${Name}
PRIVATE
@@ -338,20 +341,6 @@ function(AddObject Name Type)
PRIVATE
"-fcolor-diagnostics")
endif()
endfunction()
function(AddLibrary Name Type)
add_library(${Name} ${Type} $<TARGET_OBJECTS:${PROJECT_NAME}_object>)
target_link_libraries(${Name} ${LIBS})
set_target_properties(${Name} PROPERTIES OUTPUT_NAME FEXCore)
set_target_properties(${Name} PROPERTIES C_VISIBILITY_PRESET hidden)
set_target_properties(${Name} PROPERTIES CXX_VISIBILITY_PRESET hidden)
set_target_properties(${Name} PROPERTIES VISIBILITY_INLINES_HIDDEN TRUE)
set_target_properties(${Name} PROPERTIES VERSION ${FEXCore_VERSION} SOVERSION ${FEXCore_VERSION})
target_include_directories(${Name} PUBLIC "${CMAKE_CURRENT_BINARY_DIR}")
target_include_directories(${Name} PUBLIC "${PROJECT_SOURCE_DIR}/include/")
target_include_directories(${Name} PUBLIC "${CMAKE_BINARY_DIR}/include/")
if (CMAKE_BUILD_TYPE MATCHES "RELEASE")
target_link_options(${Name}
@@ -363,6 +352,29 @@ function(AddLibrary Name Type)
endif()
endfunction()
# Build FEXCore_Config static library
add_library(FEXCore_Base STATIC ${FEXCORE_BASE_SRCS})
target_link_libraries(FEXCore_Base fmt::fmt tiny-json)
AddDefaultOptionsToTarget(FEXCore_Base)
function(AddObject Name Type)
add_library(${Name} ${Type} ${SRCS})
add_dependencies(${Name} IR_INC)
target_link_libraries(${Name} FEXCore_Base ${LIBS})
AddDefaultOptionsToTarget(${Name})
set_target_properties(${Name} PROPERTIES OUTPUT_NAME FEXCore)
endfunction()
function(AddLibrary Name Type)
add_library(${Name} ${Type} $<TARGET_OBJECTS:${PROJECT_NAME}_object>)
target_link_libraries(${Name} FEXCore_Base ${LIBS})
set_target_properties(${Name} PROPERTIES OUTPUT_NAME FEXCore)
AddDefaultOptionsToTarget(${Name})
endfunction()
AddObject(${PROJECT_NAME}_object OBJECT)
AddLibrary(${PROJECT_NAME} STATIC)
AddLibrary(${PROJECT_NAME}_shared SHARED)
+4 -4
View File
@@ -23,7 +23,7 @@ namespace FEXCore {
// Linux perf format is very straightforward
// `<HostPtr> <Size> <Name>\n`
fmt::print(fp.get(), "{:x} {:x} JIT_0x{:x}_{:x}\n", HostAddr, CodeSize, GuestAddr, HostAddr);
fmt::print(fp.get(), "{} {:x} JIT_0x{:x}_{}\n", HostAddr, CodeSize, GuestAddr, HostAddr);
}
void JITSymbols::Register(const void *HostAddr, uint32_t CodeSize, std::string_view Name) {
@@ -31,7 +31,7 @@ namespace FEXCore {
// Linux perf format is very straightforward
// `<HostPtr> <Size> <Name>\n`
fmt::print(fp.get(), "{:x} {:x} {}_{:x}\n", HostAddr, CodeSize, Name, HostAddr);
fmt::print(fp.get(), "{} {:x} {}_{}\n", HostAddr, CodeSize, Name, HostAddr);
}
void JITSymbols::RegisterNamedRegion(const void *HostAddr, uint32_t CodeSize, std::string_view Name) {
@@ -39,7 +39,7 @@ namespace FEXCore {
// Linux perf format is very straightforward
// `<HostPtr> <Size> <Name>\n`
fmt::print(fp.get(), "{:x} {:x} {}\n", HostAddr, CodeSize, Name);
fmt::print(fp.get(), "{} {:x} {}\n", HostAddr, CodeSize, Name);
}
void JITSymbols::RegisterJITSpace(const void *HostAddr, uint32_t CodeSize) {
@@ -47,7 +47,7 @@ namespace FEXCore {
// Linux perf format is very straightforward
// `<HostPtr> <Size> <Name>\n`
fmt::print(fp.get(), "{:x} {:x} FEXJIT\n", HostAddr, CodeSize);
fmt::print(fp.get(), "{} {:x} FEXJIT\n", HostAddr, CodeSize);
}
} // namespace FEXCore
+264 -8
View File
@@ -57,35 +57,131 @@ struct X80SoftFloat {
// Ops
static X80SoftFloat FADD(X80SoftFloat const &lhs, X80SoftFloat const &rhs) {
#ifdef DEBUG_X86_FLOAT
BIGFLOAT Result;
asm (R"(
fninit;
fldt %[rhs]; # st1
fldt %[lhs]; # st0
faddp;
fstpt %[result];
)"
: [result] "=m" (Result)
: [lhs] "m" (lhs)
, [rhs] "m" (rhs)
: "st", "st(1)");
return Result;
#else
return extF80_add(lhs, rhs);
#endif
}
static X80SoftFloat FSUB(X80SoftFloat const &lhs, X80SoftFloat const &rhs) {
#ifdef DEBUG_X86_FLOAT
BIGFLOAT Result;
asm (R"(
fninit;
fldt %[rhs]; # st1
fldt %[lhs]; # st0
fsubp;
fstpt %[result];
)"
: [result] "=m" (Result)
: [lhs] "m" (lhs)
, [rhs] "m" (rhs)
: "st", "st(1)");
return Result;
#else
return extF80_sub(lhs, rhs);
#endif
}
static X80SoftFloat FMUL(X80SoftFloat const &lhs, X80SoftFloat const &rhs) {
#ifdef DEBUG_X86_FLOAT
BIGFLOAT Result;
asm (R"(
fninit;
fldt %[rhs]; # st1
fldt %[lhs]; # st0
fmulp;
fstpt %[result];
)"
: [result] "=m" (Result)
: [lhs] "m" (lhs)
, [rhs] "m" (rhs)
: "st", "st(1)");
return Result;
#else
return extF80_mul(lhs, rhs);
#endif
}
static X80SoftFloat FDIV(X80SoftFloat const &lhs, X80SoftFloat const &rhs) {
#ifdef DEBUG_X86_FLOAT
BIGFLOAT Result;
asm (R"(
fninit;
fldt %[rhs]; # st1
fldt %[lhs]; # st0
fdivp;
fstpt %[result];
)"
: [result] "=m" (Result)
: [lhs] "m" (lhs)
, [rhs] "m" (rhs)
: "st", "st(1)");
return Result;
#else
return extF80_div(lhs, rhs);
#endif
}
static X80SoftFloat FREM(X80SoftFloat const &lhs, X80SoftFloat const &rhs) {
X80SoftFloat Rem = extF80_rem(lhs, rhs);
if (SignBit(Rem)) {
Rem = extF80_add(Rem, rhs);
}
else {
Rem.Sign = SignBit(lhs);
}
#if defined(DEBUG_X86_FLOAT)
BIGFLOAT Result;
asm (R"(
fninit;
fldt %[rhs]; # st1
fldt %[lhs]; # st0
fprem;
fstpt %[result];
ffreep %%st(0);
)"
: [result] "=m" (Result)
: [lhs] "m" (lhs)
, [rhs] "m" (rhs)
: "st", "st(1)");
return Rem;
return Result;
#else
return extF80_rem(lhs, rhs);
#endif
}
static X80SoftFloat FREM1(X80SoftFloat const &lhs, X80SoftFloat const &rhs) {
#if defined(DEBUG_X86_FLOAT)
BIGFLOAT Result;
asm (R"(
fninit;
fldt %[rhs]; # st1
fldt %[lhs]; # st0
fprem1;
fstpt %[result];
ffreep %%st(0);
)"
: [result] "=m" (Result)
: [lhs] "m" (lhs)
, [rhs] "m" (rhs)
: "st", "st(1)");
return Result;
#else
return extF80_rem(lhs, rhs);
#endif
}
static X80SoftFloat FRNDINT(X80SoftFloat const &lhs) {
@@ -93,15 +189,47 @@ struct X80SoftFloat {
}
static X80SoftFloat FXTRACT_SIG(X80SoftFloat const &lhs) {
#if defined(DEBUG_X86_FLOAT)
BIGFLOAT Result;
asm (R"(
fninit;
fldt %[lhs]; # st0
fxtract;
fstpt %[result];
ffreep %%st(0);
)"
: [result] "=m" (Result)
: [lhs] "m" (lhs)
: "st", "st(1)");
return Result;
#else
X80SoftFloat Tmp = lhs;
Tmp.Exponent = 0x3FFF;
Tmp.Sign = lhs.Sign;
return Tmp;
#endif
}
static X80SoftFloat FXTRACT_EXP(X80SoftFloat const &lhs) {
#if defined(DEBUG_X86_FLOAT)
BIGFLOAT Result;
asm (R"(
fninit;
fldt %[lhs]; # st0
fxtract;
ffreep %%st(0);
fstpt %[result];
)"
: [result] "=m" (Result)
: [lhs] "m" (lhs)
: "st", "st(1)");
return Result;
#else
int32_t TrueExp = lhs.Exponent - ExponentBias;
return i32_to_extF80(TrueExp);
#endif
}
static void FCMP(X80SoftFloat const &lhs, X80SoftFloat const &rhs, bool *eq, bool *lt, bool *nan) {
@@ -112,61 +240,189 @@ struct X80SoftFloat {
static X80SoftFloat FSCALE(X80SoftFloat const &lhs, X80SoftFloat const &rhs) {
WARN_ONCE_FMT("x87: Application used FSCALE which may have accuracy problems");
#ifdef DEBUG_X86_FLOAT
BIGFLOAT Result;
asm (R"(
fninit;
fldt %[rhs]; # st1
fldt %[lhs]; # st0
fscale; # st0 = st0 * 2^(rdint(st1))
fstpt %[result];
ffreep %%st(0);
)"
: [result] "=m" (Result)
: [lhs] "m" (lhs)
, [rhs] "m" (rhs)
: "st", "st(1)");
return Result;
#else
X80SoftFloat Int = FRNDINT(rhs);
BIGFLOAT Src2_d = Int;
Src2_d = exp2l(Src2_d);
X80SoftFloat Src2_X80 = Src2_d;
X80SoftFloat Result = extF80_mul(lhs, Src2_X80);
return Result;
#endif
}
static X80SoftFloat F2XM1(X80SoftFloat const &lhs) {
WARN_ONCE_FMT("x87: Application used F2XM1 which may have accuracy problems");
#ifdef DEBUG_X86_FLOAT
BIGFLOAT Result;
asm (R"(
fninit;
fldt %[lhs]; # st0
f2xm1; # st0 = 2^st(0) - 1
fstpt %[result];
)"
: [result] "=m" (Result)
: [lhs] "m" (lhs)
: "st");
return Result;
#else
BIGFLOAT Src1_d = lhs;
BIGFLOAT Result = exp2l(Src1_d);
Result -= 1.0;
return Result;
#endif
}
static X80SoftFloat FYL2X(X80SoftFloat const &lhs, X80SoftFloat const &rhs) {
WARN_ONCE_FMT("x87: Application used FYL2X which may have accuracy problems");
#ifdef DEBUG_X86_FLOAT
BIGFLOAT Result;
asm (R"(
fninit;
fldt %[rhs]; # st(1)
fldt %[lhs]; # st(0)
fyl2x; # st(1) * log2l(st(0))
fstpt %[result];
)"
: [result] "=m" (Result)
: [lhs] "m" (lhs)
, [rhs] "m" (rhs)
: "st", "st(1)");
return Result;
#else
BIGFLOAT Src1_d = lhs;
BIGFLOAT Src2_d = rhs;
BIGFLOAT Tmp = Src2_d * log2l(Src1_d);
return Tmp;
#endif
}
static X80SoftFloat FATAN(X80SoftFloat const &lhs, X80SoftFloat const &rhs) {
WARN_ONCE_FMT("x87: Application used FATAN which may have accuracy problems");
#ifdef DEBUG_X86_FLOAT
BIGFLOAT Result;
asm (R"(
fninit;
fldt %[lhs];
fldt %[rhs];
fpatan;
fstpt %[result];
)"
: [result] "=m" (Result)
: [lhs] "m" (lhs)
, [rhs] "m" (rhs)
: "st", "st(1)");
return Result;
#else
BIGFLOAT Src1_d = lhs;
BIGFLOAT Src2_d = rhs;
BIGFLOAT Tmp = atan2l(Src1_d, Src2_d);
return Tmp;
#endif
}
static X80SoftFloat FTAN(X80SoftFloat const &lhs) {
WARN_ONCE_FMT("x87: Application used FTAN which may have accuracy problems");
#ifdef DEBUG_X86_FLOAT
BIGFLOAT Result;
asm (R"(
fninit;
fldt %[lhs]; # st0
fptan;
ffreep %%st(0);
fstpt %[result];
)"
: [result] "=m" (Result)
: [lhs] "m" (lhs)
: "st");
return Result;
#else
BIGFLOAT Src_d = lhs;
Src_d = tanl(Src_d);
return Src_d;
#endif
}
static X80SoftFloat FSIN(X80SoftFloat const &lhs) {
WARN_ONCE_FMT("x87: Application used FSIN which may have accuracy problems");
#ifdef DEBUG_X86_FLOAT
BIGFLOAT Result;
asm (R"(
fninit;
fldt %[lhs]; # st0
fsin;
fstpt %[result];
)"
: [result] "=m" (Result)
: [lhs] "m" (lhs)
: "st");
return Result;
#else
BIGFLOAT Src_d = lhs;
Src_d = sinl(Src_d);
return Src_d;
#endif
}
static X80SoftFloat FCOS(X80SoftFloat const &lhs) {
WARN_ONCE_FMT("x87: Application used FCOS which may have accuracy problems");
#ifdef DEBUG_X86_FLOAT
BIGFLOAT Result;
asm (R"(
fninit;
fldt %[lhs]; # st0
fcos;
fstpt %[result];
)"
: [result] "=m" (Result)
: [lhs] "m" (lhs)
: "st");
return Result;
#else
BIGFLOAT Src_d = lhs;
Src_d = cosl(Src_d);
return Src_d;
#endif
}
static X80SoftFloat FSQRT(X80SoftFloat const &lhs) {
#ifdef DEBUG_X86_FLOAT
BIGFLOAT Result;
asm (R"(
fninit;
fldt %[lhs]; # st0
fsqrt;
fstpt %[result];
)"
: [result] "=m" (Result)
: [lhs] "m" (lhs)
: "st");
return Result;
#else
return extF80_sqrt(lhs);
#endif
}
operator float() const {
+22 -1
View File
@@ -416,7 +416,9 @@ namespace JSON {
Meta->Load();
// Do configuration option fix ups after everything is reloaded
if (FEXCore::Config::Exists(FEXCore::Config::CONFIG_THREADS)) {
{
// Always fix up the number of threads and create the configuration
// Otherwise the application could receive zero as the number of threads
FEX_CONFIG_OPT(Cores, THREADS);
if (Cores == 0) {
// When the number of emulated CPU cores is zero then auto detect
@@ -424,6 +426,25 @@ namespace JSON {
}
}
if (FEXCore::Config::Exists(FEXCore::Config::CONFIG_CORE)) {
// Sanitize Core option
FEX_CONFIG_OPT(Core, CORE);
#if (_M_X86_64)
constexpr uint32_t MaxCoreNumber = 2;
#else
constexpr uint32_t MaxCoreNumber = 1;
#endif
#ifdef INTERPRETER_ENABLED
constexpr uint32_t MinCoreNumber = 0;
#else
constexpr uint32_t MinCoreNumber = 1;
#endif
if (Core > MaxCoreNumber || Core < MinCoreNumber) {
// Sanitize the core option by setting the core to the JIT if invalid
FEXCore::Config::EraseSet(FEXCore::Config::CONFIG_CORE, std::to_string(FEXCore::Config::CONFIG_IRJIT));
}
}
std::string ContainerPrefix { FindContainerPrefix() };
auto ExpandPathIfExists = [&ContainerPrefix](FEXCore::Config::ConfigOption Config, std::string PathName) {
auto NewPath = ExpandPath(ContainerPrefix, PathName);
+1 -1
View File
@@ -32,7 +32,7 @@
},
"Threads": {
"Type": "uint32",
"Default": "1",
"Default": "0",
"ShortArg": "T",
"Desc": [
"Number of physical hardware threads to tell the process we have.",
+20 -12
View File
@@ -51,7 +51,7 @@ namespace FEXCore::Context {
CTX->CustomExitHandler = std::move(handler);
}
ExitHandler GetExitHandler(FEXCore::Context::Context *CTX) {
ExitHandler GetExitHandler(const FEXCore::Context::Context *CTX) {
return CTX->CustomExitHandler;
}
@@ -71,23 +71,23 @@ namespace FEXCore::Context {
return CTX->RunUntilExit();
}
int GetProgramStatus(FEXCore::Context::Context *CTX) {
int GetProgramStatus(const FEXCore::Context::Context *CTX) {
return CTX->GetProgramStatus();
}
FEXCore::Context::ExitReason GetExitReason(FEXCore::Context::Context *CTX) {
FEXCore::Context::ExitReason GetExitReason(const FEXCore::Context::Context *CTX) {
return CTX->ParentThread->ExitReason;
}
bool IsDone(FEXCore::Context::Context *CTX) {
bool IsDone(const FEXCore::Context::Context *CTX) {
return CTX->IsPaused();
}
void GetCPUState(FEXCore::Context::Context *CTX, FEXCore::Core::CPUState *State) {
void GetCPUState(const FEXCore::Context::Context *CTX, FEXCore::Core::CPUState *State) {
memcpy(State, CTX->ParentThread->CurrentFrame, sizeof(FEXCore::Core::CPUState));
}
void SetCPUState(FEXCore::Context::Context *CTX, FEXCore::Core::CPUState *State) {
void SetCPUState(FEXCore::Context::Context *CTX, const FEXCore::Core::CPUState *State) {
memcpy(CTX->ParentThread->CurrentFrame, State, sizeof(FEXCore::Core::CPUState));
}
@@ -115,11 +115,11 @@ namespace FEXCore::Context {
}
void RegisterHostSignalHandler(FEXCore::Context::Context *CTX, int Signal, HostSignalDelegatorFunction Func, bool Required) {
CTX->RegisterHostSignalHandler(Signal, Func, Required);
CTX->RegisterHostSignalHandler(Signal, std::move(Func), Required);
}
void RegisterFrontendHostSignalHandler(FEXCore::Context::Context *CTX, int Signal, HostSignalDelegatorFunction Func, bool Required) {
CTX->RegisterFrontendHostSignalHandler(Signal, Func, Required);
CTX->RegisterFrontendHostSignalHandler(Signal, std::move(Func), Required);
}
FEXCore::Core::InternalThreadState* CreateThread(FEXCore::Context::Context *CTX, FEXCore::Core::CPUState *NewThreadState, uint64_t ParentTID) {
@@ -162,12 +162,20 @@ namespace FEXCore::Context {
return CTX->CPUID.RunFunction(Function, Leaf);
}
void SetAOTIRLoader(FEXCore::Context::Context *CTX, std::function<int(const std::string&)> CacheReader) {
CTX->AOTIRLoader = CacheReader;
FEX_DEFAULT_VISIBILITY FEXCore::CPUID::FunctionResults RunCPUIDFunctionName(FEXCore::Context::Context *CTX, uint32_t Function, uint32_t Leaf, uint32_t CPU) {
return CTX->CPUID.RunFunctionName(Function, Leaf, CPU);
}
void SetAOTIRWriter(FEXCore::Context::Context *CTX, std::function<std::unique_ptr<std::ostream>(const std::string&)> CacheWriter) {
CTX->AOTIRWriter = CacheWriter;
void SetAOTIRLoader(FEXCore::Context::Context *CTX, std::function<int(const std::string&)> CacheReader) {
CTX->SetAOTIRLoader(CacheReader);
}
void SetAOTIRWriter(FEXCore::Context::Context *CTX, std::function<std::unique_ptr<std::ofstream>(const std::string&)> CacheWriter) {
CTX->SetAOTIRWriter(CacheWriter);
}
void SetAOTIRRenamer(FEXCore::Context::Context *CTX, std::function<void(const std::string&)> CacheRenamer) {
CTX->SetAOTIRRenamer(CacheRenamer);
}
void FinalizeAOTIRCache(FEXCore::Context::Context *CTX) {
+22 -68
View File
@@ -4,6 +4,7 @@
#include "Interface/Core/CPUID.h"
#include "Interface/Core/HostFeatures.h"
#include "Interface/Core/X86HelperGen.h"
#include "Interface/IR/AOTIR.h"
#include <FEXCore/Config/Config.h>
#include <FEXCore/Core/Context.h>
#include <FEXCore/Core/CoreState.h>
@@ -57,38 +58,6 @@ namespace FEXCore::Context {
MODE_SINGLESTEP = 1,
};
struct AOTIRInlineEntry {
uint64_t GuestHash;
uint64_t GuestLength;
/* RAData followed by IRData */
uint8_t InlineData[0];
IR::RegisterAllocationData *GetRAData();
IR::IRListView *GetIRData();
};
struct AOTIRInlineIndexEntry {
uint64_t GuestStart;
uint64_t DataOffset;
};
struct AOTIRInlineIndex {
uint64_t Count;
uint64_t DataBase;
AOTIRInlineIndexEntry Entries[0];
AOTIRInlineEntry *Find(uint64_t GuestStart);
AOTIRInlineEntry *GetInlineEntry(uint64_t DataOffset);
};
struct AOTIRCaptureCacheEntry {
std::unique_ptr<std::ostream> Stream;
std::map<uint64_t, uint64_t> Index;
void AppendAOTIRCaptureCache(uint64_t GuestRIP, uint64_t Start, uint64_t Length, uint64_t Hash, FEXCore::IR::IRListView *IRList, FEXCore::IR::RegisterAllocationData *RAData);
};
struct Context {
friend class FEXCore::HLE::SyscallHandler;
#ifdef JIT_ARM64
@@ -156,32 +125,6 @@ namespace FEXCore::Context {
CustomCPUFactoryType CustomCPUFactory;
FEXCore::Context::ExitHandler CustomExitHandler;
struct AOTIRCacheEntry {
AOTIRInlineIndex *Array;
void *mapping;
size_t size;
};
std::unordered_map<std::string, AOTIRCacheEntry> AOTIRCache;
std::function<int(const std::string&)> AOTIRLoader;
std::function<std::unique_ptr<std::ostream>(const std::string&)> AOTIRWriter;
std::unordered_map<std::string, AOTIRCaptureCacheEntry> AOTIRCaptureCache;
struct AddrToFileEntry {
uint64_t Start;
uint64_t Len;
uint64_t Offset;
std::string fileid;
std::string filename;
void *CachedFileEntry;
bool ContainsCode;
};
using AddrToFileMapType = std::map<uint64_t, AddrToFileEntry>;
AddrToFileMapType AddrToFile;
std::map<std::string, std::string> FilesWithCode;
AddrToFileMapType::iterator FindAddrForFile(uint64_t Entry, uint64_t Length);
#ifdef BLOCKSTATS
std::unique_ptr<FEXCore::BlockSamplingData> BlockData;
#endif
@@ -253,9 +196,6 @@ namespace FEXCore::Context {
// same as CompileBlock, but aborts on failure
void CompileBlockJit(FEXCore::Core::CpuStateFrame *Frame, uint64_t GuestRIP);
bool LoadAOTIRCache(int streamfd);
void FinalizeAOTIRCache();
void WriteFilesWithCode(std::function<void(const std::string& fileid, const std::string& filename)> Writer);
/**
* @brief Initializes the JIT compilers for the thread
*
@@ -337,6 +277,26 @@ namespace FEXCore::Context {
// Public for threading
void ExecutionThread(FEXCore::Core::InternalThreadState *Thread);
void FinalizeAOTIRCache() {
IRCaptureCache.FinalizeAOTIRCache();
}
void WriteFilesWithCode(std::function<void(const std::string& fileid, const std::string& filename)> Writer) {
IRCaptureCache.WriteFilesWithCode(Writer);
}
void SetAOTIRLoader(std::function<int(const std::string&)> CacheReader) {
IRCaptureCache.SetAOTIRLoader(CacheReader);
}
void SetAOTIRWriter(std::function<std::unique_ptr<std::ofstream>(const std::string&)> CacheWriter) {
IRCaptureCache.SetAOTIRWriter(CacheWriter);
}
void SetAOTIRRenamer(std::function<void(const std::string&)> CacheRenamer) {
IRCaptureCache.SetAOTIRRenamer(CacheRenamer);
}
protected:
void ClearCodeCache(FEXCore::Core::InternalThreadState *Thread, bool AlsoClearIRCache);
@@ -362,13 +322,7 @@ namespace FEXCore::Context {
std::mutex ExitMutex;
std::unique_ptr<GdbServer> DebugServer;
std::shared_mutex AOTIRCacheLock;
std::shared_mutex AOTIRCaptureCacheWriteoutLock;
std::atomic<bool> AOTIRCaptureCacheWriteoutFlusing;
std::queue<std::function<void()>> AOTIRCaptureCacheWriteoutQueue;
void AOTIRCaptureCacheWriteoutQueue_Flush();
void AOTIRCaptureCacheWriteoutQueue_Append(const std::function<void()> &fn);
IR::AOTIRCaptureCache IRCaptureCache;
bool StartPaused = false;
FEX_CONFIG_OPT(AppFilename, APP_FILENAME);
@@ -535,7 +535,6 @@ uint64_t HandleCASPAL_ARMv8(void *_ucontext, void *_info, uint32_t Instr) {
}
bool HandleAtomicVectorStore(void *_ucontext, void *_info, uint32_t Instr) {
mcontext_t* mcontext = &reinterpret_cast<ucontext_t*>(_ucontext)->uc_mcontext;
siginfo_t* info = reinterpret_cast<siginfo_t*>(_info);
if (info->si_code != BUS_ADRALN) {
@@ -1,4 +1,5 @@
#include "Interface/Core/ArchHelpers/Arm64Emitter.h"
#include "Interface/Context/Context.h"
#include <FEXCore/Core/CoreState.h>
#include <FEXCore/Utils/LogManager.h>
@@ -15,53 +16,44 @@ namespace FEXCore::CPU {
#define STATE x28
// We want vixl to not allocate a default buffer. Jit and dispatcher will manually create one.
Arm64Emitter::Arm64Emitter(size_t size) : vixl::aarch64::Assembler(size, vixl::aarch64::PositionDependentCode) {
Arm64Emitter::Arm64Emitter(FEXCore::Context::Context *ctx, size_t size) : vixl::aarch64::Assembler(size, vixl::aarch64::PositionDependentCode) {
CPU.SetUp();
auto Features = vixl::CPUFeatures::InferFromOS();
SupportsAtomics = Features.Has(vixl::CPUFeatures::Feature::kAtomics);
// RCPC is bugged on Snapdragon 865
// Causes glibc cond16 test to immediately throw assert
// __pthread_mutex_cond_lock: Assertion `mutex->__data.__owner == 0'
SupportsRCPC = false; //Features.Has(vixl::CPUFeatures::Feature::kRCpc);
if (SupportsAtomics) {
if (ctx->HostFeatures.SupportsAtomics) {
// Hypervisor can hide this on the c630?
Features.Combine(vixl::CPUFeatures::Feature::kLORegions);
}
SetCPUFeatures(Features);
if (!SupportsAtomics) {
WARN_ONCE_FMT("Host CPU doesn't support atomics. Expect bad performance");
}
#ifdef _M_ARM_64
// We need to get the CPU's cache line size
// We expect sane targets that have correct cacheline sizes across clusters
uint64_t CTR;
__asm volatile ("mrs %[ctr], ctr_el0"
: [ctr] "=r"(CTR));
DCacheLineSize = 4 << ((CTR >> 16) & 0xF);
ICacheLineSize = 4 << (CTR & 0xF);
#endif
}
void Arm64Emitter::LoadConstant(vixl::aarch64::Register Reg, uint64_t Constant) {
void Arm64Emitter::LoadConstant(vixl::aarch64::Register Reg, uint64_t Constant, bool NOPPad) {
bool Is64Bit = Reg.IsX();
int Segments = Is64Bit ? 4 : 2;
if (Is64Bit && ((~Constant)>> 16) == 0) {
movn(Reg, (~Constant) & 0xFFFF);
if (NOPPad) {
nop(); nop(); nop();
}
return;
}
int NumMoves = 1;
movz(Reg, (Constant) & 0xFFFF, 0);
for (int i = 1; i < Segments; ++i) {
uint16_t Part = (Constant >> (i * 16)) & 0xFFFF;
if (Part) {
movk(Reg, Part, i * 16);
++NumMoves;
}
}
if (NOPPad) {
for (int i = NumMoves; i < Segments; ++i) {
nop();
}
}
}
@@ -70,7 +62,7 @@ void Arm64Emitter::PushCalleeSavedRegisters() {
// We need to save pairs of registers
// We save r19-r30
MemOperand PairOffset(sp, -16, PreIndex);
const std::array<std::pair<vixl::aarch64::XRegister, vixl::aarch64::XRegister>, 6> CalleeSaved = {{
const std::array<std::pair<vixl::aarch64::Register, vixl::aarch64::Register>, 6> CalleeSaved = {{
{x19, x20},
{x21, x22},
{x23, x24},
@@ -133,7 +125,7 @@ void Arm64Emitter::PopCalleeSavedRegisters() {
}
MemOperand PairOffset(sp, 16, PostIndex);
const std::array<std::pair<vixl::aarch64::XRegister, vixl::aarch64::XRegister>, 6> CalleeSaved = {{
const std::array<std::pair<vixl::aarch64::Register, vixl::aarch64::Register>, 6> CalleeSaved = {{
{x29, x30},
{x27, x28},
{x25, x26},
@@ -148,51 +140,75 @@ void Arm64Emitter::PopCalleeSavedRegisters() {
}
void Arm64Emitter::SpillStaticRegs(bool FPRs, uint32_t SpillMask) {
void Arm64Emitter::SpillStaticRegs(bool FPRs, uint32_t GPRSpillMask, uint32_t FPRSpillMask) {
if (StaticRegisterAllocation()) {
for (size_t i = 0; i < SRA64.size(); i+=2) {
auto Reg1 = SRA64[i];
auto Reg2 = SRA64[i+1];
if (((1U << Reg1.GetCode()) & SpillMask) &&
((1U << Reg2.GetCode()) & SpillMask)) {
if (((1U << Reg1.GetCode()) & GPRSpillMask) &&
((1U << Reg2.GetCode()) & GPRSpillMask)) {
stp(Reg1, Reg2, MemOperand(STATE, offsetof(FEXCore::Core::CpuStateFrame, State.gregs[i])));
}
else if (((1U << Reg1.GetCode()) & SpillMask)) {
else if (((1U << Reg1.GetCode()) & GPRSpillMask)) {
str(Reg1, MemOperand(STATE, offsetof(FEXCore::Core::CpuStateFrame, State.gregs[i])));
}
else if (((1U << Reg2.GetCode()) & SpillMask)) {
else if (((1U << Reg2.GetCode()) & GPRSpillMask)) {
str(Reg2, MemOperand(STATE, offsetof(FEXCore::Core::CpuStateFrame, State.gregs[i+1])));
}
}
if (FPRs) {
for (size_t i = 0; i < SRAFPR.size(); i+=2) {
stp(SRAFPR[i].Q(), SRAFPR[i+1].Q(), MemOperand(STATE, offsetof(FEXCore::Core::CpuStateFrame, State.xmm[i][0])));
auto Reg1 = SRAFPR[i];
auto Reg2 = SRAFPR[i+1];
if (((1U << Reg1.GetCode()) & FPRSpillMask) &&
((1U << Reg2.GetCode()) & FPRSpillMask)) {
stp(Reg1.Q(), Reg2.Q(), MemOperand(STATE, offsetof(FEXCore::Core::CpuStateFrame, State.xmm[i][0])));
}
else if (((1U << Reg1.GetCode()) & FPRSpillMask)) {
str(Reg1.Q(), MemOperand(STATE, offsetof(FEXCore::Core::CpuStateFrame, State.xmm[i][0])));
}
else if (((1U << Reg2.GetCode()) & FPRSpillMask)) {
str(Reg2.Q(), MemOperand(STATE, offsetof(FEXCore::Core::CpuStateFrame, State.xmm[i+1][0])));
}
}
}
}
}
void Arm64Emitter::FillStaticRegs(bool FPRs, uint32_t FillMask) {
void Arm64Emitter::FillStaticRegs(bool FPRs, uint32_t GPRFillMask, uint32_t FPRFillMask) {
if (StaticRegisterAllocation()) {
for (size_t i = 0; i < SRA64.size(); i+=2) {
auto Reg1 = SRA64[i];
auto Reg2 = SRA64[i+1];
if (((1U << Reg1.GetCode()) & FillMask) &&
((1U << Reg2.GetCode()) & FillMask)) {
if (((1U << Reg1.GetCode()) & GPRFillMask) &&
((1U << Reg2.GetCode()) & GPRFillMask)) {
ldp(Reg1, Reg2, MemOperand(STATE, offsetof(FEXCore::Core::CpuStateFrame, State.gregs[i])));
}
else if (((1U << Reg1.GetCode()) & FillMask)) {
else if (((1U << Reg1.GetCode()) & GPRFillMask)) {
ldr(Reg1, MemOperand(STATE, offsetof(FEXCore::Core::CpuStateFrame, State.gregs[i])));
}
else if (((1U << Reg2.GetCode()) & FillMask)) {
else if (((1U << Reg2.GetCode()) & GPRFillMask)) {
ldr(Reg2, MemOperand(STATE, offsetof(FEXCore::Core::CpuStateFrame, State.gregs[i+1])));
}
}
if (FPRs) {
for (size_t i = 0; i < SRAFPR.size(); i+=2) {
ldp(SRAFPR[i].Q(), SRAFPR[i+1].Q(), MemOperand(STATE, offsetof(FEXCore::Core::CpuStateFrame, State.xmm[i][0])));
auto Reg1 = SRAFPR[i];
auto Reg2 = SRAFPR[i+1];
if (((1U << Reg1.GetCode()) & FPRFillMask) &&
((1U << Reg2.GetCode()) & FPRFillMask)) {
ldp(Reg1.Q(), Reg2.Q(), MemOperand(STATE, offsetof(FEXCore::Core::CpuStateFrame, State.xmm[i][0])));
}
else if (((1U << Reg1.GetCode()) & FPRFillMask)) {
ldr(Reg1.Q(), MemOperand(STATE, offsetof(FEXCore::Core::CpuStateFrame, State.xmm[i][0])));
}
else if (((1U << Reg2.GetCode()) & FPRFillMask)) {
ldr(Reg2.Q(), MemOperand(STATE, offsetof(FEXCore::Core::CpuStateFrame, State.xmm[i+1][0])));
}
}
}
}
@@ -58,15 +58,18 @@ const std::array<aarch64::VRegister, 12> RAFPR = {
// be used by both Arm64 JIT and ARM64 Dispatcher
class Arm64Emitter : public vixl::aarch64::Assembler {
protected:
Arm64Emitter(size_t size);
Arm64Emitter(FEXCore::Context::Context *ctx, size_t size);
vixl::aarch64::CPU CPU;
bool SupportsAtomics{};
bool SupportsRCPC{};
void LoadConstant(vixl::aarch64::Register Reg, uint64_t Constant, bool NOPPad = false);
void SpillStaticRegs(bool FPRs = true, uint32_t GPRSpillMask = ~0U, uint32_t FPRSpillMask = ~0U);
void FillStaticRegs(bool FPRs = true, uint32_t GPRFillMask = ~0U, uint32_t FPRFillMask = ~0U);
void LoadConstant(vixl::aarch64::Register Reg, uint64_t Constant);
void SpillStaticRegs(bool FPRs = true, uint32_t SpillMask = ~0U);
void FillStaticRegs(bool FPRs = true, uint32_t FillMask = ~0U);
static constexpr uint32_t CALLER_GPR_MASK = 0b0011'1111'1111'1111'1111;
// This isn't technically true because the lower 64-bits of v8..v15 are callee saved
// We can't guarantee only the lower 64bits are used so flush everything
static constexpr uint32_t CALLER_FPR_MASK = ~0U;
void PushDynamicRegsAndLR();
void PopDynamicRegsAndLR();
@@ -79,9 +82,6 @@ protected:
uint32_t SpillSlots{};
uint32_t DCacheLineSize{};
uint32_t ICacheLineSize{};
FEX_CONFIG_OPT(StaticRegisterAllocation, SRA);
};
@@ -24,6 +24,7 @@ struct X86ContextBackup {
uint64_t GPRs[23];
FEXCore::x86_64::_libc_fpstate FPRState;
uint64_t sa_mask;
bool FaultToTopAndGeneratedException;
// Guest state
int Signal;
@@ -46,6 +47,7 @@ struct ArmContextBackup {
uint32_t FPCR;
__uint128_t FPRs[32];
uint64_t sa_mask;
bool FaultToTopAndGeneratedException;
// Guest state
int Signal;
+378 -69
View File
@@ -5,14 +5,16 @@ desc: Handles presented capability bits for guest cpu
$end_info$
*/
#include <FEXCore/Config/Config.h>
#include <FEXCore/Core/CPUID.h>
#include "Common/StringConv.h"
#include "Interface/Context/Context.h"
#include "Interface/Core/CPUID.h"
#include "Interface/Core/HostFeatures.h"
#include "Utils/FileLoading.h"
#include <FEXCore/Config/Config.h>
#include <FEXCore/Core/CPUID.h>
#include <FEXHeaderUtils/Syscalls.h>
#include "git_version.h"
#include <cstring>
@@ -21,6 +23,66 @@ $end_info$
#endif
namespace FEXCore {
namespace ProductNames {
#ifdef _M_ARM_64
static const char ARM_UNKNOWN[] = "Unknown ARM CPU";
static const char ARM_A57[] = "Cortex-A57";
static const char ARM_A72[] = "Cortex-A72";
static const char ARM_A73[] = "Cortex-A73";
static const char ARM_A75[] = "Cortex-A75";
static const char ARM_A76[] = "Cortex-A76";
static const char ARM_A76AE[] = "Cortex-A76AE";
static const char ARM_V1[] = "Neoverse V1";
static const char ARM_A77[] = "Cortex-A77";
static const char ARM_A78[] = "Cortex-A78";
static const char ARM_A78AE[] = "Cortex-A78AE";
static const char ARM_A78C[] = "Cortex-A78C";
static const char ARM_A710[] = "Cortex-A710";
static const char ARM_X1[] = "Cortex-X1";
static const char ARM_X2[] = "Cortex-X2";
static const char ARM_N1[] = "Neoverse N1";
static const char ARM_N2[] = "Neoverse N2";
static const char ARM_E1[] = "Neoverse E1";
static const char ARM_A35[] = "Cortex-A35";
static const char ARM_A53[] = "Cortex-A53";
static const char ARM_A55[] = "Cortex-A55";
static const char ARM_A65[] = "Cortex-A65";
static const char ARM_A510[] = "Cortex-A510";
static const char ARM_Kryo200[] = "Kryo 2xx";
static const char ARM_Kryo300[] = "Kryo 3xx";
static const char ARM_Kryo400[] = "Kryo 4xx/5xx";
static const char ARM_Kryo200S[] = "Kryo 2xx S";
static const char ARM_Kryo300S[] = "Kryo 3xx S";
static const char ARM_Kryo400S[] = "Kryo 4xx/5xx S";
static const char ARM_Denver[] = "Nvidia Denver";
static const char ARM_Carmel[] = "Nvidia Carmel";
static const char ARM_Firestorm[] = "Apple Firestorm";
static const char ARM_Icestorm[] = "Apple Icestorm";
#else
static const char UNKNOWN[] = "Unknown CPU";
#endif
}
static uint32_t GetCPUID() {
uint32_t CPU{};
FHU::Syscalls::getcpu(&CPU, nullptr);
return CPU;
}
static uint32_t CalculateNumberOfCPUs() {
size_t CPUs = 1;
while(std::filesystem::exists("/sys/devices/system/cpu/cpu" + std::to_string(CPUs))) {
CPUs++;
}
return CPUs;
}
constexpr uint32_t SUPPORTS_AVX = 0;
// #define CPUID_AMD
#ifdef CPUID_AMD
@@ -49,60 +111,242 @@ static uint32_t GetCycleCounterFrequency() {
return Result;
}
static bool GetHostHybridFlag() {
int MaxCPUs = 64;
size_t AllocSize = CPU_ALLOC_SIZE(MaxCPUs);
cpu_set_t *Set = CPU_ALLOC(MaxCPUs);
CPU_ZERO_S(AllocSize, Set);
void CPUIDEmu::SetupHostHybridFlag() {
size_t CPUs = CalculateNumberOfCPUs();
PerCPUData.resize(CPUs);
int Result{};
for (;;) {
Result = sched_getaffinity(0, AllocSize, Set);
if (Result == 0 ||
(Result == -1 && errno != EINVAL)) {
break;
}
MaxCPUs <<= 1;
CPU_FREE(Set);
Set = CPU_ALLOC(MaxCPUs);
AllocSize = CPU_ALLOC_SIZE(MaxCPUs);
CPU_ZERO_S(AllocSize, Set);
}
if (Result != 0) {
return false;
}
int CPUs = CPU_COUNT_S(AllocSize, Set);
bool Hybrid = false;
uint64_t MIDR{};
for (int i = 0; i < CPUs; ++i) {
if (CPU_ISSET_S(i, AllocSize, Set)) {
std::error_code ec{};
std::string MIDRPath = "/sys/devices/system/cpu/cpu" + std::to_string(i) + "/regs/identification/midr_el1";
if (std::filesystem::exists(MIDRPath, ec)) {
std::vector<char> Data{};
// Needs to be a fixed size since depending on kernel it will try to read a full page of data and fail
// Only read 18 bytes for a 64bit value prefixed with 0x
if (FEXCore::FileLoading::LoadFile(Data, MIDRPath, 18)) {
uint64_t NewMIDR{};
if (FEXCore::StrConv::Conv(&Data.at(0), &NewMIDR)) {
if (MIDR != 0 && MIDR != NewMIDR) {
// CPU mismatch, claim hybrid
Hybrid = true;
break;
}
MIDR = NewMIDR;
for (size_t i = 0; i < CPUs; ++i) {
std::error_code ec{};
std::string MIDRPath = "/sys/devices/system/cpu/cpu" + std::to_string(i) + "/regs/identification/midr_el1";
if (std::filesystem::exists(MIDRPath, ec)) {
std::vector<char> Data{};
// Needs to be a fixed size since depending on kernel it will try to read a full page of data and fail
// Only read 18 bytes for a 64bit value prefixed with 0x
if (FEXCore::FileLoading::LoadFile(Data, MIDRPath, 18)) {
uint64_t NewMIDR{};
std::string_view MIDRView(&Data.at(0), 18);
if (FEXCore::StrConv::Conv(MIDRView, &NewMIDR)) {
if (MIDR != 0 && MIDR != NewMIDR) {
// CPU mismatch, claim hybrid
Hybrid = true;
}
// Truncate to 32-bits, top 32-bits are all reserved in MIDR
PerCPUData[i].ProductName = ProductNames::ARM_UNKNOWN;
PerCPUData[i].MIDR = NewMIDR;
MIDR = NewMIDR;
}
}
}
}
CPU_FREE(Set);
return Hybrid;
struct CPUMIDR {
uint8_t Implementer;
uint16_t Part;
bool DefaultBig; // Defaults to a big core
const char *ProductName{};
};
// CPU priority order
// This is mostly arbitrary but will sort by some sort of CPU priority by performance
// Relative list so things they will commonly end up in big.little configurations sort of relate
static constexpr std::array<CPUMIDR, 35> CPUMIDRs = {{
// Typically big CPU cores
{0x61, 0x023, 1, ProductNames::ARM_Firestorm}, // Apple M1 Firestorm
{0x41, 0xd49, 1, ProductNames::ARM_N2}, // N2
{0x41, 0xd4b, 1, ProductNames::ARM_A78C}, // A78C
{0x41, 0xd4a, 1, ProductNames::ARM_E1}, // E1
{0x41, 0xd49, 1, ProductNames::ARM_N2}, // N2
{0x41, 0xd48, 1, ProductNames::ARM_X2}, // X2
{0x41, 0xd47, 1, ProductNames::ARM_A710}, // A710
{0x41, 0xd44, 1, ProductNames::ARM_X1}, // X1
{0x41, 0xd42, 1, ProductNames::ARM_A78AE}, // A78AE
{0x41, 0xd41, 1, ProductNames::ARM_A78}, // A78
{0x41, 0xd40, 1, ProductNames::ARM_V1}, // V1
{0x41, 0xd0e, 1, ProductNames::ARM_A76AE}, // A76AE
{0x41, 0xd0d, 1, ProductNames::ARM_A77}, // A77
{0x41, 0xd0c, 1, ProductNames::ARM_N1}, // N1
{0x41, 0xd0b, 1, ProductNames::ARM_A76}, // A76
{0x51, 0x804, 1, ProductNames::ARM_Kryo400}, // Kryo 4xx Gold (A76 based)
{0x41, 0xd0a, 1, ProductNames::ARM_A75}, // A75
{0x51, 0x802, 1, ProductNames::ARM_Kryo300}, // Kryo 3xx Gold (A75 based)
{0x41, 0xd09, 1, ProductNames::ARM_A73}, // A73
{0x51, 0x800, 1, ProductNames::ARM_Kryo200}, // Kryo 2xx Gold (A73 based)
{0x41, 0xd08, 1, ProductNames::ARM_A72}, // A72
{0x4e, 0x004, 1, ProductNames::ARM_Carmel}, // Carmel
// Denver rated above A57 to match TX2 weirdness
{0x4e, 0x003, 1, ProductNames::ARM_Denver}, // Denver
{0x41, 0xd07, 1, ProductNames::ARM_A57}, // A57
// Typically Little CPU cores
{0x61, 0x022, 0, ProductNames::ARM_Icestorm}, // Apple M1 Icestorm
{0x41, 0xd46, 0, ProductNames::ARM_A510}, // A510
{0x41, 0xd06, 0, ProductNames::ARM_A65}, // A65
{0x41, 0xd05, 0, ProductNames::ARM_A55}, // A55
{0x51, 0x805, 0, ProductNames::ARM_Kryo400S}, // Kryo 4xx/5xx Silver (A55 based)
{0x51, 0x803, 0, ProductNames::ARM_Kryo300S}, // Kryo 3xx Silver (A55 based)
{0x41, 0xd03, 0, ProductNames::ARM_A53}, // A53
{0x51, 0x801, 0, ProductNames::ARM_Kryo200S}, // Kryo 2xx Silver (A53 based)
{0x41, 0xd04, 0, ProductNames::ARM_A35}, // A35
{0x41, 0, 0, ProductNames::ARM_UNKNOWN}, // Invalid CPU or Apple CPU inside Parallels VM
{0x0, 0, 0, ProductNames::ARM_UNKNOWN}, // Invalid starting point is lowest ranked
}};
auto FindDefinedMIDR = [](uint32_t MIDR) -> const CPUMIDR* {
uint8_t Implementer = MIDR >> 24;
uint16_t Part = (MIDR >> 4) & 0xFFF;
for (auto &MIDROption : CPUMIDRs) {
if (MIDROption.Implementer == Implementer &&
MIDROption.Part == Part) {
return &MIDROption;
}
}
return nullptr;
};
if (Hybrid) {
// Walk the MIDRs and calculate big little designs
std::vector<const CPUMIDR*> BigCores;
std::vector<const CPUMIDR*> LittleCores;
// Separate CPU cores out to big or little selected
for (size_t i = 0; i < CPUs; ++i) {
uint32_t MIDR = PerCPUData[i].MIDR;
auto MIDROption = FindDefinedMIDR(MIDR);
if (MIDROption) {
// Found one
if (MIDROption->DefaultBig) {
BigCores.emplace_back(MIDROption);
}
else {
LittleCores.emplace_back(MIDROption);
}
}
else {
// If we didn't insert this MIDR then claim it is a little core.
LittleCores.emplace_back(&CPUMIDRs.back());
}
}
if (LittleCores.empty()) {
// If we only ended up with big cores then we need to move some to be little cores
uint32_t LowestMIDR = ~0U;
uint32_t LowestMIDRIdx = 0;
// Walk all the big cores
for (size_t i = 0; i < BigCores.size(); ++i) {
uint8_t Implementer = BigCores[i]->Implementer;
uint16_t Part = BigCores[i]->Part;
// Walk our list of CPUMIDRs to find the most little core
for (size_t j = LowestMIDRIdx; j < CPUMIDRs.size(); ++j) {
auto &MIDROption = CPUMIDRs[i];
if ((MIDROption.Implementer == Implementer &&
MIDROption.Part == Part) ||
(MIDROption.Implementer == 0 &&
MIDROption.Part == 0)) {
LowestMIDRIdx = j;
LowestMIDR = MIDR;
break;
}
}
}
// Now we WILL have found a big core to demote to little status
// Demote them
std::erase_if(BigCores, [&LittleCores, LowestMIDR](auto *Entry) {
// Demote by erase copy to little array
uint8_t Implementer = LowestMIDR >> 24;
uint16_t Part = (LowestMIDR >> 4) & 0xFFF;
if (Entry->Implementer == Implementer &&
Entry->Part == Part) {
// Add it to the BigCore list
LittleCores.emplace_back(Entry);
return true;
}
return false;
});
}
if (BigCores.empty()) {
// We never found a CPU core we understand
// Grab the first core, consider it as little, move everything else to Big
uint32_t LittleMIDR = PerCPUData[0].MIDR;
// Now walk the little cores and move them to Big if they don't match
std::erase_if(LittleCores, [&BigCores, LittleMIDR](auto *Entry) {
// You're promoted now
uint8_t Implementer = LittleMIDR >> 24;
uint16_t Part = (LittleMIDR >> 4) & 0xFFF;
if (Entry->Implementer != Implementer ||
Entry->Part != Part) {
// Add it to the BigCore list
BigCores.emplace_back(Entry);
return true;
}
return false;
});
}
// Now walk the per CPU data one more time and set if it is big or little
for (auto &Data : PerCPUData) {
uint8_t Implementer = Data.MIDR >> 24;
uint16_t Part = (Data.MIDR >> 4) & 0xFFF;
bool FoundBig{};
const CPUMIDR *MIDR{};
for (auto Big : BigCores) {
if (Big->Implementer == Implementer &&
Big->Part == Part) {
FoundBig = true;
MIDR = Big;
break;
}
}
if (!FoundBig) {
for (auto Little : LittleCores) {
if (Little->Implementer == Implementer &&
Little->Part == Part) {
MIDR = Little;
break;
}
}
}
Data.IsBig = FoundBig;
if (MIDR) {
Data.ProductName = MIDR->ProductName ?: ProductNames::ARM_UNKNOWN;
}
else {
Data.ProductName = ProductNames::ARM_UNKNOWN;
}
}
}
else {
// If we aren't hybrid then just claim everything is big
for (size_t i = 0; i < CPUs; ++i) {
uint32_t MIDR = PerCPUData[i].MIDR;
auto MIDROption = FindDefinedMIDR(MIDR);
PerCPUData[i].IsBig = true;
if (MIDROption) {
PerCPUData[i].ProductName = MIDROption->ProductName ?: ProductNames::ARM_UNKNOWN;
}
else {
PerCPUData[i].ProductName = ProductNames::ARM_UNKNOWN;
}
}
}
}
#else
@@ -119,16 +363,21 @@ static uint32_t GetCycleCounterFrequency() {
return 0;
}
static bool GetHostHybridFlag() {
void CPUIDEmu::SetupHostHybridFlag() {
uint32_t eax, ebx, ecx, edx;
__cpuid(0, eax, ebx, ecx, edx);
if (eax >= 0x7) {
__cpuid(0x7, eax, ebx, ecx, edx);
// Bit 15 of edx claims hybrid CPU
return (edx & (1U << 15)) != 0;
Hybrid = (edx & (1U << 15)) != 0;
}
return false;
size_t CPUs = CalculateNumberOfCPUs();
PerCPUData.resize(CPUs);
for (size_t i = 0; i < CPUs; ++i) {
PerCPUData[i].IsBig = true;
PerCPUData[i].ProductName = ProductNames::UNKNOWN;
}
}
#endif
@@ -155,6 +404,8 @@ FEXCore::CPUID::FunctionResults CPUIDEmu::Function_0h(uint32_t Leaf) {
FEXCore::CPUID::FunctionResults CPUIDEmu::Function_01h(uint32_t Leaf) {
FEXCore::CPUID::FunctionResults Res{};
uint32_t CoreCount = Cores();
// XXX: Enable once the rest of the SSE4.2 instructions are emulated
uint32_t SupportsSSE42 = CTX->HostFeatures.SupportsCRC && false ? 1 : 0;
Res.eax = FAMILY_IDENTIFIER;
@@ -184,7 +435,7 @@ FEXCore::CPUID::FunctionResults CPUIDEmu::Function_01h(uint32_t Leaf) {
(0 << 17) | // Process-context identifiers
(0 << 18) | // Prefetching from memory mapped device
(1 << 19) | // SSE4.1
(0 << 20) | // SSE4.2
(SupportsSSE42 << 20) | // SSE4.2
(0 << 21) | // X2APIC
(1 << 22) | // MOVBE
(1 << 23) | // POPCNT
@@ -194,7 +445,7 @@ FEXCore::CPUID::FunctionResults CPUIDEmu::Function_01h(uint32_t Leaf) {
(0 << 27) | // OSXSAVE
(SUPPORTS_AVX << 28) | // AVX
(0 << 29) | // F16C
(0 << 30) | // RDRAND
(CTX->HostFeatures.SupportsRAND << 30) | // RDRAND
(0 << 31); // Hypervisor always returns zero
Res.edx =
@@ -386,7 +637,7 @@ FEXCore::CPUID::FunctionResults CPUIDEmu::Function_07h(uint32_t Leaf) {
(0 << 5) | // AVX2 support
(1 << 6) | // FPU data pointer updated only on exception
(1 << 7) | // SMEP support
(0 << 8) | // BMI2
(1 << 8) | // BMI2
(0 << 9) | // Enhanced REP MOVSB/STOSB
(1 << 10) | // INVPCID for system software control of process-context
(0 << 11) | // Restricted transactional memory
@@ -396,7 +647,7 @@ FEXCore::CPUID::FunctionResults CPUIDEmu::Function_07h(uint32_t Leaf) {
(0 << 15) | // Intel Resource Directory Technology Allocation
(0 << 16) | // Reserved
(0 << 17) | // Reserved
(0 << 18) | // RDSEED
(CTX->HostFeatures.SupportsRAND << 18) | // RDSEED
(1 << 19) | // ADCX and ADOX instructions
(0 << 20) | // SMAP Supervisor mode access prevention and CLAC/STAC instructions
(0 << 21) | // Reserved
@@ -549,6 +800,40 @@ FEXCore::CPUID::FunctionResults CPUIDEmu::Function_15h(uint32_t Leaf) {
return Res;
}
FEXCore::CPUID::FunctionResults CPUIDEmu::Function_1Ah(uint32_t Leaf) {
FEXCore::CPUID::FunctionResults Res{};
if (Hybrid) {
uint32_t CPU = GetCPUID();
auto &Data = PerCPUData[CPU];
// 0x40 is a big CPU
// 0x20 is a little CPU
Res.eax |= (Data.IsBig ? 0x40 : 0x20) << 24;
}
return Res;
}
// Hypervisor CPUID information leaf
FEXCore::CPUID::FunctionResults CPUIDEmu::Function_4000_0000h(uint32_t Leaf) {
FEXCore::CPUID::FunctionResults Res{};
// Maximum supported hypervisor leafs
// We only expose the information leaf
//
// Common courtesy to follow VMWare's "Hypervisor CPUID Interface proposal"
// 4000_0000h - Information leaf. Advertising to the software which hypervisor this is
// 4000_0001h - 4000_000Fh - Hypervisor specific leafs. FEX can use these for anything
// 4000_0010h - 4000_00FFh - "Generic Leafs" - Try not to overwrite, other hypervisors might expect information in these
//
// CPUID documentation information:
// 4000_0000h - 4FFF_FFFFh - No existing or future CPU will return information in this range
// Reserved entirely for VMs to do whatever they want.
Res.eax = 0x40000000;
// EBX, EDX, ECX become the hypervisor ID signature
constexpr static char HypervisorID[12] = "FEXIFEXIEMU";
memcpy(&Res.ebx, HypervisorID, sizeof(HypervisorID));
return Res;
}
// Highest extended function implemented
FEXCore::CPUID::FunctionResults CPUIDEmu::Function_8000_0000h(uint32_t Leaf) {
FEXCore::CPUID::FunctionResults Res{};
@@ -636,35 +921,52 @@ FEXCore::CPUID::FunctionResults CPUIDEmu::Function_8000_0001h(uint32_t Leaf) {
(1 << 24) | // FXSAVE/FXRSTOR
(1 << 25) | // FXSAVE/FXRSTOR Optimizations
(0 << 26) | // 1 gigabit pages
(0 << 27) | // RDTSCP
(1 << 27) | // RDTSCP
(0 << 28) | // Reserved
(1 << 29) | // Long Mode
(0 << 30) | // 3DNow! Extensions
(0 << 31); // 3DNow!
(1 << 30) | // 3DNow! Extensions
(1 << 31); // 3DNow!
return Res;
}
constexpr char ProcessorBrand[48] = {
constexpr char ProcessorBrand[32] = {
GIT_DESCRIBE_STRING
"\0"
};
constexpr ssize_t DESCRIBE_STR_SIZE = std::char_traits<char>::length(GIT_DESCRIBE_STRING);
static_assert(DESCRIBE_STR_SIZE < 32);
//Processor brand string
FEXCore::CPUID::FunctionResults CPUIDEmu::Function_8000_0002h(uint32_t Leaf) {
FEXCore::CPUID::FunctionResults Res{};
memcpy(&Res, &ProcessorBrand[0], sizeof(FEXCore::CPUID::FunctionResults));
return Res;
return Function_8000_0002h(Leaf, GetCPUID());
}
FEXCore::CPUID::FunctionResults CPUIDEmu::Function_8000_0003h(uint32_t Leaf) {
FEXCore::CPUID::FunctionResults Res{};
memcpy(&Res, &ProcessorBrand[16], sizeof(FEXCore::CPUID::FunctionResults));
return Res;
return Function_8000_0003h(Leaf, GetCPUID());
}
FEXCore::CPUID::FunctionResults CPUIDEmu::Function_8000_0004h(uint32_t Leaf) {
return Function_8000_0004h(Leaf, GetCPUID());
}
FEXCore::CPUID::FunctionResults CPUIDEmu::Function_8000_0002h(uint32_t Leaf, uint32_t CPU) {
FEXCore::CPUID::FunctionResults Res{};
memcpy(&Res, &ProcessorBrand[32], sizeof(FEXCore::CPUID::FunctionResults));
memset(&Res, ' ', sizeof(FEXCore::CPUID::FunctionResults));
memcpy(&Res, &ProcessorBrand[0], std::min(16L, DESCRIBE_STR_SIZE));
return Res;
}
FEXCore::CPUID::FunctionResults CPUIDEmu::Function_8000_0003h(uint32_t Leaf, uint32_t CPU) {
FEXCore::CPUID::FunctionResults Res{};
memset(&Res, ' ', sizeof(FEXCore::CPUID::FunctionResults));
memcpy(&Res, &ProcessorBrand[16], std::max(0L, DESCRIBE_STR_SIZE - 16));
return Res;
}
FEXCore::CPUID::FunctionResults CPUIDEmu::Function_8000_0004h(uint32_t Leaf, uint32_t CPU) {
FEXCore::CPUID::FunctionResults Res{};
auto &Data = PerCPUData[CPU];
memcpy(&Res, Data.ProductName, std::min(strlen(Data.ProductName), sizeof(FEXCore::CPUID::FunctionResults)));
return Res;
}
@@ -757,7 +1059,7 @@ FEXCore::CPUID::FunctionResults CPUIDEmu::Function_8000_0008h(uint32_t Leaf) {
Res.ebx =
(0 << 2) | // XSaveErPtr: Saving and restoring error pointers
(0 << 1) | // IRPerf: Instructions retired count support
(0 << 0); // CLZERO support
(CTX->HostFeatures.SupportsCLZERO << 0); // CLZERO support
uint32_t CoreCount = Cores() - 1;
Res.ecx =
@@ -920,6 +1222,13 @@ void CPUIDEmu::Init(FEXCore::Context::Context *ctx) {
// 0x16: Processor frequency information
// 0x17: SoC vendor attribute enumeration
// 0x1A: Hybrid Information Sub-leaf
#ifndef CPUID_AMD
RegisterFunction(0x1A, &CPUIDEmu::Function_1Ah);
#endif
// Hypervisor CPUID information leaf
RegisterFunction(0x4000'0000, &CPUIDEmu::Function_4000_0000h);
// Largest extended function number
RegisterFunction(0x8000'0000, &CPUIDEmu::Function_8000_0000h);
// Processor vendor
@@ -960,7 +1269,7 @@ void CPUIDEmu::Init(FEXCore::Context::Context *ctx) {
// 0x8000'001F: AMD Secure Encryption
// Setup some state tracking
Hybrid = GetHostHybridFlag();
SetupHostHybridFlag();
}
}
+33
View File
@@ -3,6 +3,7 @@
#include <cstdint>
#include <unordered_map>
#include <utility>
#include <vector>
#include <FEXCore/Core/CPUID.h>
#include <FEXCore/Config/Config.h>
@@ -23,6 +24,10 @@ private:
constexpr static uint32_t CPUID_VENDOR_AMD3 = 0x444D4163; // "cAMD"
public:
// X86 cacheline size effectively has to be hardcoded to 64
// if we report anything differently then applications are likely to break
constexpr static uint64_t CACHELINE_SIZE = 64;
void Init(FEXCore::Context::Context *ctx);
FEXCore::CPUID::FunctionResults RunFunction(uint32_t Function, uint32_t Leaf) {
@@ -34,6 +39,16 @@ public:
return (this->*Handler->second)(Leaf);
}
FEXCore::CPUID::FunctionResults RunFunctionName(uint32_t Function, uint32_t Leaf, uint32_t CPU) {
if (Function == 0x8000'0002U)
return Function_8000_0002h(Leaf, CPU % PerCPUData.size());
else if (Function == 0x8000'0003U)
return Function_8000_0003h(Leaf, CPU % PerCPUData.size());
else
return Function_8000_0004h(Leaf, CPU % PerCPUData.size());
}
private:
FEXCore::Context::Context *CTX;
bool Hybrid{};
@@ -45,6 +60,14 @@ private:
}
std::unordered_map<uint32_t, FunctionHandler> FunctionHandlers;
struct CPUData {
const char *ProductName{};
#ifdef _M_ARM_64
uint32_t MIDR{};
#endif
bool IsBig{};
};
std::vector<CPUData> PerCPUData{};
// Functions
FEXCore::CPUID::FunctionResults Function_0h(uint32_t Leaf);
@@ -55,11 +78,18 @@ private:
FEXCore::CPUID::FunctionResults Function_07h(uint32_t Leaf);
FEXCore::CPUID::FunctionResults Function_0Dh(uint32_t Leaf);
FEXCore::CPUID::FunctionResults Function_15h(uint32_t Leaf);
FEXCore::CPUID::FunctionResults Function_1Ah(uint32_t Leaf);
FEXCore::CPUID::FunctionResults Function_4000_0000h(uint32_t Leaf);
FEXCore::CPUID::FunctionResults Function_8000_0000h(uint32_t Leaf);
FEXCore::CPUID::FunctionResults Function_8000_0001h(uint32_t Leaf);
FEXCore::CPUID::FunctionResults Function_8000_0002h(uint32_t Leaf);
FEXCore::CPUID::FunctionResults Function_8000_0003h(uint32_t Leaf);
FEXCore::CPUID::FunctionResults Function_8000_0004h(uint32_t Leaf);
FEXCore::CPUID::FunctionResults Function_8000_0002h(uint32_t Leaf, uint32_t CPU);
FEXCore::CPUID::FunctionResults Function_8000_0003h(uint32_t Leaf, uint32_t CPU);
FEXCore::CPUID::FunctionResults Function_8000_0004h(uint32_t Leaf, uint32_t CPU);
FEXCore::CPUID::FunctionResults Function_8000_0005h(uint32_t Leaf);
FEXCore::CPUID::FunctionResults Function_8000_0006h(uint32_t Leaf);
FEXCore::CPUID::FunctionResults Function_8000_0007h(uint32_t Leaf);
@@ -68,5 +98,8 @@ private:
FEXCore::CPUID::FunctionResults Function_8000_0019h(uint32_t Leaf);
FEXCore::CPUID::FunctionResults Function_8000_001Dh(uint32_t Leaf);
FEXCore::CPUID::FunctionResults Function_Reserved(uint32_t Leaf);
void SetupHostHybridFlag();
};
}
+114 -426
View File
@@ -41,6 +41,7 @@ $end_info$
#include <FEXCore/Utils/Event.h>
#include <FEXCore/Utils/LogManager.h>
#include <FEXCore/Utils/Threads.h>
#include <FEXHeaderUtils/Syscalls.h>
#include <algorithm>
#include <array>
@@ -50,6 +51,7 @@ $end_info$
#include <cstdint>
#include <filesystem>
#include <functional>
#include <fstream>
#include <map>
#include <memory>
#include <mutex>
@@ -61,7 +63,6 @@ $end_info$
#include <string.h>
#include <string>
#include <string_view>
#include <sstream>
#include <sys/mman.h>
#include <sys/stat.h>
#include <sys/syscall.h>
@@ -141,54 +142,8 @@ std::string_view const& GetGRegName(unsigned Reg) {
} // namespace FEXCore::Core
namespace FEXCore::Context {
void Context::AOTIRCaptureCacheWriteoutQueue_Flush() {
{
std::shared_lock lk{AOTIRCaptureCacheWriteoutLock};
if (AOTIRCaptureCacheWriteoutQueue.size() == 0) {
AOTIRCaptureCacheWriteoutFlusing.store(false);
return;
}
}
for (;;) {
AOTIRCaptureCacheWriteoutLock.lock();
std::function<void()> fn = std::move(AOTIRCaptureCacheWriteoutQueue.front());
bool MaybeEmpty = false;
AOTIRCaptureCacheWriteoutQueue.pop();
MaybeEmpty = AOTIRCaptureCacheWriteoutQueue.size() == 0;
AOTIRCaptureCacheWriteoutLock.unlock();
fn();
if (MaybeEmpty) {
std::shared_lock lk{AOTIRCaptureCacheWriteoutLock};
if (AOTIRCaptureCacheWriteoutQueue.size() == 0) {
AOTIRCaptureCacheWriteoutFlusing.store(false);
return;
}
}
}
LOGMAN_MSG_A_FMT("Must never get here");
}
void Context::AOTIRCaptureCacheWriteoutQueue_Append(const std::function<void()> &fn) {
bool Flush = false;
{
std::unique_lock lk{AOTIRCaptureCacheWriteoutLock};
AOTIRCaptureCacheWriteoutQueue.push(fn);
if (AOTIRCaptureCacheWriteoutQueue.size() > 10000) {
Flush = true;
}
}
bool test_val = false;
if (Flush && AOTIRCaptureCacheWriteoutFlusing.compare_exchange_strong(test_val, true)) {
AOTIRCaptureCacheWriteoutQueue_Flush();
}
}
Context::Context() {
Context::Context()
: IRCaptureCache {this} {
#ifdef BLOCKSTATS
BlockData = std::make_unique<FEXCore::BlockSamplingData>();
#endif
@@ -211,10 +166,6 @@ namespace FEXCore::Context {
}
Threads.clear();
}
for (auto &Mod: AOTIRCache) {
FEXCore::Allocator::munmap(Mod.second.mapping, Mod.second.size);
}
}
static FEXCore::Core::CPUState CreateDefaultCPUState() {
@@ -323,7 +274,7 @@ namespace FEXCore::Context {
Thread->SignalReason.store(FEXCore::Core::SignalEvent::Pause);
if (Thread->RunningEvents.Running.load()) {
// Only attempt to stop this thread if it is running
tgkill(Thread->ThreadManager.PID, Thread->ThreadManager.TID, SignalDelegator::SIGNAL_FOR_PAUSE);
FHU::Syscalls::tgkill(Thread->ThreadManager.PID, Thread->ThreadManager.TID, SignalDelegator::SIGNAL_FOR_PAUSE);
}
}
}
@@ -386,7 +337,7 @@ namespace FEXCore::Context {
}
void Context::Stop(bool IgnoreCurrentThread) {
pid_t tid = gettid();
pid_t tid = FHU::Syscalls::gettid();
FEXCore::Core::InternalThreadState* CurrentThread{};
// Tell all the threads that they should stop
@@ -426,20 +377,25 @@ namespace FEXCore::Context {
void Context::StopThread(FEXCore::Core::InternalThreadState *Thread) {
if (Thread->RunningEvents.Running.exchange(false)) {
Thread->SignalReason.store(FEXCore::Core::SignalEvent::Stop);
tgkill(Thread->ThreadManager.PID, Thread->ThreadManager.TID, SignalDelegator::SIGNAL_FOR_PAUSE);
FHU::Syscalls::tgkill(Thread->ThreadManager.PID, Thread->ThreadManager.TID, SignalDelegator::SIGNAL_FOR_PAUSE);
}
}
void Context::SignalThread(FEXCore::Core::InternalThreadState *Thread, FEXCore::Core::SignalEvent Event) {
if (Thread->RunningEvents.Running.load()) {
Thread->SignalReason.store(Event);
tgkill(Thread->ThreadManager.PID, Thread->ThreadManager.TID, SignalDelegator::SIGNAL_FOR_PAUSE);
FHU::Syscalls::tgkill(Thread->ThreadManager.PID, Thread->ThreadManager.TID, SignalDelegator::SIGNAL_FOR_PAUSE);
}
}
FEXCore::Context::ExitReason Context::RunUntilExit() {
if(!StartPaused)
Run();
if(!StartPaused) {
// We will only have one thread at this point, but just in case run notify everything
std::lock_guard lk(ThreadCreationMutex);
for (auto &Thread : Threads) {
Thread->StartRunning.NotifyAll();
}
}
ExecutionThread(ParentThread);
while(true) {
@@ -474,7 +430,6 @@ namespace FEXCore::Context {
};
LocalLoader->AddIR(IRHandler);
}
struct ExecutionThreadHandler {
@@ -502,7 +457,7 @@ namespace FEXCore::Context {
void Context::InitializeThreadTLSData(FEXCore::Core::InternalThreadState *Thread) {
// Let's do some initial bookkeeping here
Thread->ThreadManager.TID = ::gettid();
Thread->ThreadManager.TID = FHU::Syscalls::gettid();
Thread->ThreadManager.PID = ::getpid();
SignalDelegation->RegisterTLSState(Thread);
ThunkHandler->RegisterTLSState(Thread);
@@ -538,9 +493,11 @@ namespace FEXCore::Context {
// Create CPU backend
switch (Config.Core) {
#ifdef INTERPRETER_ENABLED
case FEXCore::Config::CONFIG_INTERPRETER:
State->CPUBackend = FEXCore::CPU::CreateInterpreterCore(this, State, CompileThread);
break;
#endif
case FEXCore::Config::CONFIG_IRJIT:
State->PassManager->InsertRegisterAllocationPass(DoSRA);
@@ -665,6 +622,58 @@ namespace FEXCore::Context {
}
}
static void IRDumper(FEXCore::Core::InternalThreadState *Thread, uint64_t GuestRIP, IR::RegisterAllocationData* RA) {
FILE* f = nullptr;
bool CloseAfter = false;
const auto DumpIRStr = Thread->CTX->Config.DumpIR();
if (DumpIRStr =="stderr") {
f = stderr;
}
else if (DumpIRStr =="stdout") {
f = stdout;
}
else {
const auto fileName = fmt::format("{}/{:x}{}", DumpIRStr, GuestRIP, RA ? "-post.ir" : "-pre.ir");
f = fopen(fileName.c_str(), "w");
CloseAfter = true;
}
if (f) {
std::stringstream out;
auto NewIR = Thread->OpDispatcher->ViewIR();
FEXCore::IR::Dump(&out, &NewIR, RA);
fmt::print(f,"IR-{} 0x{:x}:\n{}\n@@@@@\n", RA ? "post" : "pre", GuestRIP, out.str());
if (CloseAfter) {
fclose(f);
}
}
};
static void ValidateIR(FEXCore::Core::InternalThreadState *Thread) {
// Convert to text, Parse, Convert to text again and make sure the texts match
std::stringstream out;
static auto compaction = IR::CreateIRCompaction();
compaction->Run(Thread->OpDispatcher.get());
auto NewIR = Thread->OpDispatcher->ViewIR();
Dump(&out, &NewIR, nullptr);
out.seekg(0);
auto reparsed = IR::Parse(&out);
if (reparsed == nullptr) {
LOGMAN_MSG_A_FMT("Failed to parse IR\n");
} else {
std::stringstream out2;
auto NewIR2 = reparsed->ViewIR();
Dump(&out2, &NewIR2, nullptr);
if (out.str() != out2.str()) {
LogMan::Msg::IFmt("one:\n {}", out.str());
LogMan::Msg::IFmt("two:\n {}", out2.str());
LOGMAN_MSG_A_FMT("Parsed IR doesn't match\n");
}
}
}
Context::GenerateIRResult Context::GenerateIR(FEXCore::Core::InternalThreadState *Thread, uint64_t GuestRIP) {
uint8_t const *GuestCode{};
GuestCode = reinterpret_cast<uint8_t const*>(GuestRIP);
@@ -734,7 +743,7 @@ namespace FEXCore::Context {
else {
if (Thread->OpDispatcher->HandledLock != IsLocked) {
HadDispatchError = true;
LogMan::Msg::EFmt("Missing LOCK HANDLER at 0x{:x}{{'{}'}}", Block.Entry + BlockInstructionsLength, TableInfo->Name);
LogMan::Msg::EFmt("Missing LOCK HANDLER at 0x{:x}{{'{}'}}", Block.Entry + BlockInstructionsLength, TableInfo->Name ?: "UND");
}
BlockInstructionsLength += DecodedInfo->InstSize;
TotalInstructionsLength += DecodedInfo->InstSize;
@@ -771,73 +780,32 @@ namespace FEXCore::Context {
Thread->OpDispatcher->Finalize();
const auto IRDumper = [Thread, GuestRIP](IR::RegisterAllocationData* RA) {
FILE* f = nullptr;
bool CloseAfter = false;
const auto DumpIRStr = Thread->CTX->Config.DumpIR();
if (DumpIRStr =="stderr") {
f = stderr;
}
else if (DumpIRStr =="stdout") {
f = stdout;
}
else {
const auto fileName = fmt::format("{}/{:x}{}", DumpIRStr, GuestRIP, RA ? "-post.ir" : "-pre.ir");
f = fopen(fileName.c_str(), "w");
CloseAfter = true;
// Debug
{
if (Thread->CTX->Config.DumpIR() != "no") {
IRDumper(Thread, GuestRIP, nullptr);
}
if (f) {
std::stringstream out;
auto NewIR = Thread->OpDispatcher->ViewIR();
FEXCore::IR::Dump(&out, &NewIR, RA);
fmt::print(f,"IR-{} 0x{:x}:\n{}\n@@@@@\n", RA ? "post" : "pre", GuestRIP, out.str());
if (CloseAfter) {
fclose(f);
}
}
};
if (Thread->CTX->Config.DumpIR() != "no") {
IRDumper(nullptr);
}
if (Thread->CTX->Config.ValidateIRarser) {
// Convert to text, Parse, Convert to text again and make sure the texts match
std::stringstream out;
static auto compaction = IR::CreateIRCompaction();
compaction->Run(Thread->OpDispatcher.get());
auto NewIR = Thread->OpDispatcher->ViewIR();
Dump(&out, &NewIR, nullptr);
out.seekg(0);
auto reparsed = IR::Parse(&out);
if (reparsed == nullptr) {
LOGMAN_MSG_A_FMT("Failed to parse IR\n");
} else {
std::stringstream out2;
auto NewIR2 = reparsed->ViewIR();
Dump(&out2, &NewIR2, nullptr);
if (out.str() != out2.str()) {
LogMan::Msg::IFmt("one:\n {}", out.str());
LogMan::Msg::IFmt("two:\n {}", out2.str());
LOGMAN_MSG_A_FMT("Parsed IR doesn't match\n");
}
if (Thread->CTX->Config.ValidateIRarser) {
ValidateIR(Thread);
}
}
// Run the passmanager over the IR from the dispatcher
Thread->PassManager->Run(Thread->OpDispatcher.get());
if (Thread->CTX->Config.DumpIR() != "no") {
IRDumper(Thread->PassManager->HasPass("RA") ? Thread->PassManager->GetPass<IR::RegisterAllocationPass>("RA")->GetAllocationData() : nullptr);
}
// Debug
{
if (Thread->CTX->Config.DumpIR() != "no") {
IRDumper(Thread, GuestRIP, Thread->PassManager->HasPass("RA") ? Thread->PassManager->GetPass<IR::RegisterAllocationPass>("RA")->GetAllocationData() : nullptr);
}
if (Thread->OpDispatcher->ShouldDump) {
std::stringstream out;
auto NewIR = Thread->OpDispatcher->ViewIR();
FEXCore::IR::Dump(&out, &NewIR, Thread->PassManager->HasPass("RA") ? Thread->PassManager->GetPass<IR::RegisterAllocationPass>("RA")->GetAllocationData() : nullptr);
LogMan::Msg::IFmt("IR 0x{:x}:\n{}\n@@@@@\n", GuestRIP, out.str());
if (Thread->OpDispatcher->ShouldDump) {
std::stringstream out;
auto NewIR = Thread->OpDispatcher->ViewIR();
FEXCore::IR::Dump(&out, &NewIR, Thread->PassManager->HasPass("RA") ? Thread->PassManager->GetPass<IR::RegisterAllocationPass>("RA")->GetAllocationData() : nullptr);
LogMan::Msg::IFmt("IR 0x{:x}:\n{}\n@@@@@\n", GuestRIP, out.str());
}
}
auto RAData = Thread->PassManager->HasPass("RA") ? Thread->PassManager->GetPass<IR::RegisterAllocationPass>("RA")->PullAllocationData() : nullptr;
@@ -855,63 +823,6 @@ namespace FEXCore::Context {
};
}
AOTIRInlineEntry *AOTIRInlineIndex::GetInlineEntry(uint64_t DataOffset) {
uintptr_t This = (uintptr_t)this;
return (AOTIRInlineEntry*)(This + DataBase + DataOffset);
}
AOTIRInlineEntry *AOTIRInlineIndex::Find(uint64_t GuestStart) {
ssize_t l = 0;
ssize_t r = Count - 1;
while (l <= r) {
size_t m = l + (r - l) / 2;
if (Entries[m].GuestStart == GuestStart)
return GetInlineEntry(Entries[m].DataOffset);
else if (Entries[m].GuestStart < GuestStart)
l = m + 1;
else
r = m - 1;
}
return nullptr;
}
IR::RegisterAllocationData *AOTIRInlineEntry::GetRAData() {
return (IR::RegisterAllocationData *)InlineData;
}
IR::IRListView *AOTIRInlineEntry::GetIRData() {
auto RAData = GetRAData();
auto Offset = RAData->Size(RAData->MapCount);
return (IR::IRListView *)&InlineData[Offset];
}
void AOTIRCaptureCacheEntry::AppendAOTIRCaptureCache(uint64_t GuestRIP, uint64_t Start, uint64_t Length, uint64_t Hash, FEXCore::IR::IRListView *IRList, FEXCore::IR::RegisterAllocationData *RAData) {
auto Inserted = Index.emplace(GuestRIP, Stream->tellp());
if (Inserted.second) {
//GuestHash
Stream->write((const char*)&Hash, sizeof(Hash));
//GuestLength
Stream->write((const char*)&Length, sizeof(Length));
// RAData (inline)
// In file, IsShared is always set
auto Shared = RAData->IsShared;
RAData->IsShared = true;
Stream->write((const char*)RAData, RAData->Size(RAData->MapCount));
RAData->IsShared = Shared;
// IRData (inline)
IRList->Serialize(*Stream);
}
}
Context::CompileCodeResult Context::CompileCode(FEXCore::Core::InternalThreadState *Thread, uint64_t GuestRIP) {
FEXCore::IR::IRListView *IRList {};
FEXCore::Core::DebugData *DebugData {};
@@ -935,55 +846,17 @@ namespace FEXCore::Context {
GeneratedIR = false;
}
// AOT IR bookkeeping and cache
{
std::shared_lock lk(AOTIRCacheLock);
auto file = AddrToFile.lower_bound(GuestRIP);
if (file != AddrToFile.begin()) {
--file;
if (!file->second.ContainsCode) {
file->second.ContainsCode = true;
FilesWithCode[file->second.fileid] = file->second.filename;
}
}
}
if (IRList == nullptr && Config.AOTIRLoad) {
std::shared_lock lk(AOTIRCacheLock);
auto file = AddrToFile.lower_bound(GuestRIP);
if (file != AddrToFile.begin()) {
--file;
auto Mod = (AOTIRInlineIndex*)file->second.CachedFileEntry;
if (Mod == nullptr) {
file->second.CachedFileEntry = Mod = AOTIRCache[file->second.fileid].Array;
}
if (Mod != nullptr)
{
auto AOTEntry = Mod->Find(GuestRIP - file->second.Start + file->second.Offset);
if (AOTEntry) {
// verify hash
auto MappedStart = GuestRIP;
auto hash = XXH3_64bits((void*)MappedStart, AOTEntry->GuestLength);
if (hash == AOTEntry->GuestHash) {
IRList = AOTEntry->GetIRData();
//LogMan::Msg::DFmt("using {} + {:x} -> {:x}\n", file->second.fileid, AOTEntry->first, GuestRIP);
RAData = AOTEntry->GetRAData();;
DebugData = new FEXCore::Core::DebugData();
StartAddr = MappedStart;
Length = AOTEntry->GuestLength;
GeneratedIR = true;
} else {
LogMan::Msg::IFmt("AOTIR: hash check failed {:x}\n", MappedStart);
}
} else {
//LogMan::Msg::IFmt("AOTIR: Failed to find {:x}, {:x}, {}\n", GuestRIP, GuestRIP - file->second.Start + file->second.Offset, file->second.fileid);
}
}
auto [IRCopy, RACopy, DebugDataCopy, _StartAddr, _Length, _GeneratedIR] = IRCaptureCache.PreGenerateIRFetch(GuestRIP, IRList);
if (_GeneratedIR) {
// Setup pointers to internal structures
IRList = IRCopy;
RAData = RACopy;
DebugData = DebugDataCopy;
StartAddr = _StartAddr;
Length = _Length;
GeneratedIR = _GeneratedIR;
}
}
@@ -1024,114 +897,6 @@ namespace FEXCore::Context {
};
}
static bool readAll(int fd, void *data, size_t size) {
int rv = read(fd, data, size);
if (rv != size)
return false;
else
return true;
}
bool Context::LoadAOTIRCache(int streamfd) {
uint64_t tag;
if (!readAll(streamfd, (char*)&tag, sizeof(tag)) || tag != 0xDEADBEEFC0D30004)
return false;
std::string Module;
uint64_t ModSize;
uint64_t IndexSize;
lseek(streamfd, -sizeof(ModSize), SEEK_END);
if (!readAll(streamfd, (char*)&ModSize, sizeof(ModSize)))
return false;
Module.resize(ModSize);
lseek(streamfd, -sizeof(ModSize) - ModSize, SEEK_END);
if (!readAll(streamfd, (char*)&Module[0], Module.size()))
return false;
lseek(streamfd, -sizeof(ModSize) - ModSize - sizeof(IndexSize), SEEK_END);
if (!readAll(streamfd, (char*)&IndexSize, sizeof(IndexSize)))
return false;
struct stat fileinfo;
if (fstat(streamfd, &fileinfo) < 0)
return false;
size_t Size = (fileinfo.st_size + 4095) & ~4095;
size_t IndexOffset = fileinfo.st_size - IndexSize -sizeof(ModSize) - ModSize - sizeof(IndexSize);
void *FilePtr = FEXCore::Allocator::mmap(nullptr, Size, PROT_READ, MAP_SHARED, streamfd, 0);
if (FilePtr == MAP_FAILED)
return false;
auto Array = (AOTIRInlineIndex *)((char*)FilePtr + IndexOffset);
AOTIRCache.insert({Module, {Array, FilePtr, Size}});
LogMan::Msg::DFmt("AOTIR: Module {} has {} functions", Module, Array->Count);
return true;
}
void Context::WriteFilesWithCode(std::function<void(const std::string& fileid, const std::string& filename)> Writer) {
std::shared_lock lk(AOTIRCacheLock);
for( const auto &File: FilesWithCode) {
Writer(File.first, File.second);
}
}
void Context::FinalizeAOTIRCache() {
AOTIRCaptureCacheWriteoutQueue_Flush();
std::unique_lock lk(AOTIRCacheLock);
for (auto& [String, Entry] : AOTIRCaptureCache) {
if (!Entry.Stream) {
continue;
}
const auto ModSize = String.size();
auto &stream = Entry.Stream;
// pad to 32 bytes
constexpr char Zero = 0;
while(stream->tellp() & 31)
stream->write(&Zero, 1);
// AOTIRInlineIndex
const auto FnCount = Entry.Index.size();
const size_t DataBase = -stream->tellp();
stream->write((const char*)&FnCount, sizeof(FnCount));
stream->write((const char*)&DataBase, sizeof(DataBase));
for (const auto& [GuestStart, DataOffset] : Entry.Index) {
//AOTIRInlineIndexEntry
// GuestStart
stream->write((const char*)&GuestStart, sizeof(GuestStart));
// DataOffset
stream->write((const char*)&DataOffset, sizeof(DataOffset));
}
// End of file header
const auto IndexSize = FnCount * sizeof(AOTIRInlineIndexEntry) + sizeof(DataBase) + sizeof(FnCount);
stream->write((const char*)&IndexSize, sizeof(IndexSize));
stream->write(String.c_str(), ModSize);
stream->write((const char*)&ModSize, sizeof(ModSize));
}
}
void Context::CompileBlockJit(FEXCore::Core::CpuStateFrame *Frame, uint64_t GuestRIP) {
auto NewBlock = CompileBlock(Frame, GuestRIP);
@@ -1143,19 +908,6 @@ namespace FEXCore::Context {
}
}
Context::AddrToFileMapType::iterator Context::FindAddrForFile(uint64_t Entry, uint64_t Length) {
// Thread safety here! We are returning an iterator to the map object
// This needs the AOTIRCacheLock locked prior to coming in to the function
auto file = AddrToFile.lower_bound(Entry);
if (file != AddrToFile.begin()) {
--file;
if (file->second.Start <= Entry && (file->second.Start + file->second.Len) >= (Entry + Length)) {
return file;
}
}
return AddrToFile.end();
}
uintptr_t Context::CompileBlock(FEXCore::Core::CpuStateFrame *Frame, uint64_t GuestRIP) {
auto Thread = Frame->Thread;
@@ -1227,56 +979,19 @@ namespace FEXCore::Context {
}
}
// Both generated ir and LibraryJITName need a named region lookup
if (GeneratedIR || Config.LibraryJITNaming()) {
std::shared_lock lk(AOTIRCacheLock);
auto file = FindAddrForFile(StartAddr, Length);
// Only go down this path if we actually found a library region
if (file != AddrToFile.end()) {
if (DebugData && Config.LibraryJITNaming()) {
Symbols.RegisterNamedRegion(CodePtr, DebugData->HostCodeSize, file->second.filename);
}
// Add to AOT cache if aot generation is enabled
if (GeneratedIR && RAData &&
(Config.AOTIRCapture() || Config.AOTIRGenerate())) {
auto hash = XXH3_64bits((void*)StartAddr, Length);
auto LocalRIP = GuestRIP - file->second.Start + file->second.Offset;
auto LocalStartAddr = StartAddr - file->second.Start + file->second.Offset;
auto fileid = file->second.fileid;
AOTIRCaptureCacheWriteoutQueue_Append([this, LocalRIP, LocalStartAddr, Length, hash, IRList, RAData, fileid]() {
auto *AotFile = &AOTIRCaptureCache[fileid];
if (!AotFile->Stream) {
AotFile->Stream = AOTIRWriter(fileid);
uint64_t tag = 0xDEADBEEFC0D30004;
AotFile->Stream->write((char*)&tag, sizeof(tag));
}
AotFile->AppendAOTIRCaptureCache(LocalRIP, LocalStartAddr, Length, hash, IRList, RAData);
});
if (Config.AOTIRGenerate()) {
// cleanup memory and early exit here -- we're not running the application
if (DecrementRefCount)
--Thread->CompileBlockReentrantRefCount;
Thread->CPUBackend->ClearCache();
return (uintptr_t)CodePtr;
}
}
}
// Insert to caches if we generated IR
if (GeneratedIR) {
// Add to thread local ir cache
Core::LocalIREntry Entry = {StartAddr, Length, decltype(Entry.IR)(IRList), decltype(Entry.RAData)(RAData), decltype(Entry.DebugData)(DebugData)};
Thread->LocalIRCache.insert({GuestRIP, std::move(Entry)});
}
if (IRCaptureCache.PostCompileCode(
Thread,
CodePtr,
GuestRIP,
StartAddr,
Length,
RAData,
IRList,
DebugData,
GeneratedIR,
DecrementRefCount)) {
// Early exit
return (uintptr_t)CodePtr;
}
if (DecrementRefCount)
@@ -1406,38 +1121,11 @@ namespace FEXCore::Context {
}
void Context::AddNamedRegion(uintptr_t Base, uintptr_t Size, uintptr_t Offset, const std::string &filename) {
// TODO: Support overlapping maps and region splitting
auto base_filename = std::filesystem::path(filename).filename().string();
if (!base_filename.empty()) {
auto filename_hash = XXH3_64bits(filename.c_str(), filename.size());
auto fileid = base_filename + "-" + std::to_string(filename_hash) + "-";
// append optimization flags to the fileid
fileid += (Config.SMCChecks == FEXCore::Config::CONFIG_SMC_FULL) ? "S" : "s";
fileid += Config.TSOEnabled ? "T" : "t";
fileid += Config.ABILocalFlags ? "L" : "l";
fileid += Config.ABINoPF ? "p" : "P";
std::unique_lock lk(AOTIRCacheLock);
AddrToFile.insert({ Base, { Base, Size, Offset, fileid, filename, nullptr, false} });
if (Config.AOTIRLoad && !AOTIRCache.contains(fileid) && AOTIRLoader) {
auto streamfd = AOTIRLoader(fileid);
if (streamfd != -1) {
LoadAOTIRCache(streamfd);
close(streamfd);
}
}
}
IRCaptureCache.AddNamedRegion(Base, Size, Offset, filename);
}
void Context::RemoveNamedRegion(uintptr_t Base, uintptr_t Size) {
std::unique_lock lk(AOTIRCacheLock);
// TODO: Support partial removing
AddrToFile.erase(Base);
IRCaptureCache.RemoveNamedRegion(Base, Size);
}
void ConfigureAOTGen(FEXCore::Core::InternalThreadState *Thread, std::set<uint64_t> *ExternalBranches, uint64_t SectionMaxAddress) {
@@ -11,6 +11,8 @@
#include <FEXCore/Core/CoreState.h>
#include <FEXCore/Core/X86Enums.h>
#include <FEXCore/Debug/InternalThreadState.h>
#include <FEXHeaderUtils/Syscalls.h>
#include <array>
#include <bit>
#include <cmath>
@@ -37,7 +39,7 @@ static constexpr size_t MAX_DISPATCHER_CODE_SIZE = 4096;
#define STATE x28
Arm64Dispatcher::Arm64Dispatcher(FEXCore::Context::Context *ctx, FEXCore::Core::InternalThreadState *Thread, DispatcherConfig &config)
: Dispatcher(ctx, Thread), Arm64Emitter(MAX_DISPATCHER_CODE_SIZE) {
: Dispatcher(ctx, Thread), Arm64Emitter(ctx, MAX_DISPATCHER_CODE_SIZE) {
SRAEnabled = config.StaticRegisterAssignment;
SetAllowAssembler(true);
@@ -51,10 +53,7 @@ Arm64Dispatcher::Arm64Dispatcher(FEXCore::Context::Context *ctx, FEXCore::Core::
// Ptr();
// }
uint64_t VirtualMemorySize = Thread->LookupCache->GetVirtualMemorySize();
Literal l_VirtualMemory {VirtualMemorySize};
Literal l_PagePtr {Thread->LookupCache->GetPagePointer()};
Literal l_L1Ptr {Thread->LookupCache->GetL1Pointer()};
Literal l_CTX {reinterpret_cast<uintptr_t>(CTX)};
Literal l_Sleep {reinterpret_cast<uint64_t>(SleepThread)};
Literal l_CompileBlock {GetCompileBlockPtr()};
@@ -97,7 +96,7 @@ Arm64Dispatcher::Arm64Dispatcher(FEXCore::Context::Context *ctx, FEXCore::Core::
auto RipReg = x2;
// L1 Cache
ldr(x0, &l_L1Ptr);
ldr(x0, MemOperand(STATE, offsetof(FEXCore::Core::CpuStateFrame, Pointers.AArch64.L1Pointer)));
and_(x3, RipReg, LookupCache::L1_ENTRIES_MASK);
add(x0, x0, Operand(x3, Shift::LSL, 4));
@@ -119,11 +118,12 @@ Arm64Dispatcher::Arm64Dispatcher(FEXCore::Context::Context *ctx, FEXCore::Core::
ldr(x0, &l_PagePtr);
// Mask the address by the virtual address size so we can check for aliases
uint64_t VirtualMemorySize = Thread->LookupCache->GetVirtualMemorySize();
if (std::popcount(VirtualMemorySize) == 1) {
and_(x3, RipReg, Thread->LookupCache->GetVirtualMemorySize() - 1);
and_(x3, RipReg, VirtualMemorySize - 1);
}
else {
ldr(x3, &l_VirtualMemory);
LoadConstant(x3, VirtualMemorySize);
and_(x3, RipReg, x3);
}
@@ -157,7 +157,7 @@ Arm64Dispatcher::Arm64Dispatcher(FEXCore::Context::Context *ctx, FEXCore::Core::
// If we've made it here then we have a real compiled block
{
// update L1 cache
ldr(x0, &l_L1Ptr);
ldr(x0, MemOperand(STATE, offsetof(FEXCore::Core::CpuStateFrame, Pointers.AArch64.L1Pointer)));
and_(x1, RipReg, LookupCache::L1_ENTRIES_MASK);
add(x0, x0, Operand(x1, Shift::LSL, 4));
@@ -337,6 +337,31 @@ Arm64Dispatcher::Arm64Dispatcher(FEXCore::Context::Context *ctx, FEXCore::Core::
hlt(0);
}
{
// Guest Overflow handler
// Needs to be distinct from the SignalHandlerReturnAddress
OverflowExceptionInstructionAddress = GetCursorAddress<uint64_t>();
if (SRAEnabled)
SpillStaticRegs();
LoadConstant(x0, reinterpret_cast<uint64_t>(&SynchronousFaultData));
LoadConstant(w1, 1);
strb(w1, MemOperand(x0, offsetof(Dispatcher::SynchronousFaultDataStruct, FaultToTopAndGeneratedException)));
LoadConstant(w1, X86State::X86_TRAPNO_OF);
str(w1, MemOperand(x0, offsetof(Dispatcher::SynchronousFaultDataStruct, TrapNo)));
LoadConstant(w1, 0x80);
str(w1, MemOperand(x0, offsetof(Dispatcher::SynchronousFaultDataStruct, si_code)));
LoadConstant(x1, 0);
str(w1, MemOperand(x0, offsetof(Dispatcher::SynchronousFaultDataStruct, err_code)));
// hlt/udf = SIGILL
// brk = SIGTRAP
// ??? = SIGSEGV
// Force a SIGSEGV by loading zero
ldr(x1, MemOperand(x1));
}
{
ThreadPauseHandlerAddressSpillSRA = GetCursorAddress<uint64_t>();
if (SRAEnabled)
@@ -411,9 +436,7 @@ Arm64Dispatcher::Arm64Dispatcher(FEXCore::Context::Context *ctx, FEXCore::Core::
b(&LoopTop);
}
place(&l_VirtualMemory);
place(&l_PagePtr);
place(&l_L1Ptr);
place(&l_CTX);
place(&l_Sleep);
place(&l_CompileBlock);
@@ -428,12 +451,26 @@ Arm64Dispatcher::Arm64Dispatcher(FEXCore::Context::Context *ctx, FEXCore::Core::
GetBuffer()->SetExecutable();
if (CTX->Config.BlockJITNaming()) {
std::string Name = "Dispatch_" + std::to_string(::gettid());
std::string Name = "Dispatch_" + std::to_string(FHU::Syscalls::gettid());
CTX->Symbols.Register(reinterpret_cast<void*>(DispatchPtr), End - reinterpret_cast<uint64_t>(DispatchPtr), Name);
}
if (CTX->Config.GlobalJITNaming()) {
CTX->Symbols.RegisterJITSpace(reinterpret_cast<void*>(DispatchPtr), End - reinterpret_cast<uint64_t>(DispatchPtr));
}
// Setup dispatcher specific pointers that need to be accessed from JIT code
{
auto &Pointers = ThreadState->CurrentFrame->Pointers.AArch64;
Pointers.DispatcherLoopTop = AbsoluteLoopTopAddress;
Pointers.DispatcherLoopTopFillSRA = AbsoluteLoopTopAddressFillSRA;
Pointers.ThreadStopHandlerSpillSRA = ThreadStopHandlerAddressSpillSRA;
Pointers.ThreadPauseHandlerSpillSRA = ThreadPauseHandlerAddressSpillSRA;
Pointers.UnimplementedInstructionHandler = UnimplementedInstructionAddress;
Pointers.OverflowExceptionHandler = OverflowExceptionInstructionAddress;
Pointers.SignalReturnHandler = SignalHandlerReturnAddress;
Pointers.L1Pointer = Thread->LookupCache->GetL1Pointer();
}
}
void Arm64Dispatcher::SpillSRA(void *ucontext, uint32_t IgnoreMask) {
@@ -16,9 +16,9 @@
#include <atomic>
#include <condition_variable>
#include <bits/types/siginfo_t.h>
#include <csignal>
#include <cstring>
#include <signal.h>
namespace FEXCore::CPU {
@@ -81,6 +81,10 @@ ArchHelpers::Context::ContextBackup* Dispatcher::StoreThreadState(int Signal, vo
Context->UContextLocation = 0;
Context->SigInfoLocation = 0;
// Store fault to top status and then reset it
Context->FaultToTopAndGeneratedException = SynchronousFaultData.FaultToTopAndGeneratedException;
SynchronousFaultData.FaultToTopAndGeneratedException = false;
return Context;
}
@@ -118,7 +122,8 @@ void Dispatcher::RestoreThreadState(void *ucontext) {
[[maybe_unused]] auto *guest_siginfo = reinterpret_cast<siginfo_t*>(Context->SigInfoLocation);
// If the guest modified the RIP then we need to take special precautions here
if (Context->OriginalRIP != guest_uctx->uc_mcontext.gregs[FEXCore::x86_64::FEX_REG_RIP]) {
if (Context->OriginalRIP != guest_uctx->uc_mcontext.gregs[FEXCore::x86_64::FEX_REG_RIP] ||
Context->FaultToTopAndGeneratedException) {
// Hack! Go back to the top of the dispatcher top
// This is only safe inside the JIT rather than anything outside of it
ArchHelpers::Context::SetPc(ucontext, AbsoluteLoopTopAddressFillSRA);
@@ -127,6 +132,14 @@ void Dispatcher::RestoreThreadState(void *ucontext) {
Frame->State.rip = guest_uctx->uc_mcontext.gregs[FEXCore::x86_64::FEX_REG_RIP];
// XXX: Full context setting
uint32_t eflags = guest_uctx->uc_mcontext.gregs[FEXCore::x86_64::FEX_REG_EFL];
for (size_t i = 0; i < 32; ++i) {
Frame->State.flags[i] = (eflags & (1U << i)) ? 1 : 0;
}
Frame->State.flags[1] = 1;
Frame->State.flags[9] = 1;
#define COPY_REG(x) \
Frame->State.gregs[X86State::REG_##x] = guest_uctx->uc_mcontext.gregs[FEXCore::x86_64::FEX_REG_##x];
COPY_REG(R8);
@@ -146,13 +159,29 @@ void Dispatcher::RestoreThreadState(void *ucontext) {
COPY_REG(RCX);
COPY_REG(RSP);
#undef COPY_REG
FEXCore::x86_64::_libc_fpstate *fpstate = reinterpret_cast<FEXCore::x86_64::_libc_fpstate*>(guest_uctx->uc_mcontext.fpregs);
// Copy float registers
memcpy(Frame->State.mm, fpstate->_st, sizeof(Frame->State.mm));
memcpy(Frame->State.xmm, fpstate->_xmm, sizeof(Frame->State.xmm));
// FCW store default
Frame->State.FCW = fpstate->fcw;
Frame->State.FTW = fpstate->ftw;
// Deconstruct FSW
Frame->State.flags[FEXCore::X86State::X87FLAG_C0_LOC] = (fpstate->fsw >> 8) & 1;
Frame->State.flags[FEXCore::X86State::X87FLAG_C1_LOC] = (fpstate->fsw >> 9) & 1;
Frame->State.flags[FEXCore::X86State::X87FLAG_C2_LOC] = (fpstate->fsw >> 10) & 1;
Frame->State.flags[FEXCore::X86State::X87FLAG_C3_LOC] = (fpstate->fsw >> 14) & 1;
Frame->State.flags[FEXCore::X86State::X87FLAG_TOP_LOC] = (fpstate->fsw >> 11) & 0b111;
}
}
else {
auto *guest_uctx = reinterpret_cast<FEXCore::x86::ucontext_t*>(Context->UContextLocation);
[[maybe_unused]] auto *guest_siginfo = reinterpret_cast<FEXCore::x86::siginfo_t*>(Context->SigInfoLocation);
// If the guest modified the RIP then we need to take special precautions here
if (Context->OriginalRIP != guest_uctx->uc_mcontext.gregs[FEXCore::x86::FEX_REG_EIP]) {
if (Context->OriginalRIP != guest_uctx->uc_mcontext.gregs[FEXCore::x86::FEX_REG_EIP] ||
Context->FaultToTopAndGeneratedException) {
// Hack! Go back to the top of the dispatcher top
// This is only safe inside the JIT rather than anything outside of it
ArchHelpers::Context::SetPc(ucontext, AbsoluteLoopTopAddressFillSRA);
@@ -160,17 +189,54 @@ void Dispatcher::RestoreThreadState(void *ucontext) {
ArchHelpers::Context::SetState(ucontext, reinterpret_cast<uint64_t>(Frame));
// XXX: Full context setting
// First 32-bytes of flags is EFLAGS broken out
uint32_t eflags = guest_uctx->uc_mcontext.gregs[FEXCore::x86::FEX_REG_EFL];
for (size_t i = 0; i < 32; ++i) {
Frame->State.flags[i] = (eflags & (1U << i)) ? 1 : 0;
}
Frame->State.flags[1] = 1;
Frame->State.flags[9] = 1;
Frame->State.rip = guest_uctx->uc_mcontext.gregs[FEXCore::x86::FEX_REG_EIP];
Frame->State.cs = guest_uctx->uc_mcontext.gregs[FEXCore::x86::FEX_REG_CS];
Frame->State.ds = guest_uctx->uc_mcontext.gregs[FEXCore::x86::FEX_REG_DS];
Frame->State.es = guest_uctx->uc_mcontext.gregs[FEXCore::x86::FEX_REG_ES];
Frame->State.fs = guest_uctx->uc_mcontext.gregs[FEXCore::x86::FEX_REG_FS];
Frame->State.gs = guest_uctx->uc_mcontext.gregs[FEXCore::x86::FEX_REG_GS];
Frame->State.ss = guest_uctx->uc_mcontext.gregs[FEXCore::x86::FEX_REG_SS];
#define COPY_REG(x) \
Frame->State.gregs[X86State::REG_##x] = guest_uctx->uc_mcontext.gregs[FEXCore::x86::FEX_REG_##x];
COPY_REG(RDI);
COPY_REG(RSI);
COPY_REG(RBP);
COPY_REG(RBX);
COPY_REG(RDX);
COPY_REG(RAX);
COPY_REG(RCX);
COPY_REG(RSP);
COPY_REG(RDI);
COPY_REG(RSI);
COPY_REG(RBP);
COPY_REG(RBX);
COPY_REG(RDX);
COPY_REG(RAX);
COPY_REG(RCX);
COPY_REG(RSP);
#undef COPY_REG
FEXCore::x86::_libc_fpstate *fpstate = reinterpret_cast<FEXCore::x86::_libc_fpstate*>(guest_uctx->uc_mcontext.fpregs);
// Copy float registers
for (size_t i = 0; i < 8; ++i) {
// 32-bit st register size is only 10 bytes. Not padded to 16byte like x86-64
memcpy(&Frame->State.mm[i], &fpstate->_st[i], 10);
}
// Extended XMM state
memcpy(fpstate->_xmm, Frame->State.xmm, sizeof(Frame->State.xmm));
// FCW store default
Frame->State.FCW = fpstate->fcw;
Frame->State.FTW = fpstate->ftw;
// Deconstruct FSW
Frame->State.flags[FEXCore::X86State::X87FLAG_C0_LOC] = (fpstate->fsw >> 8) & 1;
Frame->State.flags[FEXCore::X86State::X87FLAG_C1_LOC] = (fpstate->fsw >> 9) & 1;
Frame->State.flags[FEXCore::X86State::X87FLAG_C2_LOC] = (fpstate->fsw >> 10) & 1;
Frame->State.flags[FEXCore::X86State::X87FLAG_C3_LOC] = (fpstate->fsw >> 14) & 1;
Frame->State.flags[FEXCore::X86State::X87FLAG_TOP_LOC] = (fpstate->fsw >> 11) & 0b111;
}
}
}
@@ -267,7 +333,10 @@ bool Dispatcher::HandleGuestSignal(int Signal, void *info, void *ucontext, Guest
// This can also happen if we have put a signal on hold, then we just reenabled the signal
// So we are in the syscall handler
// Only throw a log message in this case
LogMan::Msg::EFmt("Signals in dispatcher have unsynchronized context");
if constexpr (false) {
// XXX: Messages in the signal handler can cause us to crash
LogMan::Msg::EFmt("Signals in dispatcher have unsynchronized context");
}
}
}
}
@@ -334,8 +403,23 @@ bool Dispatcher::HandleGuestSignal(int Signal, void *info, void *ucontext, Guest
guest_uctx->uc_mcontext.gregs[FEXCore::x86_64::FEX_REG_RIP] = Frame->State.rip;
guest_uctx->uc_mcontext.gregs[FEXCore::x86_64::FEX_REG_EFL] = 0;
guest_uctx->uc_mcontext.gregs[FEXCore::x86_64::FEX_REG_CSGSFS] = 0;
guest_uctx->uc_mcontext.gregs[FEXCore::x86_64::FEX_REG_ERR] = ConvertSignalToError(Signal, HostSigInfo);
guest_uctx->uc_mcontext.gregs[FEXCore::x86_64::FEX_REG_TRAPNO] = ConvertSignalToTrapNo(Signal, HostSigInfo);
// aarch64 and x86_64 siginfo_t matches. We can just copy this over
// SI_USER could also potentially have random data in it, needs to be bit perfect
// For guest faults we don't have a real way to reconstruct state to a real guest RIP
*guest_siginfo = *HostSigInfo;
if (ContextBackup->FaultToTopAndGeneratedException) {
guest_uctx->uc_mcontext.gregs[FEXCore::x86_64::FEX_REG_TRAPNO] = SynchronousFaultData.TrapNo;
guest_uctx->uc_mcontext.gregs[FEXCore::x86_64::FEX_REG_ERR] = SynchronousFaultData.err_code;
// Overwrite si_code
guest_siginfo->si_code = SynchronousFaultData.si_code;
}
else {
guest_uctx->uc_mcontext.gregs[FEXCore::x86_64::FEX_REG_TRAPNO] = ConvertSignalToTrapNo(Signal, HostSigInfo);
guest_uctx->uc_mcontext.gregs[FEXCore::x86_64::FEX_REG_ERR] = ConvertSignalToError(Signal, HostSigInfo);
}
guest_uctx->uc_mcontext.gregs[FEXCore::x86_64::FEX_REG_OLDMASK] = 0;
guest_uctx->uc_mcontext.gregs[FEXCore::x86_64::FEX_REG_CR2] = 0;
@@ -380,11 +464,6 @@ bool Dispatcher::HandleGuestSignal(int Signal, void *info, void *ucontext, Guest
guest_uctx->uc_stack.ss_sp = GuestStack->ss_sp;
guest_uctx->uc_stack.ss_size = GuestStack->ss_size;
// aarch64 and x86_64 siginfo_t matches. We can just copy this over
// SI_USER could also potentially have random data in it, needs to be bit perfect
// For guest faults we don't have a real way to reconstruct state to a real guest RIP
*guest_siginfo = *HostSigInfo;
Frame->State.gregs[X86State::REG_RSI] = SigInfoLocation;
Frame->State.gregs[X86State::REG_RDX] = UContextLocation;
}
@@ -421,8 +500,16 @@ bool Dispatcher::HandleGuestSignal(int Signal, void *info, void *ucontext, Guest
guest_uctx->uc_mcontext.gregs[FEXCore::x86::FEX_REG_FS] = Frame->State.fs;
guest_uctx->uc_mcontext.gregs[FEXCore::x86::FEX_REG_ES] = Frame->State.es;
guest_uctx->uc_mcontext.gregs[FEXCore::x86::FEX_REG_DS] = Frame->State.ds;
guest_uctx->uc_mcontext.gregs[FEXCore::x86::FEX_REG_TRAPNO] = ConvertSignalToTrapNo(Signal, HostSigInfo);
if (ContextBackup->FaultToTopAndGeneratedException) {
guest_uctx->uc_mcontext.gregs[FEXCore::x86::FEX_REG_TRAPNO] = SynchronousFaultData.TrapNo;
guest_siginfo->si_code = SynchronousFaultData.si_code;
guest_uctx->uc_mcontext.gregs[FEXCore::x86::FEX_REG_ERR] = SynchronousFaultData.err_code;
}
else {
guest_uctx->uc_mcontext.gregs[FEXCore::x86::FEX_REG_TRAPNO] = ConvertSignalToTrapNo(Signal, HostSigInfo);
guest_siginfo->si_code = HostSigInfo->si_code;
guest_uctx->uc_mcontext.gregs[FEXCore::x86::FEX_REG_ERR] = ConvertSignalToError(Signal, HostSigInfo);
}
guest_uctx->uc_mcontext.gregs[FEXCore::x86::FEX_REG_EIP] = Frame->State.rip;
guest_uctx->uc_mcontext.gregs[FEXCore::x86::FEX_REG_CS] = Frame->State.cs;
guest_uctx->uc_mcontext.gregs[FEXCore::x86::FEX_REG_EFL] = 0;
@@ -470,7 +557,6 @@ bool Dispatcher::HandleGuestSignal(int Signal, void *info, void *ucontext, Guest
// These three elements are in every siginfo
guest_siginfo->si_signo = HostSigInfo->si_signo;
guest_siginfo->si_errno = HostSigInfo->si_errno;
guest_siginfo->si_code = HostSigInfo->si_code;
switch (Signal) {
case SIGSEGV:
@@ -5,8 +5,8 @@
#include "Interface/Context/Context.h"
#include "Interface/Core/ArchHelpers/MContext.h"
#include <bits/types/stack_t.h>
#include <cstdint>
#include <signal.h>
#include <stddef.h>
#include <stack>
#include <tuple>
@@ -49,12 +49,19 @@ public:
uint64_t ExitFunctionLinkerAddress{};
uint64_t SignalHandlerReturnAddress{};
uint64_t UnimplementedInstructionAddress{};
uint64_t OverflowExceptionInstructionAddress{};
uint64_t PauseReturnInstruction{};
/** @} */
uint32_t SignalHandlerRefCounter{};
struct SynchronousFaultDataStruct {
bool FaultToTopAndGeneratedException{};
uint32_t TrapNo;
uint32_t err_code;
uint32_t si_code;
} SynchronousFaultData;
uint64_t Start{};
uint64_t End{};
@@ -11,6 +11,7 @@
#include <FEXCore/Core/CPUBackend.h>
#include <FEXCore/Debug/InternalThreadState.h>
#include <FEXCore/Utils/Allocator.h>
#include <FEXHeaderUtils/Syscalls.h>
#include <cmath>
#include <memory>
@@ -94,7 +95,7 @@ X86Dispatcher::X86Dispatcher(FEXCore::Context::Context *ctx, FEXCore::Core::Inte
mov(rdx, qword [STATE + offsetof(FEXCore::Core::CPUState, rip)]);
// L1 Cache
mov(r13, Thread->LookupCache->GetL1Pointer());
mov(r13, qword [STATE + offsetof(FEXCore::Core::CpuStateFrame, Pointers.X86.L1Pointer)]);
mov(rax, rdx);
and_(rax, LookupCache::L1_ENTRIES_MASK);
@@ -113,8 +114,9 @@ X86Dispatcher::X86Dispatcher(FEXCore::Context::Context *ctx, FEXCore::Core::Inte
mov(r13, Thread->LookupCache->GetPagePointer());
// Full lookup
uint64_t VirtualMemorySize = Thread->LookupCache->GetVirtualMemorySize();
mov(rax, rdx);
mov(rbx, Thread->LookupCache->GetVirtualMemorySize() - 1);
mov(rbx, VirtualMemorySize - 1);
and_(rax, rbx);
shr(rax, 12);
@@ -141,8 +143,7 @@ X86Dispatcher::X86Dispatcher(FEXCore::Context::Context *ctx, FEXCore::Core::Inte
je(NoBlock);
// Update L1
mov(r13, Thread->LookupCache->GetL1Pointer());
mov(r13, qword [STATE + offsetof(FEXCore::Core::CpuStateFrame, Pointers.X86.L1Pointer)]);
mov(rcx, rdx);
and_(rcx, LookupCache::L1_ENTRIES_MASK);
shl(rcx, 1);
@@ -251,8 +252,7 @@ X86Dispatcher::X86Dispatcher(FEXCore::Context::Context *ctx, FEXCore::Core::Inte
// XXX: XMM?
// Make sure to adjust the refcounter so we don't clear the cache now
mov(rax, reinterpret_cast<uint64_t>(&SignalHandlerRefCounter));
add(dword [rax], 1);
add(qword [STATE + offsetof(FEXCore::Core::CpuStateFrame, Pointers.X86.SignalHandlerRefCountPointer)], 1);
// Now push the callback return trampoline to the guest stack
// Guest will be misaligned because calling a thunk won't correct the guest's stack once we call the callback from the host
@@ -284,6 +284,24 @@ X86Dispatcher::X86Dispatcher(FEXCore::Context::Context *ctx, FEXCore::Core::Inte
ud2();
}
{
// Guest Overflow handler
// Needs to be distinct from the SignalHandlerReturnAddress
OverflowExceptionInstructionAddress = getCurr<uint64_t>();
// ud2 = SIGILL
// int3 = SIGTRAP
// hlt = SIGSEGV
mov(rax, reinterpret_cast<uint64_t>(&SynchronousFaultData));
add(byte [rax + offsetof(Dispatcher::SynchronousFaultDataStruct, FaultToTopAndGeneratedException)], 1);
mov(dword [rax + offsetof(Dispatcher::SynchronousFaultDataStruct, TrapNo)], X86State::X86_TRAPNO_OF);
mov(dword [rax + offsetof(Dispatcher::SynchronousFaultDataStruct, err_code)], 0);
mov(dword [rax + offsetof(Dispatcher::SynchronousFaultDataStruct, si_code)], 0x80);
hlt();
}
{
ReturnPtr = getCurr<FEXCore::Context::Context::IntCallbackReturn>();
// using CallbackReturn = FEX_NAKED void(*)(FEXCore::Core::InternalThreadState *Thread, volatile void *Host_RSP);
@@ -312,12 +330,26 @@ X86Dispatcher::X86Dispatcher(FEXCore::Context::Context *ctx, FEXCore::Core::Inte
End = Start + getSize();
if (CTX->Config.BlockJITNaming()) {
std::string Name = "Dispatch_" + std::to_string(::gettid());
std::string Name = "Dispatch_" + std::to_string(FHU::Syscalls::gettid());
CTX->Symbols.Register(reinterpret_cast<void*>(Start), End-Start, Name);
}
if (CTX->Config.GlobalJITNaming()) {
CTX->Symbols.RegisterJITSpace(reinterpret_cast<void*>(Start), End-Start);
}
// Setup dispatcher specific pointers that need to be accessed from JIT code
{
auto &Pointers = ThreadState->CurrentFrame->Pointers.X86;
Pointers.DispatcherLoopTop = AbsoluteLoopTopAddress;
Pointers.DispatcherLoopTopFillSRA = AbsoluteLoopTopAddressFillSRA;
Pointers.ThreadStopHandler = ThreadStopHandlerAddress;
Pointers.ThreadPauseHandler = ThreadPauseHandlerAddress;
Pointers.UnimplementedInstructionHandler = UnimplementedInstructionAddress;
Pointers.OverflowExceptionHandler = OverflowExceptionInstructionAddress;
Pointers.SignalReturnHandler = SignalHandlerReturnAddress;
Pointers.L1Pointer = Thread->LookupCache->GetL1Pointer();
}
}
X86Dispatcher::~X86Dispatcher() {
+28 -22
View File
@@ -399,12 +399,12 @@ bool Decoder::NormalOp(FEXCore::X86Tables::X86InstInfo const *Info, uint16_t Op,
}
if (Info->Type == FEXCore::X86Tables::TYPE_UNKNOWN) {
LogMan::Msg::DFmt("Unknown instruction: {} 0x{:04x} 0x{:x}", Info->Name, Op, DecodeInst->PC);
LogMan::Msg::DFmt("Unknown instruction: {} 0x{:04x} 0x{:x}", Info->Name ?: "UND", Op, DecodeInst->PC);
return false;
}
if (Info->Type == FEXCore::X86Tables::TYPE_INVALID) {
LogMan::Msg::DFmt("Invalid or Unknown instruction: {} 0x{:04x} 0x{:x}", Info->Name, Op, DecodeInst->PC);
LogMan::Msg::DFmt("Invalid or Unknown instruction: {} 0x{:04x} 0x{:x}", Info->Name ?: "UND", Op, DecodeInst->PC);
return false;
}
@@ -583,8 +583,11 @@ bool Decoder::NormalOp(FEXCore::X86Tables::X86InstInfo const *Info, uint16_t Op,
return false;
}
else {
auto Disp = DecodeModRMs_Disp[Has16BitAddressing];
(this->*Disp)(&NonGPR, ModRM);
// Only decode if we haven't pre-decoded
if (NonGPR.IsNone()) {
auto Disp = DecodeModRMs_Disp[Has16BitAddressing];
(this->*Disp)(&NonGPR, ModRM);
}
}
return true;
@@ -664,7 +667,7 @@ bool Decoder::NormalOp(FEXCore::X86Tables::X86InstInfo const *Info, uint16_t Op,
}
LOGMAN_THROW_A_FMT(Bytes == 0, "Inst at 0x{:x}: 0x{:04x} '{}' Had an instruction of size {} with {} remaining",
DecodeInst->PC, DecodeInst->OP, DecodeInst->TableInfo->Name, InstructionSize, Bytes);
DecodeInst->PC, DecodeInst->OP, DecodeInst->TableInfo->Name ?: "UND", InstructionSize, Bytes);
DecodeInst->InstSize = InstructionSize;
return true;
}
@@ -680,12 +683,12 @@ bool Decoder::NormalOpHeader(FEXCore::X86Tables::X86InstInfo const *Info, uint16
}
if (Info->Type == FEXCore::X86Tables::TYPE_UNKNOWN) {
LogMan::Msg::DFmt("Unknown instruction: {} 0x{:04x} 0x{:x}", Info->Name, Op, DecodeInst->PC);
LogMan::Msg::DFmt("Unknown instruction: {} 0x{:04x} 0x{:x}", Info->Name ?: "UND", Op, DecodeInst->PC);
return false;
}
if (Info->Type == FEXCore::X86Tables::TYPE_INVALID) {
LogMan::Msg::DFmt("Invalid or Unknown instruction: {} 0x{:04x} 0x{:x}", Info->Name, Op, DecodeInst->PC);
LogMan::Msg::DFmt("Invalid or Unknown instruction: {} 0x{:04x} 0x{:x}", Info->Name ?: "UND", Op, DecodeInst->PC);
return false;
}
@@ -857,7 +860,7 @@ bool Decoder::DecodeInstruction(uint64_t PC) {
case 0x0F: {// Escape Op
uint8_t EscapeOp = ReadByte();
switch (EscapeOp) {
case 0x0F: { // 3DNow!
case 0x0F: [[unlikely]] { // 3DNow!
// 3DNow! Instruction Encoding: 0F 0F [ModRM] [SIB] [Displacement] [Opcode]
// Decode ModRM
uint8_t ModRMByte = ReadByte();
@@ -870,8 +873,12 @@ bool Decoder::DecodeInstruction(uint64_t PC) {
const bool Has16BitAddressing = !CTX->Config.Is64BitMode &&
DecodeInst->Flags & DecodeFlags::FLAG_ADDRESS_SIZE;
auto Disp = DecodeModRMs_Disp[Has16BitAddressing];
(this->*Disp)(&DecodeInst->Src[0], ModRM);
// All 3DNow! instructions have the second argument as the rm handler
// We need to decode it upfront to get the displacement out of the way
if (ModRM.mod != 0b11) {
auto Disp = DecodeModRMs_Disp[Has16BitAddressing];
(this->*Disp)(&DecodeInst->Src[0], ModRM);
}
// Take a peek at the op just past the displacement
uint8_t LocalOp = ReadByte();
@@ -880,20 +887,19 @@ bool Decoder::DecodeInstruction(uint64_t PC) {
}
case 0x38: { // F38 Table!
constexpr uint16_t PF_38_NONE = 0;
constexpr uint16_t PF_38_66 = 1;
constexpr uint16_t PF_38_F2 = 2;
constexpr uint16_t PF_38_F3 = 3;
constexpr uint16_t PF_38_66 = (1U << 0);
constexpr uint16_t PF_38_F2 = (1U << 1);
constexpr uint16_t PF_38_F3 = (1U << 2);
uint16_t Prefix = PF_38_NONE;
if (DecodeInst->LastEscapePrefix == 0xF2) {
// Repeat prefix or instruction-specific
Prefix = PF_38_F2;
} else if (DecodeInst->LastEscapePrefix == 0xF3) {
// Repeat prefix or instruction-specific
Prefix = PF_38_F3;
} else if (DecodeInst->LastEscapePrefix == 0x66) {
// Operand size
Prefix = PF_38_66;
if (DecodeInst->Flags & DecodeFlags::FLAG_OPERAND_SIZE) {
Prefix |= PF_38_66;
}
if (DecodeInst->Flags & DecodeFlags::FLAG_REPNE_PREFIX) {
Prefix |= PF_38_F2;
}
if (DecodeInst->Flags & DecodeFlags::FLAG_REP_PREFIX) {
Prefix |= PF_38_F3;
}
uint16_t LocalOp = (Prefix << 8) | ReadByte();
+1 -1
View File
@@ -35,7 +35,7 @@ public:
uint64_t DecodedMinAddress {};
uint64_t DecodedMaxAddress {~0ULL};
void SetSectionMaxAddress(uint64_t v) { SectionMaxAddress = v; }
void SetExternalBranches(std::set<uint64_t> *v) { ExternalBranches = v; }
private:
+7 -6
View File
@@ -480,7 +480,7 @@ std::string buildMemoryMap() {
xml << "<?xml version='1.0'?>\n";
xml << "<!DOCTYPE memory-map>\n";
xml << "<memory-map>";
xml << "<memory-map>\n";
std::fstream fs("/proc/self/maps", std::fstream::in | std::fstream::binary);
std::string Line;
@@ -494,7 +494,7 @@ std::string buildMemoryMap() {
}
}
xml << "</memory-map>";
xml << "</memory-map>\n";
xml << std::flush;
@@ -595,20 +595,20 @@ GdbServer::HandledPacketType GdbServer::handleXfer(const std::string &packet) {
ThreadString = ss.str();
}
return {encode(ThreadString.substr(offset, length)), HandledPacketType::TYPE_ACK};
return {encode(ThreadString), HandledPacketType::TYPE_ACK};
}
if (object == "memory-map") {
if (offset == 0) {
MemoryMapString = buildMemoryMap();
}
return {encode(MemoryMapString.substr(offset, length)), HandledPacketType::TYPE_ACK};
return {encode(MemoryMapString), HandledPacketType::TYPE_ACK};
}
if (object == "osdata") {
if (offset == 0) {
OSDataString = buildOSData();
}
return {encode(OSDataString.substr(offset, length)), HandledPacketType::TYPE_ACK};
return {encode(OSDataString), HandledPacketType::TYPE_ACK};
}
return {"", HandledPacketType::TYPE_UNKNOWN};
@@ -970,7 +970,8 @@ GdbServer::HandledPacketType GdbServer::handleThreadOp(const std::string &packet
GdbServer::HandledPacketType GdbServer::handleBreakpoint(const std::string &packet) {
auto ss = std::istringstream(packet);
bool Set{};
// Don't do anything with set breakpoints yet
[[maybe_unused]] bool Set{};
uint64_t Addr;
uint64_t Type;
Set = ss.get() == 'Z';
+94
View File
@@ -1,3 +1,4 @@
#include "Interface/Core/CPUID.h"
#include "Interface/Core/HostFeatures.h"
#ifdef _M_ARM_64
@@ -13,14 +14,107 @@
namespace FEXCore {
// Data Zero Prohibited flag
// 0b0 = ZVA/GVA/GZVA permitted
// 0b1 = ZVA/GVA/GZVA prohibited
constexpr uint32_t DCZID_DZP_MASK = 0b1'0000;
// Log2 of the blocksize in 32-bit words
constexpr uint32_t DCZID_BS_MASK = 0b0'1111;
#ifdef _M_ARM_64
static uint32_t GetDCZID() {
uint64_t Result{};
__asm("mrs %[Res], DCZID_EL0"
: [Res] "=r" (Result));
return Result;
}
static uint32_t GetFPCR() {
uint64_t Result{};
__asm ("mrs %[Res], FPCR"
: [Res] "=r" (Result));
return Result;
}
static void SetFPCR(uint64_t Value) {
__asm ("msr FPCR, %[Value]"
:: [Value] "r" (Value));
}
#else
static uint32_t GetDCZID() {
// Return unsupported
return DCZID_DZP_MASK;
}
#endif
HostFeatures::HostFeatures() {
#ifdef _M_ARM_64
auto Features = vixl::CPUFeatures::InferFromOS();
SupportsAES = Features.Has(vixl::CPUFeatures::Feature::kAES);
SupportsCRC = Features.Has(vixl::CPUFeatures::Feature::kCRC32);
SupportsAtomics = Features.Has(vixl::CPUFeatures::Feature::kAtomics);
SupportsRAND = Features.Has(vixl::CPUFeatures::Feature::kRNG);
// Only supported when FEAT_AFP is supported
SupportsFlushInputsToZero = Features.Has(vixl::CPUFeatures::Feature::kAFP);
// RCPC is bugged on Snapdragon 865
// Causes glibc cond16 test to immediately throw assert
// __pthread_mutex_cond_lock: Assertion `mutex->__data.__owner == 0'
SupportsRCPC = false; //Features.Has(vixl::CPUFeatures::Feature::kRCpc);
// We need to get the CPU's cache line size
// We expect sane targets that have correct cacheline sizes across clusters
uint64_t CTR;
__asm volatile ("mrs %[ctr], ctr_el0"
: [ctr] "=r"(CTR));
DCacheLineSize = 4 << ((CTR >> 16) & 0xF);
ICacheLineSize = 4 << (CTR & 0xF);
if (!SupportsAtomics) {
WARN_ONCE_FMT("Host CPU doesn't support atomics. Expect bad performance");
}
#endif
#ifdef _M_X86_64
Xbyak::util::Cpu Features{};
SupportsAES = Features.has(Xbyak::util::Cpu::tAESNI);
SupportsCRC = Features.has(Xbyak::util::Cpu::tSSE42);
SupportsRAND = Features.has(Xbyak::util::Cpu::tRDRAND) && Features.has(Xbyak::util::Cpu::tRDSEED);
SupportsFlushInputsToZero = true;
SupportsFloatExceptions = true;
#else
// Test if this CPU supports float exception trapping by attempting to enable
// On unsupported these bits are architecturally defined as RAZ/WI
constexpr uint32_t ExceptionEnableTraps =
(1U << 8) | // Invalid Operation float exception trap enable
(1U << 9) | // Divide by zero float exception trap enable
(1U << 10) | // Overflow float exception trap enable
(1U << 11) | // Underflow float exception trap enable
(1U << 12) | // Inexact float exception trap enable
(1U << 15); // Input Denormal float exception trap enable
uint32_t OriginalFPCR = GetFPCR();
uint32_t FPCR = OriginalFPCR | ExceptionEnableTraps;
SetFPCR(FPCR);
FPCR = GetFPCR();
SupportsFloatExceptions = (FPCR & ExceptionEnableTraps) == ExceptionEnableTraps;
// Set FPCR back to original just in case anything changed
SetFPCR(OriginalFPCR);
#endif
// Check if we can support cacheline clears
uint32_t DCZID = GetDCZID();
if ((DCZID & DCZID_DZP_MASK) == 0) {
uint32_t DCZID_Log2 = DCZID & DCZID_BS_MASK;
uint32_t DCZID_Bytes = (1 << DCZID_Log2) * sizeof(uint32_t);
// If the DC ZVA size matches the emulated cache line size
// This means we can use the instruction
SupportsCLZERO = DCZID_Bytes == CPUIDEmu::CACHELINE_SIZE;
}
}
}
+19
View File
@@ -1,9 +1,28 @@
#pragma once
#include <cstdint>
namespace FEXCore {
class HostFeatures final {
public:
HostFeatures();
/**
* @brief Backend features that change how codegen is generated from IR
*
* Specifically things that affect the IR->Codegen process
* Not the x86->IR process
*/
uint32_t DCacheLineSize{};
uint32_t ICacheLineSize{};
bool SupportsAES{};
bool SupportsCRC{};
bool SupportsCLZERO{};
bool SupportsAtomics{};
bool SupportsRCPC{};
bool SupportsRAND{};
// Float exception behaviour
bool SupportsFlushInputsToZero{};
bool SupportsFloatExceptions{};
};
}
+62 -11
View File
@@ -18,7 +18,7 @@ namespace FEXCore::CPU {
DEF_OP(TruncElementPair) {
auto Op = IROp->C<IR::IROp_TruncElementPair>();
switch (Op->Size) {
switch (IROp->Size) {
case 4: {
uint64_t *Src = GetSrc<uint64_t*>(Data->SSAData, Op->Header.Args[0]);
uint64_t Result{};
@@ -27,7 +27,7 @@ DEF_OP(TruncElementPair) {
GD = Result;
break;
}
default: LOGMAN_MSG_A_FMT("Unhandled Truncation size: {}", Op->Size); break;
default: LOGMAN_MSG_A_FMT("Unhandled Truncation size: {}", IROp->Size); break;
}
}
@@ -38,7 +38,13 @@ DEF_OP(Constant) {
DEF_OP(EntrypointOffset) {
auto Op = IROp->C<IR::IROp_EntrypointOffset>();
GD = Data->CurrentEntry + Op->Offset;
uint64_t Mask = ~0ULL;
uint8_t OpSize = IROp->Size;
if (OpSize == 4) {
Mask = 0xFFFF'FFFFULL;
}
GD = (Data->CurrentEntry + Op->Offset) & Mask;
}
DEF_OP(InlineConstant) {
@@ -493,6 +499,54 @@ DEF_OP(Extr) {
}
}
DEF_OP(PDep) {
const auto Op = IROp->C<IR::IROp_PExt>();
const auto OpSize = IROp->Size;
if (OpSize != 4 && OpSize != 8) {
LOGMAN_MSG_A_FMT("Unknown PDep Size: {}\n", OpSize);
return;
}
const uint64_t Input = OpSize == 4 ? *GetSrc<uint32_t*>(Data->SSAData, Op->Args(0))
: *GetSrc<uint64_t*>(Data->SSAData, Op->Args(0));
uint64_t Mask = OpSize == 4 ? *GetSrc<uint32_t*>(Data->SSAData, Op->Args(1))
: *GetSrc<uint64_t*>(Data->SSAData, Op->Args(1));
uint64_t Result = 0;
for (uint64_t Index = 0; Mask > 0; Index++) {
const uint64_t Offset = std::countr_zero(Mask);
Mask &= Mask - 1;
Result |= ((Input >> Index) & 1) << Offset;
}
GD = Result;
}
DEF_OP(PExt) {
const auto Op = IROp->C<IR::IROp_PExt>();
const auto OpSize = IROp->Size;
if (OpSize != 4 && OpSize != 8) {
LOGMAN_MSG_A_FMT("Unknown PExt Size: {}\n", OpSize);
return;
}
const uint64_t Input = OpSize == 4 ? *GetSrc<uint32_t*>(Data->SSAData, Op->Args(0))
: *GetSrc<uint64_t*>(Data->SSAData, Op->Args(0));
uint64_t Mask = OpSize == 4 ? *GetSrc<uint32_t*>(Data->SSAData, Op->Args(1))
: *GetSrc<uint64_t*>(Data->SSAData, Op->Args(1));
uint64_t Result = 0;
for (uint64_t Offset = 0; Mask > 0; Offset++) {
const uint64_t Index = std::countr_zero(Mask);
Mask &= Mask - 1;
Result |= ((Input >> Index) & 1) << Offset;
}
GD = Result;
}
DEF_OP(LDiv) {
auto Op = IROp->C<IR::IROp_LDiv>();
uint8_t OpSize = IROp->Size;
@@ -787,9 +841,8 @@ DEF_OP(Bfi) {
DEF_OP(Bfe) {
auto Op = IROp->C<IR::IROp_Bfe>();
uint8_t OpSize = IROp->Size;
LOGMAN_THROW_A_FMT(OpSize <= 8, "OpSize is too large for BFE: {}", OpSize);
LOGMAN_THROW_A_FMT(IROp->Size <= 8, "OpSize is too large for BFE: {}", IROp->Size);
uint64_t SourceMask = (1ULL << Op->Width) - 1;
if (Op->Width == 64)
SourceMask = ~0ULL;
@@ -800,9 +853,8 @@ DEF_OP(Bfe) {
DEF_OP(Sbfe) {
auto Op = IROp->C<IR::IROp_Sbfe>();
uint8_t OpSize = IROp->Size;
LOGMAN_THROW_A_FMT(OpSize <= 8, "OpSize is too large for SBFE: {}", OpSize);
LOGMAN_THROW_A_FMT(IROp->Size <= 8, "OpSize is too large for SBFE: {}", IROp->Size);
int64_t Src = *GetSrc<int64_t*>(Data->SSAData, Op->Header.Args[0]);
uint64_t ShiftLeftAmount = (64 - (Op->Width + Op->lsb));
uint64_t ShiftRightAmount = ShiftLeftAmount + Op->lsb;
@@ -841,15 +893,14 @@ DEF_OP(Select) {
DEF_OP(VExtractToGPR) {
auto Op = IROp->C<IR::IROp_VExtractToGPR>();
uint8_t OpSize = IROp->Size;
uint32_t SourceSize = GetOpSize(Data->CurrentIR, Op->Header.Args[0]);
LOGMAN_THROW_A_FMT(OpSize <= 16, "OpSize is too large for VExtractToGPR: {}", OpSize);
LOGMAN_THROW_A_FMT(IROp->Size <= 16, "OpSize is too large for VExtractToGPR: {}", IROp->Size);
if (SourceSize == 16) {
__uint128_t SourceMask = (1ULL << (Op->Header.ElementSize * 8)) - 1;
uint64_t Shift = Op->Header.ElementSize * Op->Idx * 8;
uint64_t Shift = Op->Header.ElementSize * Op->Index * 8;
if (Op->Header.ElementSize == 8)
SourceMask = ~0ULL;
@@ -860,7 +911,7 @@ DEF_OP(VExtractToGPR) {
}
else {
uint64_t SourceMask = (1ULL << (Op->Header.ElementSize * 8)) - 1;
uint64_t Shift = Op->Header.ElementSize * Op->Idx * 8;
uint64_t Shift = Op->Header.ElementSize * Op->Index * 8;
if (Op->Header.ElementSize == 8)
SourceMask = ~0ULL;
+112 -113
View File
@@ -313,30 +313,29 @@ uint64_t AtomicCompareAndSwap(uint64_t expected, uint64_t desired, uint64_t *add
#define DEF_OP(x) void InterpreterOps::Op_##x(IR::IROp_Header *IROp, IROpData *Data, IR::NodeID Node)
DEF_OP(CASPair) {
auto Op = IROp->C<IR::IROp_CASPair>();
uint8_t OpSize = IROp->Size;
// Size is the size of each pair element
switch (OpSize) {
switch (IROp->ElementSize) {
case 4: {
GD = AtomicCompareAndSwap(
*GetSrc<uint64_t*>(Data->SSAData, Op->Header.Args[0]),
*GetSrc<uint64_t*>(Data->SSAData, Op->Header.Args[1]),
*GetSrc<uint64_t**>(Data->SSAData, Op->Header.Args[2])
*GetSrc<uint64_t*>(Data->SSAData, Op->Expected),
*GetSrc<uint64_t*>(Data->SSAData, Op->Desired),
*GetSrc<uint64_t**>(Data->SSAData, Op->Addr)
);
break;
}
case 8: {
std::atomic<__uint128_t> *MemData = *GetSrc<std::atomic<__uint128_t> **>(Data->SSAData, Op->Header.Args[2]);
std::atomic<__uint128_t> *MemData = *GetSrc<std::atomic<__uint128_t> **>(Data->SSAData, Op->Addr);
__uint128_t Src1 = *GetSrc<__uint128_t*>(Data->SSAData, Op->Header.Args[0]);
__uint128_t Src2 = *GetSrc<__uint128_t*>(Data->SSAData, Op->Header.Args[1]);
__uint128_t Src1 = *GetSrc<__uint128_t*>(Data->SSAData, Op->Expected);
__uint128_t Src2 = *GetSrc<__uint128_t*>(Data->SSAData, Op->Desired);
__uint128_t Expected = Src1;
bool Result = MemData->compare_exchange_strong(Expected, Src2);
memcpy(GDP, Result ? &Src1 : &Expected, 16);
break;
}
default: LOGMAN_MSG_A_FMT("Unknown CAS size: {}", OpSize); break;
default: LOGMAN_MSG_A_FMT("Unknown CAS size: {}", IROp->ElementSize); break;
}
}
@@ -347,33 +346,33 @@ DEF_OP(CAS) {
switch (OpSize) {
case 1: {
GD = AtomicCompareAndSwap(
*GetSrc<uint8_t*>(Data->SSAData, Op->Header.Args[0]),
*GetSrc<uint8_t*>(Data->SSAData, Op->Header.Args[1]),
*GetSrc<uint8_t**>(Data->SSAData, Op->Header.Args[2])
*GetSrc<uint8_t*>(Data->SSAData, Op->Expected),
*GetSrc<uint8_t*>(Data->SSAData, Op->Desired),
*GetSrc<uint8_t**>(Data->SSAData, Op->Addr)
);
break;
}
case 2: {
GD = AtomicCompareAndSwap(
*GetSrc<uint16_t*>(Data->SSAData, Op->Header.Args[0]),
*GetSrc<uint16_t*>(Data->SSAData, Op->Header.Args[1]),
*GetSrc<uint16_t**>(Data->SSAData, Op->Header.Args[2])
*GetSrc<uint16_t*>(Data->SSAData, Op->Expected),
*GetSrc<uint16_t*>(Data->SSAData, Op->Desired),
*GetSrc<uint16_t**>(Data->SSAData, Op->Addr)
);
break;
}
case 4: {
GD = AtomicCompareAndSwap(
*GetSrc<uint32_t*>(Data->SSAData, Op->Header.Args[0]),
*GetSrc<uint32_t*>(Data->SSAData, Op->Header.Args[1]),
*GetSrc<uint32_t**>(Data->SSAData, Op->Header.Args[2])
*GetSrc<uint32_t*>(Data->SSAData, Op->Expected),
*GetSrc<uint32_t*>(Data->SSAData, Op->Desired),
*GetSrc<uint32_t**>(Data->SSAData, Op->Addr)
);
break;
}
case 8: {
GD = AtomicCompareAndSwap(
*GetSrc<uint64_t*>(Data->SSAData, Op->Header.Args[0]),
*GetSrc<uint64_t*>(Data->SSAData, Op->Header.Args[1]),
*GetSrc<uint64_t**>(Data->SSAData, Op->Header.Args[2])
*GetSrc<uint64_t*>(Data->SSAData, Op->Expected),
*GetSrc<uint64_t*>(Data->SSAData, Op->Desired),
*GetSrc<uint64_t**>(Data->SSAData, Op->Addr)
);
break;
}
@@ -385,26 +384,26 @@ DEF_OP(AtomicAdd) {
auto Op = IROp->C<IR::IROp_AtomicAdd>();
switch (IROp->Size) {
case 1: {
std::atomic<uint8_t> *MemData = *GetSrc<std::atomic<uint8_t> **>(Data->SSAData, Op->Header.Args[0]);
uint8_t Src = *GetSrc<uint8_t*>(Data->SSAData, Op->Header.Args[1]);
std::atomic<uint8_t> *MemData = *GetSrc<std::atomic<uint8_t> **>(Data->SSAData, Op->Addr);
uint8_t Src = *GetSrc<uint8_t*>(Data->SSAData, Op->Value);
*MemData += Src;
break;
}
case 2: {
std::atomic<uint16_t> *MemData = *GetSrc<std::atomic<uint16_t> **>(Data->SSAData, Op->Header.Args[0]);
uint16_t Src = *GetSrc<uint16_t*>(Data->SSAData, Op->Header.Args[1]);
std::atomic<uint16_t> *MemData = *GetSrc<std::atomic<uint16_t> **>(Data->SSAData, Op->Addr);
uint16_t Src = *GetSrc<uint16_t*>(Data->SSAData, Op->Value);
*MemData += Src;
break;
}
case 4: {
std::atomic<uint32_t> *MemData = *GetSrc<std::atomic<uint32_t> **>(Data->SSAData, Op->Header.Args[0]);
uint32_t Src = *GetSrc<uint32_t*>(Data->SSAData, Op->Header.Args[1]);
std::atomic<uint32_t> *MemData = *GetSrc<std::atomic<uint32_t> **>(Data->SSAData, Op->Addr);
uint32_t Src = *GetSrc<uint32_t*>(Data->SSAData, Op->Value);
*MemData += Src;
break;
}
case 8: {
std::atomic<uint64_t> *MemData = *GetSrc<std::atomic<uint64_t> **>(Data->SSAData, Op->Header.Args[0]);
uint64_t Src = *GetSrc<uint64_t*>(Data->SSAData, Op->Header.Args[1]);
std::atomic<uint64_t> *MemData = *GetSrc<std::atomic<uint64_t> **>(Data->SSAData, Op->Addr);
uint64_t Src = *GetSrc<uint64_t*>(Data->SSAData, Op->Value);
*MemData += Src;
break;
}
@@ -416,26 +415,26 @@ DEF_OP(AtomicSub) {
auto Op = IROp->C<IR::IROp_AtomicSub>();
switch (IROp->Size) {
case 1: {
std::atomic<uint8_t> *MemData = *GetSrc<std::atomic<uint8_t> **>(Data->SSAData, Op->Header.Args[0]);
uint8_t Src = *GetSrc<uint8_t*>(Data->SSAData, Op->Header.Args[1]);
std::atomic<uint8_t> *MemData = *GetSrc<std::atomic<uint8_t> **>(Data->SSAData, Op->Addr);
uint8_t Src = *GetSrc<uint8_t*>(Data->SSAData, Op->Value);
*MemData -= Src;
break;
}
case 2: {
std::atomic<uint16_t> *MemData = *GetSrc<std::atomic<uint16_t> **>(Data->SSAData, Op->Header.Args[0]);
uint16_t Src = *GetSrc<uint16_t*>(Data->SSAData, Op->Header.Args[1]);
std::atomic<uint16_t> *MemData = *GetSrc<std::atomic<uint16_t> **>(Data->SSAData, Op->Addr);
uint16_t Src = *GetSrc<uint16_t*>(Data->SSAData, Op->Value);
*MemData -= Src;
break;
}
case 4: {
std::atomic<uint32_t> *MemData = *GetSrc<std::atomic<uint32_t> **>(Data->SSAData, Op->Header.Args[0]);
uint32_t Src = *GetSrc<uint32_t*>(Data->SSAData, Op->Header.Args[1]);
std::atomic<uint32_t> *MemData = *GetSrc<std::atomic<uint32_t> **>(Data->SSAData, Op->Addr);
uint32_t Src = *GetSrc<uint32_t*>(Data->SSAData, Op->Value);
*MemData -= Src;
break;
}
case 8: {
std::atomic<uint64_t> *MemData = *GetSrc<std::atomic<uint64_t> **>(Data->SSAData, Op->Header.Args[0]);
uint64_t Src = *GetSrc<uint64_t*>(Data->SSAData, Op->Header.Args[1]);
std::atomic<uint64_t> *MemData = *GetSrc<std::atomic<uint64_t> **>(Data->SSAData, Op->Addr);
uint64_t Src = *GetSrc<uint64_t*>(Data->SSAData, Op->Value);
*MemData -= Src;
break;
}
@@ -447,26 +446,26 @@ DEF_OP(AtomicAnd) {
auto Op = IROp->C<IR::IROp_AtomicAnd>();
switch (IROp->Size) {
case 1: {
std::atomic<uint8_t> *MemData = *GetSrc<std::atomic<uint8_t> **>(Data->SSAData, Op->Header.Args[0]);
uint8_t Src = *GetSrc<uint8_t*>(Data->SSAData, Op->Header.Args[1]);
std::atomic<uint8_t> *MemData = *GetSrc<std::atomic<uint8_t> **>(Data->SSAData, Op->Addr);
uint8_t Src = *GetSrc<uint8_t*>(Data->SSAData, Op->Value);
*MemData &= Src;
break;
}
case 2: {
std::atomic<uint16_t> *MemData = *GetSrc<std::atomic<uint16_t> **>(Data->SSAData, Op->Header.Args[0]);
uint16_t Src = *GetSrc<uint16_t*>(Data->SSAData, Op->Header.Args[1]);
std::atomic<uint16_t> *MemData = *GetSrc<std::atomic<uint16_t> **>(Data->SSAData, Op->Addr);
uint16_t Src = *GetSrc<uint16_t*>(Data->SSAData, Op->Value);
*MemData &= Src;
break;
}
case 4: {
std::atomic<uint32_t> *MemData = *GetSrc<std::atomic<uint32_t> **>(Data->SSAData, Op->Header.Args[0]);
uint32_t Src = *GetSrc<uint32_t*>(Data->SSAData, Op->Header.Args[1]);
std::atomic<uint32_t> *MemData = *GetSrc<std::atomic<uint32_t> **>(Data->SSAData, Op->Addr);
uint32_t Src = *GetSrc<uint32_t*>(Data->SSAData, Op->Value);
*MemData &= Src;
break;
}
case 8: {
std::atomic<uint64_t> *MemData = *GetSrc<std::atomic<uint64_t> **>(Data->SSAData, Op->Header.Args[0]);
uint64_t Src = *GetSrc<uint64_t*>(Data->SSAData, Op->Header.Args[1]);
std::atomic<uint64_t> *MemData = *GetSrc<std::atomic<uint64_t> **>(Data->SSAData, Op->Addr);
uint64_t Src = *GetSrc<uint64_t*>(Data->SSAData, Op->Value);
*MemData &= Src;
break;
}
@@ -478,26 +477,26 @@ DEF_OP(AtomicOr) {
auto Op = IROp->C<IR::IROp_AtomicOr>();
switch (IROp->Size) {
case 1: {
std::atomic<uint8_t> *MemData = *GetSrc<std::atomic<uint8_t> **>(Data->SSAData, Op->Header.Args[0]);
uint8_t Src = *GetSrc<uint8_t*>(Data->SSAData, Op->Header.Args[1]);
std::atomic<uint8_t> *MemData = *GetSrc<std::atomic<uint8_t> **>(Data->SSAData, Op->Addr);
uint8_t Src = *GetSrc<uint8_t*>(Data->SSAData, Op->Value);
*MemData |= Src;
break;
}
case 2: {
std::atomic<uint16_t> *MemData = *GetSrc<std::atomic<uint16_t> **>(Data->SSAData, Op->Header.Args[0]);
uint16_t Src = *GetSrc<uint16_t*>(Data->SSAData, Op->Header.Args[1]);
std::atomic<uint16_t> *MemData = *GetSrc<std::atomic<uint16_t> **>(Data->SSAData, Op->Addr);
uint16_t Src = *GetSrc<uint16_t*>(Data->SSAData, Op->Value);
*MemData |= Src;
break;
}
case 4: {
std::atomic<uint32_t> *MemData = *GetSrc<std::atomic<uint32_t> **>(Data->SSAData, Op->Header.Args[0]);
uint32_t Src = *GetSrc<uint32_t*>(Data->SSAData, Op->Header.Args[1]);
std::atomic<uint32_t> *MemData = *GetSrc<std::atomic<uint32_t> **>(Data->SSAData, Op->Addr);
uint32_t Src = *GetSrc<uint32_t*>(Data->SSAData, Op->Value);
*MemData |= Src;
break;
}
case 8: {
std::atomic<uint64_t> *MemData = *GetSrc<std::atomic<uint64_t> **>(Data->SSAData, Op->Header.Args[0]);
uint64_t Src = *GetSrc<uint64_t*>(Data->SSAData, Op->Header.Args[1]);
std::atomic<uint64_t> *MemData = *GetSrc<std::atomic<uint64_t> **>(Data->SSAData, Op->Addr);
uint64_t Src = *GetSrc<uint64_t*>(Data->SSAData, Op->Value);
*MemData |= Src;
break;
}
@@ -509,26 +508,26 @@ DEF_OP(AtomicXor) {
auto Op = IROp->C<IR::IROp_AtomicXor>();
switch (IROp->Size) {
case 1: {
std::atomic<uint8_t> *MemData = *GetSrc<std::atomic<uint8_t> **>(Data->SSAData, Op->Header.Args[0]);
uint8_t Src = *GetSrc<uint8_t*>(Data->SSAData, Op->Header.Args[1]);
std::atomic<uint8_t> *MemData = *GetSrc<std::atomic<uint8_t> **>(Data->SSAData, Op->Addr);
uint8_t Src = *GetSrc<uint8_t*>(Data->SSAData, Op->Value);
*MemData ^= Src;
break;
}
case 2: {
std::atomic<uint16_t> *MemData = *GetSrc<std::atomic<uint16_t> **>(Data->SSAData, Op->Header.Args[0]);
uint16_t Src = *GetSrc<uint16_t*>(Data->SSAData, Op->Header.Args[1]);
std::atomic<uint16_t> *MemData = *GetSrc<std::atomic<uint16_t> **>(Data->SSAData, Op->Addr);
uint16_t Src = *GetSrc<uint16_t*>(Data->SSAData, Op->Value);
*MemData ^= Src;
break;
}
case 4: {
std::atomic<uint32_t> *MemData = *GetSrc<std::atomic<uint32_t> **>(Data->SSAData, Op->Header.Args[0]);
uint32_t Src = *GetSrc<uint32_t*>(Data->SSAData, Op->Header.Args[1]);
std::atomic<uint32_t> *MemData = *GetSrc<std::atomic<uint32_t> **>(Data->SSAData, Op->Addr);
uint32_t Src = *GetSrc<uint32_t*>(Data->SSAData, Op->Value);
*MemData ^= Src;
break;
}
case 8: {
std::atomic<uint64_t> *MemData = *GetSrc<std::atomic<uint64_t> **>(Data->SSAData, Op->Header.Args[0]);
uint64_t Src = *GetSrc<uint64_t*>(Data->SSAData, Op->Header.Args[1]);
std::atomic<uint64_t> *MemData = *GetSrc<std::atomic<uint64_t> **>(Data->SSAData, Op->Addr);
uint64_t Src = *GetSrc<uint64_t*>(Data->SSAData, Op->Value);
*MemData ^= Src;
break;
}
@@ -540,29 +539,29 @@ DEF_OP(AtomicSwap) {
auto Op = IROp->C<IR::IROp_AtomicSwap>();
switch (IROp->Size) {
case 1: {
std::atomic<uint8_t> *MemData = *GetSrc<std::atomic<uint8_t> **>(Data->SSAData, Op->Header.Args[0]);
uint8_t Src = *GetSrc<uint8_t*>(Data->SSAData, Op->Header.Args[1]);
std::atomic<uint8_t> *MemData = *GetSrc<std::atomic<uint8_t> **>(Data->SSAData, Op->Addr);
uint8_t Src = *GetSrc<uint8_t*>(Data->SSAData, Op->Value);
uint8_t Previous = MemData->exchange(Src);
GD = Previous;
break;
}
case 2: {
std::atomic<uint16_t> *MemData = *GetSrc<std::atomic<uint16_t> **>(Data->SSAData, Op->Header.Args[0]);
uint16_t Src = *GetSrc<uint16_t*>(Data->SSAData, Op->Header.Args[1]);
std::atomic<uint16_t> *MemData = *GetSrc<std::atomic<uint16_t> **>(Data->SSAData, Op->Addr);
uint16_t Src = *GetSrc<uint16_t*>(Data->SSAData, Op->Value);
uint16_t Previous = MemData->exchange(Src);
GD = Previous;
break;
}
case 4: {
std::atomic<uint32_t> *MemData = *GetSrc<std::atomic<uint32_t> **>(Data->SSAData, Op->Header.Args[0]);
uint32_t Src = *GetSrc<uint32_t*>(Data->SSAData, Op->Header.Args[1]);
std::atomic<uint32_t> *MemData = *GetSrc<std::atomic<uint32_t> **>(Data->SSAData, Op->Addr);
uint32_t Src = *GetSrc<uint32_t*>(Data->SSAData, Op->Value);
uint32_t Previous = MemData->exchange(Src);
GD = Previous;
break;
}
case 8: {
std::atomic<uint64_t> *MemData = *GetSrc<std::atomic<uint64_t> **>(Data->SSAData, Op->Header.Args[0]);
uint64_t Src = *GetSrc<uint64_t*>(Data->SSAData, Op->Header.Args[1]);
std::atomic<uint64_t> *MemData = *GetSrc<std::atomic<uint64_t> **>(Data->SSAData, Op->Addr);
uint64_t Src = *GetSrc<uint64_t*>(Data->SSAData, Op->Value);
uint64_t Previous = MemData->exchange(Src);
GD = Previous;
break;
@@ -575,29 +574,29 @@ DEF_OP(AtomicFetchAdd) {
auto Op = IROp->C<IR::IROp_AtomicFetchAdd>();
switch (IROp->Size) {
case 1: {
std::atomic<uint8_t> *MemData = *GetSrc<std::atomic<uint8_t> **>(Data->SSAData, Op->Header.Args[0]);
uint8_t Src = *GetSrc<uint8_t*>(Data->SSAData, Op->Header.Args[1]);
std::atomic<uint8_t> *MemData = *GetSrc<std::atomic<uint8_t> **>(Data->SSAData, Op->Addr);
uint8_t Src = *GetSrc<uint8_t*>(Data->SSAData, Op->Value);
uint8_t Previous = MemData->fetch_add(Src);
GD = Previous;
break;
}
case 2: {
std::atomic<uint16_t> *MemData = *GetSrc<std::atomic<uint16_t> **>(Data->SSAData, Op->Header.Args[0]);
uint16_t Src = *GetSrc<uint16_t*>(Data->SSAData, Op->Header.Args[1]);
std::atomic<uint16_t> *MemData = *GetSrc<std::atomic<uint16_t> **>(Data->SSAData, Op->Addr);
uint16_t Src = *GetSrc<uint16_t*>(Data->SSAData, Op->Value);
uint16_t Previous = MemData->fetch_add(Src);
GD = Previous;
break;
}
case 4: {
std::atomic<uint32_t> *MemData = *GetSrc<std::atomic<uint32_t> **>(Data->SSAData, Op->Header.Args[0]);
uint32_t Src = *GetSrc<uint32_t*>(Data->SSAData, Op->Header.Args[1]);
std::atomic<uint32_t> *MemData = *GetSrc<std::atomic<uint32_t> **>(Data->SSAData, Op->Addr);
uint32_t Src = *GetSrc<uint32_t*>(Data->SSAData, Op->Value);
uint32_t Previous = MemData->fetch_add(Src);
GD = Previous;
break;
}
case 8: {
std::atomic<uint64_t> *MemData = *GetSrc<std::atomic<uint64_t> **>(Data->SSAData, Op->Header.Args[0]);
uint64_t Src = *GetSrc<uint64_t*>(Data->SSAData, Op->Header.Args[1]);
std::atomic<uint64_t> *MemData = *GetSrc<std::atomic<uint64_t> **>(Data->SSAData, Op->Addr);
uint64_t Src = *GetSrc<uint64_t*>(Data->SSAData, Op->Value);
uint64_t Previous = MemData->fetch_add(Src);
GD = Previous;
break;
@@ -610,29 +609,29 @@ DEF_OP(AtomicFetchSub) {
auto Op = IROp->C<IR::IROp_AtomicFetchSub>();
switch (IROp->Size) {
case 1: {
std::atomic<uint8_t> *MemData = *GetSrc<std::atomic<uint8_t> **>(Data->SSAData, Op->Header.Args[0]);
uint8_t Src = *GetSrc<uint8_t*>(Data->SSAData, Op->Header.Args[1]);
std::atomic<uint8_t> *MemData = *GetSrc<std::atomic<uint8_t> **>(Data->SSAData, Op->Addr);
uint8_t Src = *GetSrc<uint8_t*>(Data->SSAData, Op->Value);
uint8_t Previous = MemData->fetch_sub(Src);
GD = Previous;
break;
}
case 2: {
std::atomic<uint16_t> *MemData = *GetSrc<std::atomic<uint16_t> **>(Data->SSAData, Op->Header.Args[0]);
uint16_t Src = *GetSrc<uint16_t*>(Data->SSAData, Op->Header.Args[1]);
std::atomic<uint16_t> *MemData = *GetSrc<std::atomic<uint16_t> **>(Data->SSAData, Op->Addr);
uint16_t Src = *GetSrc<uint16_t*>(Data->SSAData, Op->Value);
uint16_t Previous = MemData->fetch_sub(Src);
GD = Previous;
break;
}
case 4: {
std::atomic<uint32_t> *MemData = *GetSrc<std::atomic<uint32_t> **>(Data->SSAData, Op->Header.Args[0]);
uint32_t Src = *GetSrc<uint32_t*>(Data->SSAData, Op->Header.Args[1]);
std::atomic<uint32_t> *MemData = *GetSrc<std::atomic<uint32_t> **>(Data->SSAData, Op->Addr);
uint32_t Src = *GetSrc<uint32_t*>(Data->SSAData, Op->Value);
uint32_t Previous = MemData->fetch_sub(Src);
GD = Previous;
break;
}
case 8: {
std::atomic<uint64_t> *MemData = *GetSrc<std::atomic<uint64_t> **>(Data->SSAData, Op->Header.Args[0]);
uint64_t Src = *GetSrc<uint64_t*>(Data->SSAData, Op->Header.Args[1]);
std::atomic<uint64_t> *MemData = *GetSrc<std::atomic<uint64_t> **>(Data->SSAData, Op->Addr);
uint64_t Src = *GetSrc<uint64_t*>(Data->SSAData, Op->Value);
uint64_t Previous = MemData->fetch_sub(Src);
GD = Previous;
break;
@@ -645,29 +644,29 @@ DEF_OP(AtomicFetchAnd) {
auto Op = IROp->C<IR::IROp_AtomicFetchAnd>();
switch (IROp->Size) {
case 1: {
std::atomic<uint8_t> *MemData = *GetSrc<std::atomic<uint8_t> **>(Data->SSAData, Op->Header.Args[0]);
uint8_t Src = *GetSrc<uint8_t*>(Data->SSAData, Op->Header.Args[1]);
std::atomic<uint8_t> *MemData = *GetSrc<std::atomic<uint8_t> **>(Data->SSAData, Op->Addr);
uint8_t Src = *GetSrc<uint8_t*>(Data->SSAData, Op->Value);
uint8_t Previous = MemData->fetch_and(Src);
GD = Previous;
break;
}
case 2: {
std::atomic<uint16_t> *MemData = *GetSrc<std::atomic<uint16_t> **>(Data->SSAData, Op->Header.Args[0]);
uint16_t Src = *GetSrc<uint16_t*>(Data->SSAData, Op->Header.Args[1]);
std::atomic<uint16_t> *MemData = *GetSrc<std::atomic<uint16_t> **>(Data->SSAData, Op->Addr);
uint16_t Src = *GetSrc<uint16_t*>(Data->SSAData, Op->Value);
uint16_t Previous = MemData->fetch_and(Src);
GD = Previous;
break;
}
case 4: {
std::atomic<uint32_t> *MemData = *GetSrc<std::atomic<uint32_t> **>(Data->SSAData, Op->Header.Args[0]);
uint32_t Src = *GetSrc<uint32_t*>(Data->SSAData, Op->Header.Args[1]);
std::atomic<uint32_t> *MemData = *GetSrc<std::atomic<uint32_t> **>(Data->SSAData, Op->Addr);
uint32_t Src = *GetSrc<uint32_t*>(Data->SSAData, Op->Value);
uint32_t Previous = MemData->fetch_and(Src);
GD = Previous;
break;
}
case 8: {
std::atomic<uint64_t> *MemData = *GetSrc<std::atomic<uint64_t> **>(Data->SSAData, Op->Header.Args[0]);
uint64_t Src = *GetSrc<uint64_t*>(Data->SSAData, Op->Header.Args[1]);
std::atomic<uint64_t> *MemData = *GetSrc<std::atomic<uint64_t> **>(Data->SSAData, Op->Addr);
uint64_t Src = *GetSrc<uint64_t*>(Data->SSAData, Op->Value);
uint64_t Previous = MemData->fetch_and(Src);
GD = Previous;
break;
@@ -680,29 +679,29 @@ DEF_OP(AtomicFetchOr) {
auto Op = IROp->C<IR::IROp_AtomicFetchOr>();
switch (IROp->Size) {
case 1: {
std::atomic<uint8_t> *MemData = *GetSrc<std::atomic<uint8_t> **>(Data->SSAData, Op->Header.Args[0]);
uint8_t Src = *GetSrc<uint8_t*>(Data->SSAData, Op->Header.Args[1]);
std::atomic<uint8_t> *MemData = *GetSrc<std::atomic<uint8_t> **>(Data->SSAData, Op->Addr);
uint8_t Src = *GetSrc<uint8_t*>(Data->SSAData, Op->Value);
uint8_t Previous = MemData->fetch_or(Src);
GD = Previous;
break;
}
case 2: {
std::atomic<uint16_t> *MemData = *GetSrc<std::atomic<uint16_t> **>(Data->SSAData, Op->Header.Args[0]);
uint16_t Src = *GetSrc<uint16_t*>(Data->SSAData, Op->Header.Args[1]);
std::atomic<uint16_t> *MemData = *GetSrc<std::atomic<uint16_t> **>(Data->SSAData, Op->Addr);
uint16_t Src = *GetSrc<uint16_t*>(Data->SSAData, Op->Value);
uint16_t Previous = MemData->fetch_or(Src);
GD = Previous;
break;
}
case 4: {
std::atomic<uint32_t> *MemData = *GetSrc<std::atomic<uint32_t> **>(Data->SSAData, Op->Header.Args[0]);
uint32_t Src = *GetSrc<uint32_t*>(Data->SSAData, Op->Header.Args[1]);
std::atomic<uint32_t> *MemData = *GetSrc<std::atomic<uint32_t> **>(Data->SSAData, Op->Addr);
uint32_t Src = *GetSrc<uint32_t*>(Data->SSAData, Op->Value);
uint32_t Previous = MemData->fetch_or(Src);
GD = Previous;
break;
}
case 8: {
std::atomic<uint64_t> *MemData = *GetSrc<std::atomic<uint64_t> **>(Data->SSAData, Op->Header.Args[0]);
uint64_t Src = *GetSrc<uint64_t*>(Data->SSAData, Op->Header.Args[1]);
std::atomic<uint64_t> *MemData = *GetSrc<std::atomic<uint64_t> **>(Data->SSAData, Op->Addr);
uint64_t Src = *GetSrc<uint64_t*>(Data->SSAData, Op->Value);
uint64_t Previous = MemData->fetch_or(Src);
GD = Previous;
break;
@@ -715,29 +714,29 @@ DEF_OP(AtomicFetchXor) {
auto Op = IROp->C<IR::IROp_AtomicFetchXor>();
switch (IROp->Size) {
case 1: {
std::atomic<uint8_t> *MemData = *GetSrc<std::atomic<uint8_t> **>(Data->SSAData, Op->Header.Args[0]);
uint8_t Src = *GetSrc<uint8_t*>(Data->SSAData, Op->Header.Args[1]);
std::atomic<uint8_t> *MemData = *GetSrc<std::atomic<uint8_t> **>(Data->SSAData, Op->Addr);
uint8_t Src = *GetSrc<uint8_t*>(Data->SSAData, Op->Value);
uint8_t Previous = MemData->fetch_xor(Src);
GD = Previous;
break;
}
case 2: {
std::atomic<uint16_t> *MemData = *GetSrc<std::atomic<uint16_t> **>(Data->SSAData, Op->Header.Args[0]);
uint16_t Src = *GetSrc<uint16_t*>(Data->SSAData, Op->Header.Args[1]);
std::atomic<uint16_t> *MemData = *GetSrc<std::atomic<uint16_t> **>(Data->SSAData, Op->Addr);
uint16_t Src = *GetSrc<uint16_t*>(Data->SSAData, Op->Value);
uint16_t Previous = MemData->fetch_xor(Src);
GD = Previous;
break;
}
case 4: {
std::atomic<uint32_t> *MemData = *GetSrc<std::atomic<uint32_t> **>(Data->SSAData, Op->Header.Args[0]);
uint32_t Src = *GetSrc<uint32_t*>(Data->SSAData, Op->Header.Args[1]);
std::atomic<uint32_t> *MemData = *GetSrc<std::atomic<uint32_t> **>(Data->SSAData, Op->Addr);
uint32_t Src = *GetSrc<uint32_t*>(Data->SSAData, Op->Value);
uint32_t Previous = MemData->fetch_xor(Src);
GD = Previous;
break;
}
case 8: {
std::atomic<uint64_t> *MemData = *GetSrc<std::atomic<uint64_t> **>(Data->SSAData, Op->Header.Args[0]);
uint64_t Src = *GetSrc<uint64_t*>(Data->SSAData, Op->Header.Args[1]);
std::atomic<uint64_t> *MemData = *GetSrc<std::atomic<uint64_t> **>(Data->SSAData, Op->Addr);
uint64_t Src = *GetSrc<uint64_t*>(Data->SSAData, Op->Value);
uint64_t Previous = MemData->fetch_xor(Src);
GD = Previous;
break;
@@ -751,22 +750,22 @@ DEF_OP(AtomicFetchNeg) {
switch (IROp->Size) {
case 1: {
using Type = uint8_t;
GD = AtomicFetchNeg(*GetSrc<Type**>(Data->SSAData, Op->Header.Args[0]));
GD = AtomicFetchNeg(*GetSrc<Type**>(Data->SSAData, Op->Addr));
break;
}
case 2: {
using Type = uint16_t;
GD = AtomicFetchNeg(*GetSrc<Type**>(Data->SSAData, Op->Header.Args[0]));
GD = AtomicFetchNeg(*GetSrc<Type**>(Data->SSAData, Op->Addr));
break;
}
case 4: {
using Type = uint32_t;
GD = AtomicFetchNeg(*GetSrc<Type**>(Data->SSAData, Op->Header.Args[0]));
GD = AtomicFetchNeg(*GetSrc<Type**>(Data->SSAData, Op->Addr));
break;
}
case 8: {
using Type = uint64_t;
GD = AtomicFetchNeg(*GetSrc<Type**>(Data->SSAData, Op->Header.Args[0]));
GD = AtomicFetchNeg(*GetSrc<Type**>(Data->SSAData, Op->Addr));
break;
}
default: LOGMAN_MSG_A_FMT("Unhandled Atomic size: {}", IROp->Size);
@@ -33,10 +33,6 @@ DEF_OP(GuestCallIndirect) {
LogMan::Msg::DFmt("Unimplemented");
}
DEF_OP(GuestReturn) {
LogMan::Msg::DFmt("Unimplemented");
}
DEF_OP(SignalReturn) {
SignalReturn(Data->State);
}
@@ -19,7 +19,7 @@ DEF_OP(VInsGPR) {
__uint128_t Src1 = *GetSrc<__uint128_t*>(Data->SSAData, Op->Header.Args[0]);
__uint128_t Src2 = *GetSrc<__uint128_t*>(Data->SSAData, Op->Header.Args[1]);
uint64_t Offset = Op->Index * Op->Header.ElementSize * 8;
uint64_t Offset = Op->DestIdx * Op->Header.ElementSize * 8;
__uint128_t Mask = (1ULL << (Op->Header.ElementSize * 8)) - 1;
if (Op->Header.ElementSize == 8) {
Mask = ~0ULL;
@@ -298,6 +298,63 @@ namespace AES {
}
}
namespace CRC32 {
// CRC32 per byte lookup table.
constexpr std::array<uint32_t, 256> CRC32CTable = []() consteval {
std::array<uint32_t, 256> Table{};
// Clang 11.x doesn't support bitreverse as a consteval
// constexpr uint32_t Polynomial = 0x1EDC6F41;
constexpr uint32_t PolynomialRev = 0x82F63B78; //__builtin_bitreverse32(Polynomial);
for (size_t Char = 0; Char < std::size(Table); ++Char) {
uint32_t CurrentChar = Char;
for (size_t i = 0; i < 8; ++i) {
if (CurrentChar & 1) {
CurrentChar = (CurrentChar >> 1) ^ PolynomialRev;
}
else {
CurrentChar >>= 1;
}
}
Table[Char] = CurrentChar;
}
return Table;
}();
uint32_t crc32cb(uint32_t Accumulator, uint8_t data) {
Accumulator = CRC32CTable[(uint8_t)Accumulator ^ data] ^ Accumulator >> 8;
return Accumulator;
}
uint32_t crc32ch(uint32_t Accumulator, uint16_t data) {
Accumulator = CRC32CTable[(uint8_t)Accumulator ^ ((data >> 0) & 0xFF)] ^ Accumulator >> 8;
Accumulator = CRC32CTable[(uint8_t)Accumulator ^ ((data >> 8) & 0xFF)] ^ Accumulator >> 8;
return Accumulator;
}
uint32_t crc32cw(uint32_t Accumulator, uint32_t data) {
Accumulator = CRC32CTable[(uint8_t)Accumulator ^ ((data >> 0) & 0xFF)] ^ Accumulator >> 8;
Accumulator = CRC32CTable[(uint8_t)Accumulator ^ ((data >> 8) & 0xFF)] ^ Accumulator >> 8;
Accumulator = CRC32CTable[(uint8_t)Accumulator ^ ((data >> 16) & 0xFF)] ^ Accumulator >> 8;
Accumulator = CRC32CTable[(uint8_t)Accumulator ^ ((data >> 24) & 0xFF)] ^ Accumulator >> 8;
return Accumulator;
}
uint32_t crc32cx(uint32_t Accumulator, uint64_t data) {
Accumulator = CRC32CTable[(uint8_t)Accumulator ^ ((data >> 0) & 0xFF)] ^ Accumulator >> 8;
Accumulator = CRC32CTable[(uint8_t)Accumulator ^ ((data >> 8) & 0xFF)] ^ Accumulator >> 8;
Accumulator = CRC32CTable[(uint8_t)Accumulator ^ ((data >> 16) & 0xFF)] ^ Accumulator >> 8;
Accumulator = CRC32CTable[(uint8_t)Accumulator ^ ((data >> 24) & 0xFF)] ^ Accumulator >> 8;
Accumulator = CRC32CTable[(uint8_t)Accumulator ^ ((data >> 32) & 0xFF)] ^ Accumulator >> 8;
Accumulator = CRC32CTable[(uint8_t)Accumulator ^ ((data >> 40) & 0xFF)] ^ Accumulator >> 8;
Accumulator = CRC32CTable[(uint8_t)Accumulator ^ ((data >> 48) & 0xFF)] ^ Accumulator >> 8;
Accumulator = CRC32CTable[(uint8_t)Accumulator ^ ((data >> 56) & 0xFF)] ^ Accumulator >> 8;
return Accumulator;
}
}
namespace FEXCore::CPU {
#define DEF_OP(x) void InterpreterOps::Op_##x(IR::IROp_Header *IROp, IROpData *Data, IR::NodeID Node)
@@ -429,6 +486,33 @@ DEF_OP(AESKeyGenAssist) {
memcpy(GDP, &Tmp, sizeof(Tmp));
}
DEF_OP(CRC32) {
auto Op = IROp->C<IR::IROp_CRC32>();
uint32_t Src1 = *GetSrc<uint32_t*>(Data->SSAData, Op->Src1);
uint8_t *Src2 = GetSrc<uint8_t*>(Data->SSAData, Op->Src2);
uint32_t Tmp{};
switch (Op->SrcSize) {
case 1:
Tmp = CRC32::crc32cb(Src1, *(uint8_t*)Src2);
break;
case 2:
Tmp = CRC32::crc32ch(Src1, *(uint16_t*)Src2);
break;
case 4:
Tmp = CRC32::crc32cw(Src1, *(uint32_t*)Src2);
break;
case 8:
Tmp = CRC32::crc32cx(Src1, *(uint64_t*)Src2);
break;
default:
LOGMAN_MSG_A_FMT("Unknown CRC32C size: {}", Op->SrcSize);
break;
}
memcpy(GDP, &Tmp, sizeof(Tmp));
}
#undef DEF_OP
} // namespace FEXCore::CPU
@@ -158,7 +158,7 @@ DEF_OP(F80CVTINT) {
DEF_OP(F80CVTTO) {
auto Op = IROp->C<IR::IROp_F80CVTTo>();
switch (Op->Size) {
switch (Op->SrcSize) {
case 4: {
float Src = *GetSrc<float *>(Data->SSAData, Op->Header.Args[0]);
X80SoftFloat Tmp = Src;
@@ -171,14 +171,14 @@ DEF_OP(F80CVTTO) {
memcpy(GDP, &Tmp, sizeof(X80SoftFloat));
break;
}
default: LogMan::Msg::DFmt("Unhandled size: {}", Op->Size);
default: LogMan::Msg::DFmt("Unhandled size: {}", Op->SrcSize);
}
}
DEF_OP(F80CVTTOINT) {
auto Op = IROp->C<IR::IROp_F80CVTToInt>();
switch (Op->Size) {
switch (Op->SrcSize) {
case 2: {
int16_t Src = *GetSrc<int16_t*>(Data->SSAData, Op->Header.Args[0]);
X80SoftFloat Tmp = Src;
@@ -191,7 +191,7 @@ DEF_OP(F80CVTTOINT) {
memcpy(GDP, &Tmp, sizeof(X80SoftFloat));
break;
}
default: LogMan::Msg::DFmt("Unhandled size: {}", Op->Size);
default: LogMan::Msg::DFmt("Unhandled size: {}", Op->SrcSize);
}
}
@@ -323,7 +323,7 @@ DEF_OP(F80BCDLOAD) {
DEF_OP(F80BCDSTORE) {
auto Op = IROp->C<IR::IROp_F80BCDStore>();
X80SoftFloat Src1 = *GetSrc<X80SoftFloat*>(Data->SSAData, Op->Header.Args[0]);
X80SoftFloat Src1 = X80SoftFloat::FRNDINT(*GetSrc<X80SoftFloat*>(Data->SSAData, Op->Header.Args[0]));
bool Negative = Src1.Sign;
// Clear the Sign bit
@@ -227,6 +227,8 @@ struct OpHandlers<IR::OP_F80BCDSTORE> {
static X80SoftFloat handle(X80SoftFloat Src1) {
bool Negative = Src1.Sign;
Src1 = X80SoftFloat::FRNDINT(Src1);
// Clear the Sign bit
Src1.Sign = 0;
@@ -11,7 +11,6 @@
#include <FEXCore/Utils/LogManager.h>
#include <memory>
#include <bits/types/stack_t.h>
#include <signal.h>
#include <stdint.h>
#include <unordered_map>
@@ -0,0 +1,272 @@
#include "FEXCore/Core/CoreState.h"
#include "Interface/Core/Interpreter/InterpreterOps.h"
#include "Interface/Core/Interpreter/F80Ops.h"
#include <cstddef>
#include <cstdint>
namespace FEXCore::CPU {
template<typename R, typename... Args>
static FallbackInfo GetFallbackInfo(R(*fn)(Args...), FEXCore::Core::FallbackHandlerIndex HandlerIndex) {
return {FABI_UNKNOWN, (void*)fn, HandlerIndex};
}
template<>
FallbackInfo GetFallbackInfo(X80SoftFloat(*fn)(float), FEXCore::Core::FallbackHandlerIndex HandlerIndex) {
return {FABI_F80_F32, (void*)fn, HandlerIndex};
}
template<>
FallbackInfo GetFallbackInfo(X80SoftFloat(*fn)(double), FEXCore::Core::FallbackHandlerIndex HandlerIndex) {
return {FABI_F80_F64, (void*)fn, HandlerIndex};
}
template<>
FallbackInfo GetFallbackInfo(X80SoftFloat(*fn)(int16_t), FEXCore::Core::FallbackHandlerIndex HandlerIndex) {
return {FABI_F80_I16, (void*)fn, HandlerIndex};
}
template<>
FallbackInfo GetFallbackInfo(void(*fn)(uint16_t), FEXCore::Core::FallbackHandlerIndex HandlerIndex) {
return {FABI_VOID_U16, (void*)fn, HandlerIndex};
}
template<>
FallbackInfo GetFallbackInfo(X80SoftFloat(*fn)(int32_t), FEXCore::Core::FallbackHandlerIndex HandlerIndex) {
return {FABI_F80_I32, (void*)fn, HandlerIndex};
}
template<>
FallbackInfo GetFallbackInfo(float(*fn)(X80SoftFloat), FEXCore::Core::FallbackHandlerIndex HandlerIndex) {
return {FABI_F32_F80, (void*)fn, HandlerIndex};
}
template<>
FallbackInfo GetFallbackInfo(double(*fn)(X80SoftFloat), FEXCore::Core::FallbackHandlerIndex HandlerIndex) {
return {FABI_F64_F80, (void*)fn, HandlerIndex};
}
template<>
FallbackInfo GetFallbackInfo(int16_t(*fn)(X80SoftFloat), FEXCore::Core::FallbackHandlerIndex HandlerIndex) {
return {FABI_I16_F80, (void*)fn, HandlerIndex};
}
template<>
FallbackInfo GetFallbackInfo(int32_t(*fn)(X80SoftFloat), FEXCore::Core::FallbackHandlerIndex HandlerIndex) {
return {FABI_I32_F80, (void*)fn, HandlerIndex};
}
template<>
FallbackInfo GetFallbackInfo(int64_t(*fn)(X80SoftFloat), FEXCore::Core::FallbackHandlerIndex HandlerIndex) {
return {FABI_I64_F80, (void*)fn, HandlerIndex};
}
template<>
FallbackInfo GetFallbackInfo(uint64_t(*fn)(X80SoftFloat, X80SoftFloat), FEXCore::Core::FallbackHandlerIndex HandlerIndex) {
return {FABI_I64_F80_F80, (void*)fn, HandlerIndex};
}
template<>
FallbackInfo GetFallbackInfo(X80SoftFloat(*fn)(X80SoftFloat), FEXCore::Core::FallbackHandlerIndex HandlerIndex) {
return {FABI_F80_F80, (void*)fn, HandlerIndex};
}
template<>
FallbackInfo GetFallbackInfo(X80SoftFloat(*fn)(X80SoftFloat, X80SoftFloat), FEXCore::Core::FallbackHandlerIndex HandlerIndex) {
return {FABI_F80_F80_F80, (void*)fn, HandlerIndex};
}
void InterpreterOps::FillFallbackIndexPointers(uint64_t *Info) {
Info[Core::OPINDEX_F80LOADFCW] = reinterpret_cast<uint64_t>(GetFallbackInfo(&FEXCore::CPU::OpHandlers<IR::OP_F80LOADFCW>::handle, Core::OPINDEX_F80LOADFCW).fn);
Info[Core::OPINDEX_F80CVTTO_4] = reinterpret_cast<uint64_t>(GetFallbackInfo(&FEXCore::CPU::OpHandlers<IR::OP_F80CVTTO>::handle4, Core::OPINDEX_F80CVTTO_4).fn);
Info[Core::OPINDEX_F80CVTTO_8] = reinterpret_cast<uint64_t>(GetFallbackInfo(&FEXCore::CPU::OpHandlers<IR::OP_F80CVTTO>::handle8, Core::OPINDEX_F80CVTTO_8).fn);
Info[Core::OPINDEX_F80CVT_4] = reinterpret_cast<uint64_t>(GetFallbackInfo(&FEXCore::CPU::OpHandlers<IR::OP_F80CVT>::handle4, Core::OPINDEX_F80CVT_4).fn);
Info[Core::OPINDEX_F80CVT_8] = reinterpret_cast<uint64_t>(GetFallbackInfo(&FEXCore::CPU::OpHandlers<IR::OP_F80CVT>::handle8, Core::OPINDEX_F80CVT_8).fn);
Info[Core::OPINDEX_F80CVTINT_2] = reinterpret_cast<uint64_t>(GetFallbackInfo(&FEXCore::CPU::OpHandlers<IR::OP_F80CVTINT>::handle2, Core::OPINDEX_F80CVTINT_2).fn);
Info[Core::OPINDEX_F80CVTINT_4] = reinterpret_cast<uint64_t>(GetFallbackInfo(&FEXCore::CPU::OpHandlers<IR::OP_F80CVTINT>::handle4, Core::OPINDEX_F80CVTINT_4).fn);
Info[Core::OPINDEX_F80CVTINT_8] = reinterpret_cast<uint64_t>(GetFallbackInfo(&FEXCore::CPU::OpHandlers<IR::OP_F80CVTINT>::handle8, Core::OPINDEX_F80CVTINT_8).fn);
Info[Core::OPINDEX_F80CVTINT_TRUNC2] = reinterpret_cast<uint64_t>(GetFallbackInfo(&FEXCore::CPU::OpHandlers<IR::OP_F80CVTINT>::handle2t, Core::OPINDEX_F80CVTINT_TRUNC2).fn);
Info[Core::OPINDEX_F80CVTINT_TRUNC4] = reinterpret_cast<uint64_t>(GetFallbackInfo(&FEXCore::CPU::OpHandlers<IR::OP_F80CVTINT>::handle4t, Core::OPINDEX_F80CVTINT_TRUNC4).fn);
Info[Core::OPINDEX_F80CVTINT_TRUNC8] = reinterpret_cast<uint64_t>(GetFallbackInfo(&FEXCore::CPU::OpHandlers<IR::OP_F80CVTINT>::handle8t, Core::OPINDEX_F80CVTINT_TRUNC8).fn);
Info[Core::OPINDEX_F80CMP_0] = reinterpret_cast<uint64_t>(GetFallbackInfo(&FEXCore::CPU::OpHandlers<IR::OP_F80CMP>::handle<0>, Core::OPINDEX_F80CMP_0).fn);
Info[Core::OPINDEX_F80CMP_1] = reinterpret_cast<uint64_t>(GetFallbackInfo(&FEXCore::CPU::OpHandlers<IR::OP_F80CMP>::handle<1>, Core::OPINDEX_F80CMP_1).fn);
Info[Core::OPINDEX_F80CMP_2] = reinterpret_cast<uint64_t>(GetFallbackInfo(&FEXCore::CPU::OpHandlers<IR::OP_F80CMP>::handle<2>, Core::OPINDEX_F80CMP_2).fn);
Info[Core::OPINDEX_F80CMP_3] = reinterpret_cast<uint64_t>(GetFallbackInfo(&FEXCore::CPU::OpHandlers<IR::OP_F80CMP>::handle<3>, Core::OPINDEX_F80CMP_3).fn);
Info[Core::OPINDEX_F80CMP_4] = reinterpret_cast<uint64_t>(GetFallbackInfo(&FEXCore::CPU::OpHandlers<IR::OP_F80CMP>::handle<4>, Core::OPINDEX_F80CMP_4).fn);
Info[Core::OPINDEX_F80CMP_5] = reinterpret_cast<uint64_t>(GetFallbackInfo(&FEXCore::CPU::OpHandlers<IR::OP_F80CMP>::handle<5>, Core::OPINDEX_F80CMP_5).fn);
Info[Core::OPINDEX_F80CMP_6] = reinterpret_cast<uint64_t>(GetFallbackInfo(&FEXCore::CPU::OpHandlers<IR::OP_F80CMP>::handle<6>, Core::OPINDEX_F80CMP_6).fn);
Info[Core::OPINDEX_F80CMP_7] = reinterpret_cast<uint64_t>(GetFallbackInfo(&FEXCore::CPU::OpHandlers<IR::OP_F80CMP>::handle<7>, Core::OPINDEX_F80CMP_7).fn);
Info[Core::OPINDEX_F80CVTTOINT_2] = reinterpret_cast<uint64_t>(GetFallbackInfo(&FEXCore::CPU::OpHandlers<IR::OP_F80CVTTOINT>::handle2, Core::OPINDEX_F80CVTTOINT_2).fn);
Info[Core::OPINDEX_F80CVTTOINT_4] = reinterpret_cast<uint64_t>(GetFallbackInfo(&FEXCore::CPU::OpHandlers<IR::OP_F80CVTTOINT>::handle4, Core::OPINDEX_F80CVTTOINT_4).fn);
// Unary
Info[Core::OPINDEX_F80ROUND] = reinterpret_cast<uint64_t>(GetFallbackInfo(&FEXCore::CPU::OpHandlers<IR::OP_F80ROUND>::handle, Core::OPINDEX_F80ROUND).fn);
Info[Core::OPINDEX_F80F2XM1] = reinterpret_cast<uint64_t>(GetFallbackInfo(&FEXCore::CPU::OpHandlers<IR::OP_F80F2XM1>::handle, Core::OPINDEX_F80F2XM1).fn);
Info[Core::OPINDEX_F80TAN] = reinterpret_cast<uint64_t>(GetFallbackInfo(&FEXCore::CPU::OpHandlers<IR::OP_F80TAN>::handle, Core::OPINDEX_F80TAN).fn);
Info[Core::OPINDEX_F80SQRT] = reinterpret_cast<uint64_t>(GetFallbackInfo(&FEXCore::CPU::OpHandlers<IR::OP_F80SQRT>::handle, Core::OPINDEX_F80SQRT).fn);
Info[Core::OPINDEX_F80SIN] = reinterpret_cast<uint64_t>(GetFallbackInfo(&FEXCore::CPU::OpHandlers<IR::OP_F80SIN>::handle, Core::OPINDEX_F80SIN).fn);
Info[Core::OPINDEX_F80COS] = reinterpret_cast<uint64_t>(GetFallbackInfo(&FEXCore::CPU::OpHandlers<IR::OP_F80COS>::handle, Core::OPINDEX_F80COS).fn);
Info[Core::OPINDEX_F80XTRACT_EXP] = reinterpret_cast<uint64_t>(GetFallbackInfo(&FEXCore::CPU::OpHandlers<IR::OP_F80XTRACT_EXP>::handle, Core::OPINDEX_F80XTRACT_EXP).fn);
Info[Core::OPINDEX_F80XTRACT_SIG] = reinterpret_cast<uint64_t>(GetFallbackInfo(&FEXCore::CPU::OpHandlers<IR::OP_F80XTRACT_SIG>::handle, Core::OPINDEX_F80XTRACT_SIG).fn);
Info[Core::OPINDEX_F80BCDSTORE] = reinterpret_cast<uint64_t>(GetFallbackInfo(&FEXCore::CPU::OpHandlers<IR::OP_F80BCDSTORE>::handle, Core::OPINDEX_F80BCDSTORE).fn);
Info[Core::OPINDEX_F80BCDLOAD] = reinterpret_cast<uint64_t>(GetFallbackInfo(&FEXCore::CPU::OpHandlers<IR::OP_F80BCDLOAD>::handle, Core::OPINDEX_F80BCDLOAD).fn);
// Binary
Info[Core::OPINDEX_F80ADD] = reinterpret_cast<uint64_t>(GetFallbackInfo(&FEXCore::CPU::OpHandlers<IR::OP_F80ADD>::handle, Core::OPINDEX_F80ADD).fn);
Info[Core::OPINDEX_F80SUB] = reinterpret_cast<uint64_t>(GetFallbackInfo(&FEXCore::CPU::OpHandlers<IR::OP_F80SUB>::handle, Core::OPINDEX_F80SUB).fn);
Info[Core::OPINDEX_F80MUL] = reinterpret_cast<uint64_t>(GetFallbackInfo(&FEXCore::CPU::OpHandlers<IR::OP_F80MUL>::handle, Core::OPINDEX_F80MUL).fn);
Info[Core::OPINDEX_F80DIV] = reinterpret_cast<uint64_t>(GetFallbackInfo(&FEXCore::CPU::OpHandlers<IR::OP_F80DIV>::handle, Core::OPINDEX_F80DIV).fn);
Info[Core::OPINDEX_F80FYL2X] = reinterpret_cast<uint64_t>(GetFallbackInfo(&FEXCore::CPU::OpHandlers<IR::OP_F80FYL2X>::handle, Core::OPINDEX_F80FYL2X).fn);
Info[Core::OPINDEX_F80ATAN] = reinterpret_cast<uint64_t>(GetFallbackInfo(&FEXCore::CPU::OpHandlers<IR::OP_F80ATAN>::handle, Core::OPINDEX_F80ATAN).fn);
Info[Core::OPINDEX_F80FPREM1] = reinterpret_cast<uint64_t>(GetFallbackInfo(&FEXCore::CPU::OpHandlers<IR::OP_F80FPREM1>::handle, Core::OPINDEX_F80FPREM1).fn);
Info[Core::OPINDEX_F80FPREM] = reinterpret_cast<uint64_t>(GetFallbackInfo(&FEXCore::CPU::OpHandlers<IR::OP_F80FPREM>::handle, Core::OPINDEX_F80FPREM).fn);
Info[Core::OPINDEX_F80SCALE] = reinterpret_cast<uint64_t>(GetFallbackInfo(&FEXCore::CPU::OpHandlers<IR::OP_F80SCALE>::handle, Core::OPINDEX_F80SCALE).fn);
}
bool InterpreterOps::GetFallbackHandler(IR::IROp_Header *IROp, FallbackInfo *Info) {
uint8_t OpSize = IROp->Size;
switch(IROp->Op) {
case IR::OP_F80LOADFCW: {
*Info = GetFallbackInfo(&FEXCore::CPU::OpHandlers<IR::OP_F80LOADFCW>::handle, Core::OPINDEX_F80LOADFCW);
return true;
}
case IR::OP_F80CVTTO: {
auto Op = IROp->C<IR::IROp_F80CVTTo>();
switch (Op->SrcSize) {
case 4: {
*Info = GetFallbackInfo(&FEXCore::CPU::OpHandlers<IR::OP_F80CVTTO>::handle4, Core::OPINDEX_F80CVTTO_4);
return true;
}
case 8: {
*Info = GetFallbackInfo(&FEXCore::CPU::OpHandlers<IR::OP_F80CVTTO>::handle8, Core::OPINDEX_F80CVTTO_8);
return true;
}
default: LogMan::Msg::DFmt("Unhandled size: {}", OpSize);
}
break;
}
case IR::OP_F80CVT: {
switch (OpSize) {
case 4: {
*Info = GetFallbackInfo(&FEXCore::CPU::OpHandlers<IR::OP_F80CVT>::handle4, Core::OPINDEX_F80CVT_4);
return true;
}
case 8: {
*Info = GetFallbackInfo(&FEXCore::CPU::OpHandlers<IR::OP_F80CVT>::handle8, Core::OPINDEX_F80CVT_8);
return true;
}
default: LogMan::Msg::DFmt("Unhandled size: {}", OpSize);
}
break;
}
case IR::OP_F80CVTINT: {
auto Op = IROp->C<IR::IROp_F80CVTInt>();
switch (OpSize) {
case 2: {
if (Op->Truncate) {
*Info = GetFallbackInfo(&FEXCore::CPU::OpHandlers<IR::OP_F80CVTINT>::handle2t, Core::OPINDEX_F80CVTINT_TRUNC2);
}
else {
*Info = GetFallbackInfo(&FEXCore::CPU::OpHandlers<IR::OP_F80CVTINT>::handle2, Core::OPINDEX_F80CVTINT_2);
}
return true;
}
case 4: {
if (Op->Truncate) {
*Info = GetFallbackInfo(&FEXCore::CPU::OpHandlers<IR::OP_F80CVTINT>::handle4t, Core::OPINDEX_F80CVTINT_TRUNC4);
}
else {
*Info = GetFallbackInfo(&FEXCore::CPU::OpHandlers<IR::OP_F80CVTINT>::handle4, Core::OPINDEX_F80CVTINT_4);
}
return true;
}
case 8: {
if (Op->Truncate) {
*Info = GetFallbackInfo(&FEXCore::CPU::OpHandlers<IR::OP_F80CVTINT>::handle8t, Core::OPINDEX_F80CVTINT_TRUNC8);
}
else {
*Info = GetFallbackInfo(&FEXCore::CPU::OpHandlers<IR::OP_F80CVTINT>::handle8, Core::OPINDEX_F80CVTINT_8);
}
return true;
}
default: LogMan::Msg::DFmt("Unhandled size: {}", OpSize);
}
break;
}
case IR::OP_F80CMP: {
auto Op = IROp->C<IR::IROp_F80Cmp>();
static constexpr std::array handlers{
&FEXCore::CPU::OpHandlers<IR::OP_F80CMP>::handle<0>,
&FEXCore::CPU::OpHandlers<IR::OP_F80CMP>::handle<1>,
&FEXCore::CPU::OpHandlers<IR::OP_F80CMP>::handle<2>,
&FEXCore::CPU::OpHandlers<IR::OP_F80CMP>::handle<3>,
&FEXCore::CPU::OpHandlers<IR::OP_F80CMP>::handle<4>,
&FEXCore::CPU::OpHandlers<IR::OP_F80CMP>::handle<5>,
&FEXCore::CPU::OpHandlers<IR::OP_F80CMP>::handle<6>,
&FEXCore::CPU::OpHandlers<IR::OP_F80CMP>::handle<7>,
};
*Info = GetFallbackInfo(handlers[Op->Flags], (Core::FallbackHandlerIndex)(Core::OPINDEX_F80CMP_0 + Op->Flags));
return true;
}
case IR::OP_F80CVTTOINT: {
auto Op = IROp->C<IR::IROp_F80CVTToInt>();
switch (Op->SrcSize) {
case 2: {
*Info = GetFallbackInfo(&FEXCore::CPU::OpHandlers<IR::OP_F80CVTTOINT>::handle2, Core::OPINDEX_F80CVTTOINT_2);
return true;
}
case 4: {
*Info = GetFallbackInfo(&FEXCore::CPU::OpHandlers<IR::OP_F80CVTTOINT>::handle4, Core::OPINDEX_F80CVTTOINT_4);
return true;
}
default: LogMan::Msg::DFmt("Unhandled size: {}", OpSize);
}
break;
}
#define COMMON_X87_OP(OP) \
case IR::OP_F80##OP: { \
*Info = GetFallbackInfo(&FEXCore::CPU::OpHandlers<IR::OP_F80##OP>::handle, Core::OPINDEX_F80##OP); \
return true; \
}
// Unary
COMMON_X87_OP(ROUND)
COMMON_X87_OP(F2XM1)
COMMON_X87_OP(TAN)
COMMON_X87_OP(SQRT)
COMMON_X87_OP(SIN)
COMMON_X87_OP(COS)
COMMON_X87_OP(XTRACT_EXP)
COMMON_X87_OP(XTRACT_SIG)
COMMON_X87_OP(BCDSTORE)
COMMON_X87_OP(BCDLOAD)
// Binary
COMMON_X87_OP(ADD)
COMMON_X87_OP(SUB)
COMMON_X87_OP(MUL)
COMMON_X87_OP(DIV)
COMMON_X87_OP(FYL2X)
COMMON_X87_OP(ATAN)
COMMON_X87_OP(FPREM1)
COMMON_X87_OP(FPREM)
COMMON_X87_OP(SCALE)
default:
break;
}
return false;
}
}
@@ -73,6 +73,8 @@ constexpr OpHandlerArray InterpreterOpHandlers = [] {
REGISTER_OP(ASHR, Ashr);
REGISTER_OP(ROR, Ror);
REGISTER_OP(EXTR, Extr);
REGISTER_OP(PDEP, PDep);
REGISTER_OP(PEXT, PExt);
REGISTER_OP(LDIV, LDiv);
REGISTER_OP(LUDIV, LUDiv);
REGISTER_OP(LREM, LRem);
@@ -112,7 +114,6 @@ constexpr OpHandlerArray InterpreterOpHandlers = [] {
// Branch ops
REGISTER_OP(GUESTCALLDIRECT, GuestCallDirect);
REGISTER_OP(GUESTCALLINDIRECT, GuestCallIndirect);
REGISTER_OP(GUESTRETURN, GuestReturn);
REGISTER_OP(SIGNALRETURN, SignalReturn);
REGISTER_OP(CALLBACKRETURN, CallbackReturn);
REGISTER_OP(EXITFUNCTION, ExitFunction);
@@ -157,6 +158,7 @@ constexpr OpHandlerArray InterpreterOpHandlers = [] {
REGISTER_OP(VLOADMEMELEMENT, VLoadMemElement);
REGISTER_OP(VSTOREMEMELEMENT, VStoreMemElement);
REGISTER_OP(CACHELINECLEAR, CacheLineClear);
REGISTER_OP(CACHELINEZERO, CacheLineZero);
// Misc ops
REGISTER_OP(DUMMY, NoOp);
@@ -172,6 +174,8 @@ constexpr OpHandlerArray InterpreterOpHandlers = [] {
REGISTER_OP(GETROUNDINGMODE, GetRoundingMode);
REGISTER_OP(SETROUNDINGMODE, SetRoundingMode);
REGISTER_OP(INVALIDATEFLAGS, NoOp);
REGISTER_OP(PROCESSORID, ProcessorID);
REGISTER_OP(RDRAND, RDRAND);
// Move ops
REGISTER_OP(EXTRACTELEMENTPAIR, ExtractElementPair);
@@ -181,8 +185,6 @@ constexpr OpHandlerArray InterpreterOpHandlers = [] {
// Vector ops
REGISTER_OP(VECTORZERO, VectorZero);
REGISTER_OP(VECTORIMM, VectorImm);
REGISTER_OP(CREATEVECTOR2, CreateVector2);
REGISTER_OP(CREATEVECTOR4, CreateVector4);
REGISTER_OP(SPLATVECTOR2, SplatVector);
REGISTER_OP(SPLATVECTOR4, SplatVector);
REGISTER_OP(VMOV, VMov);
@@ -271,6 +273,7 @@ constexpr OpHandlerArray InterpreterOpHandlers = [] {
REGISTER_OP(VSMULL2, VSMull2);
REGISTER_OP(VUABDL, VUABDL);
REGISTER_OP(VTBL1, VTBL1);
REGISTER_OP(VREV64, VRev64);
// Encryption ops
REGISTER_OP(VAESIMC, AESImc);
@@ -279,6 +282,7 @@ constexpr OpHandlerArray InterpreterOpHandlers = [] {
REGISTER_OP(VAESDEC, AESDec);
REGISTER_OP(VAESDECLAST, AESDecLast);
REGISTER_OP(VAESKEYGENASSIST, AESKeyGenAssist);
REGISTER_OP(CRC32, CRC32);
// F80 ops
REGISTER_OP(F80LOADFCW, F80LOADFCW);
@@ -317,205 +321,6 @@ void InterpreterOps::Op_Unhandled(FEXCore::IR::IROp_Header *IROp, IROpData *Data
void InterpreterOps::Op_NoOp(FEXCore::IR::IROp_Header *IROp, IROpData *Data, IR::NodeID Node) {
}
template<typename R, typename... Args>
static FallbackInfo GetFallbackInfo(R(*fn)(Args...)) {
return {FABI_UNKNOWN, (void*)fn};
}
template<>
FallbackInfo GetFallbackInfo(X80SoftFloat(*fn)(float)) {
return {FABI_F80_F32, (void*)fn};
}
template<>
FallbackInfo GetFallbackInfo(X80SoftFloat(*fn)(double)) {
return {FABI_F80_F64, (void*)fn};
}
template<>
FallbackInfo GetFallbackInfo(X80SoftFloat(*fn)(int16_t)) {
return {FABI_F80_I16, (void*)fn};
}
template<>
FallbackInfo GetFallbackInfo(void(*fn)(uint16_t)) {
return {FABI_VOID_U16, (void*)fn};
}
template<>
FallbackInfo GetFallbackInfo(X80SoftFloat(*fn)(int32_t)) {
return {FABI_F80_I32, (void*)fn};
}
template<>
FallbackInfo GetFallbackInfo(float(*fn)(X80SoftFloat)) {
return {FABI_F32_F80, (void*)fn};
}
template<>
FallbackInfo GetFallbackInfo(double(*fn)(X80SoftFloat)) {
return {FABI_F64_F80, (void*)fn};
}
template<>
FallbackInfo GetFallbackInfo(int16_t(*fn)(X80SoftFloat)) {
return {FABI_I16_F80, (void*)fn};
}
template<>
FallbackInfo GetFallbackInfo(int32_t(*fn)(X80SoftFloat)) {
return {FABI_I32_F80, (void*)fn};
}
template<>
FallbackInfo GetFallbackInfo(int64_t(*fn)(X80SoftFloat)) {
return {FABI_I64_F80, (void*)fn};
}
template<>
FallbackInfo GetFallbackInfo(uint64_t(*fn)(X80SoftFloat, X80SoftFloat)) {
return {FABI_I64_F80_F80, (void*)fn};
}
template<>
FallbackInfo GetFallbackInfo(X80SoftFloat(*fn)(X80SoftFloat)) {
return {FABI_F80_F80, (void*)fn};
}
template<>
FallbackInfo GetFallbackInfo(X80SoftFloat(*fn)(X80SoftFloat, X80SoftFloat)) {
return {FABI_F80_F80_F80, (void*)fn};
}
bool InterpreterOps::GetFallbackHandler(IR::IROp_Header *IROp, FallbackInfo *Info) {
uint8_t OpSize = IROp->Size;
switch(IROp->Op) {
case IR::OP_F80LOADFCW: {
*Info = GetFallbackInfo(&FEXCore::CPU::OpHandlers<IR::OP_F80LOADFCW>::handle);
return true;
}
case IR::OP_F80CVTTO: {
auto Op = IROp->C<IR::IROp_F80CVTTo>();
switch (Op->Size) {
case 4: {
*Info = GetFallbackInfo(&FEXCore::CPU::OpHandlers<IR::OP_F80CVTTO>::handle4);
return true;
}
case 8: {
*Info = GetFallbackInfo(&FEXCore::CPU::OpHandlers<IR::OP_F80CVTTO>::handle8);
return true;
}
default: LogMan::Msg::DFmt("Unhandled size: {}", OpSize);
}
break;
}
case IR::OP_F80CVT: {
switch (OpSize) {
case 4: {
*Info = GetFallbackInfo(&FEXCore::CPU::OpHandlers<IR::OP_F80CVT>::handle4);
return true;
}
case 8: {
*Info = GetFallbackInfo(&FEXCore::CPU::OpHandlers<IR::OP_F80CVT>::handle8);
return true;
}
default: LogMan::Msg::DFmt("Unhandled size: {}", OpSize);
}
break;
}
case IR::OP_F80CVTINT: {
auto Op = IROp->C<IR::IROp_F80CVTInt>();
switch (OpSize) {
case 2: {
*Info = GetFallbackInfo(Op->Truncate ? &FEXCore::CPU::OpHandlers<IR::OP_F80CVTINT>::handle2t : &FEXCore::CPU::OpHandlers<IR::OP_F80CVTINT>::handle2);
return true;
}
case 4: {
*Info = GetFallbackInfo(Op->Truncate ? &FEXCore::CPU::OpHandlers<IR::OP_F80CVTINT>::handle4t : &FEXCore::CPU::OpHandlers<IR::OP_F80CVTINT>::handle4);
return true;
}
case 8: {
*Info = GetFallbackInfo(Op->Truncate ? &FEXCore::CPU::OpHandlers<IR::OP_F80CVTINT>::handle8t : &FEXCore::CPU::OpHandlers<IR::OP_F80CVTINT>::handle8);
return true;
}
default: LogMan::Msg::DFmt("Unhandled size: {}", OpSize);
}
break;
}
case IR::OP_F80CMP: {
auto Op = IROp->C<IR::IROp_F80Cmp>();
static constexpr std::array handlers{
&FEXCore::CPU::OpHandlers<IR::OP_F80CMP>::handle<0>,
&FEXCore::CPU::OpHandlers<IR::OP_F80CMP>::handle<1>,
&FEXCore::CPU::OpHandlers<IR::OP_F80CMP>::handle<2>,
&FEXCore::CPU::OpHandlers<IR::OP_F80CMP>::handle<3>,
&FEXCore::CPU::OpHandlers<IR::OP_F80CMP>::handle<4>,
&FEXCore::CPU::OpHandlers<IR::OP_F80CMP>::handle<5>,
&FEXCore::CPU::OpHandlers<IR::OP_F80CMP>::handle<6>,
&FEXCore::CPU::OpHandlers<IR::OP_F80CMP>::handle<7>,
};
*Info = GetFallbackInfo(handlers[Op->Flags]);
return true;
}
case IR::OP_F80CVTTOINT: {
auto Op = IROp->C<IR::IROp_F80CVTToInt>();
switch (Op->Size) {
case 2: {
*Info = GetFallbackInfo(&FEXCore::CPU::OpHandlers<IR::OP_F80CVTTOINT>::handle2);
return true;
}
case 4: {
*Info = GetFallbackInfo(&FEXCore::CPU::OpHandlers<IR::OP_F80CVTTOINT>::handle4);
return true;
}
default: LogMan::Msg::DFmt("Unhandled size: {}", OpSize);
}
break;
}
#define COMMON_X87_OP(OP) \
case IR::OP_F80##OP: { \
*Info = GetFallbackInfo(&FEXCore::CPU::OpHandlers<IR::OP_F80##OP>::handle); \
return true; \
}
// Unary
COMMON_X87_OP(ROUND)
COMMON_X87_OP(F2XM1)
COMMON_X87_OP(TAN)
COMMON_X87_OP(SQRT)
COMMON_X87_OP(SIN)
COMMON_X87_OP(COS)
COMMON_X87_OP(XTRACT_EXP)
COMMON_X87_OP(XTRACT_SIG)
COMMON_X87_OP(BCDSTORE)
COMMON_X87_OP(BCDLOAD)
// Binary
COMMON_X87_OP(ADD)
COMMON_X87_OP(SUB)
COMMON_X87_OP(MUL)
COMMON_X87_OP(DIV)
COMMON_X87_OP(FYL2X)
COMMON_X87_OP(ATAN)
COMMON_X87_OP(FPREM1)
COMMON_X87_OP(FPREM)
COMMON_X87_OP(SCALE)
default:
break;
}
return false;
}
void InterpreterOps::InterpretIR(FEXCore::Core::InternalThreadState *Thread, uint64_t Entry, FEXCore::IR::IRListView *CurrentIR, FEXCore::Core::DebugData *DebugData) {
volatile void *StackEntry = alloca(0);
@@ -1,6 +1,7 @@
#pragma once
#include <stdint.h>
#include <FEXCore/Core/CoreState.h>
#include <FEXCore/IR/IR.h>
#include <FEXCore/IR/IntrusiveIRList.h>
@@ -38,12 +39,14 @@ namespace FEXCore::CPU {
struct FallbackInfo {
FallbackABI ABI;
void *fn;
FEXCore::Core::FallbackHandlerIndex HandlerIndex;
};
class InterpreterOps {
public:
static void InterpretIR(FEXCore::Core::InternalThreadState *Thread, uint64_t Entry, FEXCore::IR::IRListView *CurrentIR, FEXCore::Core::DebugData *DebugData);
static void FillFallbackIndexPointers(uint64_t *Info);
static bool GetFallbackHandler(IR::IROp_Header *IROp, FallbackInfo *Info);
struct IROpData {
@@ -96,6 +99,8 @@ namespace FEXCore::CPU {
DEF_OP(Rol);
DEF_OP(Ror);
DEF_OP(Extr);
DEF_OP(PDep);
DEF_OP(PExt);
DEF_OP(LDiv);
DEF_OP(LUDiv);
DEF_OP(LRem);
@@ -137,7 +142,6 @@ namespace FEXCore::CPU {
///< Branch ops
DEF_OP(GuestCallDirect);
DEF_OP(GuestCallIndirect);
DEF_OP(GuestReturn);
DEF_OP(SignalReturn);
DEF_OP(CallbackReturn);
DEF_OP(ExitFunction);
@@ -180,6 +184,7 @@ namespace FEXCore::CPU {
DEF_OP(VLoadMemElement);
DEF_OP(VStoreMemElement);
DEF_OP(CacheLineClear);
DEF_OP(CacheLineZero);
///< Misc ops
DEF_OP(EndBlock);
@@ -190,6 +195,8 @@ namespace FEXCore::CPU {
DEF_OP(Print);
DEF_OP(GetRoundingMode);
DEF_OP(SetRoundingMode);
DEF_OP(ProcessorID);
DEF_OP(RDRAND);
///< Move ops
DEF_OP(ExtractElementPair);
@@ -199,8 +206,6 @@ namespace FEXCore::CPU {
///< Vector ops
DEF_OP(VectorZero);
DEF_OP(VectorImm);
DEF_OP(CreateVector2);
DEF_OP(CreateVector4);
DEF_OP(SplatVector);
DEF_OP(VMov);
DEF_OP(VAnd);
@@ -286,6 +291,7 @@ namespace FEXCore::CPU {
DEF_OP(VSMull2);
DEF_OP(VUABDL);
DEF_OP(VTBL1);
DEF_OP(VRev64);
///< Encryption ops
DEF_OP(AESImc);
@@ -294,6 +300,7 @@ namespace FEXCore::CPU {
DEF_OP(AESDec);
DEF_OP(AESDecLast);
DEF_OP(AESKeyGenAssist);
DEF_OP(CRC32);
///< F80 ops
DEF_OP(F80LOADFCW);
@@ -264,5 +264,22 @@ DEF_OP(CacheLineClear) {
CacheLineFlush(MemData);
}
DEF_OP(CacheLineZero) {
auto Op = IROp->C<IR::IROp_CacheLineZero>();
uintptr_t MemData = *GetSrc<uintptr_t*>(Data->SSAData, Op->Addr);
// Force cacheline alignment
MemData = MemData & ~(CPUIDEmu::CACHELINE_SIZE - 1);
using DataType = uint64_t;
DataType *MemData64 = reinterpret_cast<DataType*>(MemData);
// 64-byte cache line zero
for (size_t i = 0; i < (CPUIDEmu::CACHELINE_SIZE / sizeof(DataType)); ++i) {
MemData64[i] = 0;
}
}
#undef DEF_OP
} // namespace FEXCore::CPU
@@ -8,10 +8,13 @@ $end_info$
#include "Interface/Core/Interpreter/InterpreterOps.h"
#include "Interface/Core/Interpreter/InterpreterDefines.h"
#include <FEXHeaderUtils/Syscalls.h>
#include <cstdint>
#ifdef _M_X86_64
#include <xmmintrin.h>
#endif
#include <sys/random.h>
namespace FEXCore::CPU {
[[noreturn]]
@@ -46,7 +49,7 @@ DEF_OP(Break) {
StopThread(Data->State);
break;
case FEXCore::IR::Break_InvalidInstruction:
tgkill(Data->State->ThreadManager.PID, Data->State->ThreadManager.TID, SIGILL);
FHU::Syscalls::tgkill(Data->State->ThreadManager.PID, Data->State->ThreadManager.TID, SIGILL);
break;
default: LOGMAN_MSG_A_FMT("Unknown Break Reason: {}", Op->Reason); break;
}
@@ -140,6 +143,20 @@ DEF_OP(Print) {
LOGMAN_MSG_A_FMT("Unknown value size: {}", OpSize);
}
DEF_OP(ProcessorID) {
uint32_t CPU, CPUNode;
FHU::Syscalls::getcpu(&CPU, &CPUNode);
GD = (CPUNode << 12) | CPU;
}
DEF_OP(RDRAND) {
// We are ignoring Op->GetReseeded in the interpreter
uint64_t *DstPtr = GetDest<uint64_t*>(Data->SSAData, Node);
ssize_t Result = ::getrandom(&DstPtr[0], 8, 0);
// Second result is if we managed to read a valid random number or not
DstPtr[1] = Result == 8 ? 1 : 0;
}
#undef DEF_OP
} // namespace FEXCore::CPU
@@ -26,8 +26,8 @@ DEF_OP(CreateElementPair) {
uint8_t *Dst = GetDest<uint8_t*>(Data->SSAData, Node);
memcpy(Dst, Src_Lower, Op->Header.Size);
memcpy(Dst + Op->Header.Size, Src_Upper, Op->Header.Size);
memcpy(Dst, Src_Lower, IROp->ElementSize);
memcpy(Dst + IROp->ElementSize, Src_Upper, IROp->ElementSize);
}
DEF_OP(Mov) {
@@ -7,7 +7,9 @@ $end_info$
#include "Interface/Core/Interpreter/InterpreterClass.h"
#include "Interface/Core/Interpreter/InterpreterOps.h"
#include "Interface/Core/Interpreter/InterpreterDefines.h"
#include <FEXCore/Utils/BitUtils.h>
#include <bit>
#include <cstdint>
namespace FEXCore::CPU {
@@ -37,39 +39,6 @@ DEF_OP(VectorImm) {
memcpy(GDP, Tmp, OpSize);
}
DEF_OP(CreateVector2) {
auto Op = IROp->C<IR::IROp_CreateVector2>();
uint8_t OpSize = IROp->Size;
LOGMAN_THROW_A_FMT(OpSize <= 16, "Can't handle a vector of size: {}", OpSize);
void *Src1 = GetSrc<void*>(Data->SSAData, Op->Header.Args[0]);
void *Src2 = GetSrc<void*>(Data->SSAData, Op->Header.Args[1]);
uint8_t Tmp[16];
uint8_t ElementSize = OpSize / 2;
#define CREATE_VECTOR(elementsize, type) \
case elementsize: { \
auto *Dst_d = reinterpret_cast<type*>(Tmp); \
auto *Src1_d = reinterpret_cast<type*>(Src1); \
auto *Src2_d = reinterpret_cast<type*>(Src2); \
Dst_d[0] = *Src1_d; \
Dst_d[1] = *Src2_d; \
break; \
}
switch (ElementSize) {
CREATE_VECTOR(1, uint8_t)
CREATE_VECTOR(2, uint16_t)
CREATE_VECTOR(4, uint32_t)
CREATE_VECTOR(8, uint64_t)
default: LOGMAN_MSG_A_FMT("Unknown Element Size: {}", ElementSize); break;
}
#undef CREATE_VECTOR
memcpy(GDP, Tmp, OpSize);
}
DEF_OP(CreateVector4) {
LOGMAN_MSG_A_FMT("Unimplemented");
}
DEF_OP(SplatVector) {
auto Op = IROp->C<IR::IROp_SplatVector2>();
uint8_t OpSize = IROp->Size;
@@ -1401,10 +1370,9 @@ DEF_OP(VInsScalarElement) {
DEF_OP(VExtractElement) {
auto Op = IROp->C<IR::IROp_VExtractElement>();
uint8_t OpSize = IROp->Size;
uint32_t SourceSize = GetOpSize(Data->CurrentIR, Op->Header.Args[0]);
LOGMAN_THROW_A_FMT(OpSize <= 16, "OpSize is too large for VExtractElement: {}", OpSize);
LOGMAN_THROW_A_FMT(IROp->Size <= 16, "OpSize is too large for VExtractElement: {}", IROp->Size);
if (SourceSize == 16) {
__uint128_t SourceMask = (1ULL << (Op->Header.ElementSize * 8)) - 1;
uint64_t Shift = Op->Header.ElementSize * Op->Index * 8;
@@ -1930,6 +1898,39 @@ DEF_OP(VTBL1) {
memcpy(GDP, Tmp, OpSize);
}
DEF_OP(VRev64) {
auto Op = IROp->C<IR::IROp_VRev64>();
uint8_t OpSize = IROp->Size;
void *Src = GetSrc<void*>(Data->SSAData, Op->Header.Args[0]);
uint8_t Tmp[16];
uint8_t Elements = OpSize / 8;
// The element working size is always 64-bit
// The defined element size in the op is the operating size of the element swapping
auto Func8 = [](auto a) { return BSwap64(a); };
auto Func16 = [](auto a) {
return (a >> 48) | // Element[3] -> Element[0]
((a >> 16) & 0xFFFF'0000U) | // Element[2] -> Element[1]
((a << 16) & 0xFFFF'0000'0000ULL) | // Element[1] -> Element[2]
(a << 48); // Element[0] -> Element[3]
};
auto Func32 = [](auto a) {
return (a >> 32) | (a << 32);
};
switch (Op->Header.ElementSize) {
DO_VECTOR_1SRC_OP(1, uint64_t, Func8)
DO_VECTOR_1SRC_OP(2, uint64_t, Func16)
DO_VECTOR_1SRC_OP(4, uint64_t, Func32)
default: LOGMAN_MSG_A_FMT("Unknown Element Size: {}", Op->Header.ElementSize); break;
}
memcpy(GDP, Tmp, Op->Header.Size);
}
#undef DEF_OP
} // namespace FEXCore::CPU
+142 -36
View File
@@ -8,29 +8,9 @@ $end_info$
#include "Interface/IR/Passes/RegisterAllocationPass.h"
namespace FEXCore::CPU {
static uint64_t LUDIV(uint64_t SrcHigh, uint64_t SrcLow, uint64_t Divisor) {
__uint128_t Source = (static_cast<__uint128_t>(SrcHigh) << 64) | SrcLow;
__uint128_t Res = Source / Divisor;
return Res;
}
static int64_t LDIV(int64_t SrcHigh, int64_t SrcLow, int64_t Divisor) {
__int128_t Source = (static_cast<__int128_t>(SrcHigh) << 64) | SrcLow;
__int128_t Res = Source / Divisor;
return Res;
}
static uint64_t LUREM(uint64_t SrcHigh, uint64_t SrcLow, uint64_t Divisor) {
__uint128_t Source = (static_cast<__uint128_t>(SrcHigh) << 64) | SrcLow;
__uint128_t Res = Source % Divisor;
return Res;
}
static int64_t LREM(int64_t SrcHigh, int64_t SrcLow, int64_t Divisor) {
__int128_t Source = (static_cast<__int128_t>(SrcHigh) << 64) | SrcLow;
__int128_t Res = Source % Divisor;
return Res;
}
#define GRD(Node) (IROp->Size <= 4 ? GetDst<RA_32>(Node) : GetDst<RA_64>(Node))
#define GRS(Node) (IROp->Size <= 4 ? GetReg<RA_32>(Node) : GetReg<RA_64>(Node))
using namespace vixl;
using namespace vixl::aarch64;
@@ -38,7 +18,7 @@ using namespace vixl::aarch64;
DEF_OP(TruncElementPair) {
auto Op = IROp->C<IR::IROp_TruncElementPair>();
switch (Op->Size) {
switch (IROp->Size) {
case 4: {
auto Dst = GetSrcPair<RA_32>(Node);
auto Src = GetSrcPair<RA_32>(Op->Header.Args[0].ID());
@@ -46,7 +26,7 @@ DEF_OP(TruncElementPair) {
mov(Dst.second, Src.second);
break;
}
default: LOGMAN_MSG_A_FMT("Unhandled Truncation size: {}", Op->Size); break;
default: LOGMAN_MSG_A_FMT("Unhandled Truncation size: {}", IROp->Size); break;
}
}
@@ -61,7 +41,13 @@ DEF_OP(EntrypointOffset) {
auto Constant = Entry + Op->Offset;
auto Dst = GetReg<RA_64>(Node);
LoadConstant(Dst, Constant);
uint64_t Mask = ~0ULL;
uint8_t OpSize = IROp->Size;
if (OpSize == 4) {
Mask = 0xFFFF'FFFFULL;
}
LoadConstant(Dst, Constant & Mask);
}
DEF_OP(InlineConstant) {
@@ -80,8 +66,6 @@ DEF_OP(CycleCounter) {
#endif
}
#define GRS(Node) (IROp->Size <= 4 ? GetReg<RA_32>(Node) : GetReg<RA_64>(Node))
DEF_OP(Add) {
auto Op = IROp->C<IR::IROp_Add>();
uint8_t OpSize = IROp->Size;
@@ -511,6 +495,125 @@ DEF_OP(Extr) {
}
}
DEF_OP(PDep) {
auto Op = IROp->C<IR::IROp_PExt>();
const auto OpSize = IROp->Size;
const Register Input = GRS(Op->Args(0).ID());
const Register Mask = GRS(Op->Args(1).ID());
const Register Dest = GRS(Node);
const Register ShiftedBitReg = OpSize <= 4 ? TMP1.W() : TMP1;
const Register BitReg = OpSize <= 4 ? TMP2.W() : TMP2;
const Register SubMaskReg = OpSize <= 4 ? TMP3.W() : TMP3;
const Register IndexReg = OpSize <= 4 ? TMP4.W() : TMP4;
const Register SizedZero = OpSize <= 4 ? Register{wzr} : Register{xzr};
const Register InputReg = OpSize <= 4 ? SRA64[0].W() : SRA64[0];
const Register MaskReg = OpSize <= 4 ? SRA64[1].W() : SRA64[1];
const Register DestReg = OpSize <= 4 ? SRA64[2].W() : SRA64[2];
const auto SpillCode = 1U << InputReg.GetCode() |
1U << MaskReg.GetCode() |
1U << DestReg.GetCode();
aarch64::Label EarlyExit;
aarch64::Label NextBit;
aarch64::Label Done;
cbz(Mask, &EarlyExit);
mov(IndexReg, SizedZero);
// We sadly need to spill regs for this for the time being
// TODO: Remove when scratch registers can be allocated
// explicitly.
SpillStaticRegs(false, SpillCode);
mov(InputReg, Input);
mov(MaskReg, Mask);
mov(DestReg, SizedZero);
// Main loop
bind(&NextBit);
rbit(ShiftedBitReg, MaskReg);
clz(ShiftedBitReg, ShiftedBitReg);
lsrv(BitReg, InputReg, IndexReg);
and_(BitReg, BitReg, 1);
sub(SubMaskReg, MaskReg, 1);
add(IndexReg, IndexReg, 1);
ands(MaskReg, MaskReg, SubMaskReg);
lslv(ShiftedBitReg, BitReg, ShiftedBitReg);
orr(DestReg, DestReg, ShiftedBitReg);
b(&NextBit, Condition::ne);
// Store result in a temp so it doesn't get clobbered.
// and restore it after the re-fill below.
mov(IndexReg, DestReg);
// Restore our registers before leaving
// TODO: Also remove along with above TODO.
FillStaticRegs(false, SpillCode);
mov(Dest, IndexReg);
b(&Done);
// Early exit
bind(&EarlyExit);
mov(Dest, SizedZero);
// All done with nothing to do.
bind(&Done);
}
DEF_OP(PExt) {
auto Op = IROp->C<IR::IROp_PExt>();
const auto OpSize = IROp->Size;
const Register Input = GRS(Op->Args(0).ID());
const Register Mask = GRS(Op->Args(1).ID());
const Register Dest = GRS(Node);
const Register MaskReg = OpSize <= 4 ? TMP1.W() : TMP1;
const Register BitReg = OpSize <= 4 ? TMP2.W() : TMP2;
const Register SubMaskReg = OpSize <= 4 ? TMP3.W() : TMP3;
const Register Offset = OpSize <= 4 ? TMP4.W() : TMP4;
const Register SizedZero = OpSize <= 4 ? Register{wzr} : Register{xzr};
aarch64::Label EarlyExit;
aarch64::Label NextBit;
aarch64::Label Done;
cbz(Mask, &EarlyExit);
mov(MaskReg, Mask);
mov(Offset, SizedZero);
// We sadly need to spill a reg for this for the time being
// TODO: Remove when scratch registers can be allocated
// explicitly.
SpillStaticRegs(false, 1U << Mask.GetCode());
mov(Mask, SizedZero);
// Main loop
bind(&NextBit);
rbit(BitReg, MaskReg);
clz(BitReg, BitReg);
sub(SubMaskReg, MaskReg, 1);
ands(MaskReg, SubMaskReg, MaskReg);
lsrv(BitReg, Input, BitReg);
and_(BitReg, BitReg, 1);
lslv(BitReg, BitReg, Offset);
add(Offset, Offset, 1);
orr(Mask, BitReg, Mask);
b(&NextBit, Condition::ne);
mov(Dest, Mask);
// Restore our mask register before leaving
// TODO: Also remove along with above TODO.
FillStaticRegs(false, 1U << Mask.GetCode());
b(&Done);
// Early exit
bind(&EarlyExit);
mov(Dest, SizedZero);
// All done with nothing to do.
bind(&Done);
}
DEF_OP(LDiv) {
auto Op = IROp->C<IR::IROp_LDiv>();
uint8_t OpSize = IROp->Size;
@@ -539,7 +642,8 @@ DEF_OP(LDiv) {
mov(x1, GetReg<RA_64>(Op->Header.Args[0].ID()));
mov(x2, GetReg<RA_64>(Op->Header.Args[2].ID()));
LoadConstant(x3, reinterpret_cast<uint64_t>(LDIV));
ldr(x3, MemOperand(STATE, offsetof(FEXCore::Core::CpuStateFrame, Pointers.AArch64.LDIV)));
SpillStaticRegs();
blr(x3);
FillStaticRegs();
@@ -582,7 +686,7 @@ DEF_OP(LUDiv) {
mov(x1, GetReg<RA_64>(Op->Header.Args[0].ID()));
mov(x2, GetReg<RA_64>(Op->Header.Args[2].ID()));
LoadConstant(x3, reinterpret_cast<uint64_t>(LUDIV));
ldr(x3, MemOperand(STATE, offsetof(FEXCore::Core::CpuStateFrame, Pointers.AArch64.LUDIV)));
SpillStaticRegs();
blr(x3);
FillStaticRegs();
@@ -635,7 +739,7 @@ DEF_OP(LRem) {
mov(x1, GetReg<RA_64>(Op->Header.Args[0].ID()));
mov(x2, GetReg<RA_64>(Op->Header.Args[2].ID()));
LoadConstant(x3, reinterpret_cast<uint64_t>(LREM));
ldr(x3, MemOperand(STATE, offsetof(FEXCore::Core::CpuStateFrame, Pointers.AArch64.LREM)));
SpillStaticRegs();
blr(x3);
FillStaticRegs();
@@ -685,7 +789,7 @@ DEF_OP(LURem) {
mov(x1, GetReg<RA_64>(Op->Header.Args[0].ID()));
mov(x2, GetReg<RA_64>(Op->Header.Args[2].ID()));
LoadConstant(x3, reinterpret_cast<uint64_t>(LUREM));
ldr(x3, MemOperand(STATE, offsetof(FEXCore::Core::CpuStateFrame, Pointers.AArch64.LUREM)));
SpillStaticRegs();
blr(x3);
FillStaticRegs();
@@ -916,7 +1020,7 @@ Condition MapSelectCC(IR::CondClassType Cond) {
case FEXCore::IR::COND_FLU: return Condition::lt;
case FEXCore::IR::COND_FGE: return Condition::ge;
case FEXCore::IR::COND_FLEU:return Condition::le;
case FEXCore::IR::COND_FGT: return Condition::hi;
case FEXCore::IR::COND_FGT: return Condition::gt;
case FEXCore::IR::COND_FU: return Condition::vs;
case FEXCore::IR::COND_FNU: return Condition::vc;
case FEXCore::IR::COND_VS:
@@ -966,16 +1070,16 @@ DEF_OP(VExtractToGPR) {
uint8_t OpSize = IROp->Size;
switch (OpSize) {
case 1:
umov(GetReg<RA_32>(Node), GetSrc(Op->Header.Args[0].ID()).V16B(), Op->Idx);
umov(GetReg<RA_32>(Node), GetSrc(Op->Header.Args[0].ID()).V16B(), Op->Index);
break;
case 2:
umov(GetReg<RA_32>(Node), GetSrc(Op->Header.Args[0].ID()).V8H(), Op->Idx);
umov(GetReg<RA_32>(Node), GetSrc(Op->Header.Args[0].ID()).V8H(), Op->Index);
break;
case 4:
umov(GetReg<RA_32>(Node), GetSrc(Op->Header.Args[0].ID()).V4S(), Op->Idx);
umov(GetReg<RA_32>(Node), GetSrc(Op->Header.Args[0].ID()).V4S(), Op->Index);
break;
case 8:
umov(GetReg<RA_64>(Node), GetSrc(Op->Header.Args[0].ID()).V2D(), Op->Idx);
umov(GetReg<RA_64>(Node), GetSrc(Op->Header.Args[0].ID()).V2D(), Op->Index);
break;
default: LOGMAN_MSG_A_FMT("Unhandled ExtractElementSize: {}", OpSize);
}
@@ -1099,6 +1203,8 @@ void Arm64JITCore::RegisterALUHandlers() {
REGISTER_OP(ASHR, Ashr);
REGISTER_OP(ROR, Ror);
REGISTER_OP(EXTR, Extr);
REGISTER_OP(PDEP, PDep);
REGISTER_OP(PEXT, PExt);
REGISTER_OP(LDIV, LDiv);
REGISTER_OP(LUDIV, LUDiv);
REGISTER_OP(LREM, LRem);
+108 -112
View File
@@ -12,18 +12,17 @@ using namespace vixl::aarch64;
#define DEF_OP(x) void Arm64JITCore::Op_##x(IR::IROp_Header *IROp, IR::NodeID Node)
DEF_OP(CASPair) {
auto Op = IROp->C<IR::IROp_CASPair>();
uint8_t OpSize = IROp->Size;
// Size is the size of each pair element
auto Dst = GetSrcPair<RA_64>(Node);
auto Expected = GetSrcPair<RA_64>(Op->Header.Args[0].ID());
auto Desired = GetSrcPair<RA_64>(Op->Header.Args[1].ID());
auto MemSrc = GetReg<RA_64>(Op->Header.Args[2].ID());
auto Expected = GetSrcPair<RA_64>(Op->Expected.ID());
auto Desired = GetSrcPair<RA_64>(Op->Desired.ID());
auto MemSrc = GetReg<RA_64>(Op->Addr.ID());
if (SupportsAtomics) {
if (CTX->HostFeatures.SupportsAtomics) {
mov(TMP3, Expected.first);
mov(TMP4, Expected.second);
switch (OpSize) {
switch (IROp->ElementSize) {
case 4:
caspal(TMP3.W(), TMP4.W(), Desired.first.W(), Desired.second.W(), MemOperand(MemSrc));
mov(Dst.first.W(), TMP3.W());
@@ -34,11 +33,11 @@ DEF_OP(CASPair) {
mov(Dst.first, TMP3);
mov(Dst.second, TMP4);
break;
default: LOGMAN_MSG_A_FMT("Unsupported: {}", OpSize);
default: LOGMAN_MSG_A_FMT("Unsupported: {}", IROp->ElementSize);
}
}
else {
switch (OpSize) {
switch (IROp->ElementSize) {
case 4: {
aarch64::Label LoopTop;
aarch64::Label LoopNotExpected;
@@ -91,7 +90,7 @@ DEF_OP(CASPair) {
bind(&LoopExpected);
break;
}
default: LOGMAN_MSG_A_FMT("Unsupported: {}", OpSize);
default: LOGMAN_MSG_A_FMT("Unsupported: {}", IROp->ElementSize);
}
}
}
@@ -99,18 +98,15 @@ DEF_OP(CASPair) {
DEF_OP(CAS) {
auto Op = IROp->C<IR::IROp_CAS>();
uint8_t OpSize = IROp->Size;
// Args[0]: Expected
// Args[1]: Desired
// Args[2]: Pointer
// DataSrc = *Src1
// if (DataSrc == Src3) { *Src1 == Src2; } Src2 = DataSrc
// This will write to memory! Careful!
auto Expected = GetReg<RA_64>(Op->Header.Args[0].ID());
auto Desired = GetReg<RA_64>(Op->Header.Args[1].ID());
auto MemSrc = GetReg<RA_64>(Op->Header.Args[2].ID());
auto Expected = GetReg<RA_64>(Op->Expected.ID());
auto Desired = GetReg<RA_64>(Op->Desired.ID());
auto MemSrc = GetReg<RA_64>(Op->Addr.ID());
if (SupportsAtomics) {
if (CTX->HostFeatures.SupportsAtomics) {
mov(TMP2, Expected);
switch (OpSize) {
case 1: casalb(TMP2.W(), Desired.W(), MemOperand(MemSrc)); break;
@@ -216,14 +212,14 @@ DEF_OP(CAS) {
DEF_OP(AtomicAdd) {
auto Op = IROp->C<IR::IROp_AtomicAdd>();
auto MemSrc = GetReg<RA_64>(Op->Header.Args[0].ID());
auto MemSrc = GetReg<RA_64>(Op->Addr.ID());
if (SupportsAtomics) {
if (CTX->HostFeatures.SupportsAtomics) {
switch (IROp->Size) {
case 1: staddlb(GetReg<RA_32>(Op->Header.Args[1].ID()), MemOperand(MemSrc)); break;
case 2: staddlh(GetReg<RA_32>(Op->Header.Args[1].ID()), MemOperand(MemSrc)); break;
case 4: staddl(GetReg<RA_32>(Op->Header.Args[1].ID()), MemOperand(MemSrc)); break;
case 8: staddl(GetReg<RA_64>(Op->Header.Args[1].ID()), MemOperand(MemSrc)); break;
case 1: staddlb(GetReg<RA_32>(Op->Value.ID()), MemOperand(MemSrc)); break;
case 2: staddlh(GetReg<RA_32>(Op->Value.ID()), MemOperand(MemSrc)); break;
case 4: staddl(GetReg<RA_32>(Op->Value.ID()), MemOperand(MemSrc)); break;
case 8: staddl(GetReg<RA_64>(Op->Value.ID()), MemOperand(MemSrc)); break;
default: LOGMAN_MSG_A_FMT("Unhandled Atomic size: {}", IROp->Size);
}
}
@@ -234,7 +230,7 @@ DEF_OP(AtomicAdd) {
aarch64::Label LoopTop;
bind(&LoopTop);
ldaxrb(TMP2.W(), MemOperand(MemSrc));
add(TMP2.W(), TMP2.W(), GetReg<RA_32>(Op->Header.Args[1].ID()));
add(TMP2.W(), TMP2.W(), GetReg<RA_32>(Op->Value.ID()));
stlxrb(TMP2.W(), TMP2.W(), MemOperand(MemSrc));
cbnz(TMP2.W(), &LoopTop);
break;
@@ -243,7 +239,7 @@ DEF_OP(AtomicAdd) {
aarch64::Label LoopTop;
bind(&LoopTop);
ldaxrh(TMP2.W(), MemOperand(MemSrc));
add(TMP2.W(), TMP2.W(), GetReg<RA_32>(Op->Header.Args[1].ID()));
add(TMP2.W(), TMP2.W(), GetReg<RA_32>(Op->Value.ID()));
stlxrh(TMP2.W(), TMP2.W(), MemOperand(MemSrc));
cbnz(TMP2.W(), &LoopTop);
break;
@@ -252,7 +248,7 @@ DEF_OP(AtomicAdd) {
aarch64::Label LoopTop;
bind(&LoopTop);
ldaxr(TMP2.W(), MemOperand(MemSrc));
add(TMP2.W(), TMP2.W(), GetReg<RA_32>(Op->Header.Args[1].ID()));
add(TMP2.W(), TMP2.W(), GetReg<RA_32>(Op->Value.ID()));
stlxr(TMP2.W(), TMP2.W(), MemOperand(MemSrc));
cbnz(TMP2.W(), &LoopTop);
break;
@@ -261,7 +257,7 @@ DEF_OP(AtomicAdd) {
aarch64::Label LoopTop;
bind(&LoopTop);
ldaxr(TMP2, MemOperand(MemSrc));
add(TMP2, TMP2, GetReg<RA_64>(Op->Header.Args[1].ID()));
add(TMP2, TMP2, GetReg<RA_64>(Op->Value.ID()));
stlxr(TMP2, TMP2, MemOperand(MemSrc));
cbnz(TMP2, &LoopTop);
break;
@@ -274,10 +270,10 @@ DEF_OP(AtomicAdd) {
DEF_OP(AtomicSub) {
auto Op = IROp->C<IR::IROp_AtomicSub>();
auto MemSrc = GetReg<RA_64>(Op->Header.Args[0].ID());
auto MemSrc = GetReg<RA_64>(Op->Addr.ID());
if (SupportsAtomics) {
neg(TMP2, GetReg<RA_64>(Op->Header.Args[1].ID()));
if (CTX->HostFeatures.SupportsAtomics) {
neg(TMP2, GetReg<RA_64>(Op->Value.ID()));
switch (IROp->Size) {
case 1: staddlb(TMP2.W(), MemOperand(MemSrc)); break;
case 2: staddlh(TMP2.W(), MemOperand(MemSrc)); break;
@@ -293,7 +289,7 @@ DEF_OP(AtomicSub) {
aarch64::Label LoopTop;
bind(&LoopTop);
ldaxrb(TMP2.W(), MemOperand(MemSrc));
sub(TMP2.W(), TMP2.W(), GetReg<RA_32>(Op->Header.Args[1].ID()));
sub(TMP2.W(), TMP2.W(), GetReg<RA_32>(Op->Value.ID()));
stlxrb(TMP2.W(), TMP2.W(), MemOperand(MemSrc));
cbnz(TMP2.W(), &LoopTop);
break;
@@ -302,7 +298,7 @@ DEF_OP(AtomicSub) {
aarch64::Label LoopTop;
bind(&LoopTop);
ldaxrh(TMP2.W(), MemOperand(MemSrc));
sub(TMP2.W(), TMP2.W(), GetReg<RA_32>(Op->Header.Args[1].ID()));
sub(TMP2.W(), TMP2.W(), GetReg<RA_32>(Op->Value.ID()));
stlxrh(TMP2.W(), TMP2.W(), MemOperand(MemSrc));
cbnz(TMP2.W(), &LoopTop);
break;
@@ -311,7 +307,7 @@ DEF_OP(AtomicSub) {
aarch64::Label LoopTop;
bind(&LoopTop);
ldaxr(TMP2.W(), MemOperand(MemSrc));
sub(TMP2.W(), TMP2.W(), GetReg<RA_32>(Op->Header.Args[1].ID()));
sub(TMP2.W(), TMP2.W(), GetReg<RA_32>(Op->Value.ID()));
stlxr(TMP2.W(), TMP2.W(), MemOperand(MemSrc));
cbnz(TMP2.W(), &LoopTop);
break;
@@ -320,7 +316,7 @@ DEF_OP(AtomicSub) {
aarch64::Label LoopTop;
bind(&LoopTop);
ldaxr(TMP2, MemOperand(MemSrc));
sub(TMP2, TMP2, GetReg<RA_64>(Op->Header.Args[1].ID()));
sub(TMP2, TMP2, GetReg<RA_64>(Op->Value.ID()));
stlxr(TMP2, TMP2, MemOperand(MemSrc));
cbnz(TMP2, &LoopTop);
break;
@@ -333,10 +329,10 @@ DEF_OP(AtomicSub) {
DEF_OP(AtomicAnd) {
auto Op = IROp->C<IR::IROp_AtomicAnd>();
auto MemSrc = GetReg<RA_64>(Op->Header.Args[0].ID());
auto MemSrc = GetReg<RA_64>(Op->Addr.ID());
if (SupportsAtomics) {
mvn(TMP2, GetReg<RA_64>(Op->Header.Args[1].ID()));
if (CTX->HostFeatures.SupportsAtomics) {
mvn(TMP2, GetReg<RA_64>(Op->Value.ID()));
switch (IROp->Size) {
case 1: stclrlb(TMP2.W(), MemOperand(MemSrc)); break;
case 2: stclrlh(TMP2.W(), MemOperand(MemSrc)); break;
@@ -352,7 +348,7 @@ DEF_OP(AtomicAnd) {
aarch64::Label LoopTop;
bind(&LoopTop);
ldaxrb(TMP2.W(), MemOperand(MemSrc));
and_(TMP2.W(), TMP2.W(), GetReg<RA_32>(Op->Header.Args[1].ID()));
and_(TMP2.W(), TMP2.W(), GetReg<RA_32>(Op->Value.ID()));
stlxrb(TMP2.W(), TMP2.W(), MemOperand(MemSrc));
cbnz(TMP2.W(), &LoopTop);
break;
@@ -361,7 +357,7 @@ DEF_OP(AtomicAnd) {
aarch64::Label LoopTop;
bind(&LoopTop);
ldaxrh(TMP2.W(), MemOperand(MemSrc));
and_(TMP2.W(), TMP2.W(), GetReg<RA_32>(Op->Header.Args[1].ID()));
and_(TMP2.W(), TMP2.W(), GetReg<RA_32>(Op->Value.ID()));
stlxrh(TMP2.W(), TMP2.W(), MemOperand(MemSrc));
cbnz(TMP2.W(), &LoopTop);
break;
@@ -370,7 +366,7 @@ DEF_OP(AtomicAnd) {
aarch64::Label LoopTop;
bind(&LoopTop);
ldaxr(TMP2.W(), MemOperand(MemSrc));
and_(TMP2.W(), TMP2.W(), GetReg<RA_32>(Op->Header.Args[1].ID()));
and_(TMP2.W(), TMP2.W(), GetReg<RA_32>(Op->Value.ID()));
stlxr(TMP2.W(), TMP2.W(), MemOperand(MemSrc));
cbnz(TMP2.W(), &LoopTop);
break;
@@ -379,7 +375,7 @@ DEF_OP(AtomicAnd) {
aarch64::Label LoopTop;
bind(&LoopTop);
ldaxr(TMP2, MemOperand(MemSrc));
and_(TMP2, TMP2, GetReg<RA_64>(Op->Header.Args[1].ID()));
and_(TMP2, TMP2, GetReg<RA_64>(Op->Value.ID()));
stlxr(TMP2, TMP2, MemOperand(MemSrc));
cbnz(TMP2, &LoopTop);
break;
@@ -392,14 +388,14 @@ DEF_OP(AtomicAnd) {
DEF_OP(AtomicOr) {
auto Op = IROp->C<IR::IROp_AtomicOr>();
auto MemSrc = GetReg<RA_64>(Op->Header.Args[0].ID());
auto MemSrc = GetReg<RA_64>(Op->Addr.ID());
if (SupportsAtomics) {
if (CTX->HostFeatures.SupportsAtomics) {
switch (IROp->Size) {
case 1: stsetlb(GetReg<RA_32>(Op->Header.Args[1].ID()), MemOperand(MemSrc)); break;
case 2: stsetlh(GetReg<RA_32>(Op->Header.Args[1].ID()), MemOperand(MemSrc)); break;
case 4: stsetl(GetReg<RA_32>(Op->Header.Args[1].ID()), MemOperand(MemSrc)); break;
case 8: stsetl(GetReg<RA_64>(Op->Header.Args[1].ID()), MemOperand(MemSrc)); break;
case 1: stsetlb(GetReg<RA_32>(Op->Value.ID()), MemOperand(MemSrc)); break;
case 2: stsetlh(GetReg<RA_32>(Op->Value.ID()), MemOperand(MemSrc)); break;
case 4: stsetl(GetReg<RA_32>(Op->Value.ID()), MemOperand(MemSrc)); break;
case 8: stsetl(GetReg<RA_64>(Op->Value.ID()), MemOperand(MemSrc)); break;
default: LOGMAN_MSG_A_FMT("Unhandled Atomic size: {}", IROp->Size);
}
}
@@ -410,7 +406,7 @@ DEF_OP(AtomicOr) {
aarch64::Label LoopTop;
bind(&LoopTop);
ldaxrb(TMP2.W(), MemOperand(MemSrc));
orr(TMP2.W(), TMP2.W(), GetReg<RA_32>(Op->Header.Args[1].ID()));
orr(TMP2.W(), TMP2.W(), GetReg<RA_32>(Op->Value.ID()));
stlxrb(TMP2.W(), TMP2.W(), MemOperand(MemSrc));
cbnz(TMP2.W(), &LoopTop);
break;
@@ -419,7 +415,7 @@ DEF_OP(AtomicOr) {
aarch64::Label LoopTop;
bind(&LoopTop);
ldaxrh(TMP2.W(), MemOperand(MemSrc));
orr(TMP2.W(), TMP2.W(), GetReg<RA_32>(Op->Header.Args[1].ID()));
orr(TMP2.W(), TMP2.W(), GetReg<RA_32>(Op->Value.ID()));
stlxrh(TMP2.W(), TMP2.W(), MemOperand(MemSrc));
cbnz(TMP2.W(), &LoopTop);
break;
@@ -428,7 +424,7 @@ DEF_OP(AtomicOr) {
aarch64::Label LoopTop;
bind(&LoopTop);
ldaxr(TMP2.W(), MemOperand(MemSrc));
orr(TMP2.W(), TMP2.W(), GetReg<RA_32>(Op->Header.Args[1].ID()));
orr(TMP2.W(), TMP2.W(), GetReg<RA_32>(Op->Value.ID()));
stlxr(TMP2.W(), TMP2.W(), MemOperand(MemSrc));
cbnz(TMP2.W(), &LoopTop);
break;
@@ -437,7 +433,7 @@ DEF_OP(AtomicOr) {
aarch64::Label LoopTop;
bind(&LoopTop);
ldaxr(TMP2, MemOperand(MemSrc));
orr(TMP2, TMP2, GetReg<RA_64>(Op->Header.Args[1].ID()));
orr(TMP2, TMP2, GetReg<RA_64>(Op->Value.ID()));
stlxr(TMP2, TMP2, MemOperand(MemSrc));
cbnz(TMP2, &LoopTop);
break;
@@ -450,14 +446,14 @@ DEF_OP(AtomicOr) {
DEF_OP(AtomicXor) {
auto Op = IROp->C<IR::IROp_AtomicXor>();
auto MemSrc = GetReg<RA_64>(Op->Header.Args[0].ID());
auto MemSrc = GetReg<RA_64>(Op->Addr.ID());
if (SupportsAtomics) {
if (CTX->HostFeatures.SupportsAtomics) {
switch (IROp->Size) {
case 1: steorlb(GetReg<RA_32>(Op->Header.Args[1].ID()), MemOperand(MemSrc)); break;
case 2: steorlh(GetReg<RA_32>(Op->Header.Args[1].ID()), MemOperand(MemSrc)); break;
case 4: steorl(GetReg<RA_32>(Op->Header.Args[1].ID()), MemOperand(MemSrc)); break;
case 8: steorl(GetReg<RA_64>(Op->Header.Args[1].ID()), MemOperand(MemSrc)); break;
case 1: steorlb(GetReg<RA_32>(Op->Value.ID()), MemOperand(MemSrc)); break;
case 2: steorlh(GetReg<RA_32>(Op->Value.ID()), MemOperand(MemSrc)); break;
case 4: steorl(GetReg<RA_32>(Op->Value.ID()), MemOperand(MemSrc)); break;
case 8: steorl(GetReg<RA_64>(Op->Value.ID()), MemOperand(MemSrc)); break;
default: LOGMAN_MSG_A_FMT("Unhandled Atomic size: {}", IROp->Size);
}
}
@@ -468,7 +464,7 @@ DEF_OP(AtomicXor) {
aarch64::Label LoopTop;
bind(&LoopTop);
ldaxrb(TMP2.W(), MemOperand(MemSrc));
eor(TMP2.W(), TMP2.W(), GetReg<RA_32>(Op->Header.Args[1].ID()));
eor(TMP2.W(), TMP2.W(), GetReg<RA_32>(Op->Value.ID()));
stlxrb(TMP2.W(), TMP2.W(), MemOperand(MemSrc));
cbnz(TMP2.W(), &LoopTop);
break;
@@ -477,7 +473,7 @@ DEF_OP(AtomicXor) {
aarch64::Label LoopTop;
bind(&LoopTop);
ldaxrh(TMP2.W(), MemOperand(MemSrc));
eor(TMP2.W(), TMP2.W(), GetReg<RA_32>(Op->Header.Args[1].ID()));
eor(TMP2.W(), TMP2.W(), GetReg<RA_32>(Op->Value.ID()));
stlxrh(TMP2.W(), TMP2.W(), MemOperand(MemSrc));
cbnz(TMP2.W(), &LoopTop);
break;
@@ -486,7 +482,7 @@ DEF_OP(AtomicXor) {
aarch64::Label LoopTop;
bind(&LoopTop);
ldaxr(TMP2.W(), MemOperand(MemSrc));
eor(TMP2.W(), TMP2.W(), GetReg<RA_32>(Op->Header.Args[1].ID()));
eor(TMP2.W(), TMP2.W(), GetReg<RA_32>(Op->Value.ID()));
stlxr(TMP2.W(), TMP2.W(), MemOperand(MemSrc));
cbnz(TMP2.W(), &LoopTop);
break;
@@ -495,7 +491,7 @@ DEF_OP(AtomicXor) {
aarch64::Label LoopTop;
bind(&LoopTop);
ldaxr(TMP2, MemOperand(MemSrc));
eor(TMP2, TMP2, GetReg<RA_64>(Op->Header.Args[1].ID()));
eor(TMP2, TMP2, GetReg<RA_64>(Op->Value.ID()));
stlxr(TMP2, TMP2, MemOperand(MemSrc));
cbnz(TMP2, &LoopTop);
break;
@@ -508,10 +504,10 @@ DEF_OP(AtomicXor) {
DEF_OP(AtomicSwap) {
auto Op = IROp->C<IR::IROp_AtomicSwap>();
auto MemSrc = GetReg<RA_64>(Op->Header.Args[0].ID());
auto MemSrc = GetReg<RA_64>(Op->Addr.ID());
if (SupportsAtomics) {
mov(TMP2, GetReg<RA_64>(Op->Header.Args[1].ID()));
if (CTX->HostFeatures.SupportsAtomics) {
mov(TMP2, GetReg<RA_64>(Op->Value.ID()));
switch (IROp->Size) {
case 1: swplb(TMP2.W(), GetReg<RA_32>(Node), MemOperand(MemSrc)); break;
case 2: swplh(TMP2.W(), GetReg<RA_32>(Node), MemOperand(MemSrc)); break;
@@ -527,7 +523,7 @@ DEF_OP(AtomicSwap) {
aarch64::Label LoopTop;
bind(&LoopTop);
ldaxrb(TMP2.W(), MemOperand(MemSrc));
stlxrb(TMP4.W(), GetReg<RA_32>(Op->Header.Args[1].ID()), MemOperand(MemSrc));
stlxrb(TMP4.W(), GetReg<RA_32>(Op->Value.ID()), MemOperand(MemSrc));
cbnz(TMP4.W(), &LoopTop);
uxtb(GetReg<RA_32>(Node), TMP2.W());
break;
@@ -536,7 +532,7 @@ DEF_OP(AtomicSwap) {
aarch64::Label LoopTop;
bind(&LoopTop);
ldaxrh(TMP2.W(), MemOperand(MemSrc));
stlxrh(TMP4.W(), GetReg<RA_32>(Op->Header.Args[1].ID()), MemOperand(MemSrc));
stlxrh(TMP4.W(), GetReg<RA_32>(Op->Value.ID()), MemOperand(MemSrc));
cbnz(TMP4.W(), &LoopTop);
uxtw(GetReg<RA_32>(Node), TMP2.W());
break;
@@ -545,7 +541,7 @@ DEF_OP(AtomicSwap) {
aarch64::Label LoopTop;
bind(&LoopTop);
ldaxr(TMP2.W(), MemOperand(MemSrc));
stlxr(TMP4.W(), GetReg<RA_32>(Op->Header.Args[1].ID()), MemOperand(MemSrc));
stlxr(TMP4.W(), GetReg<RA_32>(Op->Value.ID()), MemOperand(MemSrc));
cbnz(TMP4.W(), &LoopTop);
mov(GetReg<RA_32>(Node), TMP2.W());
break;
@@ -554,7 +550,7 @@ DEF_OP(AtomicSwap) {
aarch64::Label LoopTop;
bind(&LoopTop);
ldaxr(TMP2, MemOperand(MemSrc));
stlxr(TMP4, GetReg<RA_64>(Op->Header.Args[1].ID()), MemOperand(MemSrc));
stlxr(TMP4, GetReg<RA_64>(Op->Value.ID()), MemOperand(MemSrc));
cbnz(TMP4, &LoopTop);
mov(GetReg<RA_64>(Node), TMP2.X());
break;
@@ -566,14 +562,14 @@ DEF_OP(AtomicSwap) {
DEF_OP(AtomicFetchAdd) {
auto Op = IROp->C<IR::IROp_AtomicFetchAdd>();
auto MemSrc = GetReg<RA_64>(Op->Header.Args[0].ID());
auto MemSrc = GetReg<RA_64>(Op->Addr.ID());
if (SupportsAtomics) {
if (CTX->HostFeatures.SupportsAtomics) {
switch (IROp->Size) {
case 1: ldaddalb(GetReg<RA_32>(Op->Header.Args[1].ID()), GetReg<RA_32>(Node), MemOperand(MemSrc)); break;
case 2: ldaddalh(GetReg<RA_32>(Op->Header.Args[1].ID()), GetReg<RA_32>(Node), MemOperand(MemSrc)); break;
case 4: ldaddal(GetReg<RA_32>(Op->Header.Args[1].ID()), GetReg<RA_32>(Node), MemOperand(MemSrc)); break;
case 8: ldaddal(GetReg<RA_64>(Op->Header.Args[1].ID()), GetReg<RA_64>(Node), MemOperand(MemSrc)); break;
case 1: ldaddalb(GetReg<RA_32>(Op->Value.ID()), GetReg<RA_32>(Node), MemOperand(MemSrc)); break;
case 2: ldaddalh(GetReg<RA_32>(Op->Value.ID()), GetReg<RA_32>(Node), MemOperand(MemSrc)); break;
case 4: ldaddal(GetReg<RA_32>(Op->Value.ID()), GetReg<RA_32>(Node), MemOperand(MemSrc)); break;
case 8: ldaddal(GetReg<RA_64>(Op->Value.ID()), GetReg<RA_64>(Node), MemOperand(MemSrc)); break;
default: LOGMAN_MSG_A_FMT("Unhandled Atomic size: {}", IROp->Size);
}
}
@@ -584,7 +580,7 @@ DEF_OP(AtomicFetchAdd) {
aarch64::Label LoopTop;
bind(&LoopTop);
ldaxrb(TMP2.W(), MemOperand(MemSrc));
add(TMP3.W(), TMP2.W(), GetReg<RA_32>(Op->Header.Args[1].ID()));
add(TMP3.W(), TMP2.W(), GetReg<RA_32>(Op->Value.ID()));
stlxrb(TMP4.W(), TMP3.W(), MemOperand(MemSrc));
cbnz(TMP4.W(), &LoopTop);
mov(GetReg<RA_32>(Node), TMP2.W());
@@ -594,7 +590,7 @@ DEF_OP(AtomicFetchAdd) {
aarch64::Label LoopTop;
bind(&LoopTop);
ldaxrh(TMP2.W(), MemOperand(MemSrc));
add(TMP3.W(), TMP2.W(), GetReg<RA_32>(Op->Header.Args[1].ID()));
add(TMP3.W(), TMP2.W(), GetReg<RA_32>(Op->Value.ID()));
stlxrh(TMP4.W(), TMP3.W(), MemOperand(MemSrc));
cbnz(TMP4.W(), &LoopTop);
mov(GetReg<RA_32>(Node), TMP2.W());
@@ -604,7 +600,7 @@ DEF_OP(AtomicFetchAdd) {
aarch64::Label LoopTop;
bind(&LoopTop);
ldaxr(TMP2.W(), MemOperand(MemSrc));
add(TMP3.W(), TMP2.W(), GetReg<RA_32>(Op->Header.Args[1].ID()));
add(TMP3.W(), TMP2.W(), GetReg<RA_32>(Op->Value.ID()));
stlxr(TMP4.W(), TMP3.W(), MemOperand(MemSrc));
cbnz(TMP4.W(), &LoopTop);
mov(GetReg<RA_32>(Node), TMP2.W());
@@ -614,7 +610,7 @@ DEF_OP(AtomicFetchAdd) {
aarch64::Label LoopTop;
bind(&LoopTop);
ldaxr(TMP2, MemOperand(MemSrc));
add(TMP3, TMP2, GetReg<RA_64>(Op->Header.Args[1].ID()));
add(TMP3, TMP2, GetReg<RA_64>(Op->Value.ID()));
stlxr(TMP4, TMP3, MemOperand(MemSrc));
cbnz(TMP4, &LoopTop);
mov(GetReg<RA_64>(Node), TMP2);
@@ -627,10 +623,10 @@ DEF_OP(AtomicFetchAdd) {
DEF_OP(AtomicFetchSub) {
auto Op = IROp->C<IR::IROp_AtomicFetchSub>();
auto MemSrc = GetReg<RA_64>(Op->Header.Args[0].ID());
auto MemSrc = GetReg<RA_64>(Op->Addr.ID());
if (SupportsAtomics) {
neg(TMP2, GetReg<RA_64>(Op->Header.Args[1].ID()));
if (CTX->HostFeatures.SupportsAtomics) {
neg(TMP2, GetReg<RA_64>(Op->Value.ID()));
switch (IROp->Size) {
case 1: ldaddalb(TMP2.W(), GetReg<RA_32>(Node), MemOperand(MemSrc)); break;
case 2: ldaddalh(TMP2.W(), GetReg<RA_32>(Node), MemOperand(MemSrc)); break;
@@ -646,7 +642,7 @@ DEF_OP(AtomicFetchSub) {
aarch64::Label LoopTop;
bind(&LoopTop);
ldaxrb(TMP2.W(), MemOperand(MemSrc));
sub(TMP3.W(), TMP2.W(), GetReg<RA_32>(Op->Header.Args[1].ID()));
sub(TMP3.W(), TMP2.W(), GetReg<RA_32>(Op->Value.ID()));
stlxrb(TMP4.W(), TMP3.W(), MemOperand(MemSrc));
cbnz(TMP4.W(), &LoopTop);
mov(GetReg<RA_32>(Node), TMP2.W());
@@ -656,7 +652,7 @@ DEF_OP(AtomicFetchSub) {
aarch64::Label LoopTop;
bind(&LoopTop);
ldaxrh(TMP2.W(), MemOperand(MemSrc));
sub(TMP3.W(), TMP2.W(), GetReg<RA_32>(Op->Header.Args[1].ID()));
sub(TMP3.W(), TMP2.W(), GetReg<RA_32>(Op->Value.ID()));
stlxrh(TMP4.W(), TMP3.W(), MemOperand(MemSrc));
cbnz(TMP4.W(), &LoopTop);
mov(GetReg<RA_32>(Node), TMP2.W());
@@ -666,7 +662,7 @@ DEF_OP(AtomicFetchSub) {
aarch64::Label LoopTop;
bind(&LoopTop);
ldaxr(TMP2.W(), MemOperand(MemSrc));
sub(TMP3.W(), TMP2.W(), GetReg<RA_32>(Op->Header.Args[1].ID()));
sub(TMP3.W(), TMP2.W(), GetReg<RA_32>(Op->Value.ID()));
stlxr(TMP4.W(), TMP3.W(), MemOperand(MemSrc));
cbnz(TMP4.W(), &LoopTop);
mov(GetReg<RA_32>(Node), TMP2.W());
@@ -676,7 +672,7 @@ DEF_OP(AtomicFetchSub) {
aarch64::Label LoopTop;
bind(&LoopTop);
ldaxr(TMP2, MemOperand(MemSrc));
sub(TMP3, TMP2, GetReg<RA_64>(Op->Header.Args[1].ID()));
sub(TMP3, TMP2, GetReg<RA_64>(Op->Value.ID()));
stlxr(TMP4, TMP3, MemOperand(MemSrc));
cbnz(TMP4, &LoopTop);
mov(GetReg<RA_64>(Node), TMP2);
@@ -689,10 +685,10 @@ DEF_OP(AtomicFetchSub) {
DEF_OP(AtomicFetchAnd) {
auto Op = IROp->C<IR::IROp_AtomicFetchAnd>();
auto MemSrc = GetReg<RA_64>(Op->Header.Args[0].ID());
auto MemSrc = GetReg<RA_64>(Op->Addr.ID());
if (SupportsAtomics) {
mvn(TMP2, GetReg<RA_64>(Op->Header.Args[1].ID()));
if (CTX->HostFeatures.SupportsAtomics) {
mvn(TMP2, GetReg<RA_64>(Op->Value.ID()));
switch (IROp->Size) {
case 1: ldclralb(TMP2.W(), GetReg<RA_32>(Node), MemOperand(MemSrc)); break;
case 2: ldclralh(TMP2.W(), GetReg<RA_32>(Node), MemOperand(MemSrc)); break;
@@ -708,7 +704,7 @@ DEF_OP(AtomicFetchAnd) {
aarch64::Label LoopTop;
bind(&LoopTop);
ldaxrb(TMP2.W(), MemOperand(MemSrc));
and_(TMP3.W(), TMP2.W(), GetReg<RA_32>(Op->Header.Args[1].ID()));
and_(TMP3.W(), TMP2.W(), GetReg<RA_32>(Op->Value.ID()));
stlxrb(TMP4.W(), TMP3.W(), MemOperand(MemSrc));
cbnz(TMP4.W(), &LoopTop);
mov(GetReg<RA_32>(Node), TMP2.W());
@@ -718,7 +714,7 @@ DEF_OP(AtomicFetchAnd) {
aarch64::Label LoopTop;
bind(&LoopTop);
ldaxrh(TMP2.W(), MemOperand(MemSrc));
and_(TMP3.W(), TMP2.W(), GetReg<RA_32>(Op->Header.Args[1].ID()));
and_(TMP3.W(), TMP2.W(), GetReg<RA_32>(Op->Value.ID()));
stlxrh(TMP4.W(), TMP3.W(), MemOperand(MemSrc));
cbnz(TMP4.W(), &LoopTop);
mov(GetReg<RA_32>(Node), TMP2.W());
@@ -728,7 +724,7 @@ DEF_OP(AtomicFetchAnd) {
aarch64::Label LoopTop;
bind(&LoopTop);
ldaxr(TMP2.W(), MemOperand(MemSrc));
and_(TMP3.W(), TMP2.W(), GetReg<RA_32>(Op->Header.Args[1].ID()));
and_(TMP3.W(), TMP2.W(), GetReg<RA_32>(Op->Value.ID()));
stlxr(TMP4.W(), TMP3.W(), MemOperand(MemSrc));
cbnz(TMP4.W(), &LoopTop);
mov(GetReg<RA_32>(Node), TMP2.W());
@@ -738,7 +734,7 @@ DEF_OP(AtomicFetchAnd) {
aarch64::Label LoopTop;
bind(&LoopTop);
ldaxr(TMP2, MemOperand(MemSrc));
and_(TMP3, TMP2, GetReg<RA_64>(Op->Header.Args[1].ID()));
and_(TMP3, TMP2, GetReg<RA_64>(Op->Value.ID()));
stlxr(TMP4, TMP3, MemOperand(MemSrc));
cbnz(TMP4, &LoopTop);
mov(GetReg<RA_64>(Node), TMP2);
@@ -751,14 +747,14 @@ DEF_OP(AtomicFetchAnd) {
DEF_OP(AtomicFetchOr) {
auto Op = IROp->C<IR::IROp_AtomicFetchOr>();
auto MemSrc = GetReg<RA_64>(Op->Header.Args[0].ID());
auto MemSrc = GetReg<RA_64>(Op->Addr.ID());
if (SupportsAtomics) {
if (CTX->HostFeatures.SupportsAtomics) {
switch (IROp->Size) {
case 1: ldsetalb(GetReg<RA_32>(Op->Header.Args[1].ID()), GetReg<RA_32>(Node), MemOperand(MemSrc)); break;
case 2: ldsetalh(GetReg<RA_32>(Op->Header.Args[1].ID()), GetReg<RA_32>(Node), MemOperand(MemSrc)); break;
case 4: ldsetal(GetReg<RA_32>(Op->Header.Args[1].ID()), GetReg<RA_32>(Node), MemOperand(MemSrc)); break;
case 8: ldsetal(GetReg<RA_64>(Op->Header.Args[1].ID()), GetReg<RA_64>(Node), MemOperand(MemSrc)); break;
case 1: ldsetalb(GetReg<RA_32>(Op->Value.ID()), GetReg<RA_32>(Node), MemOperand(MemSrc)); break;
case 2: ldsetalh(GetReg<RA_32>(Op->Value.ID()), GetReg<RA_32>(Node), MemOperand(MemSrc)); break;
case 4: ldsetal(GetReg<RA_32>(Op->Value.ID()), GetReg<RA_32>(Node), MemOperand(MemSrc)); break;
case 8: ldsetal(GetReg<RA_64>(Op->Value.ID()), GetReg<RA_64>(Node), MemOperand(MemSrc)); break;
default: LOGMAN_MSG_A_FMT("Unhandled Atomic size: {}", IROp->Size);
}
}
@@ -769,7 +765,7 @@ DEF_OP(AtomicFetchOr) {
aarch64::Label LoopTop;
bind(&LoopTop);
ldaxrb(TMP2.W(), MemOperand(MemSrc));
orr(TMP3.W(), TMP2.W(), GetReg<RA_32>(Op->Header.Args[1].ID()));
orr(TMP3.W(), TMP2.W(), GetReg<RA_32>(Op->Value.ID()));
stlxrb(TMP4.W(), TMP3.W(), MemOperand(MemSrc));
cbnz(TMP4.W(), &LoopTop);
mov(GetReg<RA_32>(Node), TMP2.W());
@@ -779,7 +775,7 @@ DEF_OP(AtomicFetchOr) {
aarch64::Label LoopTop;
bind(&LoopTop);
ldaxrh(TMP2.W(), MemOperand(MemSrc));
orr(TMP3.W(), TMP2.W(), GetReg<RA_32>(Op->Header.Args[1].ID()));
orr(TMP3.W(), TMP2.W(), GetReg<RA_32>(Op->Value.ID()));
stlxrh(TMP4.W(), TMP3.W(), MemOperand(MemSrc));
cbnz(TMP4.W(), &LoopTop);
mov(GetReg<RA_32>(Node), TMP2.W());
@@ -789,7 +785,7 @@ DEF_OP(AtomicFetchOr) {
aarch64::Label LoopTop;
bind(&LoopTop);
ldaxr(TMP2.W(), MemOperand(MemSrc));
orr(TMP3.W(), TMP2.W(), GetReg<RA_32>(Op->Header.Args[1].ID()));
orr(TMP3.W(), TMP2.W(), GetReg<RA_32>(Op->Value.ID()));
stlxr(TMP4.W(), TMP3.W(), MemOperand(MemSrc));
cbnz(TMP4.W(), &LoopTop);
mov(GetReg<RA_32>(Node), TMP2.W());
@@ -799,7 +795,7 @@ DEF_OP(AtomicFetchOr) {
aarch64::Label LoopTop;
bind(&LoopTop);
ldaxr(TMP2, MemOperand(MemSrc));
orr(TMP3, TMP2, GetReg<RA_64>(Op->Header.Args[1].ID()));
orr(TMP3, TMP2, GetReg<RA_64>(Op->Value.ID()));
stlxr(TMP4, TMP3, MemOperand(MemSrc));
cbnz(TMP4, &LoopTop);
mov(GetReg<RA_64>(Node), TMP2);
@@ -812,14 +808,14 @@ DEF_OP(AtomicFetchOr) {
DEF_OP(AtomicFetchXor) {
auto Op = IROp->C<IR::IROp_AtomicFetchXor>();
auto MemSrc = GetReg<RA_64>(Op->Header.Args[0].ID());
auto MemSrc = GetReg<RA_64>(Op->Addr.ID());
if (SupportsAtomics) {
if (CTX->HostFeatures.SupportsAtomics) {
switch (IROp->Size) {
case 1: ldeoralb(GetReg<RA_32>(Op->Header.Args[1].ID()), GetReg<RA_32>(Node), MemOperand(MemSrc)); break;
case 2: ldeoralh(GetReg<RA_32>(Op->Header.Args[1].ID()), GetReg<RA_32>(Node), MemOperand(MemSrc)); break;
case 4: ldeoral(GetReg<RA_32>(Op->Header.Args[1].ID()), GetReg<RA_32>(Node), MemOperand(MemSrc)); break;
case 8: ldeoral(GetReg<RA_64>(Op->Header.Args[1].ID()), GetReg<RA_64>(Node), MemOperand(MemSrc)); break;
case 1: ldeoralb(GetReg<RA_32>(Op->Value.ID()), GetReg<RA_32>(Node), MemOperand(MemSrc)); break;
case 2: ldeoralh(GetReg<RA_32>(Op->Value.ID()), GetReg<RA_32>(Node), MemOperand(MemSrc)); break;
case 4: ldeoral(GetReg<RA_32>(Op->Value.ID()), GetReg<RA_32>(Node), MemOperand(MemSrc)); break;
case 8: ldeoral(GetReg<RA_64>(Op->Value.ID()), GetReg<RA_64>(Node), MemOperand(MemSrc)); break;
default: LOGMAN_MSG_A_FMT("Unhandled Atomic size: {}", IROp->Size);
}
}
@@ -830,7 +826,7 @@ DEF_OP(AtomicFetchXor) {
aarch64::Label LoopTop;
bind(&LoopTop);
ldaxrb(TMP2.W(), MemOperand(MemSrc));
eor(TMP3.W(), TMP2.W(), GetReg<RA_32>(Op->Header.Args[1].ID()));
eor(TMP3.W(), TMP2.W(), GetReg<RA_32>(Op->Value.ID()));
stlxrb(TMP4.W(), TMP3.W(), MemOperand(MemSrc));
cbnz(TMP4.W(), &LoopTop);
mov(GetReg<RA_32>(Node), TMP2.W());
@@ -840,7 +836,7 @@ DEF_OP(AtomicFetchXor) {
aarch64::Label LoopTop;
bind(&LoopTop);
ldaxrh(TMP2.W(), MemOperand(MemSrc));
eor(TMP3.W(), TMP2.W(), GetReg<RA_32>(Op->Header.Args[1].ID()));
eor(TMP3.W(), TMP2.W(), GetReg<RA_32>(Op->Value.ID()));
stlxrh(TMP4.W(), TMP3.W(), MemOperand(MemSrc));
cbnz(TMP4.W(), &LoopTop);
mov(GetReg<RA_32>(Node), TMP2.W());
@@ -850,7 +846,7 @@ DEF_OP(AtomicFetchXor) {
aarch64::Label LoopTop;
bind(&LoopTop);
ldaxr(TMP2.W(), MemOperand(MemSrc));
eor(TMP3.W(), TMP2.W(), GetReg<RA_32>(Op->Header.Args[1].ID()));
eor(TMP3.W(), TMP2.W(), GetReg<RA_32>(Op->Value.ID()));
stlxr(TMP4.W(), TMP3.W(), MemOperand(MemSrc));
cbnz(TMP4.W(), &LoopTop);
mov(GetReg<RA_32>(Node), TMP2.W());
@@ -860,7 +856,7 @@ DEF_OP(AtomicFetchXor) {
aarch64::Label LoopTop;
bind(&LoopTop);
ldaxr(TMP2, MemOperand(MemSrc));
eor(TMP3, TMP2, GetReg<RA_64>(Op->Header.Args[1].ID()));
eor(TMP3, TMP2, GetReg<RA_64>(Op->Value.ID()));
stlxr(TMP4, TMP3, MemOperand(MemSrc));
cbnz(TMP4, &LoopTop);
mov(GetReg<RA_64>(Node), TMP2);
@@ -873,7 +869,7 @@ DEF_OP(AtomicFetchXor) {
DEF_OP(AtomicFetchNeg) {
auto Op = IROp->C<IR::IROp_AtomicFetchNeg>();
auto MemSrc = GetReg<RA_64>(Op->Header.Args[0].ID());
auto MemSrc = GetReg<RA_64>(Op->Addr.ID());
// TMP2-TMP3
switch (IROp->Size) {
@@ -4,6 +4,7 @@ tags: backend|arm64
$end_info$
*/
#include "FEXCore/IR/IR.h"
#include "Interface/Core/LookupCache.h"
#include "Interface/Core/JIT/Arm64/JITClass.h"
@@ -26,17 +27,13 @@ DEF_OP(GuestCallIndirect) {
LogMan::Msg::DFmt("Unimplemented");
}
DEF_OP(GuestReturn) {
LogMan::Msg::DFmt("Unimplemented");
}
DEF_OP(SignalReturn) {
// First we must reset the stack
ResetStack();
// Now branch to our signal return helper
// This can't be a direct branch since the code needs to live at a constant location
LoadConstant(x0, ThreadSharedData.SignalReturnInstruction);
ldr(x0, MemOperand(STATE, offsetof(FEXCore::Core::CpuStateFrame, Pointers.AArch64.SignalReturnHandler)));
br(x0);
}
@@ -49,7 +46,7 @@ DEF_OP(CallbackReturn) {
ResetStack();
// We can now lower the ref counter again
LoadConstant(x0, reinterpret_cast<uint64_t>(ThreadSharedData.SignalHandlerRefCounterPtr));
ldr(x0, MemOperand(STATE, offsetof(FEXCore::Core::CpuStateFrame, Pointers.AArch64.SignalHandlerRefCountPointer)));
ldr(w2, MemOperand(x0));
sub(w2, w2, 1);
str(w2, MemOperand(x0));
@@ -88,7 +85,7 @@ DEF_OP(ExitFunction) {
RipReg = GetReg<RA_64>(Op->Header.Args[0].ID());
// L1 Cache
LoadConstant(x0, ThreadState->LookupCache->GetL1Pointer());
ldr(x0, MemOperand(STATE, offsetof(FEXCore::Core::CpuStateFrame, Pointers.AArch64.L1Pointer)));
and_(x3, RipReg, LookupCache::L1_ENTRIES_MASK);
add(x0, x0, Operand(x3, Shift::LSL, 4));
@@ -99,7 +96,7 @@ DEF_OP(ExitFunction) {
br(x1);
bind(&FullLookup);
LoadConstant(TMP1, ThreadSharedData.Dispatcher->AbsoluteLoopTopAddress);
ldr(TMP1, MemOperand(STATE, offsetof(FEXCore::Core::CpuStateFrame, Pointers.AArch64.DispatcherLoopTop)));
str(RipReg, MemOperand(STATE, offsetof(FEXCore::Core::CpuStateFrame, State.rip)));
br(TMP1);
}
@@ -184,8 +181,16 @@ DEF_OP(Syscall) {
// X1: ThreadState
// X2: Pointer to SyscallArguments
FEXCore::IR::SyscallFlags Flags = Op->Flags;
PushDynamicRegsAndLR();
SpillStaticRegs();
if ((Flags & FEXCore::IR::SyscallFlags::NOSYNCSTATEONENTRY) != FEXCore::IR::SyscallFlags::NOSYNCSTATEONENTRY) {
SpillStaticRegs();
}
else {
// Need to spill all caller saved registers still
SpillStaticRegs(true, CALLER_GPR_MASK, CALLER_FPR_MASK);
}
uint64_t SPOffset = AlignUp(FEXCore::HLE::SyscallArguments::MAX_ARGS * 8, 16);
sub(sp, sp, SPOffset);
@@ -194,22 +199,30 @@ DEF_OP(Syscall) {
str(GetReg<RA_64>(Op->Header.Args[i].ID()), MemOperand(sp, i * 8));
}
LoadConstant(x0, reinterpret_cast<uint64_t>(CTX->SyscallHandler));
ldr(x0, MemOperand(STATE, offsetof(FEXCore::Core::CpuStateFrame, Pointers.AArch64.SyscallHandlerObj)));
ldr(x3, MemOperand(STATE, offsetof(FEXCore::Core::CpuStateFrame, Pointers.AArch64.SyscallHandlerFunc)));
mov(x1, STATE);
mov(x2, sp);
LoadConstant(x3, reinterpret_cast<uint64_t>(FEXCore::Context::HandleSyscall));
blr(x3);
add(sp, sp, SPOffset);
// Result is now in x0
// Fix the stack and any values that were stepped on
FillStaticRegs();
if ((Flags & FEXCore::IR::SyscallFlags::NOSYNCSTATEONENTRY) != FEXCore::IR::SyscallFlags::NOSYNCSTATEONENTRY &&
(Flags & FEXCore::IR::SyscallFlags::NORETURN) != FEXCore::IR::SyscallFlags::NORETURN) {
FillStaticRegs();
}
else {
// Result is now in x0
// Fix the stack and any values that were stepped on
FillStaticRegs(true, CALLER_GPR_MASK, CALLER_FPR_MASK);
}
PopDynamicRegsAndLR();
// Move result to its destination register
mov(GetReg<RA_64>(Node), x0);
if ((Flags & FEXCore::IR::SyscallFlags::NORETURN) != FEXCore::IR::SyscallFlags::NORETURN) {
// Move result to its destination register
mov(GetReg<RA_64>(Node), x0);
}
}
DEF_OP(InlineSyscall) {
@@ -318,7 +331,7 @@ DEF_OP(InlineSyscall) {
ldr(RegArgs[i].W(), MemOperand(STATE, offsetof(FEXCore::Core::CpuStateFrame, State.gregs[X86State::REG_RBX])));
}
else {
uxtw(RegArgs[i], Reg);
uxtw(RegArgs[i].W(), Reg);
}
}
}
@@ -331,7 +344,7 @@ DEF_OP(InlineSyscall) {
mov(RegArgs[i], GetReg<RA_64>(Op->Header.Args[i].ID()));
}
else {
uxtw(RegArgs[i], GetReg<RA_32>(Op->Header.Args[i].ID()));
uxtw(RegArgs[i], GetReg<RA_64>(Op->Header.Args[i].ID()));
}
}
}
@@ -340,21 +353,23 @@ DEF_OP(InlineSyscall) {
svc(0);
// On updated signal mask we can receive a signal RIGHT HERE
// Now that we are done in the syscall we need to carefully peel back the state
// First unspill the registers from before
FillStaticRegs(false, SpillMask);
if ((Op->Flags & FEXCore::IR::SyscallFlags::NORETURN) != FEXCore::IR::SyscallFlags::NORETURN) {
// Now that we are done in the syscall we need to carefully peel back the state
// First unspill the registers from before
FillStaticRegs(false, SpillMask);
// Now the registers we've spilled are back in their original host registers
// We can safely claim we are no longer in a syscall
str(xzr, MemOperand(STATE, offsetof(FEXCore::Core::CpuStateFrame, InSyscallInfo)));
// Now the registers we've spilled are back in their original host registers
// We can safely claim we are no longer in a syscall
str(xzr, MemOperand(STATE, offsetof(FEXCore::Core::CpuStateFrame, InSyscallInfo)));
// Result is now in x0
// Move result to its destination register
if (CTX->Config.Is64BitMode()) {
mov(GetReg<RA_64>(Node), x0);
}
else {
uxtw(GetReg<RA_64>(Node), w0);
// Result is now in x0
// Move result to its destination register
if (CTX->Config.Is64BitMode()) {
mov(GetReg<RA_64>(Node), x0);
}
else {
uxtw(GetReg<RA_64>(Node), x0);
}
}
}
@@ -437,7 +452,7 @@ DEF_OP(RemoveCodeEntry) {
mov(x0, STATE);
LoadConstant(x1, Entry);
LoadConstant(x2, reinterpret_cast<uintptr_t>(&Context::Context::RemoveCodeEntryFromJit));
ldr(x2, MemOperand(STATE, offsetof(FEXCore::Core::CpuStateFrame, Pointers.AArch64.RemoveCodeEntryFromJIT)));
SpillStaticRegs();
blr(x2);
FillStaticRegs();
@@ -454,19 +469,10 @@ DEF_OP(CPUID) {
// x0 = CPUID Handler
// x1 = CPUID Function
// x2 = CPUID Leaf
LoadConstant(x0, reinterpret_cast<uint64_t>(&CTX->CPUID));
ldr(x0, MemOperand(STATE, offsetof(FEXCore::Core::CpuStateFrame, Pointers.AArch64.CPUIDObj)));
ldr(x3, MemOperand(STATE, offsetof(FEXCore::Core::CpuStateFrame, Pointers.AArch64.CPUIDFunction)));
mov(x1, GetReg<RA_64>(Op->Header.Args[0].ID()));
mov(x2, GetReg<RA_64>(Op->Header.Args[1].ID()));
using ClassPtrType = FEXCore::CPUID::FunctionResults (FEXCore::CPUIDEmu::*)(uint32_t, uint32_t);
union PtrCast {
ClassPtrType ClassPtr;
uintptr_t Data;
};
PtrCast Ptr;
Ptr.ClassPtr = &FEXCore::CPUIDEmu::RunFunction;
LoadConstant(x3, Ptr.Data);
SpillStaticRegs();
blr(x3);
FillStaticRegs();
@@ -485,7 +491,6 @@ void Arm64JITCore::RegisterBranchHandlers() {
#define REGISTER_OP(op, x) OpHandlers[FEXCore::IR::IROps::OP_##op] = &Arm64JITCore::Op_##x
REGISTER_OP(GUESTCALLDIRECT, GuestCallDirect);
REGISTER_OP(GUESTCALLINDIRECT, GuestCallIndirect);
REGISTER_OP(GUESTRETURN, GuestReturn);
REGISTER_OP(SIGNALRETURN, SignalReturn);
REGISTER_OP(CALLBACKRETURN, CallbackReturn);
REGISTER_OP(EXITFUNCTION, ExitFunction);
@@ -16,19 +16,19 @@ DEF_OP(VInsGPR) {
mov(GetDst(Node), GetSrc(Op->Header.Args[0].ID()));
switch (Op->Header.ElementSize) {
case 1: {
ins(GetDst(Node).V16B(), Op->Index, GetReg<RA_32>(Op->Header.Args[1].ID()));
ins(GetDst(Node).V16B(), Op->DestIdx, GetReg<RA_32>(Op->Header.Args[1].ID()));
break;
}
case 2: {
ins(GetDst(Node).V8H(), Op->Index, GetReg<RA_32>(Op->Header.Args[1].ID()));
ins(GetDst(Node).V8H(), Op->DestIdx, GetReg<RA_32>(Op->Header.Args[1].ID()));
break;
}
case 4: {
ins(GetDst(Node).V4S(), Op->Index, GetReg<RA_32>(Op->Header.Args[1].ID()));
ins(GetDst(Node).V4S(), Op->DestIdx, GetReg<RA_32>(Op->Header.Args[1].ID()));
break;
}
case 8: {
ins(GetDst(Node).V2D(), Op->Index, GetReg<RA_64>(Op->Header.Args[1].ID()));
ins(GetDst(Node).V2D(), Op->DestIdx, GetReg<RA_64>(Op->Header.Args[1].ID()));
break;
}
default: LOGMAN_MSG_A_FMT("Unknown Element Size: {}", Op->Header.ElementSize); break;
@@ -39,11 +39,11 @@ DEF_OP(VCastFromGPR) {
auto Op = IROp->C<IR::IROp_VCastFromGPR>();
switch (Op->Header.ElementSize) {
case 1:
uxtb(TMP1.W(), GetReg<RA_32>(Op->Header.Args[0].ID()).W());
uxtb(TMP1.W(), GetReg<RA_32>(Op->Header.Args[0].ID()));
fmov(GetDst(Node).S(), TMP1.W());
break;
case 2:
uxth(TMP1.W(), GetReg<RA_32>(Op->Header.Args[0].ID()).W());
uxth(TMP1.W(), GetReg<RA_32>(Op->Header.Args[0].ID()));
fmov(GetDst(Node).S(), TMP1.W());
break;
case 4:
@@ -54,7 +54,7 @@ DEF_OP(AESDecLast) {
DEF_OP(AESKeyGenAssist) {
auto Op = IROp->C<IR::IROp_VAESKeyGenAssist>();
aarch64::Label Constant;
aarch64::Literal ConstantLiteral (0x0C030609'0306090CULL, 0x040B0E01'0B0E0104ULL);
aarch64::Label PastConstant;
// Do a "regular" AESE step
@@ -63,8 +63,7 @@ DEF_OP(AESKeyGenAssist) {
aese(VTMP1.V16B(), VTMP2.V16B());
// Do a table shuffle to undo ShiftRows
adr(TMP1.X(), &Constant);
ldr(VTMP3, MemOperand(TMP1.X()));
ldr(VTMP3, &ConstantLiteral);
// Now EOR in the RCON
if (Op->RCON) {
@@ -80,14 +79,29 @@ DEF_OP(AESKeyGenAssist) {
}
b(&PastConstant);
bind(&Constant);
dc32(0x0B0E0104);
dc32(0x040B0E01);
dc32(0x0306090C);
dc32(0x0C030609);
place(&ConstantLiteral);
bind(&PastConstant);
}
DEF_OP(CRC32) {
auto Op = IROp->C<IR::IROp_CRC32>();
switch (Op->SrcSize) {
case 1:
crc32cb(GetReg<RA_32>(Node), GetReg<RA_32>(Op->Src1.ID()), GetReg<RA_32>(Op->Src2.ID()));
break;
case 2:
crc32ch(GetReg<RA_32>(Node), GetReg<RA_32>(Op->Src1.ID()), GetReg<RA_32>(Op->Src2.ID()));
break;
case 4:
crc32cw(GetReg<RA_32>(Node), GetReg<RA_32>(Op->Src1.ID()), GetReg<RA_32>(Op->Src2.ID()));
break;
case 8:
crc32cx(GetReg<RA_32>(Node), GetReg<RA_32>(Op->Src1.ID()), GetReg<RA_64>(Op->Src2.ID()));
break;
default: LOGMAN_MSG_A_FMT("Unknown CRC32 size: {}", Op->SrcSize);
}
}
#undef DEF_OP
void Arm64JITCore::RegisterEncryptionHandlers() {
#define REGISTER_OP(op, x) OpHandlers[FEXCore::IR::IROps::OP_##op] = &Arm64JITCore::Op_##x
@@ -97,7 +111,7 @@ void Arm64JITCore::RegisterEncryptionHandlers() {
REGISTER_OP(VAESDEC, AESDec);
REGISTER_OP(VAESDECLAST, AESDecLast);
REGISTER_OP(VAESKEYGENASSIST, AESKeyGenAssist);
REGISTER_OP(CRC32, CRC32);
#undef REGISTER_OP
}
}
+86 -27
View File
@@ -21,10 +21,13 @@ $end_info$
#include "Interface/IR/Passes/RegisterAllocationPass.h"
#include "Utils/MemberFunctionToPointer.h"
#include <FEXCore/Core/X86Enums.h>
#include <FEXCore/Core/UContext.h>
#include <FEXCore/Utils/Allocator.h>
#include <FEXCore/Utils/CompilerDefs.h>
#include "Interface/Core/Interpreter/InterpreterOps.h"
#include <sys/mman.h>
@@ -32,6 +35,40 @@ $end_info$
#include <unistd.h>
#include <string.h>
namespace {
static uint64_t LUDIV(uint64_t SrcHigh, uint64_t SrcLow, uint64_t Divisor) {
__uint128_t Source = (static_cast<__uint128_t>(SrcHigh) << 64) | SrcLow;
__uint128_t Res = Source / Divisor;
return Res;
}
static int64_t LDIV(int64_t SrcHigh, int64_t SrcLow, int64_t Divisor) {
__int128_t Source = (static_cast<__int128_t>(SrcHigh) << 64) | SrcLow;
__int128_t Res = Source / Divisor;
return Res;
}
static uint64_t LUREM(uint64_t SrcHigh, uint64_t SrcLow, uint64_t Divisor) {
__uint128_t Source = (static_cast<__uint128_t>(SrcHigh) << 64) | SrcLow;
__uint128_t Res = Source % Divisor;
return Res;
}
static int64_t LREM(int64_t SrcHigh, int64_t SrcLow, int64_t Divisor) {
__int128_t Source = (static_cast<__int128_t>(SrcHigh) << 64) | SrcLow;
__int128_t Res = Source % Divisor;
return Res;
}
static void PrintValue(uint64_t Value) {
LogMan::Msg::DFmt("Value: 0x{:x}", Value);
}
static void PrintVectorValue(uint64_t Value, uint64_t ValueUpper) {
LogMan::Msg::DFmt("Value: 0x{:016x}'{:016x}", ValueUpper, Value);
}
}
namespace FEXCore::CPU {
void Arm64JITCore::CopyNecessaryDataForCompileThread(CPUBackend *Original) {
@@ -56,8 +93,7 @@ void Arm64JITCore::Op_Unhandled(IR::IROp_Header *IROp, IR::NodeID Node) {
PushDynamicRegsAndLR();
uxth(w0, GetReg<RA_32>(IROp->Args[0].ID()));
LoadConstant(x1, (uintptr_t)Info.fn);
ldr(x1, MemOperand(STATE, offsetof(FEXCore::Core::CpuStateFrame, Pointers.AArch64.FallbackHandlerPointers[Info.HandlerIndex])));
blr(x1);
PopDynamicRegsAndLR();
@@ -72,8 +108,7 @@ void Arm64JITCore::Op_Unhandled(IR::IROp_Header *IROp, IR::NodeID Node) {
PushDynamicRegsAndLR();
fmov(v0.S(), GetSrc(IROp->Args[0].ID()).S()) ;
LoadConstant(x0, (uintptr_t)Info.fn);
ldr(x0, MemOperand(STATE, offsetof(FEXCore::Core::CpuStateFrame, Pointers.AArch64.FallbackHandlerPointers[Info.HandlerIndex])));
blr(x0);
PopDynamicRegsAndLR();
@@ -92,8 +127,7 @@ void Arm64JITCore::Op_Unhandled(IR::IROp_Header *IROp, IR::NodeID Node) {
PushDynamicRegsAndLR();
mov(v0.D(), GetSrc(IROp->Args[0].ID()).D());
LoadConstant(x0, (uintptr_t)Info.fn);
ldr(x0, MemOperand(STATE, offsetof(FEXCore::Core::CpuStateFrame, Pointers.AArch64.FallbackHandlerPointers[Info.HandlerIndex])));
blr(x0);
PopDynamicRegsAndLR();
@@ -118,8 +152,7 @@ void Arm64JITCore::Op_Unhandled(IR::IROp_Header *IROp, IR::NodeID Node) {
else {
mov(w0, GetReg<RA_32>(IROp->Args[0].ID()));
}
LoadConstant(x1, (uintptr_t)Info.fn);
ldr(x1, MemOperand(STATE, offsetof(FEXCore::Core::CpuStateFrame, Pointers.AArch64.FallbackHandlerPointers[Info.HandlerIndex])));
blr(x1);
PopDynamicRegsAndLR();
@@ -140,8 +173,7 @@ void Arm64JITCore::Op_Unhandled(IR::IROp_Header *IROp, IR::NodeID Node) {
umov(x0, GetSrc(IROp->Args[0].ID()).V2D(), 0);
umov(w1, GetSrc(IROp->Args[0].ID()).V8H(), 4);
LoadConstant(x2, (uintptr_t)Info.fn);
ldr(x2, MemOperand(STATE, offsetof(FEXCore::Core::CpuStateFrame, Pointers.AArch64.FallbackHandlerPointers[Info.HandlerIndex])));
blr(x2);
PopDynamicRegsAndLR();
@@ -160,8 +192,7 @@ void Arm64JITCore::Op_Unhandled(IR::IROp_Header *IROp, IR::NodeID Node) {
umov(x0, GetSrc(IROp->Args[0].ID()).V2D(), 0);
umov(w1, GetSrc(IROp->Args[0].ID()).V8H(), 4);
LoadConstant(x2, (uintptr_t)Info.fn);
ldr(x2, MemOperand(STATE, offsetof(FEXCore::Core::CpuStateFrame, Pointers.AArch64.FallbackHandlerPointers[Info.HandlerIndex])));
blr(x2);
PopDynamicRegsAndLR();
@@ -180,8 +211,7 @@ void Arm64JITCore::Op_Unhandled(IR::IROp_Header *IROp, IR::NodeID Node) {
umov(x0, GetSrc(IROp->Args[0].ID()).V2D(), 0);
umov(w1, GetSrc(IROp->Args[0].ID()).V8H(), 4);
LoadConstant(x2, (uintptr_t)Info.fn);
ldr(x2, MemOperand(STATE, offsetof(FEXCore::Core::CpuStateFrame, Pointers.AArch64.FallbackHandlerPointers[Info.HandlerIndex])));
blr(x2);
PopDynamicRegsAndLR();
@@ -199,8 +229,7 @@ void Arm64JITCore::Op_Unhandled(IR::IROp_Header *IROp, IR::NodeID Node) {
umov(x0, GetSrc(IROp->Args[0].ID()).V2D(), 0);
umov(w1, GetSrc(IROp->Args[0].ID()).V8H(), 4);
LoadConstant(x2, (uintptr_t)Info.fn);
ldr(x2, MemOperand(STATE, offsetof(FEXCore::Core::CpuStateFrame, Pointers.AArch64.FallbackHandlerPointers[Info.HandlerIndex])));
blr(x2);
PopDynamicRegsAndLR();
@@ -218,8 +247,7 @@ void Arm64JITCore::Op_Unhandled(IR::IROp_Header *IROp, IR::NodeID Node) {
umov(x0, GetSrc(IROp->Args[0].ID()).V2D(), 0);
umov(w1, GetSrc(IROp->Args[0].ID()).V8H(), 4);
LoadConstant(x2, (uintptr_t)Info.fn);
ldr(x2, MemOperand(STATE, offsetof(FEXCore::Core::CpuStateFrame, Pointers.AArch64.FallbackHandlerPointers[Info.HandlerIndex])));
blr(x2);
PopDynamicRegsAndLR();
@@ -240,8 +268,7 @@ void Arm64JITCore::Op_Unhandled(IR::IROp_Header *IROp, IR::NodeID Node) {
umov(x2, GetSrc(IROp->Args[1].ID()).V2D(), 0);
umov(w3, GetSrc(IROp->Args[1].ID()).V8H(), 4);
LoadConstant(x4, (uintptr_t)Info.fn);
ldr(x4, MemOperand(STATE, offsetof(FEXCore::Core::CpuStateFrame, Pointers.AArch64.FallbackHandlerPointers[Info.HandlerIndex])));
blr(x4);
PopDynamicRegsAndLR();
@@ -259,8 +286,7 @@ void Arm64JITCore::Op_Unhandled(IR::IROp_Header *IROp, IR::NodeID Node) {
umov(x0, GetSrc(IROp->Args[0].ID()).V2D(), 0);
umov(w1, GetSrc(IROp->Args[0].ID()).V8H(), 4);
LoadConstant(x2, (uintptr_t)Info.fn);
ldr(x2, MemOperand(STATE, offsetof(FEXCore::Core::CpuStateFrame, Pointers.AArch64.FallbackHandlerPointers[Info.HandlerIndex])));
blr(x2);
PopDynamicRegsAndLR();
@@ -283,8 +309,7 @@ void Arm64JITCore::Op_Unhandled(IR::IROp_Header *IROp, IR::NodeID Node) {
umov(x2, GetSrc(IROp->Args[1].ID()).V2D(), 0);
umov(w3, GetSrc(IROp->Args[1].ID()).V8H(), 4);
LoadConstant(x4, (uintptr_t)Info.fn);
ldr(x4, MemOperand(STATE, offsetof(FEXCore::Core::CpuStateFrame, Pointers.AArch64.FallbackHandlerPointers[Info.HandlerIndex])));
blr(x4);
PopDynamicRegsAndLR();
@@ -333,9 +358,38 @@ void Arm64JITCore::FreeCodeBuffer(CodeBuffer Buffer) {
}
Arm64JITCore::Arm64JITCore(FEXCore::Context::Context *ctx, FEXCore::Core::InternalThreadState *Thread, bool CompileThread)
: Arm64Emitter(0)
: Arm64Emitter(ctx, 0)
, CTX {ctx}
, ThreadState {Thread} {
{
// Set up pointers that the JIT needs to load
auto &Pointers = ThreadState->CurrentFrame->Pointers.AArch64;
// Process specific
Pointers.LUDIV = reinterpret_cast<uint64_t>(LUDIV);
Pointers.LDIV = reinterpret_cast<uint64_t>(LDIV);
Pointers.LUREM = reinterpret_cast<uint64_t>(LUREM);
Pointers.LREM = reinterpret_cast<uint64_t>(LREM);
Pointers.PrintValue = reinterpret_cast<uint64_t>(PrintValue);
Pointers.PrintVectorValue = reinterpret_cast<uint64_t>(PrintVectorValue);
Pointers.RemoveCodeEntryFromJIT = reinterpret_cast<uintptr_t>(&Context::Context::RemoveCodeEntryFromJit);
Pointers.CPUIDObj = reinterpret_cast<uint64_t>(&CTX->CPUID);
{
FEXCore::Utils::MemberFunctionToPointerCast PMF(&FEXCore::CPUIDEmu::RunFunction);
Pointers.CPUIDFunction = PMF.GetConvertedPointer();
}
Pointers.SyscallHandlerObj = reinterpret_cast<uint64_t>(CTX->SyscallHandler);
Pointers.SyscallHandlerFunc = reinterpret_cast<uint64_t>(FEXCore::Context::HandleSyscall);
// Fill in the fallback handlers
InterpreterOps::FillFallbackIndexPointers(Pointers.FallbackHandlerPointers);
// Thread Specific
Pointers.SignalHandlerRefCountPointer = reinterpret_cast<uint64_t>(&Dispatcher->SignalHandlerRefCounter);
}
{
DispatcherConfig config;
config.ExitFunctionLink = reinterpret_cast<uintptr_t>(&ExitFunctionLink);
@@ -582,7 +636,12 @@ bool Arm64JITCore::IsInlineEntrypointOffset(const IR::OrderedNodeWrapper& WNode,
if (OpHeader->Op == IR::IROps::OP_INLINEENTRYPOINTOFFSET) {
auto Op = OpHeader->C<IR::IROp_InlineEntrypointOffset>();
if (Value) {
*Value = Entry + Op->Offset;
uint64_t Mask = ~0ULL;
uint8_t OpSize = OpHeader->Size;
if (OpSize == 4) {
Mask = 0xFFFF'FFFFULL;
}
*Value = (Entry + Op->Offset) & Mask;
}
return true;
} else {
@@ -668,7 +727,7 @@ void *Arm64JITCore::CompileCode(uint64_t Entry, [[maybe_unused]] FEXCore::IR::IR
str(x0, MemOperand(STATE, offsetof(FEXCore::Core::CpuStateFrame, State.rip)));
// Stop the thread
LoadConstant(x0, ThreadSharedData.Dispatcher->ThreadPauseHandlerAddressSpillSRA);
ldr(x0, MemOperand(STATE, offsetof(FEXCore::Core::CpuStateFrame, Pointers.AArch64.ThreadPauseHandlerSpillSRA)));
br(x0);
}
bind(&RunBlock);
@@ -176,6 +176,7 @@ private:
struct CompilerSharedData {
uint64_t SignalReturnInstruction{};
uint64_t UnimplementedInstructionAddress{};
uint64_t OverflowExceptionInstructionAddress{};
uint32_t *SignalHandlerRefCounterPtr{};
FEXCore::CPU::Dispatcher *Dispatcher{};
@@ -233,6 +234,8 @@ private:
DEF_OP(Rol);
DEF_OP(Ror);
DEF_OP(Extr);
DEF_OP(PDep);
DEF_OP(PExt);
DEF_OP(LDiv);
DEF_OP(LUDiv);
DEF_OP(LRem);
@@ -274,7 +277,6 @@ private:
///< Branch ops
DEF_OP(GuestCallDirect);
DEF_OP(GuestCallIndirect);
DEF_OP(GuestReturn);
DEF_OP(SignalReturn);
DEF_OP(CallbackReturn);
DEF_OP(ExitFunction);
@@ -302,7 +304,7 @@ private:
DEF_OP(GetHostFlag);
///< Memory ops
DEF_OP(LoadContext);
DEF_OP(LoadContext);
DEF_OP(StoreContext);
DEF_OP(LoadRegister);
DEF_OP(StoreRegister);
@@ -321,6 +323,7 @@ private:
DEF_OP(VLoadMemElement);
DEF_OP(VStoreMemElement);
DEF_OP(CacheLineClear);
DEF_OP(CacheLineZero);
///< Misc ops
DEF_OP(EndBlock);
@@ -331,6 +334,8 @@ private:
DEF_OP(Print);
DEF_OP(GetRoundingMode);
DEF_OP(SetRoundingMode);
DEF_OP(ProcessorID);
DEF_OP(RDRAND);
///< Move ops
DEF_OP(ExtractElementPair);
@@ -340,8 +345,6 @@ private:
///< Vector ops
DEF_OP(VectorZero);
DEF_OP(VectorImm);
DEF_OP(CreateVector2);
DEF_OP(CreateVector4);
DEF_OP(SplatVector2);
DEF_OP(SplatVector4);
DEF_OP(VMov);
@@ -429,6 +432,7 @@ private:
DEF_OP(VSMull2);
DEF_OP(VUABDL);
DEF_OP(VTBL1);
DEF_OP(VRev64);
///< Encryption ops
DEF_OP(AESImc);
@@ -437,6 +441,7 @@ private:
DEF_OP(AESDec);
DEF_OP(AESDecLast);
DEF_OP(AESKeyGenAssist);
DEF_OP(CRC32);
#undef DEF_OP
};
@@ -623,7 +623,7 @@ DEF_OP(LoadMemTSO) {
LOGMAN_MSG_A_FMT("LoadMemTSO: No offset allowed");
}
if (SupportsRCPC && Op->Class == FEXCore::IR::GPRClass) {
if (CTX->HostFeatures.SupportsRCPC && Op->Class == FEXCore::IR::GPRClass) {
if (IROp->Size == 1) {
// 8bit load is always aligned to natural alignment
auto Dst = GetReg<RA_64>(Node);
@@ -698,29 +698,29 @@ DEF_OP(LoadMemTSO) {
DEF_OP(StoreMem) {
auto Op = IROp->C<IR::IROp_StoreMem>();
auto MemReg = GetReg<RA_64>(Op->Header.Args[0].ID());
auto MemReg = GetReg<RA_64>(Op->Addr.ID());
auto MemSrc = GenerateMemOperand(IROp->Size, MemReg, Op->Offset, Op->OffsetType, Op->OffsetScale);
if (Op->Class == FEXCore::IR::GPRClass) {
switch (IROp->Size) {
case 1:
strb(GetReg<RA_64>(Op->Header.Args[1].ID()), MemSrc);
strb(GetReg<RA_64>(Op->Value.ID()), MemSrc);
break;
case 2:
strh(GetReg<RA_64>(Op->Header.Args[1].ID()), MemSrc);
strh(GetReg<RA_64>(Op->Value.ID()), MemSrc);
break;
case 4:
str(GetReg<RA_32>(Op->Header.Args[1].ID()), MemSrc);
str(GetReg<RA_32>(Op->Value.ID()), MemSrc);
break;
case 8:
str(GetReg<RA_64>(Op->Header.Args[1].ID()), MemSrc);
str(GetReg<RA_64>(Op->Value.ID()), MemSrc);
break;
default: LOGMAN_MSG_A_FMT("Unhandled StoreMem size: {}", IROp->Size);
}
}
else {
auto Src = GetSrc(Op->Header.Args[1].ID());
auto Src = GetSrc(Op->Value.ID());
switch (IROp->Size) {
case 1:
str(Src.B(), MemSrc);
@@ -744,7 +744,7 @@ DEF_OP(StoreMem) {
DEF_OP(StoreMemTSO) {
auto Op = IROp->C<IR::IROp_StoreMemTSO>();
auto MemSrc = MemOperand(GetReg<RA_64>(Op->Header.Args[0].ID()));
auto MemSrc = MemOperand(GetReg<RA_64>(Op->Addr.ID()));
if (!Op->Offset.IsInvalid()) {
LOGMAN_MSG_A_FMT("StoreMemTSO: No offset allowed");
@@ -753,19 +753,19 @@ DEF_OP(StoreMemTSO) {
if (Op->Class == FEXCore::IR::GPRClass) {
if (IROp->Size == 1) {
// 8bit load is always aligned to natural alignment
stlrb(GetReg<RA_64>(Op->Header.Args[1].ID()), MemSrc);
stlrb(GetReg<RA_64>(Op->Value.ID()), MemSrc);
}
else {
nop();
switch (IROp->Size) {
case 2:
stlrh(GetReg<RA_64>(Op->Header.Args[1].ID()), MemSrc);
stlrh(GetReg<RA_64>(Op->Value.ID()), MemSrc);
break;
case 4:
stlr(GetReg<RA_32>(Op->Header.Args[1].ID()), MemSrc);
stlr(GetReg<RA_32>(Op->Value.ID()), MemSrc);
break;
case 8:
stlr(GetReg<RA_64>(Op->Header.Args[1].ID()), MemSrc);
stlr(GetReg<RA_64>(Op->Value.ID()), MemSrc);
break;
default: LOGMAN_MSG_A_FMT("Unhandled StoreMemTSO size: {}", IROp->Size);
}
@@ -774,7 +774,7 @@ DEF_OP(StoreMemTSO) {
}
else {
dmb(InnerShareable, BarrierAll);
auto Src = GetSrc(Op->Header.Args[1].ID());
auto Src = GetSrc(Op->Value.ID());
switch (IROp->Size) {
case 1:
str(Src.B(), MemSrc);
@@ -800,7 +800,7 @@ DEF_OP(StoreMemTSO) {
DEF_OP(ParanoidLoadMemTSO) {
auto Op = IROp->C<IR::IROp_LoadMemTSO>();
auto MemSrc = MemOperand(GetReg<RA_64>(Op->Header.Args[0].ID()));
auto MemSrc = MemOperand(GetReg<RA_64>(Op->Addr.ID()));
if (!Op->Offset.IsInvalid()) {
LOGMAN_MSG_A_FMT("ParanoidLoadMemTSO: No offset allowed");
@@ -857,7 +857,7 @@ DEF_OP(ParanoidLoadMemTSO) {
DEF_OP(ParanoidStoreMemTSO) {
auto Op = IROp->C<IR::IROp_StoreMemTSO>();
auto MemSrc = MemOperand(GetReg<RA_64>(Op->Header.Args[0].ID()));
auto MemSrc = MemOperand(GetReg<RA_64>(Op->Addr.ID()));
if (!Op->Offset.IsInvalid()) {
LOGMAN_MSG_A_FMT("ParanoidStoreMemTSO: No offset allowed");
@@ -866,25 +866,25 @@ DEF_OP(ParanoidStoreMemTSO) {
if (Op->Class == FEXCore::IR::GPRClass) {
if (IROp->Size == 1) {
// 8bit load is always aligned to natural alignment
stlrb(GetReg<RA_64>(Op->Header.Args[1].ID()), MemSrc);
stlrb(GetReg<RA_64>(Op->Value.ID()), MemSrc);
}
else {
switch (IROp->Size) {
case 2:
stlrh(GetReg<RA_64>(Op->Header.Args[1].ID()), MemSrc);
stlrh(GetReg<RA_64>(Op->Value.ID()), MemSrc);
break;
case 4:
stlr(GetReg<RA_32>(Op->Header.Args[1].ID()), MemSrc);
stlr(GetReg<RA_32>(Op->Value.ID()), MemSrc);
break;
case 8:
stlr(GetReg<RA_64>(Op->Header.Args[1].ID()), MemSrc);
stlr(GetReg<RA_64>(Op->Value.ID()), MemSrc);
break;
default: LOGMAN_MSG_A_FMT("Unhandled ParanoidStoreMemTSO size: {}", IROp->Size);
}
}
}
else {
auto Src = GetSrc(Op->Header.Args[1].ID());
auto Src = GetSrc(Op->Value.ID());
if (IROp->Size == 1) {
// 8bit load is always aligned to natural alignment
mov(TMP1.W(), Src.V16B(), 0);
@@ -939,13 +939,35 @@ DEF_OP(CacheLineClear) {
// Clear dcache only
// icache doesn't matter here since the guest application shouldn't be calling clflush on JIT code.
mov(TMP1, MemReg);
for (size_t i = 0; i < std::max(1U, DCacheLineSize / 64U); ++i) {
for (size_t i = 0; i < std::max(1U, CTX->HostFeatures.DCacheLineSize / 64U); ++i) {
dc(DataCacheOp::CVAU, TMP1);
add(TMP1, TMP1, DCacheLineSize);
add(TMP1, TMP1, CTX->HostFeatures.DCacheLineSize);
}
dsb(InnerShareable, BarrierAll);
}
DEF_OP(CacheLineZero) {
auto Op = IROp->C<IR::IROp_CacheLineZero>();
auto MemReg = GetReg<RA_64>(Op->Header.Args[0].ID());
if (CTX->HostFeatures.SupportsCLZERO) {
// We can use this instruction directly
dc(DataCacheOp::ZVA, MemReg);
}
else {
// We must walk the cacheline ourselves
// Force cacheline alignment
and_(TMP1, MemReg, ~(CPUIDEmu::CACHELINE_SIZE - 1));
// This will end up being four STPs
// Depending on uarch it could be slightly more efficient in instructions emitted
// and uops to use vector pair STP, but we want the non-temporal bit specifically here
for (size_t i = 0; i < CPUIDEmu::CACHELINE_SIZE; i += 16) {
stnp(xzr, xzr, MemOperand(TMP1, i, Offset));
}
}
}
#undef DEF_OP
void Arm64JITCore::RegisterMemoryHandlers() {
#define REGISTER_OP(op, x) OpHandlers[FEXCore::IR::IROps::OP_##op] = &Arm64JITCore::Op_##x
@@ -972,6 +994,7 @@ void Arm64JITCore::RegisterMemoryHandlers() {
REGISTER_OP(VLOADMEMELEMENT, VLoadMemElement);
REGISTER_OP(VSTOREMEMELEMENT, VStoreMemElement);
REGISTER_OP(CACHELINECLEAR, CacheLineClear);
REGISTER_OP(CACHELINEZERO, CacheLineZero);
#undef REGISTER_OP
}
}
+83 -15
View File
@@ -7,14 +7,6 @@ $end_info$
#include "Interface/Core/JIT/Arm64/JITClass.h"
namespace FEXCore::CPU {
static void PrintValue(uint64_t Value) {
LogMan::Msg::DFmt("Value: 0x{:x}", Value);
}
static void PrintVectorValue(uint64_t Value, uint64_t ValueUpper) {
LogMan::Msg::DFmt("Value: 0x{:016x}'{:016x}", ValueUpper, Value);
}
using namespace vixl;
using namespace vixl::aarch64;
#define DEF_OP(x) void Arm64JITCore::Op_##x(IR::IROp_Header *IROp, IR::NodeID Node)
@@ -40,9 +32,13 @@ DEF_OP(Break) {
switch (Op->Reason) {
case FEXCore::IR::Break_Unimplemented: // Hard fault
case FEXCore::IR::Break_Interrupt: // Guest ud2
case FEXCore::IR::Break_Overflow: // overflow
hlt(4);
break;
case FEXCore::IR::Break_Overflow: // overflow
ResetStack();
ldr(TMP1, MemOperand(STATE, offsetof(FEXCore::Core::CpuStateFrame, Pointers.AArch64.OverflowExceptionHandler)));
br(TMP1);
break;
case FEXCore::IR::Break_Halt: { // HLT
// Time to quit
// Set our stack to the starting stack location
@@ -50,14 +46,13 @@ DEF_OP(Break) {
add(sp, TMP1, 0);
// Now we need to jump to the thread stop handler
LoadConstant(TMP1, ThreadSharedData.Dispatcher->ThreadStopHandlerAddressSpillSRA);
ldr(TMP1, MemOperand(STATE, offsetof(FEXCore::Core::CpuStateFrame, Pointers.AArch64.ThreadStopHandlerSpillSRA)));
br(TMP1);
break;
}
case FEXCore::IR::Break_Interrupt3: { // INT3
ResetStack();
LoadConstant(TMP1, ThreadSharedData.Dispatcher->ThreadPauseHandlerAddressSpillSRA);
ldr(TMP1, MemOperand(STATE, offsetof(FEXCore::Core::CpuStateFrame, Pointers.AArch64.ThreadPauseHandlerSpillSRA)));
br(TMP1);
break;
}
@@ -65,7 +60,7 @@ DEF_OP(Break) {
{
ResetStack();
LoadConstant(TMP1, ThreadSharedData.Dispatcher->UnimplementedInstructionAddress);
ldr(TMP1, MemOperand(STATE, offsetof(FEXCore::Core::CpuStateFrame, Pointers.AArch64.UnimplementedInstructionHandler)));
br(TMP1);
break;
@@ -139,13 +134,13 @@ DEF_OP(Print) {
if (IsGPR(Op->Header.Args[0].ID())) {
mov(x0, GetReg<RA_64>(Op->Header.Args[0].ID()));
LoadConstant(x3, reinterpret_cast<uint64_t>(PrintValue));
ldr(x3, MemOperand(STATE, offsetof(FEXCore::Core::CpuStateFrame, Pointers.AArch64.PrintValue)));
}
else {
fmov(x0, GetSrc(Op->Header.Args[0].ID()).V1D());
// Bug in vixl that source vector needs to b V1D rather than V2D?
fmov(x1, GetSrc(Op->Header.Args[0].ID()).V1D(), 1);
LoadConstant(x3, reinterpret_cast<uint64_t>(PrintVectorValue));
ldr(x3, MemOperand(STATE, offsetof(FEXCore::Core::CpuStateFrame, Pointers.AArch64.PrintVectorValue)));
}
SpillStaticRegs();
blr(x3);
@@ -154,6 +149,76 @@ DEF_OP(Print) {
PopDynamicRegsAndLR();
}
DEF_OP(ProcessorID) {
// We always need to spill x8 since we can't know if it is live at this SSA location
uint32_t SpillMask = 1U << 8;
// Ordering is incredibly important here
// We must spill any overlapping registers first THEN claim we are in a syscall without invalidating state at all
// Only spill the registers that intersect with our usage
SpillStaticRegs(false, SpillMask);
// Now that we are spilled, store in the state that we are in a syscall
// Still without overwriting registers that matter
// 16bit LoadConstant to be a single instruction
// We must always spill at least one register (x8) so this value always has a bit set
// This gives the signal handler a value to check to see if we are in a syscall at all
LoadConstant(x0, SpillMask & 0xFFFF);
str(x0, MemOperand(STATE, offsetof(FEXCore::Core::CpuStateFrame, InSyscallInfo)));
// Allocate some temporary space for storing the uint32_t CPU and Node IDs
sub(sp, sp, 16);
// Load the getcpu syscall number
LoadConstant(x8, SYS_getcpu);
// CPU pointer in x0
add(x0, sp, 0);
// Node in x1
add(x1, sp, 4);
svc(0);
// On updated signal mask we can receive a signal RIGHT HERE
// Load the values returned by the kernel
ldp(w0, w1, MemOperand(sp));
// Deallocate stack space
sub(sp, sp, 16);
// Now that we are done in the syscall we need to carefully peel back the state
// First unspill the registers from before
FillStaticRegs(false, SpillMask);
// Now the registers we've spilled are back in their original host registers
// We can safely claim we are no longer in a syscall
str(xzr, MemOperand(STATE, offsetof(FEXCore::Core::CpuStateFrame, InSyscallInfo)));
// Now store the result in the destination in the expected format
// uint32_t Res = (node << 12) | cpu;
// CPU is in w0
// Node is in w1
orr(GetReg<RA_64>(Node), x0, Operand(x1, LSL, 12));
}
DEF_OP(RDRAND) {
auto Op = IROp->C<IR::IROp_RDRAND>();
// Results are in x0, x1
// Results want to be in a i64v2 vector
auto Dst = GetSrcPair<RA_64>(Node);
if (Op->GetReseeded) {
mrs(Dst.first, RNDRRS);
}
else {
mrs(Dst.first, RNDR);
}
// If the rng number is valid then NZCV is 0b0000, otherwise NZCV is 0b0100
cset(Dst.second, Condition::ne);
}
#undef DEF_OP
void Arm64JITCore::RegisterMiscHandlers() {
#define REGISTER_OP(op, x) OpHandlers[FEXCore::IR::IROps::OP_##op] = &Arm64JITCore::Op_##x
@@ -170,6 +235,9 @@ void Arm64JITCore::RegisterMiscHandlers() {
REGISTER_OP(GETROUNDINGMODE, GetRoundingMode);
REGISTER_OP(SETROUNDINGMODE, SetRoundingMode);
REGISTER_OP(INVALIDATEFLAGS, NoOp);
REGISTER_OP(PROCESSORID, ProcessorID);
REGISTER_OP(RDRAND, RDRAND);
#undef REGISTER_OP
}
}
@@ -37,7 +37,7 @@ DEF_OP(CreateElementPair) {
aarch64::Register RegSecond;
aarch64::Register RegTmp;
switch (Op->Header.Size) {
switch (IROp->ElementSize) {
case 4: {
Dst = GetSrcPair<RA_32>(Node);
RegFirst = GetReg<RA_32>(Op->Header.Args[0].ID());
@@ -42,14 +42,6 @@ DEF_OP(VectorImm) {
}
}
DEF_OP(CreateVector2) {
LOGMAN_MSG_A_FMT("Unimplemented");
}
DEF_OP(CreateVector4) {
LOGMAN_MSG_A_FMT("Unimplemented");
}
DEF_OP(SplatVector2) {
auto Op = IROp->C<IR::IROp_SplatVector2>();
uint8_t OpSize = IROp->Size;
@@ -1758,8 +1750,7 @@ DEF_OP(VInsScalarElement) {
DEF_OP(VExtractElement) {
auto Op = IROp->C<IR::IROp_VExtractElement>();
uint8_t OpSize = IROp->Size;
switch (OpSize) {
switch (Op->Header.Size) {
case 1:
mov(GetDst(Node).B(), GetSrc(Op->Header.Args[0].ID()).V16B(), Op->Index);
break;
@@ -1772,7 +1763,7 @@ DEF_OP(VExtractElement) {
case 8:
mov(GetDst(Node).D(), GetSrc(Op->Header.Args[0].ID()).V2D(), Op->Index);
break;
default: LOGMAN_MSG_A_FMT("Unhandled VExtractElement element size: {}", OpSize);
default: LOGMAN_MSG_A_FMT("Unhandled VExtractElement element size: {}", Op->Header.Size);
}
}
@@ -2318,13 +2309,27 @@ DEF_OP(VTBL1) {
}
}
DEF_OP(VRev64) {
auto Op = IROp->C<IR::IROp_VRev64>();
uint8_t OpSize = IROp->Size;
uint8_t Elements = OpSize / Op->Header.ElementSize;
// Vector
switch (Op->Header.ElementSize) {
case 1:
case 2:
case 4:
rev64(GetDst(Node).VCast(OpSize * 8, Elements), GetSrc(Op->Header.Args[0].ID()).VCast(OpSize * 8, Elements));
break;
case 8:
default: LOGMAN_MSG_A_FMT("Invalid Element Size: {}", Op->Header.ElementSize); break;
}
}
#undef DEF_OP
void Arm64JITCore::RegisterVectorHandlers() {
#define REGISTER_OP(op, x) OpHandlers[FEXCore::IR::IROps::OP_##op] = &Arm64JITCore::Op_##x
REGISTER_OP(VECTORZERO, VectorZero);
REGISTER_OP(VECTORIMM, VectorImm);
REGISTER_OP(CREATEVECTOR2, CreateVector2);
REGISTER_OP(CREATEVECTOR4, CreateVector4);
REGISTER_OP(SPLATVECTOR2, SplatVector2);
REGISTER_OP(SPLATVECTOR4, SplatVector4);
REGISTER_OP(VMOV, VMov);
@@ -2413,6 +2418,7 @@ void Arm64JITCore::RegisterVectorHandlers() {
REGISTER_OP(VSMULL2, VSMull2);
REGISTER_OP(VUABDL, VUABDL);
REGISTER_OP(VTBL1, VTBL1);
REGISTER_OP(VREV64, VRev64);
#undef REGISTER_OP
}
}
@@ -24,7 +24,7 @@ namespace FEXCore::CPU {
DEF_OP(TruncElementPair) {
auto Op = IROp->C<IR::IROp_TruncElementPair>();
switch (Op->Size) {
switch (IROp->Size) {
case 4: {
auto Dst = GetSrcPair<RA_32>(Node);
auto Src = GetSrcPair<RA_32>(Op->Header.Args[0].ID());
@@ -32,7 +32,7 @@ DEF_OP(TruncElementPair) {
mov(Dst.second, Src.second);
break;
}
default: LOGMAN_MSG_A_FMT("Unhandled Truncation size: {}", Op->Size); break;
default: LOGMAN_MSG_A_FMT("Unhandled Truncation size: {}", IROp->Size); break;
}
}
@@ -45,7 +45,13 @@ DEF_OP(EntrypointOffset) {
auto Op = IROp->C<IR::IROp_EntrypointOffset>();
auto Constant = Entry + Op->Offset;
mov(GetDst<RA_64>(Node), Constant);
uint64_t Mask = ~0ULL;
uint8_t OpSize = IROp->Size;
if (OpSize == 4) {
Mask = 0xFFFF'FFFFULL;
}
mov(GetDst<RA_64>(Node), Constant & Mask);
}
DEF_OP(InlineConstant) {
@@ -671,6 +677,36 @@ DEF_OP(Extr) {
}
}
DEF_OP(PDep) {
const auto Op = IROp->C<IR::IROp_PExt>();
const auto OpSize = IROp->Size;
const auto Input = GRS(Op->Args(0).ID());
const auto Mask = GRS(Op->Args(1).ID());
const auto Dest = GRD(Node);
if (OpSize == 4) {
pdep(Dest.cvt32(), Input.cvt32(), Mask.cvt32());
} else {
pdep(Dest.cvt64(), Input.cvt64(), Mask.cvt64());
}
}
DEF_OP(PExt) {
const auto Op = IROp->C<IR::IROp_PExt>();
const auto OpSize = IROp->Size;
const auto Input = GRS(Op->Args(0).ID());
const auto Mask = GRS(Op->Args(1).ID());
const auto Dest = GRD(Node);
if (OpSize == 4) {
pext(Dest.cvt32(), Input.cvt32(), Mask.cvt32());
} else {
pext(Dest.cvt64(), Input.cvt64(), Mask.cvt64());
}
}
DEF_OP(LDiv) {
auto Op = IROp->C<IR::IROp_LDiv>();
uint8_t OpSize = IROp->Size;
@@ -1114,19 +1150,19 @@ DEF_OP(VExtractToGPR) {
switch (Op->Header.ElementSize) {
case 1: {
pextrb(GetDst<RA_32>(Node), GetSrc(Op->Header.Args[0].ID()), Op->Idx);
pextrb(GetDst<RA_32>(Node), GetSrc(Op->Header.Args[0].ID()), Op->Index);
break;
}
case 2: {
pextrw(GetDst<RA_32>(Node), GetSrc(Op->Header.Args[0].ID()), Op->Idx);
pextrw(GetDst<RA_32>(Node), GetSrc(Op->Header.Args[0].ID()), Op->Index);
break;
}
case 4: {
pextrd(GetDst<RA_32>(Node), GetSrc(Op->Header.Args[0].ID()), Op->Idx);
pextrd(GetDst<RA_32>(Node), GetSrc(Op->Header.Args[0].ID()), Op->Index);
break;
}
case 8: {
pextrq(GetDst<RA_64>(Node), GetSrc(Op->Header.Args[0].ID()), Op->Idx);
pextrq(GetDst<RA_64>(Node), GetSrc(Op->Header.Args[0].ID()), Op->Index);
break;
}
default: LOGMAN_MSG_A_FMT("Unknown Element Size: {}", Op->Header.ElementSize); break;
@@ -1248,6 +1284,8 @@ void X86JITCore::RegisterALUHandlers() {
REGISTER_OP(ASHR, Ashr);
REGISTER_OP(ROR, Ror);
REGISTER_OP(EXTR, Extr);
REGISTER_OP(PDEP, PDep);
REGISTER_OP(PEXT, PExt);
REGISTER_OP(LDIV, LDiv);
REGISTER_OP(LUDIV, LUDiv);
REGISTER_OP(LREM, LRem);
@@ -18,20 +18,16 @@ namespace FEXCore::CPU {
#define DEF_OP(x) void X86JITCore::Op_##x(IR::IROp_Header *IROp, IR::NodeID Node)
DEF_OP(CASPair) {
auto Op = IROp->C<IR::IROp_CAS>();
uint8_t OpSize = IROp->Size;
// Args[0]: Desired
// Args[1]: Expected
// Args[2]: Pointer
// DataSrc = *Src1
// if (DataSrc == Src3) { *Src1 == Src2; } Src2 = DataSrc
// This will write to memory! Careful!
// Third operand must be a calculated guest memory address
//OrderedNode *CASResult = _CAS(Src3, Src2, Src1);
auto Dst = GetSrcPair<RA_64>(Node);
auto Expected = GetSrcPair<RA_64>(Op->Header.Args[0].ID());
auto Desired = GetSrcPair<RA_64>(Op->Header.Args[1].ID());
auto MemSrc = GetSrc<RA_64>(Op->Header.Args[2].ID());
auto Expected = GetSrcPair<RA_64>(Op->Expected.ID());
auto Desired = GetSrcPair<RA_64>(Op->Desired.ID());
auto MemSrc = GetSrc<RA_64>(Op->Addr.ID());
Xbyak::Reg MemReg = MemSrc;
@@ -47,7 +43,7 @@ DEF_OP(CASPair) {
lock();
switch (OpSize) {
switch (IROp->ElementSize) {
case 4: {
cmpxchg8b(dword [MemReg]);
// EDX:EAX now contains the result
@@ -62,7 +58,7 @@ DEF_OP(CASPair) {
mov(Dst.second, rdx);
break;
}
default: LOGMAN_MSG_A_FMT("Unsupported: {}", OpSize);
default: LOGMAN_MSG_A_FMT("Unsupported: {}", IROp->ElementSize);
}
}
@@ -70,18 +66,15 @@ DEF_OP(CAS) {
auto Op = IROp->C<IR::IROp_CAS>();
uint8_t OpSize = IROp->Size;
// Args[0]: Desired
// Args[1]: Expected
// Args[2]: Pointer
// DataSrc = *Src1
// if (DataSrc == Src3) { *Src1 == Src2; } Src2 = DataSrc
// This will write to memory! Careful!
// Third operand must be a calculated guest memory address
//OrderedNode *CASResult = _CAS(Src3, Src2, Src1);
Xbyak::Reg MemReg = GetSrc<RA_64>(Op->Header.Args[2].ID());
Xbyak::Reg MemReg = GetSrc<RA_64>(Op->Addr.ID());
mov(rax, GetSrc<RA_64>(Op->Header.Args[0].ID()));
mov(rax, GetSrc<RA_64>(Op->Expected.ID()));
// RCX now contains pointer
// RAX contains our expected value
@@ -90,23 +83,23 @@ DEF_OP(CAS) {
lock();
switch (OpSize) {
case 1: {
cmpxchg(byte [MemReg], GetSrc<RA_8>(Op->Header.Args[1].ID()));
cmpxchg(byte [MemReg], GetSrc<RA_8>(Op->Desired.ID()));
movzx(GetDst<RA_64>(Node), al);
break;
}
case 2: {
cmpxchg(word [MemReg], GetSrc<RA_16>(Op->Header.Args[1].ID()));
cmpxchg(word [MemReg], GetSrc<RA_16>(Op->Desired.ID()));
movzx(GetDst<RA_64>(Node), ax);
break;
}
case 4: {
cmpxchg(dword [MemReg], GetSrc<RA_32>(Op->Header.Args[1].ID()));
cmpxchg(dword [MemReg], GetSrc<RA_32>(Op->Desired.ID()));
// RAX now contains the result
mov (GetDst<RA_64>(Node), eax);
break;
}
case 8: {
cmpxchg(qword [MemReg], GetSrc<RA_64>(Op->Header.Args[1].ID()));
cmpxchg(qword [MemReg], GetSrc<RA_64>(Op->Desired.ID()));
// RAX now contains the result
mov (GetDst<RA_64>(Node), rax);
break;
@@ -118,21 +111,21 @@ DEF_OP(CAS) {
DEF_OP(AtomicAdd) {
auto Op = IROp->C<IR::IROp_AtomicAdd>();
Xbyak::Reg MemReg = GetSrc<RA_64>(Op->Header.Args[0].ID());
Xbyak::Reg MemReg = GetSrc<RA_64>(Op->Addr.ID());
lock();
switch (IROp->Size) {
case 1:
add(byte [MemReg], GetSrc<RA_8>(Op->Header.Args[1].ID()));
add(byte [MemReg], GetSrc<RA_8>(Op->Value.ID()));
break;
case 2:
add(word [MemReg], GetSrc<RA_16>(Op->Header.Args[1].ID()));
add(word [MemReg], GetSrc<RA_16>(Op->Value.ID()));
break;
case 4:
add(dword [MemReg], GetSrc<RA_32>(Op->Header.Args[1].ID()));
add(dword [MemReg], GetSrc<RA_32>(Op->Value.ID()));
break;
case 8:
add(qword [MemReg], GetSrc<RA_64>(Op->Header.Args[1].ID()));
add(qword [MemReg], GetSrc<RA_64>(Op->Value.ID()));
break;
default: LOGMAN_MSG_A_FMT("Unhandled AtomicAdd size: {}", IROp->Size);
}
@@ -141,20 +134,20 @@ DEF_OP(AtomicAdd) {
DEF_OP(AtomicSub) {
auto Op = IROp->C<IR::IROp_AtomicSub>();
Xbyak::Reg MemReg = GetSrc<RA_64>(Op->Header.Args[0].ID());
Xbyak::Reg MemReg = GetSrc<RA_64>(Op->Addr.ID());
lock();
switch (IROp->Size) {
case 1:
sub(byte [MemReg], GetSrc<RA_8>(Op->Header.Args[1].ID()));
sub(byte [MemReg], GetSrc<RA_8>(Op->Value.ID()));
break;
case 2:
sub(word [MemReg], GetSrc<RA_16>(Op->Header.Args[1].ID()));
sub(word [MemReg], GetSrc<RA_16>(Op->Value.ID()));
break;
case 4:
sub(dword [MemReg], GetSrc<RA_32>(Op->Header.Args[1].ID()));
sub(dword [MemReg], GetSrc<RA_32>(Op->Value.ID()));
break;
case 8:
sub(qword [MemReg], GetSrc<RA_64>(Op->Header.Args[1].ID()));
sub(qword [MemReg], GetSrc<RA_64>(Op->Value.ID()));
break;
default: LOGMAN_MSG_A_FMT("Unhandled AtomicAdd size: {}", IROp->Size);
}
@@ -163,20 +156,20 @@ DEF_OP(AtomicSub) {
DEF_OP(AtomicAnd) {
auto Op = IROp->C<IR::IROp_AtomicAnd>();
Xbyak::Reg MemReg = GetSrc<RA_64>(Op->Header.Args[0].ID());
Xbyak::Reg MemReg = GetSrc<RA_64>(Op->Addr.ID());
lock();
switch (IROp->Size) {
case 1:
and_(byte [MemReg], GetSrc<RA_8>(Op->Header.Args[1].ID()));
and_(byte [MemReg], GetSrc<RA_8>(Op->Value.ID()));
break;
case 2:
and_(word [MemReg], GetSrc<RA_16>(Op->Header.Args[1].ID()));
and_(word [MemReg], GetSrc<RA_16>(Op->Value.ID()));
break;
case 4:
and_(dword [MemReg], GetSrc<RA_32>(Op->Header.Args[1].ID()));
and_(dword [MemReg], GetSrc<RA_32>(Op->Value.ID()));
break;
case 8:
and_(qword [MemReg], GetSrc<RA_64>(Op->Header.Args[1].ID()));
and_(qword [MemReg], GetSrc<RA_64>(Op->Value.ID()));
break;
default: LOGMAN_MSG_A_FMT("Unhandled AtomicAdd size: {}", IROp->Size);
}
@@ -185,20 +178,20 @@ DEF_OP(AtomicAnd) {
DEF_OP(AtomicOr) {
auto Op = IROp->C<IR::IROp_AtomicOr>();
Xbyak::Reg MemReg = GetSrc<RA_64>(Op->Header.Args[0].ID());
Xbyak::Reg MemReg = GetSrc<RA_64>(Op->Addr.ID());
lock();
switch (IROp->Size) {
case 1:
or_(byte [MemReg], GetSrc<RA_8>(Op->Header.Args[1].ID()));
or_(byte [MemReg], GetSrc<RA_8>(Op->Value.ID()));
break;
case 2:
or_(word [MemReg], GetSrc<RA_16>(Op->Header.Args[1].ID()));
or_(word [MemReg], GetSrc<RA_16>(Op->Value.ID()));
break;
case 4:
or_(dword [MemReg], GetSrc<RA_32>(Op->Header.Args[1].ID()));
or_(dword [MemReg], GetSrc<RA_32>(Op->Value.ID()));
break;
case 8:
or_(qword [MemReg], GetSrc<RA_64>(Op->Header.Args[1].ID()));
or_(qword [MemReg], GetSrc<RA_64>(Op->Value.ID()));
break;
default: LOGMAN_MSG_A_FMT("Unhandled AtomicAdd size: {}", IROp->Size);
}
@@ -207,20 +200,20 @@ DEF_OP(AtomicOr) {
DEF_OP(AtomicXor) {
auto Op = IROp->C<IR::IROp_AtomicXor>();
Xbyak::Reg MemReg = GetSrc<RA_64>(Op->Header.Args[0].ID());
Xbyak::Reg MemReg = GetSrc<RA_64>(Op->Addr.ID());
lock();
switch (IROp->Size) {
case 1:
xor_(byte [MemReg], GetSrc<RA_8>(Op->Header.Args[1].ID()));
xor_(byte [MemReg], GetSrc<RA_8>(Op->Value.ID()));
break;
case 2:
xor_(word [MemReg], GetSrc<RA_16>(Op->Header.Args[1].ID()));
xor_(word [MemReg], GetSrc<RA_16>(Op->Value.ID()));
break;
case 4:
xor_(dword [MemReg], GetSrc<RA_32>(Op->Header.Args[1].ID()));
xor_(dword [MemReg], GetSrc<RA_32>(Op->Value.ID()));
break;
case 8:
xor_(qword [MemReg], GetSrc<RA_64>(Op->Header.Args[1].ID()));
xor_(qword [MemReg], GetSrc<RA_64>(Op->Value.ID()));
break;
default: LOGMAN_MSG_A_FMT("Unhandled AtomicAdd size: {}", IROp->Size);
}
@@ -230,26 +223,26 @@ DEF_OP(AtomicSwap) {
auto Op = IROp->C<IR::IROp_AtomicSwap>();
Xbyak::Reg MemReg = rax;
mov(MemReg, GetSrc<RA_64>(Op->Header.Args[0].ID()));
mov(MemReg, GetSrc<RA_64>(Op->Addr.ID()));
switch (IROp->Size) {
case 1:
movzx(GetDst<RA_64>(Node), GetSrc<RA_8>(Op->Header.Args[1].ID()));
movzx(GetDst<RA_64>(Node), GetSrc<RA_8>(Op->Value.ID()));
lock();
xchg(byte [MemReg], GetDst<RA_8>(Node));
break;
case 2:
movzx(GetDst<RA_64>(Node), GetSrc<RA_16>(Op->Header.Args[1].ID()));
movzx(GetDst<RA_64>(Node), GetSrc<RA_16>(Op->Value.ID()));
lock();
xchg(word [MemReg], GetDst<RA_16>(Node));
break;
case 4:
mov(GetDst<RA_64>(Node), GetSrc<RA_32>(Op->Header.Args[1].ID()));
mov(GetDst<RA_64>(Node), GetSrc<RA_32>(Op->Value.ID()));
lock();
xchg(dword [MemReg], GetDst<RA_32>(Node));
break;
case 8:
mov(GetDst<RA_64>(Node), GetSrc<RA_64>(Op->Header.Args[1].ID()));
mov(GetDst<RA_64>(Node), GetSrc<RA_64>(Op->Value.ID()));
lock();
xchg(qword [MemReg], GetDst<RA_64>(Node));
break;
@@ -260,28 +253,28 @@ DEF_OP(AtomicSwap) {
DEF_OP(AtomicFetchAdd) {
auto Op = IROp->C<IR::IROp_AtomicFetchAdd>();
Xbyak::Reg MemReg = GetSrc<RA_64>(Op->Header.Args[0].ID());
Xbyak::Reg MemReg = GetSrc<RA_64>(Op->Addr.ID());
switch (IROp->Size) {
case 1:
movzx(rcx, GetSrc<RA_8>(Op->Header.Args[1].ID()));
movzx(rcx, GetSrc<RA_8>(Op->Value.ID()));
lock();
xadd(byte [MemReg], cl);
movzx(GetDst<RA_32>(Node), cl);
break;
case 2:
movzx(rcx, GetSrc<RA_16>(Op->Header.Args[1].ID()));
movzx(rcx, GetSrc<RA_16>(Op->Value.ID()));
lock();
xadd(word [MemReg], cx);
movzx(GetDst<RA_32>(Node), cx);
break;
case 4:
mov(ecx, GetSrc<RA_32>(Op->Header.Args[1].ID()));
mov(ecx, GetSrc<RA_32>(Op->Value.ID()));
lock();
xadd(dword [MemReg], ecx);
mov(GetDst<RA_64>(Node), ecx);
break;
case 8:
mov(rcx, GetSrc<RA_64>(Op->Header.Args[1].ID()));
mov(rcx, GetSrc<RA_64>(Op->Value.ID()));
lock();
xadd(qword [MemReg], rcx);
mov(GetDst<RA_64>(Node), rcx);
@@ -293,31 +286,31 @@ DEF_OP(AtomicFetchAdd) {
DEF_OP(AtomicFetchSub) {
auto Op = IROp->C<IR::IROp_AtomicFetchSub>();
Xbyak::Reg MemReg = GetSrc<RA_64>(Op->Header.Args[0].ID());
Xbyak::Reg MemReg = GetSrc<RA_64>(Op->Addr.ID());
switch (IROp->Size) {
case 1:
mov(cl, GetSrc<RA_8>(Op->Header.Args[1].ID()));
mov(cl, GetSrc<RA_8>(Op->Value.ID()));
neg(cl);
lock();
xadd(byte [MemReg], cl);
movzx(GetDst<RA_32>(Node), cl);
break;
case 2:
mov(cx, GetSrc<RA_16>(Op->Header.Args[1].ID()));
mov(cx, GetSrc<RA_16>(Op->Value.ID()));
neg(cx);
lock();
xadd(word [MemReg], cx);
movzx(GetDst<RA_32>(Node), cx);
break;
case 4:
mov(ecx, GetSrc<RA_32>(Op->Header.Args[1].ID()));
mov(ecx, GetSrc<RA_32>(Op->Value.ID()));
neg(ecx);
lock();
xadd(dword [MemReg], ecx);
mov(GetDst<RA_32>(Node), ecx);
break;
case 8:
mov(rcx, GetSrc<RA_64>(Op->Header.Args[1].ID()));
mov(rcx, GetSrc<RA_64>(Op->Value.ID()));
neg(rcx);
lock();
xadd(qword [MemReg], rcx);
@@ -331,7 +324,7 @@ DEF_OP(AtomicFetchAnd) {
auto Op = IROp->C<IR::IROp_AtomicFetchAnd>();
// TMP1 = rax
Xbyak::Reg MemReg = GetSrc<RA_64>(Op->Header.Args[0].ID());
Xbyak::Reg MemReg = GetSrc<RA_64>(Op->Addr.ID());
switch (IROp->Size) {
case 1: {
@@ -341,7 +334,7 @@ DEF_OP(AtomicFetchAnd) {
L(Loop);
mov(TMP2.cvt8(), TMP1.cvt8());
mov(TMP3.cvt8(), TMP1.cvt8());
and_(TMP2.cvt8(), GetSrc<RA_8>(Op->Header.Args[1].ID()));
and_(TMP2.cvt8(), GetSrc<RA_8>(Op->Value.ID()));
// Updates RAX with the value from memory
lock(); cmpxchg(byte [MemReg], TMP2.cvt8());
@@ -357,7 +350,7 @@ DEF_OP(AtomicFetchAnd) {
L(Loop);
mov(TMP2.cvt16(), TMP1.cvt16());
mov(TMP3.cvt16(), TMP1.cvt16());
and_(TMP2.cvt16(), GetSrc<RA_16>(Op->Header.Args[1].ID()));
and_(TMP2.cvt16(), GetSrc<RA_16>(Op->Value.ID()));
// Updates RAX with the value from memory
lock(); cmpxchg(word [MemReg], TMP2.cvt16());
@@ -374,7 +367,7 @@ DEF_OP(AtomicFetchAnd) {
L(Loop);
mov(TMP2.cvt32(), TMP1.cvt32());
mov(TMP3.cvt32(), TMP1.cvt32());
and_(TMP2.cvt32(), GetSrc<RA_32>(Op->Header.Args[1].ID()));
and_(TMP2.cvt32(), GetSrc<RA_32>(Op->Value.ID()));
// Updates RAX with the value from memory
lock(); cmpxchg(dword [MemReg], TMP2.cvt32());
@@ -391,7 +384,7 @@ DEF_OP(AtomicFetchAnd) {
L(Loop);
mov(TMP2.cvt64(), TMP1.cvt64());
mov(TMP3.cvt64(), TMP1.cvt64());
and_(TMP2.cvt64(), GetSrc<RA_64>(Op->Header.Args[1].ID()));
and_(TMP2.cvt64(), GetSrc<RA_64>(Op->Value.ID()));
// Updates RAX with the value from memory
lock(); cmpxchg(qword [MemReg], TMP2.cvt64());
@@ -409,7 +402,7 @@ DEF_OP(AtomicFetchOr) {
auto Op = IROp->C<IR::IROp_AtomicFetchOr>();
// TMP1 = rax
Xbyak::Reg MemReg = GetSrc<RA_64>(Op->Header.Args[0].ID());
Xbyak::Reg MemReg = GetSrc<RA_64>(Op->Addr.ID());
switch (IROp->Size) {
case 1: {
mov(TMP1.cvt8(), byte [MemReg]);
@@ -418,7 +411,7 @@ DEF_OP(AtomicFetchOr) {
L(Loop);
mov(TMP2.cvt8(), TMP1.cvt8());
mov(TMP3.cvt8(), TMP1.cvt8());
or_(TMP2.cvt8(), GetSrc<RA_8>(Op->Header.Args[1].ID()));
or_(TMP2.cvt8(), GetSrc<RA_8>(Op->Value.ID()));
// Updates RAX with the value from memory
lock(); cmpxchg(byte [MemReg], TMP2.cvt8());
@@ -434,7 +427,7 @@ DEF_OP(AtomicFetchOr) {
L(Loop);
mov(TMP2.cvt16(), TMP1.cvt16());
mov(TMP3.cvt16(), TMP1.cvt16());
or_(TMP2.cvt16(), GetSrc<RA_16>(Op->Header.Args[1].ID()));
or_(TMP2.cvt16(), GetSrc<RA_16>(Op->Value.ID()));
// Updates RAX with the value from memory
lock(); cmpxchg(word [MemReg], TMP2.cvt16());
@@ -451,7 +444,7 @@ DEF_OP(AtomicFetchOr) {
L(Loop);
mov(TMP2.cvt32(), TMP1.cvt32());
mov(TMP3.cvt32(), TMP1.cvt32());
or_(TMP2.cvt32(), GetSrc<RA_32>(Op->Header.Args[1].ID()));
or_(TMP2.cvt32(), GetSrc<RA_32>(Op->Value.ID()));
// Updates RAX with the value from memory
lock(); cmpxchg(dword [MemReg], TMP2.cvt32());
@@ -468,7 +461,7 @@ DEF_OP(AtomicFetchOr) {
L(Loop);
mov(TMP2.cvt64(), TMP1.cvt64());
mov(TMP3.cvt64(), TMP1.cvt64());
or_(TMP2.cvt64(), GetSrc<RA_64>(Op->Header.Args[1].ID()));
or_(TMP2.cvt64(), GetSrc<RA_64>(Op->Value.ID()));
// Updates RAX with the value from memory
lock(); cmpxchg(qword [MemReg], TMP2.cvt64());
@@ -486,7 +479,7 @@ DEF_OP(AtomicFetchXor) {
auto Op = IROp->C<IR::IROp_AtomicFetchXor>();
// TMP1 = rax
Xbyak::Reg MemReg = GetSrc<RA_64>(Op->Header.Args[0].ID());
Xbyak::Reg MemReg = GetSrc<RA_64>(Op->Addr.ID());
switch (IROp->Size) {
case 1: {
mov(TMP1.cvt8(), byte [MemReg]);
@@ -495,7 +488,7 @@ DEF_OP(AtomicFetchXor) {
L(Loop);
mov(TMP2.cvt8(), TMP1.cvt8());
mov(TMP3.cvt8(), TMP1.cvt8());
xor_(TMP2.cvt8(), GetSrc<RA_8>(Op->Header.Args[1].ID()));
xor_(TMP2.cvt8(), GetSrc<RA_8>(Op->Value.ID()));
// Updates RAX with the value from memory
lock(); cmpxchg(byte [MemReg], TMP2.cvt8());
@@ -511,7 +504,7 @@ DEF_OP(AtomicFetchXor) {
L(Loop);
mov(TMP2.cvt16(), TMP1.cvt16());
mov(TMP3.cvt16(), TMP1.cvt16());
xor_(TMP2.cvt16(), GetSrc<RA_16>(Op->Header.Args[1].ID()));
xor_(TMP2.cvt16(), GetSrc<RA_16>(Op->Value.ID()));
// Updates RAX with the value from memory
lock(); cmpxchg(word [MemReg], TMP2.cvt16());
@@ -528,7 +521,7 @@ DEF_OP(AtomicFetchXor) {
L(Loop);
mov(TMP2.cvt32(), TMP1.cvt32());
mov(TMP3.cvt32(), TMP1.cvt32());
xor_(TMP2.cvt32(), GetSrc<RA_32>(Op->Header.Args[1].ID()));
xor_(TMP2.cvt32(), GetSrc<RA_32>(Op->Value.ID()));
// Updates RAX with the value from memory
lock(); cmpxchg(dword [MemReg], TMP2.cvt32());
@@ -545,7 +538,7 @@ DEF_OP(AtomicFetchXor) {
L(Loop);
mov(TMP2.cvt64(), TMP1.cvt64());
mov(TMP3.cvt64(), TMP1.cvt64());
xor_(TMP2.cvt64(), GetSrc<RA_64>(Op->Header.Args[1].ID()));
xor_(TMP2.cvt64(), GetSrc<RA_64>(Op->Value.ID()));
// Updates RAX with the value from memory
lock(); cmpxchg(qword [MemReg], TMP2.cvt64());
@@ -562,7 +555,7 @@ DEF_OP(AtomicFetchXor) {
DEF_OP(AtomicFetchNeg) {
auto Op = IROp->C<IR::IROp_AtomicFetchNeg>();
Xbyak::Reg MemReg = GetSrc<RA_64>(Op->Header.Args[0].ID());
Xbyak::Reg MemReg = GetSrc<RA_64>(Op->Addr.ID());
switch (IROp->Size) {
case 1: {
mov(TMP1.cvt8(), byte [MemReg]);
@@ -37,18 +37,13 @@ DEF_OP(GuestCallIndirect) {
LogMan::Msg::DFmt("Unimplemented");
}
DEF_OP(GuestReturn) {
LogMan::Msg::DFmt("Unimplemented");
}
DEF_OP(SignalReturn) {
// Adjust the stack first for a regular return
if (SpillSlots) {
add(rsp, SpillSlots * 16); // + 8 to consume return address
}
mov(TMP1, ThreadSharedData.SignalHandlerReturnAddress);
jmp(TMP1);
jmp(qword [STATE + offsetof(FEXCore::Core::CpuStateFrame, Pointers.X86.SignalReturnHandler)]);
}
DEF_OP(CallbackReturn) {
@@ -58,8 +53,7 @@ DEF_OP(CallbackReturn) {
}
// Make sure to adjust the refcounter so we don't clear the cache now
mov(rax, reinterpret_cast<uint64_t>(ThreadSharedData.SignalHandlerRefCounterPtr));
sub(dword [rax], 1);
sub(qword [STATE + offsetof(FEXCore::Core::CpuStateFrame, Pointers.X86.SignalHandlerRefCountPointer)], 1);
// We need to adjust an additional 8 bytes to get back to the original "misaligned" RSP state
add(qword [STATE + offsetof(FEXCore::Core::CpuStateFrame, State.gregs[X86State::REG_RSP])], 8);
@@ -104,7 +98,8 @@ DEF_OP(ExitFunction) {
Xbyak::Reg RipReg = GetSrc<RA_64>(Op->NewRIP.ID());
// L1 Cache
mov(rcx, ThreadState->LookupCache->GetL1Pointer());
mov(rcx, qword [STATE + offsetof(FEXCore::Core::CpuStateFrame, Pointers.X86.L1Pointer)]);
mov(rax, RipReg);
and_(rax, LookupCache::L1_ENTRIES_MASK);
@@ -117,9 +112,8 @@ DEF_OP(ExitFunction) {
jmp(qword[LookupBase + 0]);
L(FullLookup);
mov(rax, ThreadSharedData.Dispatcher->AbsoluteLoopTopAddress);
mov(qword [STATE + offsetof(FEXCore::Core::CpuStateFrame, State.rip)], RipReg);
jmp(rax);
jmp(qword [STATE + offsetof(FEXCore::Core::CpuStateFrame, Pointers.X86.DispatcherLoopTop)]);
}
#ifdef BLOCKSTATS
@@ -187,15 +181,13 @@ DEF_OP(Syscall) {
}
mov(rsi, STATE); // Move thread in to rsi
mov(rdi, reinterpret_cast<uint64_t>(CTX->SyscallHandler));
mov(rdi, qword [STATE + offsetof(FEXCore::Core::CpuStateFrame, Pointers.X86.SyscallHandlerObj)]);
mov(rdx, rsp);
mov(rax, reinterpret_cast<uint64_t>(FEXCore::Context::HandleSyscall));
if (NumPush & 1)
sub(rsp, 8); // Align
// {rdi, rsi, rdx}
call(rax);
call(qword [STATE + offsetof(FEXCore::Core::CpuStateFrame, Pointers.X86.SyscallHandlerFunc)]);
if (NumPush & 1)
add(rsp, 8); // Align
@@ -280,9 +272,7 @@ DEF_OP(RemoveCodeEntry) {
mov(rax, Entry); // imm64 move
mov(rsi, rax);
mov(rax, reinterpret_cast<uintptr_t>(&Context::Context::RemoveCodeEntryFromJit));
call(rax);
call(qword [STATE + offsetof(FEXCore::Core::CpuStateFrame, Pointers.X86.RemoveCodeEntryFromJIT)]);
if (NumPush & 1)
add(rsp, 8); // Align
@@ -294,13 +284,6 @@ DEF_OP(RemoveCodeEntry) {
DEF_OP(CPUID) {
auto Op = IROp->C<IR::IROp_CPUID>();
using ClassPtrType = FEXCore::CPUID::FunctionResults (FEXCore::CPUIDEmu::*)(uint32_t Function, uint32_t Leaf);
union {
ClassPtrType ClassPtr;
uint64_t Raw;
} Ptr;
Ptr.ClassPtr = &CPUIDEmu::RunFunction;
for (auto &Reg : RA64)
push(Reg);
@@ -313,18 +296,15 @@ DEF_OP(CPUID) {
// rsi can be in the source registers, so copy argument to edx first
mov (edx, GetSrc<RA_32>(Op->Header.Args[1].ID()));
mov (esi, GetSrc<RA_32>(Op->Header.Args[0].ID()));
mov (rdi, reinterpret_cast<uint64_t>(&CTX->CPUID));
mov (rdi, qword [STATE + offsetof(FEXCore::Core::CpuStateFrame, Pointers.X86.CPUIDObj)]);
auto NumPush = RA64.size();
if (NumPush & 1)
sub(rsp, 8); // Align
mov(rax, Ptr.Raw);
// {rdi, rsi, rdx}
call(rax);
call(qword [STATE + offsetof(FEXCore::Core::CpuStateFrame, Pointers.X86.CPUIDFunction)]);
if (NumPush & 1)
add(rsp, 8); // Align
@@ -342,7 +322,6 @@ void X86JITCore::RegisterBranchHandlers() {
#define REGISTER_OP(op, x) OpHandlers[FEXCore::IR::IROps::OP_##op] = &X86JITCore::Op_##x
REGISTER_OP(GUESTCALLDIRECT, GuestCallDirect);
REGISTER_OP(GUESTCALLINDIRECT, GuestCallIndirect);
REGISTER_OP(GUESTRETURN, GuestReturn);
REGISTER_OP(SIGNALRETURN, SignalReturn);
REGISTER_OP(CALLBACKRETURN, CallbackReturn);
REGISTER_OP(EXITFUNCTION, ExitFunction);
@@ -18,23 +18,23 @@ namespace FEXCore::CPU {
#define DEF_OP(x) void X86JITCore::Op_##x(IR::IROp_Header *IROp, IR::NodeID Node)
DEF_OP(VInsGPR) {
auto Op = IROp->C<IR::IROp_VInsGPR>();
movapd(GetDst(Node), GetSrc(Op->Header.Args[0].ID()));
movapd(GetDst(Node), GetSrc(Op->DestVector.ID()));
switch (Op->Header.ElementSize) {
case 1: {
pinsrb(GetDst(Node), GetSrc<RA_32>(Op->Header.Args[1].ID()), Op->Index);
pinsrb(GetDst(Node), GetSrc<RA_32>(Op->Src.ID()), Op->DestIdx);
break;
}
case 2: {
pinsrw(GetDst(Node), GetSrc<RA_32>(Op->Header.Args[1].ID()), Op->Index);
pinsrw(GetDst(Node), GetSrc<RA_32>(Op->Src.ID()), Op->DestIdx);
break;
}
case 4: {
pinsrd(GetDst(Node), GetSrc<RA_32>(Op->Header.Args[1].ID()), Op->Index);
pinsrd(GetDst(Node), GetSrc<RA_32>(Op->Src.ID()), Op->DestIdx);
break;
}
case 8: {
pinsrq(GetDst(Node), GetSrc<RA_64>(Op->Header.Args[1].ID()), Op->Index);
pinsrq(GetDst(Node), GetSrc<RA_64>(Op->Src.ID()), Op->DestIdx);
break;
}
default: LOGMAN_MSG_A_FMT("Unknown Element Size: {}", Op->Header.ElementSize); break;
@@ -45,6 +45,36 @@ DEF_OP(AESKeyGenAssist) {
vaeskeygenassist(GetDst(Node), GetSrc(Op->Header.Args[0].ID()), Op->RCON);
}
DEF_OP(CRC32) {
auto Op = IROp->C<IR::IROp_CRC32>();
switch (IROp->Size) {
case 4:
mov(TMP1, GetSrc<RA_32>(Op->Header.Args[1].ID()));
mov(GetDst<RA_32>(Node), GetSrc<RA_32>(Op->Header.Args[0].ID()));
break;
case 8:
mov(TMP1, GetSrc<RA_64>(Op->Header.Args[1].ID()));
mov(GetDst<RA_64>(Node), GetSrc<RA_64>(Op->Header.Args[0].ID()));
break;
default: LOGMAN_MSG_A_FMT("Unknown CRC32 size: {}", IROp->Size);
}
switch (Op->SrcSize) {
case 1:
crc32(GetDst<RA_32>(Node).cvt32(), TMP1.cvt8());
break;
case 2:
crc32(GetDst<RA_32>(Node).cvt32(), TMP1.cvt16());
break;
case 4:
crc32(GetDst<RA_32>(Node).cvt32(), TMP1.cvt32());
break;
case 8:
crc32(GetDst<RA_64>(Node).cvt64(), TMP1.cvt64());
break;
}
}
#undef DEF_OP
void X86JITCore::RegisterEncryptionHandlers() {
#define REGISTER_OP(op, x) OpHandlers[FEXCore::IR::IROps::OP_##op] = &X86JITCore::Op_##x
@@ -54,7 +84,7 @@ void X86JITCore::RegisterEncryptionHandlers() {
REGISTER_OP(VAESDEC, AESDec);
REGISTER_OP(VAESDECLAST, AESDecLast);
REGISTER_OP(VAESKEYGENASSIST, AESKeyGenAssist);
REGISTER_OP(CRC32, CRC32);
#undef REGISTER_OP
}
}
+56 -38
View File
@@ -15,6 +15,8 @@ $end_info$
#include "Interface/IR/PassManager.h"
#include "Interface/IR/Passes/RegisterAllocationPass.h"
#include "Utils/MemberFunctionToPointer.h"
#include <FEXCore/Core/CPUBackend.h>
#include <FEXCore/Core/CoreState.h>
#include <FEXCore/Core/SignalDelegator.h>
@@ -27,7 +29,6 @@ $end_info$
#include <algorithm>
#include <array>
#include <bits/types/stack_t.h>
#include <memory>
#include <stddef.h>
#include <stdint.h>
@@ -42,6 +43,16 @@ $end_info$
// #define DEBUG_RA 1
// #define DEBUG_CYCLES
namespace {
static void PrintValue(uint64_t Value) {
LogMan::Msg::DFmt("Value: 0x{:x}", Value);
}
static void PrintVectorValue(uint64_t Value, uint64_t ValueUpper) {
LogMan::Msg::DFmt("Value: 0x{:016x}'{:016x}", ValueUpper, Value);
}
}
namespace FEXCore::CPU {
CodeBuffer AllocateNewCodeBuffer(FEXCore::Context::Context *CTX, size_t Size) {
@@ -109,9 +120,7 @@ void X86JITCore::Op_Unhandled(IR::IROp_Header *IROp, IR::NodeID Node) {
case FABI_VOID_U16: {
PushRegs();
mov(edi, GetSrc<RA_32>(IROp->Args[0].ID()));
mov(rax, (uintptr_t)Info.fn);
call(rax);
call(qword [STATE + offsetof(FEXCore::Core::CpuStateFrame, Pointers.X86.FallbackHandlerPointers[Info.HandlerIndex])]);
PopRegs();
break;
@@ -120,9 +129,7 @@ void X86JITCore::Op_Unhandled(IR::IROp_Header *IROp, IR::NodeID Node) {
PushRegs();
movss(xmm0, GetSrc(IROp->Args[0].ID()));
mov(rax, (uintptr_t)Info.fn);
call(rax);
call(qword [STATE + offsetof(FEXCore::Core::CpuStateFrame, Pointers.X86.FallbackHandlerPointers[Info.HandlerIndex])]);
PopRegs();
@@ -136,9 +143,7 @@ void X86JITCore::Op_Unhandled(IR::IROp_Header *IROp, IR::NodeID Node) {
PushRegs();
movsd(xmm0, GetSrc(IROp->Args[0].ID()));
mov(rax, (uintptr_t)Info.fn);
call(rax);
call(qword [STATE + offsetof(FEXCore::Core::CpuStateFrame, Pointers.X86.FallbackHandlerPointers[Info.HandlerIndex])]);
PopRegs();
@@ -153,9 +158,7 @@ void X86JITCore::Op_Unhandled(IR::IROp_Header *IROp, IR::NodeID Node) {
PushRegs();
mov(edi, GetSrc<RA_32>(IROp->Args[0].ID()));
mov(rax, (uintptr_t)Info.fn);
call(rax);
call(qword [STATE + offsetof(FEXCore::Core::CpuStateFrame, Pointers.X86.FallbackHandlerPointers[Info.HandlerIndex])]);
PopRegs();
@@ -171,9 +174,7 @@ void X86JITCore::Op_Unhandled(IR::IROp_Header *IROp, IR::NodeID Node) {
movq(rdi, GetSrc(IROp->Args[0].ID()));
pextrq(rsi, GetSrc(IROp->Args[0].ID()), 1);
mov(rax, (uintptr_t)Info.fn);
call(rax);
call(qword [STATE + offsetof(FEXCore::Core::CpuStateFrame, Pointers.X86.FallbackHandlerPointers[Info.HandlerIndex])]);
PopRegs();
@@ -187,9 +188,7 @@ void X86JITCore::Op_Unhandled(IR::IROp_Header *IROp, IR::NodeID Node) {
movq(rdi, GetSrc(IROp->Args[0].ID()));
pextrq(rsi, GetSrc(IROp->Args[0].ID()), 1);
mov(rax, (uintptr_t)Info.fn);
call(rax);
call(qword [STATE + offsetof(FEXCore::Core::CpuStateFrame, Pointers.X86.FallbackHandlerPointers[Info.HandlerIndex])]);
PopRegs();
@@ -203,9 +202,7 @@ void X86JITCore::Op_Unhandled(IR::IROp_Header *IROp, IR::NodeID Node) {
movq(rdi, GetSrc(IROp->Args[0].ID()));
pextrq(rsi, GetSrc(IROp->Args[0].ID()), 1);
mov(rax, (uintptr_t)Info.fn);
call(rax);
call(qword [STATE + offsetof(FEXCore::Core::CpuStateFrame, Pointers.X86.FallbackHandlerPointers[Info.HandlerIndex])]);
PopRegs();
@@ -218,9 +215,7 @@ void X86JITCore::Op_Unhandled(IR::IROp_Header *IROp, IR::NodeID Node) {
movq(rdi, GetSrc(IROp->Args[0].ID()));
pextrq(rsi, GetSrc(IROp->Args[0].ID()), 1);
mov(rax, (uintptr_t)Info.fn);
call(rax);
call(qword [STATE + offsetof(FEXCore::Core::CpuStateFrame, Pointers.X86.FallbackHandlerPointers[Info.HandlerIndex])]);
PopRegs();
@@ -233,9 +228,7 @@ void X86JITCore::Op_Unhandled(IR::IROp_Header *IROp, IR::NodeID Node) {
movq(rdi, GetSrc(IROp->Args[0].ID()));
pextrq(rsi, GetSrc(IROp->Args[0].ID()), 1);
mov(rax, (uintptr_t)Info.fn);
call(rax);
call(qword [STATE + offsetof(FEXCore::Core::CpuStateFrame, Pointers.X86.FallbackHandlerPointers[Info.HandlerIndex])]);
PopRegs();
@@ -251,9 +244,7 @@ void X86JITCore::Op_Unhandled(IR::IROp_Header *IROp, IR::NodeID Node) {
movq(rdx, GetSrc(IROp->Args[1].ID()));
pextrq(rcx, GetSrc(IROp->Args[1].ID()), 1);
mov(rax, (uintptr_t)Info.fn);
call(rax);
call(qword [STATE + offsetof(FEXCore::Core::CpuStateFrame, Pointers.X86.FallbackHandlerPointers[Info.HandlerIndex])]);
PopRegs();
@@ -266,9 +257,7 @@ void X86JITCore::Op_Unhandled(IR::IROp_Header *IROp, IR::NodeID Node) {
movq(rdi, GetSrc(IROp->Args[0].ID()));
pextrq(rsi, GetSrc(IROp->Args[0].ID()), 1);
mov(rax, (uintptr_t)Info.fn);
call(rax);
call(qword [STATE + offsetof(FEXCore::Core::CpuStateFrame, Pointers.X86.FallbackHandlerPointers[Info.HandlerIndex])]);
PopRegs();
@@ -286,9 +275,7 @@ void X86JITCore::Op_Unhandled(IR::IROp_Header *IROp, IR::NodeID Node) {
movq(rdx, GetSrc(IROp->Args[1].ID()));
pextrq(rcx, GetSrc(IROp->Args[1].ID()), 1);
mov(rax, (uintptr_t)Info.fn);
call(rax);
call(qword [STATE + offsetof(FEXCore::Core::CpuStateFrame, Pointers.X86.FallbackHandlerPointers[Info.HandlerIndex])]);
PopRegs();
@@ -317,6 +304,31 @@ X86JITCore::X86JITCore(FEXCore::Context::Context *ctx, FEXCore::Core::InternalTh
, ThreadState {Thread}
, InitialCodeBuffer {Buffer}
{
{
// Set up pointers that the JIT needs to load
auto &Pointers = ThreadState->CurrentFrame->Pointers.X86;
// Process specific
Pointers.PrintValue = reinterpret_cast<uint64_t>(PrintValue);
Pointers.PrintVectorValue = reinterpret_cast<uint64_t>(PrintVectorValue);
Pointers.RemoveCodeEntryFromJIT = reinterpret_cast<uintptr_t>(&Context::Context::RemoveCodeEntryFromJit);
Pointers.CPUIDObj = reinterpret_cast<uint64_t>(&CTX->CPUID);
{
FEXCore::Utils::MemberFunctionToPointerCast PMF(&FEXCore::CPUIDEmu::RunFunction);
Pointers.CPUIDFunction = PMF.GetConvertedPointer();
}
Pointers.SyscallHandlerObj = reinterpret_cast<uint64_t>(CTX->SyscallHandler);
Pointers.SyscallHandlerFunc = reinterpret_cast<uint64_t>(FEXCore::Context::HandleSyscall);
// Fill in the fallback handlers
InterpreterOps::FillFallbackIndexPointers(Pointers.FallbackHandlerPointers);
// Thread Specific
Pointers.SignalHandlerRefCountPointer = reinterpret_cast<uint64_t>(&Dispatcher->SignalHandlerRefCounter);
}
CurrentCodeBuffer = &InitialCodeBuffer;
RAPass = Thread->PassManager->GetPass<IR::RegisterAllocationPass>("RA");
@@ -358,6 +370,7 @@ X86JITCore::X86JITCore(FEXCore::Context::Context *ctx, FEXCore::Core::InternalTh
ThreadSharedData.SignalHandlerRefCounterPtr = &Dispatcher->SignalHandlerRefCounter;
ThreadSharedData.SignalHandlerReturnAddress = Dispatcher->SignalHandlerReturnAddress;
ThreadSharedData.UnimplementedInstructionAddress = Dispatcher->UnimplementedInstructionAddress;
ThreadSharedData.OverflowExceptionInstructionAddress = Dispatcher->OverflowExceptionInstructionAddress;
ThreadSharedData.Dispatcher = Dispatcher.get();
@@ -552,7 +565,12 @@ bool X86JITCore::IsInlineEntrypointOffset(const IR::OrderedNodeWrapper& WNode, u
if (OpHeader->Op == IR::IROps::OP_INLINEENTRYPOINTOFFSET) {
auto Op = OpHeader->C<IR::IROp_InlineEntrypointOffset>();
if (Value) {
*Value = Entry + Op->Offset;
uint64_t Mask = ~0ULL;
uint8_t OpSize = OpHeader->Size;
if (OpSize == 4) {
Mask = 0xFFFF'FFFFULL;
}
*Value = (Entry + Op->Offset) & Mask;
}
return true;
} else {
@@ -167,7 +167,7 @@ private:
struct CompilerSharedData {
uint64_t SignalHandlerReturnAddress{};
uint64_t UnimplementedInstructionAddress{};
uint64_t OverflowExceptionInstructionAddress{};
uint32_t *SignalHandlerRefCounterPtr{};
FEXCore::CPU::Dispatcher *Dispatcher{};
@@ -233,6 +233,8 @@ private:
DEF_OP(Rol);
DEF_OP(Ror);
DEF_OP(Extr);
DEF_OP(PDep);
DEF_OP(PExt);
DEF_OP(LDiv);
DEF_OP(LUDiv);
DEF_OP(LRem);
@@ -274,7 +276,6 @@ private:
///< Branch ops
DEF_OP(GuestCallDirect);
DEF_OP(GuestCallIndirect);
DEF_OP(GuestReturn);
DEF_OP(SignalReturn);
DEF_OP(CallbackReturn);
DEF_OP(ExitFunction);
@@ -315,6 +316,7 @@ private:
DEF_OP(VLoadMemElement);
DEF_OP(VStoreMemElement);
DEF_OP(CacheLineClear);
DEF_OP(CacheLineZero);
///< Misc ops
DEF_OP(EndBlock);
@@ -325,6 +327,8 @@ private:
DEF_OP(Print);
DEF_OP(GetRoundingMode);
DEF_OP(SetRoundingMode);
DEF_OP(ProcessorID);
DEF_OP(RDRAND);
///< Move ops
DEF_OP(ExtractElementPair);
@@ -334,8 +338,6 @@ private:
///< Vector ops
DEF_OP(VectorZero);
DEF_OP(VectorImm);
DEF_OP(CreateVector2);
DEF_OP(CreateVector4);
DEF_OP(SplatVector);
DEF_OP(VMov);
DEF_OP(VAnd);
@@ -422,6 +424,7 @@ private:
DEF_OP(VSMull2);
DEF_OP(VUABDL);
DEF_OP(VTBL1);
DEF_OP(VRev64);
///< Encryption ops
DEF_OP(AESImc);
@@ -430,6 +433,7 @@ private:
DEF_OP(AESDec);
DEF_OP(AESDecLast);
DEF_OP(AESKeyGenAssist);
DEF_OP(CRC32);
#undef DEF_OP
};
@@ -23,7 +23,7 @@ DEF_OP(LoadContext) {
auto Op = IROp->C<IR::IROp_LoadContext>();
uint8_t OpSize = IROp->Size;
if (Op->Class.Val == 0) {
if (Op->Class == IR::GPRClass) {
switch (OpSize) {
case 1: {
movzx(GetDst<RA_32>(Node), byte [STATE + Op->Offset]);
@@ -84,23 +84,23 @@ DEF_OP(StoreContext) {
auto Op = IROp->C<IR::IROp_StoreContext>();
uint8_t OpSize = IROp->Size;
if (Op->Class.Val == 0) {
if (Op->Class == IR::GPRClass) {
switch (OpSize) {
case 1: {
mov(byte [STATE + Op->Offset], GetSrc<RA_8>(Op->Header.Args[0].ID()));
mov(byte [STATE + Op->Offset], GetSrc<RA_8>(Op->Value.ID()));
}
break;
case 2: {
mov(word [STATE + Op->Offset], GetSrc<RA_16>(Op->Header.Args[0].ID()));
mov(word [STATE + Op->Offset], GetSrc<RA_16>(Op->Value.ID()));
}
break;
case 4: {
mov(dword [STATE + Op->Offset], GetSrc<RA_32>(Op->Header.Args[0].ID()));
mov(dword [STATE + Op->Offset], GetSrc<RA_32>(Op->Value.ID()));
}
break;
case 8: {
mov(qword [STATE + Op->Offset], GetSrc<RA_64>(Op->Header.Args[0].ID()));
mov(qword [STATE + Op->Offset], GetSrc<RA_64>(Op->Value.ID()));
}
break;
case 16:
@@ -112,27 +112,27 @@ DEF_OP(StoreContext) {
else {
switch (OpSize) {
case 1: {
pextrb(byte [STATE + Op->Offset], GetSrc(Op->Header.Args[0].ID()), 0);
pextrb(byte [STATE + Op->Offset], GetSrc(Op->Value.ID()), 0);
}
break;
case 2: {
pextrw(word [STATE + Op->Offset], GetSrc(Op->Header.Args[0].ID()), 0);
pextrw(word [STATE + Op->Offset], GetSrc(Op->Value.ID()), 0);
}
break;
case 4: {
vmovd(dword [STATE + Op->Offset], GetSrc(Op->Header.Args[0].ID()));
vmovd(dword [STATE + Op->Offset], GetSrc(Op->Value.ID()));
}
break;
case 8: {
vmovq(qword [STATE + Op->Offset], GetSrc(Op->Header.Args[0].ID()));
vmovq(qword [STATE + Op->Offset], GetSrc(Op->Value.ID()));
}
break;
case 16: {
if (Op->Offset % 16 == 0)
movaps(xword [STATE + Op->Offset], GetSrc(Op->Header.Args[0].ID()));
movaps(xword [STATE + Op->Offset], GetSrc(Op->Value.ID()));
else
movups(xword [STATE + Op->Offset], GetSrc(Op->Header.Args[0].ID()));
movups(xword [STATE + Op->Offset], GetSrc(Op->Value.ID()));
}
break;
default: LOGMAN_MSG_A_FMT("Unhandled StoreContext size: {}", OpSize);
@@ -143,9 +143,9 @@ DEF_OP(StoreContext) {
DEF_OP(LoadContextIndexed) {
auto Op = IROp->C<IR::IROp_LoadContextIndexed>();
size_t size = IROp->Size;
Reg index = GetSrc<RA_64>(Op->Header.Args[0].ID());
Reg index = GetSrc<RA_64>(Op->Index.ID());
if (Op->Class.Val == 0) {
if (Op->Class == IR::GPRClass) {
switch (Op->Stride) {
case 1:
case 2:
@@ -245,11 +245,11 @@ DEF_OP(LoadContextIndexed) {
DEF_OP(StoreContextIndexed) {
auto Op = IROp->C<IR::IROp_StoreContextIndexed>();
Reg index = GetSrc<RA_64>(Op->Header.Args[1].ID());
Reg index = GetSrc<RA_64>(Op->Index.ID());
size_t size = IROp->Size;
if (Op->Class.Val == 0) {
auto value = GetSrc<RA_64>(Op->Header.Args[0].ID());
if (Op->Class == IR::GPRClass) {
auto value = GetSrc<RA_64>(Op->Value.ID());
lea(rax, dword [STATE + Op->BaseOffset]);
switch (Op->Stride) {
@@ -269,7 +269,7 @@ DEF_OP(StoreContextIndexed) {
}
}
else {
auto value = GetSrc(Op->Header.Args[0].ID());
auto value = GetSrc(Op->Value.ID());
switch (Op->Stride) {
case 1:
case 2:
@@ -339,19 +339,19 @@ DEF_OP(SpillRegister) {
if (Op->Class == FEXCore::IR::GPRClass) {
switch (OpSize) {
case 1: {
mov(byte [rsp + SlotOffset], GetSrc<RA_8>(Op->Header.Args[0].ID()));
mov(byte [rsp + SlotOffset], GetSrc<RA_8>(Op->Value.ID()));
break;
}
case 2: {
mov(word [rsp + SlotOffset], GetSrc<RA_16>(Op->Header.Args[0].ID()));
mov(word [rsp + SlotOffset], GetSrc<RA_16>(Op->Value.ID()));
break;
}
case 4: {
mov(dword [rsp + SlotOffset], GetSrc<RA_32>(Op->Header.Args[0].ID()));
mov(dword [rsp + SlotOffset], GetSrc<RA_32>(Op->Value.ID()));
break;
}
case 8: {
mov(qword [rsp + SlotOffset], GetSrc<RA_64>(Op->Header.Args[0].ID()));
mov(qword [rsp + SlotOffset], GetSrc<RA_64>(Op->Value.ID()));
break;
}
default: LOGMAN_MSG_A_FMT("Unhandled SpillRegister size: {}", OpSize);
@@ -359,15 +359,15 @@ DEF_OP(SpillRegister) {
} else if (Op->Class == FEXCore::IR::FPRClass) {
switch (OpSize) {
case 4: {
movss(dword [rsp + SlotOffset], GetSrc(Op->Header.Args[0].ID()));
movss(dword [rsp + SlotOffset], GetSrc(Op->Value.ID()));
break;
}
case 8: {
movsd(qword [rsp + SlotOffset], GetSrc(Op->Header.Args[0].ID()));
movsd(qword [rsp + SlotOffset], GetSrc(Op->Value.ID()));
break;
}
case 16: {
movaps(xword [rsp + SlotOffset], GetSrc(Op->Header.Args[0].ID()));
movaps(xword [rsp + SlotOffset], GetSrc(Op->Value.ID()));
break;
}
default: LOGMAN_MSG_A_FMT("Unhandled SpillRegister size: {}", OpSize);
@@ -435,7 +435,7 @@ DEF_OP(LoadFlag) {
DEF_OP(StoreFlag) {
auto Op = IROp->C<IR::IROp_StoreFlag>();
mov (rax, GetSrc<RA_64>(Op->Header.Args[0].ID()));
mov (rax, GetSrc<RA_64>(Op->Value.ID()));
mov(byte [STATE + (offsetof(FEXCore::Core::CPUState, flags[0]) + Op->Flag)], al);
}
@@ -469,7 +469,7 @@ DEF_OP(LoadMem) {
auto MemPtr = GenerateModRM(MemReg, Op->Offset, Op->OffsetType, Op->OffsetScale);
if (Op->Class.Val == 0) {
if (Op->Class == IR::GPRClass) {
auto Dst = GetDst<RA_64>(Node);
switch (IROp->Size) {
@@ -537,19 +537,19 @@ DEF_OP(StoreMem) {
auto MemPtr = GenerateModRM(MemReg, Op->Offset, Op->OffsetType, Op->OffsetScale);
if (Op->Class.Val == 0) {
if (Op->Class == IR::GPRClass) {
switch (IROp->Size) {
case 1:
mov(byte [MemPtr], GetSrc<RA_8>(Op->Header.Args[1].ID()));
mov(byte [MemPtr], GetSrc<RA_8>(Op->Value.ID()));
break;
case 2:
mov(word [MemPtr], GetSrc<RA_16>(Op->Header.Args[1].ID()));
mov(word [MemPtr], GetSrc<RA_16>(Op->Value.ID()));
break;
case 4:
mov(dword [MemPtr], GetSrc<RA_32>(Op->Header.Args[1].ID()));
mov(dword [MemPtr], GetSrc<RA_32>(Op->Value.ID()));
break;
case 8:
mov(qword [MemPtr], GetSrc<RA_64>(Op->Header.Args[1].ID()));
mov(qword [MemPtr], GetSrc<RA_64>(Op->Value.ID()));
break;
default: LOGMAN_MSG_A_FMT("Unhandled StoreMem size: {}", IROp->Size);
}
@@ -557,22 +557,22 @@ DEF_OP(StoreMem) {
else {
switch (IROp->Size) {
case 1:
pextrb(byte [MemPtr], GetSrc(Op->Header.Args[1].ID()), 0);
pextrb(byte [MemPtr], GetSrc(Op->Value.ID()), 0);
break;
case 2:
pextrw(word [MemPtr], GetSrc(Op->Header.Args[1].ID()), 0);
pextrw(word [MemPtr], GetSrc(Op->Value.ID()), 0);
break;
case 4:
vmovd(dword [MemPtr], GetSrc(Op->Header.Args[1].ID()));
vmovd(dword [MemPtr], GetSrc(Op->Value.ID()));
break;
case 8:
vmovq(qword [MemPtr], GetSrc(Op->Header.Args[1].ID()));
vmovq(qword [MemPtr], GetSrc(Op->Value.ID()));
break;
case 16:
if (IROp->Size == Op->Align)
movups(xword [MemPtr], GetSrc(Op->Header.Args[1].ID()));
movups(xword [MemPtr], GetSrc(Op->Value.ID()));
else
movups(xword [MemPtr], GetSrc(Op->Header.Args[1].ID()));
movups(xword [MemPtr], GetSrc(Op->Value.ID()));
break;
default: LOGMAN_MSG_A_FMT("Unhandled StoreMem size: {}", IROp->Size);
}
@@ -595,6 +595,23 @@ DEF_OP(CacheLineClear) {
clflush(ptr [MemReg]);
}
DEF_OP(CacheLineZero) {
auto Op = IROp->C<IR::IROp_CacheLineZero>();
Xbyak::Reg MemReg = GetSrc<RA_64>(Op->Addr.ID());
// Align by cacheline
mov (TMP1, CPUIDEmu::CACHELINE_SIZE - 1);
andn(TMP1, TMP1, MemReg.cvt64());
xor_(TMP2, TMP2);
using DataType = uint64_t;
// 64-byte cache line zero
for (size_t i = 0; i < CPUIDEmu::CACHELINE_SIZE; i += sizeof(DataType)) {
mov (qword [TMP1 + i], TMP2);
}
}
#undef DEF_OP
void X86JITCore::RegisterMemoryHandlers() {
#define REGISTER_OP(op, x) OpHandlers[FEXCore::IR::IROps::OP_##op] = &X86JITCore::Op_##x
@@ -615,6 +632,7 @@ void X86JITCore::RegisterMemoryHandlers() {
REGISTER_OP(VLOADMEMELEMENT, VLoadMemElement);
REGISTER_OP(VSTOREMEMELEMENT, VStoreMemElement);
REGISTER_OP(CACHELINECLEAR, CacheLineClear);
REGISTER_OP(CACHELINEZERO, CacheLineZero);
#undef REGISTER_OP
}
}
+36 -23
View File
@@ -18,14 +18,6 @@ $end_info$
#include <xbyak/xbyak.h>
namespace FEXCore::CPU {
static void PrintValue(uint64_t Value) {
LogMan::Msg::DFmt("Value: 0x{:x}", Value);
}
static void PrintVectorValue(uint64_t Value, uint64_t ValueUpper) {
LogMan::Msg::DFmt("Value: 0x{:016x}'{:016x}", ValueUpper, Value);
}
#define DEF_OP(x) void X86JITCore::Op_##x(IR::IROp_Header *IROp, IR::NodeID Node)
DEF_OP(Fence) {
@@ -49,17 +41,19 @@ DEF_OP(Break) {
switch (Op->Reason) {
case FEXCore::IR::Break_Unimplemented: // Hard fault
case FEXCore::IR::Break_Interrupt: // Guest ud2
case FEXCore::IR::Break_Overflow: // overflow
ud2();
break;
case FEXCore::IR::Break_Overflow: // overflow
// Need to be outside of JIT cache space to ensure cache clearing correctness
jmp(qword [STATE + offsetof(FEXCore::Core::CpuStateFrame, Pointers.X86.OverflowExceptionHandler)]);
break;
case FEXCore::IR::Break_Halt: { // HLT
// Time to quit
// Set our stack to the starting stack location
mov(rsp, qword [STATE + offsetof(FEXCore::Core::CpuStateFrame, ReturningStackLocation)]);
// Now we need to jump to the thread stop handler
mov(TMP1, ThreadSharedData.Dispatcher->ThreadStopHandlerAddress);
jmp(TMP1);
jmp(qword [STATE + offsetof(FEXCore::Core::CpuStateFrame, Pointers.X86.ThreadStopHandler)]);
break;
}
case FEXCore::IR::Break_Interrupt3: // INT3
@@ -71,8 +65,7 @@ DEF_OP(Break) {
}
// This jump target needs to be a constant offset here
mov(TMP1, ThreadSharedData.Dispatcher->ThreadPauseHandlerAddress);
jmp(TMP1);
jmp(qword [STATE + offsetof(FEXCore::Core::CpuStateFrame, Pointers.X86.ThreadPauseHandler)]);
}
else {
// If we don't have a gdb server attached then....crash?
@@ -80,8 +73,7 @@ DEF_OP(Break) {
mov(rsp, qword [STATE + offsetof(FEXCore::Core::CpuStateFrame, ReturningStackLocation)]);
// Now we need to jump to the thread stop handler
mov(TMP1, ThreadSharedData.Dispatcher->ThreadStopHandlerAddress);
jmp(TMP1);
jmp(qword [STATE + offsetof(FEXCore::Core::CpuStateFrame, Pointers.X86.ThreadStopHandler)]);
}
break;
}
@@ -92,9 +84,7 @@ DEF_OP(Break) {
}
// Need to be outside of JIT cache space to ensure cache clearing correctness
mov(TMP1, ThreadSharedData.Dispatcher->UnimplementedInstructionAddress);
jmp(TMP1);
jmp(qword [STATE + offsetof(FEXCore::Core::CpuStateFrame, Pointers.X86.UnimplementedInstructionHandler)]);
break;
}
default: LOGMAN_MSG_A_FMT("Unknown Break reason: {}", Op->Reason);
@@ -140,21 +130,42 @@ DEF_OP(Print) {
PushRegs();
if (IsGPR(Op->Header.Args[0].ID())) {
mov (rdi, GetSrc<RA_64>(Op->Header.Args[0].ID()));
mov(rax, reinterpret_cast<uintptr_t>(PrintValue));
call(qword [STATE + offsetof(FEXCore::Core::CpuStateFrame, Pointers.X86.PrintValue)]);
}
else {
pextrq(rdi, GetSrc(Op->Header.Args[0].ID()), 0);
pextrq(rsi, GetSrc(Op->Header.Args[0].ID()), 1);
mov(rax, reinterpret_cast<uintptr_t>(PrintVectorValue));
call(qword [STATE + offsetof(FEXCore::Core::CpuStateFrame, Pointers.X86.PrintVectorValue)]);
}
call(rax);
PopRegs();
}
DEF_OP(ProcessorID) {
// Cyclecounter in EDX:EAX
// IA32_TSC_AUX in ECX
rdtscp();
mov (GetDst<RA_32>(Node), ecx);
}
DEF_OP(RDRAND) {
auto Op = IROp->C<IR::IROp_RDRAND>();
auto Dst = GetSrcPair<RA_64>(Node);
if (Op->GetReseeded) {
rdrand(Dst.first);
}
else {
rdseed(Dst.first);
}
// In the case of RDRAND or RDSEED returning a valid number then CF = 1, else 0
mov (Dst.second, 0);
setc(Dst.second.cvt8());
}
#undef DEF_OP
void X86JITCore::RegisterMiscHandlers() {
#define REGISTER_OP(op, x) OpHandlers[FEXCore::IR::IROps::OP_##op] = &X86JITCore::Op_##x
@@ -171,6 +182,8 @@ void X86JITCore::RegisterMiscHandlers() {
REGISTER_OP(GETROUNDINGMODE, GetRoundingMode);
REGISTER_OP(SETROUNDINGMODE, SetRoundingMode);
REGISTER_OP(INVALIDATEFLAGS, NoOp);
REGISTER_OP(PROCESSORID, ProcessorID);
REGISTER_OP(RDRAND, RDRAND);
#undef REGISTER_OP
}
}
@@ -42,7 +42,7 @@ DEF_OP(CreateElementPair) {
Xbyak::Reg RegSecond;
Xbyak::Reg RegTmp;
switch (Op->Header.Size) {
switch (IROp->ElementSize) {
case 4: {
Dst = GetSrcPair<RA_32>(Node);
RegFirst = GetSrc<RA_32>(Op->Header.Args[0].ID());
@@ -67,14 +67,6 @@ DEF_OP(VectorImm) {
}
}
DEF_OP(CreateVector2) {
LOGMAN_MSG_A_FMT("Unimplemented");
}
DEF_OP(CreateVector4) {
LOGMAN_MSG_A_FMT("Unimplemented");
}
DEF_OP(SplatVector) {
auto Op = IROp->C<IR::IROp_SplatVector2>();
uint8_t OpSize = IROp->Size;
@@ -1545,7 +1537,7 @@ DEF_OP(VInsScalarElement) {
DEF_OP(VExtractElement) {
auto Op = IROp->C<IR::IROp_VExtractElement>();
switch (Op->Header.ElementSize) {
switch (Op->Header.Size) {
case 1: {
pextrb(eax, GetSrc(Op->Header.Args[0].ID()), Op->Index);
pinsrb(GetDst(Node), eax, 0);
@@ -1566,7 +1558,7 @@ DEF_OP(VExtractElement) {
pinsrq(GetDst(Node), rax, 0);
break;
}
default: LOGMAN_MSG_A_FMT("Unknown Element Size: {}", Op->Header.ElementSize); break;
default: LOGMAN_MSG_A_FMT("Unknown Element Size: {}", Op->Header.Size); break;
}
}
@@ -2151,13 +2143,79 @@ DEF_OP(VTBL1) {
}
}
DEF_OP(VRev64) {
auto Op = IROp->C<IR::IROp_VDupElement>();
switch (Op->Header.ElementSize) {
case 1: {
mov(rax, 0x00'01'02'03'04'05'06'07); // Lower
vmovq(xmm15, rax);
if (IROp->Size == 16) {
// Full 8bit byteswap in each 64-bit element
mov(rcx, 0x08'09'0A'0B'0C'0D'0E'0F); // Upper
pinsrq(xmm15, rcx, 1);
}
else {
// 8byte, upper bits get zero
// Full 8bit byteswap in each 64-bit element
mov(rcx, 0x80'80'80'80'80'80'80'80); // Upper
pinsrq(xmm15, rcx, 1);
}
vpshufb(GetDst(Node), GetSrc(Op->Header.Args[0].ID()), xmm15);
break;
}
case 2: {
// Full 16-bit byteswap in each 64-bit element
mov(rax, 0x01'00'03'02'05'04'07'06); // Lower
vmovq(xmm15, rax);
if (IROp->Size == 16) {
mov(rcx, 0x09'08'0B'0A'0D'0C'0F'0E); // Upper
pinsrq(xmm15, rcx, 1);
}
else {
// 8byte, upper bits get zero
// Full 8bit byteswap in each 64-bit element
mov(rcx, 0x80'80'80'80'80'80'80'80); // Upper
pinsrq(xmm15, rcx, 1);
}
vpshufb(GetDst(Node), GetSrc(Op->Header.Args[0].ID()), xmm15);
break;
}
case 4: {
if (IROp->Size == 16) {
vpshufd(GetDst(Node),
GetSrc(Op->Header.Args[0].ID()),
(0b11 << 0) |
(0b10 << 2) |
(0b01 << 4) |
(0b00 << 6));
}
else {
vpshufd(GetDst(Node),
GetSrc(Op->Header.Args[0].ID()),
(0b01 << 0) |
(0b00 << 2) |
(0b11 << 4) | // Last two don't matter, will be overwritten with zero
(0b11 << 6));
// Zero upper 64-bits
mov(rcx, 0);
pinsrq(GetDst(Node), rcx, 1);
}
break;
}
default: LOGMAN_MSG_A_FMT("Unknown Element Size: {}", Op->Header.ElementSize); break;
}
}
#undef DEF_OP
void X86JITCore::RegisterVectorHandlers() {
#define REGISTER_OP(op, x) OpHandlers[FEXCore::IR::IROps::OP_##op] = &X86JITCore::Op_##x
REGISTER_OP(VECTORZERO, VectorZero);
REGISTER_OP(VECTORIMM, VectorImm);
REGISTER_OP(CREATEVECTOR2, CreateVector2);
REGISTER_OP(CREATEVECTOR4, CreateVector4);
REGISTER_OP(SPLATVECTOR2, SplatVector);
REGISTER_OP(SPLATVECTOR4, SplatVector);
REGISTER_OP(VMOV, VMov);
@@ -2246,6 +2304,7 @@ void X86JITCore::RegisterVectorHandlers() {
REGISTER_OP(VSMULL2, VSMull2);
REGISTER_OP(VUABDL, VUABDL);
REGISTER_OP(VTBL1, VTBL1);
REGISTER_OP(VREV64, VRev64);
#undef REGISTER_OP
}
}
File diff suppressed because it is too large. Load diff
+602 -25
View File
@@ -14,6 +14,7 @@
#include <FEXCore/Utils/LogManager.h>
#include <cstdint>
#include <fmt/format.h>
#include <map>
#include <stddef.h>
#include <utility>
@@ -35,6 +36,38 @@ enum class SelectionFlag {
};
public:
enum class FlagsGenerationType : uint8_t {
TYPE_NONE,
TYPE_ADC,
TYPE_SBB,
TYPE_SUB,
TYPE_ADD,
TYPE_MUL,
TYPE_UMUL,
TYPE_LOGICAL,
TYPE_LSHL,
TYPE_LSHLI,
TYPE_LSHR,
TYPE_LSHRI,
TYPE_ASHR,
TYPE_ASHRI,
TYPE_ROR,
TYPE_RORI,
TYPE_ROL,
TYPE_ROLI,
TYPE_FCMP,
TYPE_BEXTR,
TYPE_BLSI,
TYPE_BLSMSK,
TYPE_BLSR,
TYPE_POPCOUNT,
TYPE_BZHI,
TYPE_TZCNT,
TYPE_LZCNT,
TYPE_BITSELECT,
TYPE_RDRAND,
};
SelectionFlag flagsOp{};
uint8_t flagsOpSize{};
OrderedNode* flagsOpDest{};
@@ -87,6 +120,9 @@ public:
// cmp qword [rdi-8], 0
// jne .label
if (LastOp && !BlockSetRIP) {
// Calculate flags first
CalculateDeferredFlags();
auto it = JumpTargets.find(NextRIP);
if (it == JumpTargets.end()) {
@@ -101,6 +137,11 @@ public:
return true;
}
}
if (LastOp) {
LOGMAN_THROW_A_FMT(IsDeferredFlagsStored(), "FinishOp: Deferred flags weren't generated at end of block");
}
BlockSetRIP = false;
return false;
@@ -233,6 +274,8 @@ public:
void XADDOp(OpcodeArgs);
void PopcountOp(OpcodeArgs);
void XLATOp(OpcodeArgs);
template<bool Reseed>
void RDRANDOp(OpcodeArgs);
enum class Segment {
FS,
@@ -256,9 +299,14 @@ public:
template<FEXCore::IR::IROps IROp, size_t ElementSize>
void VectorALUOp(OpcodeArgs);
template<FEXCore::IR::IROps IROp, size_t ElementSize>
void VectorALUROp(OpcodeArgs);
template<FEXCore::IR::IROps IROp, size_t ElementSize>
void VectorScalarALUOp(OpcodeArgs);
template<FEXCore::IR::IROps IROp, size_t ElementSize, bool Scalar>
void VectorUnaryOp(OpcodeArgs);
template<FEXCore::IR::IROps IROp, size_t ElementSize>
void VectorUnaryDuplicateOp(OpcodeArgs);
void MOVQOp(OpcodeArgs);
template<size_t ElementSize>
void PADDQOp(OpcodeArgs);
@@ -338,6 +386,8 @@ public:
void BMI2Shift(OpcodeArgs);
void BZHI(OpcodeArgs);
void MULX(OpcodeArgs);
void PDEP(OpcodeArgs);
void PEXT(OpcodeArgs);
void RORX(OpcodeArgs);
// ADX Ops
@@ -443,6 +493,17 @@ public:
template<size_t ElementSize>
void ADDSUBPOp(OpcodeArgs);
void PFNACCOp(OpcodeArgs);
void PFPNACCOp(OpcodeArgs);
void PSWAPDOp(OpcodeArgs);
template<uint8_t CompType>
void VPFCMPOp(OpcodeArgs);
void PI2FWOp(OpcodeArgs);
void PF2IWOp(OpcodeArgs);
void PMULHRWOp(OpcodeArgs);
void PMADDWD(OpcodeArgs);
void PMADDUBSW(OpcodeArgs);
@@ -469,6 +530,8 @@ public:
void FenceOp(OpcodeArgs);
void StoreFenceOrCLFlush(OpcodeArgs);
void CLZeroOp(OpcodeArgs);
void RDTSCPOp(OpcodeArgs);
void PSADBW(OpcodeArgs);
@@ -496,6 +559,8 @@ public:
void MPSADBWOp(OpcodeArgs);
void CRC32(OpcodeArgs);
void UnimplementedOp(OpcodeArgs);
void InvalidOp(OpcodeArgs);
@@ -515,7 +580,7 @@ private:
OrderedNode *AppendSegmentOffset(OrderedNode *Value, uint32_t Flags, uint32_t DefaultPrefix = 0, bool Override = false);
OrderedNode *GetDynamicPC(FEXCore::X86Tables::DecodedOp const& Op, int64_t Offset = 0);
OrderedNode *GetRelocatedPC(FEXCore::X86Tables::DecodedOp const& Op, int64_t Offset = 0);
OrderedNode *LoadSource(FEXCore::IR::RegisterClassType Class, FEXCore::X86Tables::DecodedOp const& Op, FEXCore::X86Tables::DecodedOperand const& Operand, uint32_t Flags, int8_t Align, bool LoadData = true, bool ForceLoad = false);
OrderedNode *LoadSource_WithOpSize(FEXCore::IR::RegisterClassType Class, FEXCore::X86Tables::DecodedOp const& Op, FEXCore::X86Tables::DecodedOperand const& Operand, uint8_t OpSize, uint32_t Flags, int8_t Align, bool LoadData = true, bool ForceLoad = false);
void StoreResult_WithOpSize(FEXCore::IR::RegisterClassType Class, FEXCore::X86Tables::DecodedOp Op, FEXCore::X86Tables::DecodedOperand const& Operand, OrderedNode *const Src, uint8_t OpSize, int8_t Align);
@@ -553,23 +618,524 @@ private:
OrderedNode *SelectCC(uint8_t OP, OrderedNode *TrueValue, OrderedNode *FalseValue);
void GenerateFlags_ADC(FEXCore::X86Tables::DecodedOp Op, OrderedNode *Res, OrderedNode *Src1, OrderedNode *Src2, OrderedNode *CF);
void GenerateFlags_SBB(FEXCore::X86Tables::DecodedOp Op, OrderedNode *Res, OrderedNode *Src1, OrderedNode *Src2, OrderedNode *CF);
void GenerateFlags_SUB(FEXCore::X86Tables::DecodedOp Op, OrderedNode *Res, OrderedNode *Src1, OrderedNode *Src2, bool UpdateCF = true);
void GenerateFlags_ADD(FEXCore::X86Tables::DecodedOp Op, OrderedNode *Res, OrderedNode *Src1, OrderedNode *Src2, bool UpdateCF = true);
void GenerateFlags_MUL(FEXCore::X86Tables::DecodedOp Op, OrderedNode *Res, OrderedNode *High);
void GenerateFlags_UMUL(FEXCore::X86Tables::DecodedOp Op, OrderedNode *High);
void GenerateFlags_Logical(FEXCore::X86Tables::DecodedOp Op, OrderedNode *Res, OrderedNode *Src1, OrderedNode *Src2);
void GenerateFlags_ShiftLeft(FEXCore::X86Tables::DecodedOp Op, OrderedNode *Res, OrderedNode *Src1, OrderedNode *Src2);
void GenerateFlags_ShiftLeftImmediate(FEXCore::X86Tables::DecodedOp Op, OrderedNode *Res, OrderedNode *Src1, uint64_t Shift);
void GenerateFlags_ShiftRight(FEXCore::X86Tables::DecodedOp Op, OrderedNode *Res, OrderedNode *Src1, OrderedNode *Src2);
void GenerateFlags_ShiftRightImmediate(FEXCore::X86Tables::DecodedOp Op, OrderedNode *Res, OrderedNode *Src1, uint64_t Shift);
void GenerateFlags_SignShiftRight(FEXCore::X86Tables::DecodedOp Op, OrderedNode *Res, OrderedNode *Src1, OrderedNode *Src2);
void GenerateFlags_SignShiftRightImmediate(FEXCore::X86Tables::DecodedOp Op, OrderedNode *Res, OrderedNode *Src1, uint64_t Shift);
void GenerateFlags_RotateRight(FEXCore::X86Tables::DecodedOp Op, OrderedNode *Res, OrderedNode *Src1, OrderedNode *Src2);
void GenerateFlags_RotateLeft(FEXCore::X86Tables::DecodedOp Op, OrderedNode *Res, OrderedNode *Src1, OrderedNode *Src2);
void GenerateFlags_RotateRightImmediate(FEXCore::X86Tables::DecodedOp Op, OrderedNode *Res, OrderedNode *Src1, uint64_t Shift);
void GenerateFlags_RotateLeftImmediate(FEXCore::X86Tables::DecodedOp Op, OrderedNode *Res, OrderedNode *Src1, uint64_t Shift);
/**
* @name Deferred RFLAG calculation and generation.
*
* Only handles the six flags that ALU ops typically generate.
* Specifically: CF, PF, AF, ZF, SF, OF
* These six flags are heavily generated through basic ALU ops and balloon the IR if not early eliminated.
* This tracking structure only tracks single blocks and requires RFLAGS calculation at block-ending ops.
* Some flags generating ALU ops only touch part of the registers, In these cases it will do calculation up front.
* This means we still need our IR passes to eliminate all redundant flags accesses but this light OpcodeDispatcher optimization
* doesn't take it to that level.
* @{ */
// Deferred flag generation tracking structure.
// This structure is used to track RFlags from ALU ops for invalidation.
//
// Future ideas: Use an invalidation mask to do partial generation of flags.
// Particularly for the instructions that don't do the full set of flags calculations.
// These instructions currently calculate the deferred RFLAGS immediately then overwrite rflags state.
// RCLSE IR pass will catch and remove redundant rflags stores like this currently.
struct DeferredFlagData {
// What type of flags to generate
FlagsGenerationType Type {FlagsGenerationType::TYPE_NONE};
// Source size of the op
uint8_t SrcSize;
// Every flag generation type has a result
OrderedNode *Res{};
union {
// UMUL, BEXTR, BLSI, BLSMSK, POPCOUNT, TZCNT, LZCNT, BITSELECT, RDRAND
struct {
} NoSource;
// MUL, BLSR, BZHI
struct {
OrderedNode *Src1;
} OneSource;
// Logical, LSHL, LSHR, ASHR, ROR, ROL
struct {
OrderedNode *Src1;
OrderedNode *Src2;
} TwoSource;
// ADC, SBB
struct {
OrderedNode *Src1;
OrderedNode *Src2;
OrderedNode *Src3;
} ThreeSource;
// LSHLI, LSHRI, ASHRI, RORI, ROLI
struct {
OrderedNode *Src1;
uint64_t Imm;
} OneSrcImmediate;
// ADD, SUB
struct {
OrderedNode *Src1;
OrderedNode *Src2;
bool UpdateCF;
} TwoSrcImmediate;
} Sources{};
};
DeferredFlagData CurrentDeferredFlags{};
/**
* @brief Takes the current deferred flag state and stores the result in to RFLAGS.
*
* Once executed there will no longer be any deferred flag state and RFLAGS will have the correct flags in it.
* Necessary to do when leaving a IR block, or if an instruction is doing a partial overwrite of the flags.
*/
void CalculateDeferredFlags(uint32_t FlagsToCalculateMask = ~0U);
/**
* @brief Invalidates the current deferred flags structure.
*
* If the emulated instruction is going to overwrite all of the flags but isn't tracked using the deferred flag system
* then use this function to stop tracking the current active deferred flags.
*/
void InvalidateDeferredFlags() {
CurrentDeferredFlags.Type = FlagsGenerationType::TYPE_NONE;
}
/**
* @brief Checks if there is any deferred flag state active.
*
* @return True if RFLAGs contains the flags. False if deferred flags is tracking the data.
*/
bool IsDeferredFlagsStored() const {
return CurrentDeferredFlags.Type == FlagsGenerationType::TYPE_NONE;
}
/**
* @name These functions are used by the deferred flag handling while it is calculating and storing flags in to RFLAGs.
* @{ */
void CalculcateFlags_ADC(uint8_t SrcSize, OrderedNode *Res, OrderedNode *Src1, OrderedNode *Src2, OrderedNode *CF);
void CalculcateFlags_SBB(uint8_t SrcSize, OrderedNode *Res, OrderedNode *Src1, OrderedNode *Src2, OrderedNode *CF);
void CalculcateFlags_SUB(uint8_t SrcSize, OrderedNode *Res, OrderedNode *Src1, OrderedNode *Src2, bool UpdateCF = true);
void CalculcateFlags_ADD(uint8_t SrcSize, OrderedNode *Res, OrderedNode *Src1, OrderedNode *Src2, bool UpdateCF = true);
void CalculcateFlags_MUL(uint8_t SrcSize, OrderedNode *Res, OrderedNode *High);
void CalculcateFlags_UMUL(OrderedNode *High);
void CalculcateFlags_Logical(uint8_t SrcSize, OrderedNode *Res, OrderedNode *Src1, OrderedNode *Src2);
void CalculcateFlags_ShiftLeft(uint8_t SrcSize, OrderedNode *Res, OrderedNode *Src1, OrderedNode *Src2);
void CalculcateFlags_ShiftLeftImmediate(uint8_t SrcSize, OrderedNode *Res, OrderedNode *Src1, uint64_t Shift);
void CalculcateFlags_ShiftRight(uint8_t SrcSize, OrderedNode *Res, OrderedNode *Src1, OrderedNode *Src2);
void CalculcateFlags_ShiftRightImmediate(uint8_t SrcSize, OrderedNode *Res, OrderedNode *Src1, uint64_t Shift);
void CalculcateFlags_SignShiftRight(uint8_t SrcSize, OrderedNode *Res, OrderedNode *Src1, OrderedNode *Src2);
void CalculcateFlags_SignShiftRightImmediate(uint8_t SrcSize, OrderedNode *Res, OrderedNode *Src1, uint64_t Shift);
void CalculcateFlags_RotateRight(uint8_t SrcSize, OrderedNode *Res, OrderedNode *Src1, OrderedNode *Src2);
void CalculcateFlags_RotateLeft(uint8_t SrcSize, OrderedNode *Res, OrderedNode *Src1, OrderedNode *Src2);
void CalculcateFlags_RotateRightImmediate(uint8_t SrcSize, OrderedNode *Res, OrderedNode *Src1, uint64_t Shift);
void CalculcateFlags_RotateLeftImmediate(uint8_t SrcSize, OrderedNode *Res, OrderedNode *Src1, uint64_t Shift);
void CalculcateFlags_FCMP(uint8_t SrcSize, OrderedNode *Res, OrderedNode *Src1, OrderedNode *Src2);
void CalculcateFlags_BEXTR(OrderedNode *Src);
void CalculcateFlags_BLSI(uint8_t SrcSize, OrderedNode *Src);
void CalculcateFlags_BLSMSK(OrderedNode *Src);
void CalculcateFlags_BLSR(uint8_t SrcSize, OrderedNode *Res, OrderedNode *Src);
void CalculcateFlags_POPCOUNT(OrderedNode *Src);
void CalculcateFlags_BZHI(uint8_t SrcSize, OrderedNode *Result, OrderedNode *Src);
void CalculcateFlags_TZCNT(OrderedNode *Src);
void CalculcateFlags_LZCNT(uint8_t SrcSize, OrderedNode *Src);
void CalculcateFlags_BITSELECT(OrderedNode *Src);
void CalculcateFlags_RDRAND(OrderedNode *Src);
/** @} */
/**
* @name These functions generated deferred RFLAGs tracking.
*
* Depending on the operation it may force a RFLAGs calculation before storing the new deferred state.
* @{ */
void GenerateFlags_ADC(FEXCore::X86Tables::DecodedOp Op, OrderedNode *Res, OrderedNode *Src1, OrderedNode *Src2, OrderedNode *CF) {
CurrentDeferredFlags = DeferredFlagData {
.Type = FlagsGenerationType::TYPE_ADC,
.SrcSize = GetSrcSize(Op),
.Res = Res,
.Sources = {
.ThreeSource = {
.Src1 = Src1,
.Src2 = Src2,
.Src3 = CF,
},
},
};
}
void GenerateFlags_SBB(FEXCore::X86Tables::DecodedOp Op, OrderedNode *Res, OrderedNode *Src1, OrderedNode *Src2, OrderedNode *CF) {
CurrentDeferredFlags = DeferredFlagData {
.Type = FlagsGenerationType::TYPE_SBB,
.SrcSize = GetSrcSize(Op),
.Res = Res,
.Sources = {
.ThreeSource = {
.Src1 = Src1,
.Src2 = Src2,
.Src3 = CF,
},
},
};
}
void GenerateFlags_SUB(FEXCore::X86Tables::DecodedOp Op, OrderedNode *Res, OrderedNode *Src1, OrderedNode *Src2, bool UpdateCF = true) {
if (!UpdateCF) {
// If we aren't updating CF then we need to calculate flags. Invalidation mask would make this not required.
CalculateDeferredFlags();
}
CurrentDeferredFlags = DeferredFlagData {
.Type = FlagsGenerationType::TYPE_SUB,
.SrcSize = GetSrcSize(Op),
.Res = Res,
.Sources = {
.TwoSrcImmediate = {
.Src1 = Src1,
.Src2 = Src2,
.UpdateCF = UpdateCF,
},
},
};
}
void GenerateFlags_ADD(FEXCore::X86Tables::DecodedOp Op, OrderedNode *Res, OrderedNode *Src1, OrderedNode *Src2, bool UpdateCF = true) {
if (!UpdateCF) {
// If we aren't updating CF then we need to calculate flags. Invalidation mask would make this not required.
CalculateDeferredFlags();
}
CurrentDeferredFlags = DeferredFlagData {
.Type = FlagsGenerationType::TYPE_ADD,
.SrcSize = GetSrcSize(Op),
.Res = Res,
.Sources = {
.TwoSrcImmediate = {
.Src1 = Src1,
.Src2 = Src2,
.UpdateCF = UpdateCF,
},
},
};
}
void GenerateFlags_MUL(FEXCore::X86Tables::DecodedOp Op, OrderedNode *Res, OrderedNode *High) {
CurrentDeferredFlags = DeferredFlagData {
.Type = FlagsGenerationType::TYPE_MUL,
.SrcSize = GetSrcSize(Op),
.Res = Res,
.Sources = {
.OneSource = {
.Src1 = High,
},
},
};
}
void GenerateFlags_UMUL(FEXCore::X86Tables::DecodedOp Op, OrderedNode *High) {
CurrentDeferredFlags = DeferredFlagData {
.Type = FlagsGenerationType::TYPE_UMUL,
.SrcSize = GetSrcSize(Op),
.Res = High,
};
}
void GenerateFlags_Logical(FEXCore::X86Tables::DecodedOp Op, OrderedNode *Res, OrderedNode *Src1, OrderedNode *Src2) {
CurrentDeferredFlags = DeferredFlagData {
.Type = FlagsGenerationType::TYPE_LOGICAL,
.SrcSize = GetSrcSize(Op),
.Res = Res,
.Sources = {
.TwoSource = {
.Src1 = Src1,
.Src2 = Src2,
},
},
};
}
void GenerateFlags_ShiftLeft(FEXCore::X86Tables::DecodedOp Op, OrderedNode *Res, OrderedNode *Src1, OrderedNode *Src2) {
// Flags need to be used, generate incoming flags first.
CalculateDeferredFlags();
CurrentDeferredFlags = DeferredFlagData {
.Type = FlagsGenerationType::TYPE_LSHL,
.SrcSize = GetSrcSize(Op),
.Res = Res,
.Sources = {
.TwoSource = {
.Src1 = Src1,
.Src2 = Src2,
},
},
};
}
void GenerateFlags_ShiftRight(FEXCore::X86Tables::DecodedOp Op, OrderedNode *Res, OrderedNode *Src1, OrderedNode *Src2) {
// Flags need to be used, generate incoming flags first.
CalculateDeferredFlags();
CurrentDeferredFlags = DeferredFlagData {
.Type = FlagsGenerationType::TYPE_LSHR,
.SrcSize = GetSrcSize(Op),
.Res = Res,
.Sources = {
.TwoSource = {
.Src1 = Src1,
.Src2 = Src2,
},
},
};
}
void GenerateFlags_SignShiftRight(FEXCore::X86Tables::DecodedOp Op, OrderedNode *Res, OrderedNode *Src1, OrderedNode *Src2) {
// Flags need to be used, generate incoming flags first.
CalculateDeferredFlags();
CurrentDeferredFlags = DeferredFlagData {
.Type = FlagsGenerationType::TYPE_ASHR,
.SrcSize = GetSrcSize(Op),
.Res = Res,
.Sources = {
.TwoSource = {
.Src1 = Src1,
.Src2 = Src2,
},
},
};
}
void GenerateFlags_ShiftLeftImmediate(FEXCore::X86Tables::DecodedOp Op, OrderedNode *Res, OrderedNode *Src1, uint64_t Shift) {
// No flags changed if shift is zero.
if (Shift == 0) return;
CurrentDeferredFlags = DeferredFlagData {
.Type = FlagsGenerationType::TYPE_LSHLI,
.SrcSize = GetSrcSize(Op),
.Res = Res,
.Sources = {
.OneSrcImmediate = {
.Src1 = Src1,
.Imm = Shift,
},
},
};
}
void GenerateFlags_SignShiftRightImmediate(FEXCore::X86Tables::DecodedOp Op, OrderedNode *Res, OrderedNode *Src1, uint64_t Shift) {
// No flags changed if shift is zero.
if (Shift == 0) return;
CurrentDeferredFlags = DeferredFlagData {
.Type = FlagsGenerationType::TYPE_ASHRI,
.SrcSize = GetSrcSize(Op),
.Res = Res,
.Sources = {
.OneSrcImmediate = {
.Src1 = Src1,
.Imm = Shift,
},
},
};
}
void GenerateFlags_ShiftRightImmediate(FEXCore::X86Tables::DecodedOp Op, OrderedNode *Res, OrderedNode *Src1, uint64_t Shift) {
// No flags changed if shift is zero.
if (Shift == 0) return;
CurrentDeferredFlags = DeferredFlagData {
.Type = FlagsGenerationType::TYPE_LSHRI,
.SrcSize = GetSrcSize(Op),
.Res = Res,
.Sources = {
.OneSrcImmediate = {
.Src1 = Src1,
.Imm = Shift,
},
},
};
}
void GenerateFlags_RotateRight(FEXCore::X86Tables::DecodedOp Op, OrderedNode *Res, OrderedNode *Src1, OrderedNode *Src2) {
// Doesn't set all the flags, needs to calculate.
CalculateDeferredFlags();
CurrentDeferredFlags = DeferredFlagData {
.Type = FlagsGenerationType::TYPE_ROR,
.SrcSize = GetSrcSize(Op),
.Res = Res,
.Sources = {
.TwoSource = {
.Src1 = Src1,
.Src2 = Src2,
},
},
};
}
void GenerateFlags_RotateLeft(FEXCore::X86Tables::DecodedOp Op, OrderedNode *Res, OrderedNode *Src1, OrderedNode *Src2) {
// Doesn't set all the flags, needs to calculate.
CalculateDeferredFlags();
CurrentDeferredFlags = DeferredFlagData {
.Type = FlagsGenerationType::TYPE_ROL,
.SrcSize = GetSrcSize(Op),
.Res = Res,
.Sources = {
.TwoSource = {
.Src1 = Src1,
.Src2 = Src2,
},
},
};
}
void GenerateFlags_RotateRightImmediate(FEXCore::X86Tables::DecodedOp Op, OrderedNode *Res, OrderedNode *Src1, uint64_t Shift) {
if (Shift == 0) return;
// Doesn't set all the flags, needs to calculate.
CalculateDeferredFlags();
CurrentDeferredFlags = DeferredFlagData {
.Type = FlagsGenerationType::TYPE_RORI,
.SrcSize = GetSrcSize(Op),
.Res = Res,
.Sources = {
.OneSrcImmediate = {
.Src1 = Src1,
.Imm = Shift,
},
},
};
}
void GenerateFlags_RotateLeftImmediate(FEXCore::X86Tables::DecodedOp Op, OrderedNode *Res, OrderedNode *Src1, uint64_t Shift) {
if (Shift == 0) return;
// Doesn't set all the flags, needs to calculate.
CalculateDeferredFlags();
CurrentDeferredFlags = DeferredFlagData {
.Type = FlagsGenerationType::TYPE_ROLI,
.SrcSize = GetSrcSize(Op),
.Res = Res,
.Sources = {
.OneSrcImmediate = {
.Src1 = Src1,
.Imm = Shift,
},
}
};
}
void GenerateFlags_FCMP(FEXCore::X86Tables::DecodedOp Op, OrderedNode *Res, OrderedNode *Src1, OrderedNode *Src2) {
CurrentDeferredFlags = DeferredFlagData {
.Type = FlagsGenerationType::TYPE_FCMP,
.SrcSize = GetSrcSize(Op),
.Res = Res,
.Sources = {
.TwoSource = {
.Src1 = Src1,
.Src2 = Src2,
},
}
};
}
void GenerateFlags_BEXTR(FEXCore::X86Tables::DecodedOp Op, OrderedNode *Src) {
CurrentDeferredFlags = DeferredFlagData {
.Type = FlagsGenerationType::TYPE_BEXTR,
.SrcSize = GetSrcSize(Op),
.Res = Src,
};
}
void GenerateFlags_BLSI(FEXCore::X86Tables::DecodedOp Op, OrderedNode *Src) {
CurrentDeferredFlags = DeferredFlagData {
.Type = FlagsGenerationType::TYPE_BLSI,
.SrcSize = GetSrcSize(Op),
.Res = Src,
};
}
void GenerateFlags_BLSMSK(FEXCore::X86Tables::DecodedOp Op, OrderedNode *Src) {
CurrentDeferredFlags = DeferredFlagData {
.Type = FlagsGenerationType::TYPE_BLSMSK,
.SrcSize = GetSrcSize(Op),
.Res = Src,
};
}
void GenerateFlags_BLSR(FEXCore::X86Tables::DecodedOp Op, OrderedNode *Res, OrderedNode *Src) {
CurrentDeferredFlags = DeferredFlagData {
.Type = FlagsGenerationType::TYPE_BLSR,
.SrcSize = GetSrcSize(Op),
.Res = Res,
.Sources = {
.OneSource = {
.Src1 = Src,
},
},
};
}
void GenerateFlags_POPCOUNT(FEXCore::X86Tables::DecodedOp Op, OrderedNode *Src) {
CurrentDeferredFlags = DeferredFlagData {
.Type = FlagsGenerationType::TYPE_POPCOUNT,
.SrcSize = GetSrcSize(Op),
.Res = Src,
};
}
void GenerateFlags_BZHI(FEXCore::X86Tables::DecodedOp Op, OrderedNode *Result, OrderedNode *Src) {
CurrentDeferredFlags = DeferredFlagData {
.Type = FlagsGenerationType::TYPE_BZHI,
.SrcSize = GetSrcSize(Op),
.Res = Result,
.Sources = {
.OneSource = {
.Src1 = Src,
},
},
};
}
void GenerateFlags_TZCNT(FEXCore::X86Tables::DecodedOp Op, OrderedNode *Src) {
CurrentDeferredFlags = DeferredFlagData {
.Type = FlagsGenerationType::TYPE_TZCNT,
.SrcSize = GetSrcSize(Op),
.Res = Src,
};
}
void GenerateFlags_LZCNT(FEXCore::X86Tables::DecodedOp Op, OrderedNode *Src) {
CurrentDeferredFlags = DeferredFlagData {
.Type = FlagsGenerationType::TYPE_LZCNT,
.SrcSize = GetSrcSize(Op),
.Res = Src,
};
}
void GenerateFlags_BITSELECT(FEXCore::X86Tables::DecodedOp Op, OrderedNode *Src) {
CurrentDeferredFlags = DeferredFlagData {
.Type = FlagsGenerationType::TYPE_BITSELECT,
.SrcSize = GetSrcSize(Op),
.Res = Src,
};
}
void GenerateFlags_RDRAND(FEXCore::X86Tables::DecodedOp Op, OrderedNode *Src) {
CurrentDeferredFlags = DeferredFlagData {
.Type = FlagsGenerationType::TYPE_RDRAND,
.SrcSize = GetSrcSize(Op),
.Res = Src,
};
}
/** @} */
/** @} */
OrderedNode * GetX87Top();
enum class X87Tag {
@@ -596,24 +1162,35 @@ private:
bool Multiblock{};
uint64_t Entry;
OrderedNode* _StoreMemAutoTSO(FEXCore::IR::RegisterClassType Class, uint8_t Size, OrderedNode *ssa0, OrderedNode *ssa1, uint8_t Align = 1) {
OrderedNode* _StoreMemAutoTSO(FEXCore::IR::RegisterClassType Class, uint8_t Size, OrderedNode *Addr, OrderedNode *Value, uint8_t Align = 1) {
if (CTX->Config.TSOEnabled)
return _StoreMemTSO(ssa0, ssa1, Invalid(), Align, Class, MEM_OFFSET_SXTX, 1, Size);
return _StoreMemTSO(Class, Size, Value, Addr, Invalid(), Align, MEM_OFFSET_SXTX, 1);
else
return _StoreMem(ssa0, ssa1, Invalid(), Align, Class, MEM_OFFSET_SXTX, 1, Size);
return _StoreMem(Class, Size, Value, Addr, Invalid(), Align, MEM_OFFSET_SXTX, 1);
}
OrderedNode* _LoadMemAutoTSO(FEXCore::IR::RegisterClassType Class, uint8_t Size, OrderedNode *ssa0, uint8_t Align = 1) {
if (CTX->Config.TSOEnabled)
return _LoadMemTSO(ssa0, Invalid(), Align, Class, MEM_OFFSET_SXTX, 1, Size);
return _LoadMemTSO(Class, Size, ssa0, Invalid(), Align, MEM_OFFSET_SXTX, 1);
else
return _LoadMem(ssa0, Invalid(), Align, Class, MEM_OFFSET_SXTX, 1, Size);
return _LoadMem(Class, Size, ssa0, Invalid(), Align, MEM_OFFSET_SXTX, 1);
}
};
void InstallOpcodeHandlers(Context::OperatingMode Mode);
}
template <>
struct fmt::formatter<FEXCore::IR::OpDispatchBuilder::FlagsGenerationType> : fmt::formatter<int> {
using Base = fmt::formatter<int>;
// Pass-through the underlying value, so IDs can
// be formatted like any integral value.
template <typename FormatContext>
auto format(const FEXCore::IR::OpDispatchBuilder::FlagsGenerationType& ID, FormatContext& ctx) {
return Base::format(static_cast<int>(ID), ctx);
}
};
@@ -41,8 +41,17 @@ constexpr std::array<uint32_t, 17> FlagOffsets = {
void OpDispatchBuilder::SetPackedRFLAG(bool Lower8, OrderedNode *Src) {
size_t NumFlags = FlagOffsets.size();
if (Lower8) {
// Calculate flags early.
// Could use InvalidateDeferredFlags() if we had masked invalidation.
// This is only a partial overwrite of flags since OF isn't stored here.
CalculateDeferredFlags();
NumFlags = 5;
}
else {
// We are overwriting all RFLAGS. Invalidate the deferred flag state.
InvalidateDeferredFlags();
}
auto OneConst = _Constant(1);
for (size_t i = 0; i < NumFlags; ++i) {
const auto FlagOffset = FlagOffsets[i];
@@ -52,6 +61,9 @@ void OpDispatchBuilder::SetPackedRFLAG(bool Lower8, OrderedNode *Src) {
}
OrderedNode *OpDispatchBuilder::GetPackedRFLAG(bool Lower8) {
// Calculate flags early.
CalculateDeferredFlags();
OrderedNode *Original = _Constant(2);
size_t NumFlags = FlagOffsets.size();
if (Lower8) {
@@ -68,8 +80,188 @@ OrderedNode *OpDispatchBuilder::GetPackedRFLAG(bool Lower8) {
return Original;
}
void OpDispatchBuilder::GenerateFlags_ADC(FEXCore::X86Tables::DecodedOp Op, OrderedNode *Res, OrderedNode *Src1, OrderedNode *Src2, OrderedNode *CF) {
auto Size = GetSrcSize(Op) * 8;
void OpDispatchBuilder::CalculateDeferredFlags(uint32_t FlagsToCalculateMask) {
if (CurrentDeferredFlags.Type == FlagsGenerationType::TYPE_NONE) {
// Nothing to do
return;
}
switch (CurrentDeferredFlags.Type) {
case FlagsGenerationType::TYPE_ADC:
CalculcateFlags_ADC(
CurrentDeferredFlags.SrcSize,
CurrentDeferredFlags.Res,
CurrentDeferredFlags.Sources.ThreeSource.Src1,
CurrentDeferredFlags.Sources.ThreeSource.Src2,
CurrentDeferredFlags.Sources.ThreeSource.Src3);
break;
case FlagsGenerationType::TYPE_SBB:
CalculcateFlags_SBB(
CurrentDeferredFlags.SrcSize,
CurrentDeferredFlags.Res,
CurrentDeferredFlags.Sources.ThreeSource.Src1,
CurrentDeferredFlags.Sources.ThreeSource.Src2,
CurrentDeferredFlags.Sources.ThreeSource.Src3);
break;
case FlagsGenerationType::TYPE_SUB:
CalculcateFlags_SUB(
CurrentDeferredFlags.SrcSize,
CurrentDeferredFlags.Res,
CurrentDeferredFlags.Sources.TwoSrcImmediate.Src1,
CurrentDeferredFlags.Sources.TwoSrcImmediate.Src2,
CurrentDeferredFlags.Sources.TwoSrcImmediate.UpdateCF);
break;
case FlagsGenerationType::TYPE_ADD:
CalculcateFlags_ADD(
CurrentDeferredFlags.SrcSize,
CurrentDeferredFlags.Res,
CurrentDeferredFlags.Sources.TwoSrcImmediate.Src1,
CurrentDeferredFlags.Sources.TwoSrcImmediate.Src2,
CurrentDeferredFlags.Sources.TwoSrcImmediate.UpdateCF);
break;
case FlagsGenerationType::TYPE_MUL:
CalculcateFlags_MUL(
CurrentDeferredFlags.SrcSize,
CurrentDeferredFlags.Res,
CurrentDeferredFlags.Sources.OneSource.Src1);
break;
case FlagsGenerationType::TYPE_UMUL:
CalculcateFlags_UMUL(CurrentDeferredFlags.Res);
break;
case FlagsGenerationType::TYPE_LOGICAL:
CalculcateFlags_Logical(
CurrentDeferredFlags.SrcSize,
CurrentDeferredFlags.Res,
CurrentDeferredFlags.Sources.TwoSource.Src1,
CurrentDeferredFlags.Sources.TwoSource.Src2);
break;
case FlagsGenerationType::TYPE_LSHL:
CalculcateFlags_ShiftLeft(
CurrentDeferredFlags.SrcSize,
CurrentDeferredFlags.Res,
CurrentDeferredFlags.Sources.TwoSource.Src1,
CurrentDeferredFlags.Sources.TwoSource.Src2);
break;
case FlagsGenerationType::TYPE_LSHLI:
CalculcateFlags_ShiftLeftImmediate(
CurrentDeferredFlags.SrcSize,
CurrentDeferredFlags.Res,
CurrentDeferredFlags.Sources.OneSrcImmediate.Src1,
CurrentDeferredFlags.Sources.OneSrcImmediate.Imm);
break;
case FlagsGenerationType::TYPE_LSHR:
CalculcateFlags_ShiftRight(
CurrentDeferredFlags.SrcSize,
CurrentDeferredFlags.Res,
CurrentDeferredFlags.Sources.TwoSource.Src1,
CurrentDeferredFlags.Sources.TwoSource.Src2);
break;
case FlagsGenerationType::TYPE_LSHRI:
CalculcateFlags_ShiftRightImmediate(
CurrentDeferredFlags.SrcSize,
CurrentDeferredFlags.Res,
CurrentDeferredFlags.Sources.OneSrcImmediate.Src1,
CurrentDeferredFlags.Sources.OneSrcImmediate.Imm);
break;
case FlagsGenerationType::TYPE_ASHR:
CalculcateFlags_SignShiftRight(
CurrentDeferredFlags.SrcSize,
CurrentDeferredFlags.Res,
CurrentDeferredFlags.Sources.TwoSource.Src1,
CurrentDeferredFlags.Sources.TwoSource.Src2);
break;
case FlagsGenerationType::TYPE_ASHRI:
CalculcateFlags_SignShiftRightImmediate(
CurrentDeferredFlags.SrcSize,
CurrentDeferredFlags.Res,
CurrentDeferredFlags.Sources.OneSrcImmediate.Src1,
CurrentDeferredFlags.Sources.OneSrcImmediate.Imm);
break;
case FlagsGenerationType::TYPE_ROR:
CalculcateFlags_RotateRight(
CurrentDeferredFlags.SrcSize,
CurrentDeferredFlags.Res,
CurrentDeferredFlags.Sources.TwoSource.Src1,
CurrentDeferredFlags.Sources.TwoSource.Src2);
break;
case FlagsGenerationType::TYPE_RORI:
CalculcateFlags_RotateRightImmediate(
CurrentDeferredFlags.SrcSize,
CurrentDeferredFlags.Res,
CurrentDeferredFlags.Sources.OneSrcImmediate.Src1,
CurrentDeferredFlags.Sources.OneSrcImmediate.Imm);
break;
case FlagsGenerationType::TYPE_ROL:
CalculcateFlags_RotateLeft(
CurrentDeferredFlags.SrcSize,
CurrentDeferredFlags.Res,
CurrentDeferredFlags.Sources.TwoSource.Src1,
CurrentDeferredFlags.Sources.TwoSource.Src2);
break;
case FlagsGenerationType::TYPE_ROLI:
CalculcateFlags_RotateLeftImmediate(
CurrentDeferredFlags.SrcSize,
CurrentDeferredFlags.Res,
CurrentDeferredFlags.Sources.OneSrcImmediate.Src1,
CurrentDeferredFlags.Sources.OneSrcImmediate.Imm);
break;
case FlagsGenerationType::TYPE_FCMP:
CalculcateFlags_FCMP(
CurrentDeferredFlags.SrcSize,
CurrentDeferredFlags.Res,
CurrentDeferredFlags.Sources.TwoSource.Src1,
CurrentDeferredFlags.Sources.TwoSource.Src2);
break;
case FlagsGenerationType::TYPE_BEXTR:
CalculcateFlags_BEXTR(CurrentDeferredFlags.Res);
break;
case FlagsGenerationType::TYPE_BLSI:
CalculcateFlags_BLSI(
CurrentDeferredFlags.SrcSize,
CurrentDeferredFlags.Res);
break;
case FlagsGenerationType::TYPE_BLSMSK:
CalculcateFlags_BLSMSK(CurrentDeferredFlags.Res);
break;
case FlagsGenerationType::TYPE_BLSR:
CalculcateFlags_BLSR(
CurrentDeferredFlags.SrcSize,
CurrentDeferredFlags.Res,
CurrentDeferredFlags.Sources.OneSource.Src1);
break;
case FlagsGenerationType::TYPE_POPCOUNT:
CalculcateFlags_POPCOUNT(CurrentDeferredFlags.Res);
break;
case FlagsGenerationType::TYPE_BZHI:
CalculcateFlags_BZHI(
CurrentDeferredFlags.SrcSize,
CurrentDeferredFlags.Res,
CurrentDeferredFlags.Sources.OneSource.Src1);
break;
case FlagsGenerationType::TYPE_TZCNT:
CalculcateFlags_TZCNT(CurrentDeferredFlags.Res);
break;
case FlagsGenerationType::TYPE_LZCNT:
CalculcateFlags_LZCNT(
CurrentDeferredFlags.SrcSize,
CurrentDeferredFlags.Res);
break;
case FlagsGenerationType::TYPE_BITSELECT:
CalculcateFlags_BITSELECT(CurrentDeferredFlags.Res);
break;
case FlagsGenerationType::TYPE_RDRAND:
CalculcateFlags_RDRAND(CurrentDeferredFlags.Res);
break;
case FlagsGenerationType::TYPE_NONE:
default: ERROR_AND_DIE_FMT("Unhandled flags type {}", CurrentDeferredFlags.Type);
}
// Done calculating
CurrentDeferredFlags.Type = FlagsGenerationType::TYPE_NONE;
}
void OpDispatchBuilder::CalculcateFlags_ADC(uint8_t SrcSize, OrderedNode *Res, OrderedNode *Src1, OrderedNode *Src2, OrderedNode *CF) {
auto Size = SrcSize * 8;
// AF
{
OrderedNode *AFRes = _Xor(_Xor(Src1, Src2), Res);
@@ -79,7 +271,7 @@ void OpDispatchBuilder::GenerateFlags_ADC(FEXCore::X86Tables::DecodedOp Op, Orde
// SF
{
auto SignBitConst = _Constant(GetSrcSize(Op) * 8 - 1);
auto SignBitConst = _Constant(Size - 1);
auto LshrOp = _Lshr(Res, SignBitConst);
SetRFLAG<FEXCore::X86State::RFLAG_SF_LOC>(LshrOp);
@@ -140,9 +332,7 @@ void OpDispatchBuilder::GenerateFlags_ADC(FEXCore::X86Tables::DecodedOp Op, Orde
}
}
void OpDispatchBuilder::GenerateFlags_SBB(FEXCore::X86Tables::DecodedOp Op, OrderedNode *Res, OrderedNode *Src1, OrderedNode *Src2, OrderedNode *CF) {
const auto SrcSize = GetSrcSize(Op);
void OpDispatchBuilder::CalculcateFlags_SBB(uint8_t SrcSize, OrderedNode *Res, OrderedNode *Src1, OrderedNode *Src2, OrderedNode *CF) {
// AF
{
OrderedNode *AFRes = _Xor(_Xor(Src1, Src2), Res);
@@ -212,7 +402,7 @@ void OpDispatchBuilder::GenerateFlags_SBB(FEXCore::X86Tables::DecodedOp Op, Orde
}
}
void OpDispatchBuilder::GenerateFlags_SUB(FEXCore::X86Tables::DecodedOp Op, OrderedNode *Res, OrderedNode *Src1, OrderedNode *Src2, bool UpdateCF) {
void OpDispatchBuilder::CalculcateFlags_SUB(uint8_t SrcSize, OrderedNode *Res, OrderedNode *Src1, OrderedNode *Src2, bool UpdateCF) {
// AF
{
OrderedNode *AFRes = _Xor(_Xor(Src1, Src2), Res);
@@ -222,7 +412,7 @@ void OpDispatchBuilder::GenerateFlags_SUB(FEXCore::X86Tables::DecodedOp Op, Orde
// SF
{
auto SignBitConst = _Constant(GetSrcSize(Op) * 8 - 1);
auto SignBitConst = _Constant(SrcSize * 8 - 1);
auto LshrOp = _Lshr(Res, SignBitConst);
SetRFLAG<FEXCore::X86State::RFLAG_SF_LOC>(LshrOp);
@@ -263,15 +453,13 @@ void OpDispatchBuilder::GenerateFlags_SUB(FEXCore::X86Tables::DecodedOp Op, Orde
auto XorOp2 = _Xor(Res, Src1);
OrderedNode *FinalAnd = _And(XorOp1, XorOp2);
FinalAnd = _Bfe(1, GetSrcSize(Op) * 8 - 1, FinalAnd);
FinalAnd = _Bfe(1, SrcSize * 8 - 1, FinalAnd);
SetRFLAG<FEXCore::X86State::RFLAG_OF_LOC>(FinalAnd);
}
}
void OpDispatchBuilder::GenerateFlags_ADD(FEXCore::X86Tables::DecodedOp Op, OrderedNode *Res, OrderedNode *Src1, OrderedNode *Src2, bool UpdateCF) {
const auto SrcSize = GetSrcSize(Op);
void OpDispatchBuilder::CalculcateFlags_ADD(uint8_t SrcSize, OrderedNode *Res, OrderedNode *Src1, OrderedNode *Src2, bool UpdateCF) {
// AF
{
OrderedNode *AFRes = _Xor(_Xor(Src1, Src2), Res);
@@ -339,7 +527,7 @@ void OpDispatchBuilder::GenerateFlags_ADD(FEXCore::X86Tables::DecodedOp Op, Orde
}
}
void OpDispatchBuilder::GenerateFlags_MUL(FEXCore::X86Tables::DecodedOp Op, OrderedNode *Res, OrderedNode *High) {
void OpDispatchBuilder::CalculcateFlags_MUL(uint8_t SrcSize, OrderedNode *Res, OrderedNode *High) {
// PF/AF/ZF/SF
// Undefined
{
@@ -354,7 +542,7 @@ void OpDispatchBuilder::GenerateFlags_MUL(FEXCore::X86Tables::DecodedOp Op, Orde
// CF and OF are set if the result of the operation can't be fit in to the destination register
// If the value can fit then the top bits will be zero
auto SignBit = _Sbfe(1, GetSrcSize(Op) * 8 - 1, Res);
auto SignBit = _Sbfe(1, SrcSize * 8 - 1, Res);
auto SelectOp = _Select(FEXCore::IR::COND_EQ, High, SignBit, _Constant(0), _Constant(1));
@@ -363,7 +551,7 @@ void OpDispatchBuilder::GenerateFlags_MUL(FEXCore::X86Tables::DecodedOp Op, Orde
}
}
void OpDispatchBuilder::GenerateFlags_UMUL(FEXCore::X86Tables::DecodedOp Op, OrderedNode *High) {
void OpDispatchBuilder::CalculcateFlags_UMUL(OrderedNode *High) {
// AF/SF/PF/ZF
// Undefined
{
@@ -385,7 +573,7 @@ void OpDispatchBuilder::GenerateFlags_UMUL(FEXCore::X86Tables::DecodedOp Op, Ord
}
}
void OpDispatchBuilder::GenerateFlags_Logical(FEXCore::X86Tables::DecodedOp Op, OrderedNode *Res, OrderedNode *Src1, OrderedNode *Src2) {
void OpDispatchBuilder::CalculcateFlags_Logical(uint8_t SrcSize, OrderedNode *Res, OrderedNode *Src1, OrderedNode *Src2) {
// AF
{
// Undefined
@@ -395,7 +583,7 @@ void OpDispatchBuilder::GenerateFlags_Logical(FEXCore::X86Tables::DecodedOp Op,
// SF
{
auto SignBitConst = _Constant(GetSrcSize(Op) * 8 - 1);
auto SignBitConst = _Constant(SrcSize * 8 - 1);
auto LshrOp = _Lshr(Res, SignBitConst);
SetRFLAG<FEXCore::X86State::RFLAG_SF_LOC>(LshrOp);
@@ -430,11 +618,11 @@ auto oldflag = GetRFLAG(FEXCore::X86State::flag);\
auto newval = _Select(FEXCore::IR::COND_EQ, cond, _Constant(0), oldflag, newflag);\
SetRFLAG<FEXCore::X86State::flag>(newval);
void OpDispatchBuilder::GenerateFlags_ShiftLeft(FEXCore::X86Tables::DecodedOp Op, OrderedNode *Res, OrderedNode *Src1, OrderedNode *Src2) {
void OpDispatchBuilder::CalculcateFlags_ShiftLeft(uint8_t SrcSize, OrderedNode *Res, OrderedNode *Src1, OrderedNode *Src2) {
// CF
{
// Extract the last bit shifted in to CF
auto Size = _Constant(GetSrcSize(Op) * 8);
auto Size = _Constant(SrcSize * 8);
auto ShiftAmt = _Sub(Size, Src2);
auto LastBit = _And(_Lshr(Src1, ShiftAmt), _Constant(1));
COND_FLAG_SET(Src2, RFLAG_CF_LOC, LastBit);
@@ -466,7 +654,7 @@ void OpDispatchBuilder::GenerateFlags_ShiftLeft(FEXCore::X86Tables::DecodedOp Op
// SF
{
auto val = _Bfe(1, GetSrcSize(Op) * 8 - 1, Res);
auto val = _Bfe(1, SrcSize * 8 - 1, Res);
COND_FLAG_SET(Src2, RFLAG_SF_LOC, val);
}
@@ -474,12 +662,12 @@ void OpDispatchBuilder::GenerateFlags_ShiftLeft(FEXCore::X86Tables::DecodedOp Op
{
// In the case of left shift. OF is only set from the result of <Top Source Bit> XOR <Top Result Bit>
// When Shift > 1 then OF is undefined
auto val = _Bfe(1, GetSrcSize(Op) * 8 - 1, _Xor(Src1, Res));
auto val = _Bfe(1, SrcSize * 8 - 1, _Xor(Src1, Res));
COND_FLAG_SET(Src2, RFLAG_OF_LOC, val);
}
}
void OpDispatchBuilder::GenerateFlags_ShiftRight(FEXCore::X86Tables::DecodedOp Op, OrderedNode *Res, OrderedNode *Src1, OrderedNode *Src2) {
void OpDispatchBuilder::CalculcateFlags_ShiftRight(uint8_t SrcSize, OrderedNode *Res, OrderedNode *Src1, OrderedNode *Src2) {
// CF
{
// Extract the last bit shifted in to CF
@@ -514,7 +702,7 @@ void OpDispatchBuilder::GenerateFlags_ShiftRight(FEXCore::X86Tables::DecodedOp O
// SF
{
auto val =_Bfe(1, GetSrcSize(Op) * 8 - 1, Res);
auto val =_Bfe(1, SrcSize * 8 - 1, Res);
COND_FLAG_SET(Src2, RFLAG_SF_LOC, val);
}
@@ -522,12 +710,12 @@ void OpDispatchBuilder::GenerateFlags_ShiftRight(FEXCore::X86Tables::DecodedOp O
{
// Only defined when Shift is 1 else undefined
// OF flag is set if a sign change occurred
auto val = _Bfe(1, GetSrcSize(Op) * 8 - 1, _Xor(Src1, Res));
auto val = _Bfe(1, SrcSize * 8 - 1, _Xor(Src1, Res));
COND_FLAG_SET(Src2, RFLAG_OF_LOC, val);
}
}
void OpDispatchBuilder::GenerateFlags_SignShiftRight(FEXCore::X86Tables::DecodedOp Op, OrderedNode *Res, OrderedNode *Src1, OrderedNode *Src2) {
void OpDispatchBuilder::CalculcateFlags_SignShiftRight(uint8_t SrcSize, OrderedNode *Res, OrderedNode *Src1, OrderedNode *Src2) {
// CF
{
// Extract the last bit shifted in to CF
@@ -562,7 +750,7 @@ void OpDispatchBuilder::GenerateFlags_SignShiftRight(FEXCore::X86Tables::Decoded
// SF
{
auto SignBitConst = _Constant(GetSrcSize(Op) * 8 - 1);
auto SignBitConst = _Constant(SrcSize * 8 - 1);
auto LshrOp = _Lshr(Res, SignBitConst);
COND_FLAG_SET(Src2, RFLAG_SF_LOC, LshrOp);
@@ -574,14 +762,14 @@ void OpDispatchBuilder::GenerateFlags_SignShiftRight(FEXCore::X86Tables::Decoded
}
}
void OpDispatchBuilder::GenerateFlags_ShiftLeftImmediate(FEXCore::X86Tables::DecodedOp Op, OrderedNode *Res, OrderedNode *Src1, uint64_t Shift) {
void OpDispatchBuilder::CalculcateFlags_ShiftLeftImmediate(uint8_t SrcSize, OrderedNode *Res, OrderedNode *Src1, uint64_t Shift) {
// No flags changed if shift is zero
if (Shift == 0) return;
// CF
{
// Extract the last bit shifted in to CF
SetRFLAG<FEXCore::X86State::RFLAG_CF_LOC>(_Bfe(1, GetSrcSize(Op) * 8 - Shift, Src1));
SetRFLAG<FEXCore::X86State::RFLAG_CF_LOC>(_Bfe(1, SrcSize * 8 - Shift, Src1));
}
// PF
@@ -610,20 +798,20 @@ void OpDispatchBuilder::GenerateFlags_ShiftLeftImmediate(FEXCore::X86Tables::Dec
// SF
{
auto LshrOp = _Bfe(1, GetSrcSize(Op) * 8 - 1, Res);
auto LshrOp = _Bfe(1, SrcSize * 8 - 1, Res);
SetRFLAG<FEXCore::X86State::RFLAG_SF_LOC>(LshrOp);
// OF
// In the case of left shift. OF is only set from the result of <Top Source Bit> XOR <Top Result Bit>
if (Shift == 1) {
auto SourceBit = _Bfe(1, GetSrcSize(Op) * 8 - 1, Src1);
auto SourceBit = _Bfe(1, SrcSize * 8 - 1, Src1);
SetRFLAG<FEXCore::X86State::RFLAG_OF_LOC>(_Xor(SourceBit, LshrOp));
}
}
}
void OpDispatchBuilder::GenerateFlags_SignShiftRightImmediate(FEXCore::X86Tables::DecodedOp Op, OrderedNode *Res, OrderedNode *Src1, uint64_t Shift) {
void OpDispatchBuilder::CalculcateFlags_SignShiftRightImmediate(uint8_t SrcSize, OrderedNode *Res, OrderedNode *Src1, uint64_t Shift) {
// No flags changed if shift is zero
if (Shift == 0) return;
@@ -659,7 +847,7 @@ void OpDispatchBuilder::GenerateFlags_SignShiftRightImmediate(FEXCore::X86Tables
// SF
{
auto SignBitConst = _Constant(GetSrcSize(Op) * 8 - 1);
auto SignBitConst = _Constant(SrcSize * 8 - 1);
auto LshrOp = _Lshr(Res, SignBitConst);
SetRFLAG<FEXCore::X86State::RFLAG_SF_LOC>(LshrOp);
@@ -674,7 +862,7 @@ void OpDispatchBuilder::GenerateFlags_SignShiftRightImmediate(FEXCore::X86Tables
}
}
void OpDispatchBuilder::GenerateFlags_ShiftRightImmediate(FEXCore::X86Tables::DecodedOp Op, OrderedNode *Res, OrderedNode *Src1, uint64_t Shift) {
void OpDispatchBuilder::CalculcateFlags_ShiftRightImmediate(uint8_t SrcSize, OrderedNode *Res, OrderedNode *Src1, uint64_t Shift) {
// No flags changed if shift is zero
if (Shift == 0) return;
@@ -710,7 +898,7 @@ void OpDispatchBuilder::GenerateFlags_ShiftRightImmediate(FEXCore::X86Tables::De
// SF
{
auto SignBitConst = _Constant(GetSrcSize(Op) * 8 - 1);
auto SignBitConst = _Constant(SrcSize * 8 - 1);
auto LshrOp = _Lshr(Res, SignBitConst);
SetRFLAG<FEXCore::X86State::RFLAG_SF_LOC>(LshrOp);
@@ -721,13 +909,13 @@ void OpDispatchBuilder::GenerateFlags_ShiftRightImmediate(FEXCore::X86Tables::De
// Only defined when Shift is 1 else undefined
// Is set to the MSB of the original value
if (Shift == 1) {
SetRFLAG<FEXCore::X86State::RFLAG_OF_LOC>(_Bfe(1, GetSrcSize(Op) * 8 - 1, Src1));
SetRFLAG<FEXCore::X86State::RFLAG_OF_LOC>(_Bfe(1, SrcSize * 8 - 1, Src1));
}
}
}
void OpDispatchBuilder::GenerateFlags_RotateRight(FEXCore::X86Tables::DecodedOp Op, OrderedNode *Res, OrderedNode *Src1, OrderedNode *Src2) {
auto OpSize = GetSrcSize(Op) * 8;
void OpDispatchBuilder::CalculcateFlags_RotateRight(uint8_t SrcSize, OrderedNode *Res, OrderedNode *Src1, OrderedNode *Src2) {
auto OpSize = SrcSize * 8;
// Extract the last bit shifted in to CF
auto NewCF = _Bfe(1, OpSize - 1, Res);
@@ -755,8 +943,8 @@ void OpDispatchBuilder::GenerateFlags_RotateRight(FEXCore::X86Tables::DecodedOp
}
}
void OpDispatchBuilder::GenerateFlags_RotateLeft(FEXCore::X86Tables::DecodedOp Op, OrderedNode *Res, OrderedNode *Src1, OrderedNode *Src2) {
auto OpSize = GetSrcSize(Op) * 8;
void OpDispatchBuilder::CalculcateFlags_RotateLeft(uint8_t SrcSize, OrderedNode *Res, OrderedNode *Src1, OrderedNode *Src2) {
auto OpSize = SrcSize * 8;
// Extract the last bit shifted in to CF
//auto Size = _Constant(GetSrcSize(Res) * 8);
@@ -785,10 +973,10 @@ void OpDispatchBuilder::GenerateFlags_RotateLeft(FEXCore::X86Tables::DecodedOp O
}
}
void OpDispatchBuilder::GenerateFlags_RotateRightImmediate(FEXCore::X86Tables::DecodedOp Op, OrderedNode *Res, OrderedNode *Src1, uint64_t Shift) {
void OpDispatchBuilder::CalculcateFlags_RotateRightImmediate(uint8_t SrcSize, OrderedNode *Res, OrderedNode *Src1, uint64_t Shift) {
if (Shift == 0) return;
auto OpSize = GetSrcSize(Op) * 8;
auto OpSize = SrcSize * 8;
auto NewCF = _Bfe(1, OpSize - Shift, Src1);
@@ -807,10 +995,10 @@ void OpDispatchBuilder::GenerateFlags_RotateRightImmediate(FEXCore::X86Tables::D
}
}
void OpDispatchBuilder::GenerateFlags_RotateLeftImmediate(FEXCore::X86Tables::DecodedOp Op, OrderedNode *Res, OrderedNode *Src1, uint64_t Shift) {
void OpDispatchBuilder::CalculcateFlags_RotateLeftImmediate(uint8_t SrcSize, OrderedNode *Res, OrderedNode *Src1, uint64_t Shift) {
if (Shift == 0) return;
auto OpSize = GetSrcSize(Op) * 8;
auto OpSize = SrcSize * 8;
// CF
{
@@ -827,4 +1015,264 @@ void OpDispatchBuilder::GenerateFlags_RotateLeftImmediate(FEXCore::X86Tables::De
}
}
void OpDispatchBuilder::CalculcateFlags_FCMP(uint8_t SrcSize, OrderedNode *Res, OrderedNode *Src1, OrderedNode *Src2) {
OrderedNode *HostFlag_CF = _GetHostFlag(Res, FCMP_FLAG_LT);
OrderedNode *HostFlag_ZF = _GetHostFlag(Res, FCMP_FLAG_EQ);
OrderedNode *HostFlag_Unordered = _GetHostFlag(Res, FCMP_FLAG_UNORDERED);
SetRFLAG<FEXCore::X86State::RFLAG_CF_LOC>(HostFlag_CF);
SetRFLAG<FEXCore::X86State::RFLAG_ZF_LOC>(HostFlag_ZF);
SetRFLAG<FEXCore::X86State::RFLAG_PF_LOC>(HostFlag_Unordered);
auto ZeroConst = _Constant(0);
SetRFLAG<FEXCore::X86State::RFLAG_AF_LOC>(ZeroConst);
SetRFLAG<FEXCore::X86State::RFLAG_SF_LOC>(ZeroConst);
SetRFLAG<FEXCore::X86State::RFLAG_OF_LOC>(ZeroConst);
}
void OpDispatchBuilder::CalculcateFlags_BEXTR(OrderedNode *Src) {
// Handle flag setting.
//
// All that matters primarily for this instruction is
// that we only set the ZF flag properly.
//
// CF and OF are defined as being set to zero
//
SetRFLAG<X86State::RFLAG_CF_LOC>(_Constant(0));
SetRFLAG<X86State::RFLAG_OF_LOC>(_Constant(0));
// Every other flag is considered undefined after a
// BEXTR instruction, but we opt to reliably clear them.
//
SetRFLAG<X86State::RFLAG_AF_LOC>(_Constant(0));
SetRFLAG<X86State::RFLAG_SF_LOC>(_Constant(0));
// PF
if (CTX->Config.ABINoPF) {
_InvalidateFlags(1UL << X86State::RFLAG_PF_LOC);
} else {
SetRFLAG<X86State::RFLAG_PF_LOC>(_Constant(0));
}
// ZF
auto ZeroOp = _Select(IR::COND_EQ,
Src, _Constant(0),
_Constant(1), _Constant(0));
SetRFLAG<X86State::RFLAG_ZF_LOC>(ZeroOp);
}
void OpDispatchBuilder::CalculcateFlags_BLSI(uint8_t SrcSize, OrderedNode *Src) {
// Now for the flags:
//
// Only CF, SF, ZF and OF are defined as being updated
// CF is cleared if Src is zero, otherwise it's set.
// SF is set to the value of the most significant operand bit of Result.
// OF is always cleared
// ZF is set, as usual, if Result is zero or not.
//
// AF and PF are documented as being in an undefined state after
// a BLSI operation, however, we choose to reliably clear them.
auto Zero = _Constant(0);
auto One = _Constant(1);
SetRFLAG<X86State::RFLAG_OF_LOC>(Zero);
SetRFLAG<X86State::RFLAG_AF_LOC>(Zero);
if (CTX->Config.ABINoPF) {
_InvalidateFlags(1UL << X86State::RFLAG_PF_LOC);
} else {
SetRFLAG<X86State::RFLAG_PF_LOC>(Zero);
}
// ZF
{
auto ZFOp = _Select(IR::COND_EQ,
Src, Zero,
One, Zero);
SetRFLAG<X86State::RFLAG_ZF_LOC>(ZFOp);
}
// CF
{
auto CFOp = _Select(IR::COND_EQ,
Src, Zero,
Zero, One);
SetRFLAG<X86State::RFLAG_CF_LOC>(CFOp);
}
// SF
{
auto SignBit = _Constant(SrcSize * 8 - 1);
auto SFOp = _Lshr(Src, SignBit);
SetRFLAG<X86State::RFLAG_SF_LOC>(SFOp);
}
}
void OpDispatchBuilder::CalculcateFlags_BLSMSK(OrderedNode *Src) {
// Now for the flags.
auto Zero = _Constant(0);
auto One = _Constant(1);
SetRFLAG<X86State::RFLAG_ZF_LOC>(Zero);
SetRFLAG<X86State::RFLAG_OF_LOC>(Zero);
SetRFLAG<X86State::RFLAG_AF_LOC>(Zero);
if (CTX->Config.ABINoPF) {
_InvalidateFlags(1UL << X86State::RFLAG_PF_LOC);
} else {
SetRFLAG<X86State::RFLAG_PF_LOC>(Zero);
}
auto CFOp = _Select(IR::COND_EQ,
Src, Zero,
Zero, One);
SetRFLAG<X86State::RFLAG_CF_LOC>(CFOp);
}
void OpDispatchBuilder::CalculcateFlags_BLSR(uint8_t SrcSize, OrderedNode *Result, OrderedNode *Src) {
// Now for flags.
auto Zero = _Constant(0);
auto One = _Constant(1);
SetRFLAG<X86State::RFLAG_OF_LOC>(Zero);
SetRFLAG<X86State::RFLAG_AF_LOC>(Zero);
if (CTX->Config.ABINoPF) {
_InvalidateFlags(1UL << X86State::RFLAG_PF_LOC);
} else {
SetRFLAG<X86State::RFLAG_PF_LOC>(Zero);
}
// ZF
{
auto ZFOp = _Select(IR::COND_EQ,
Result, Zero,
One, Zero);
SetRFLAG<X86State::RFLAG_ZF_LOC>(ZFOp);
}
// CF
{
auto CFOp = _Select(IR::COND_EQ,
Src, Zero,
Zero, One);
SetRFLAG<X86State::RFLAG_CF_LOC>(CFOp);
}
// SF
{
auto SignBit = _Constant(SrcSize * 8 - 1);
auto SFOp = _Lshr(Result, SignBit);
SetRFLAG<X86State::RFLAG_SF_LOC>(SFOp);
}
}
void OpDispatchBuilder::CalculcateFlags_POPCOUNT(OrderedNode *Src) {
// Set ZF
auto Zero = _Constant(0);
auto ZFResult = _Select(FEXCore::IR::COND_EQ,
Src, Zero,
_Constant(1), Zero);
// Set flags
SetRFLAG<FEXCore::X86State::RFLAG_CF_LOC>(Zero);
SetRFLAG<FEXCore::X86State::RFLAG_PF_LOC>(Zero);
SetRFLAG<FEXCore::X86State::RFLAG_AF_LOC>(Zero);
SetRFLAG<FEXCore::X86State::RFLAG_ZF_LOC>(ZFResult);
SetRFLAG<FEXCore::X86State::RFLAG_SF_LOC>(Zero);
SetRFLAG<FEXCore::X86State::RFLAG_OF_LOC>(Zero);
}
void OpDispatchBuilder::CalculcateFlags_BZHI(uint8_t SrcSize, OrderedNode *Result, OrderedNode *Src) {
// Now for the flags
auto Bounds = _Constant(SrcSize * 8- 1);
auto Zero = _Constant(0);
auto One = _Constant(1);
SetRFLAG<X86State::RFLAG_OF_LOC>(Zero);
SetRFLAG<X86State::RFLAG_AF_LOC>(Zero);
if (CTX->Config.ABINoPF) {
_InvalidateFlags(1UL << X86State::RFLAG_PF_LOC);
} else {
SetRFLAG<X86State::RFLAG_PF_LOC>(Zero);
}
// ZF
{
auto ZFOp = _Select(IR::COND_EQ,
Result, Zero,
One, Zero);
SetRFLAG<X86State::RFLAG_ZF_LOC>(ZFOp);
}
// CF
{
auto CFOp = _Select(IR::COND_UGT,
Src, Bounds,
One, Zero);
SetRFLAG<X86State::RFLAG_CF_LOC>(CFOp);
}
// SF
{
auto SFOp = _Lshr(Result, Bounds);
SetRFLAG<X86State::RFLAG_SF_LOC>(SFOp);
}
}
void OpDispatchBuilder::CalculcateFlags_TZCNT(OrderedNode *Src) {
// OF, SF, AF, PF all undefined
auto Zero = _Constant(0);
auto ZFResult = _Select(FEXCore::IR::COND_EQ,
Src, Zero,
_Constant(1), Zero);
// Set flags
SetRFLAG<FEXCore::X86State::RFLAG_CF_LOC>(ZFResult);
SetRFLAG<FEXCore::X86State::RFLAG_ZF_LOC>(_Bfe(1, 0, Src));
}
void OpDispatchBuilder::CalculcateFlags_LZCNT(uint8_t SrcSize, OrderedNode *Src) {
// OF, SF, AF, PF all undefined
auto Zero = _Constant(0);
auto ZFResult = _Select(FEXCore::IR::COND_EQ,
Src, Zero,
_Constant(1), Zero);
// Set flags
SetRFLAG<FEXCore::X86State::RFLAG_CF_LOC>(ZFResult);
SetRFLAG<FEXCore::X86State::RFLAG_ZF_LOC>(_Bfe(1, SrcSize * 8 - 1, Src));
}
void OpDispatchBuilder::CalculcateFlags_BITSELECT(OrderedNode *Src) {
// OF, SF, AF, PF, CF all undefined
auto ZeroConst = _Constant(0);
auto OneConst = _Constant(1);
// ZF is set to 1 if the source was zero
auto ZFSelectOp = _Select(FEXCore::IR::COND_EQ,
Src, ZeroConst,
OneConst, ZeroConst);
SetRFLAG<FEXCore::X86State::RFLAG_ZF_LOC>(ZFSelectOp);
}
void OpDispatchBuilder::CalculcateFlags_RDRAND(OrderedNode *Src) {
// OF, SF, ZF, AF, PF all zero
// CF is set to the incoming source
auto ZeroConst = _Constant(0);
SetRFLAG<X86State::RFLAG_OF_LOC>(ZeroConst);
SetRFLAG<X86State::RFLAG_SF_LOC>(ZeroConst);
SetRFLAG<X86State::RFLAG_ZF_LOC>(ZeroConst);
SetRFLAG<X86State::RFLAG_AF_LOC>(ZeroConst);
SetRFLAG<X86State::RFLAG_PF_LOC>(ZeroConst);
SetRFLAG<FEXCore::X86State::RFLAG_CF_LOC>(Src);
}
}
@@ -241,6 +241,8 @@ void OpDispatchBuilder::VectorALUOp<IR::OP_VCMPGT, 2>(OpcodeArgs);
template
void OpDispatchBuilder::VectorALUOp<IR::OP_VCMPGT, 4>(OpcodeArgs);
template
void OpDispatchBuilder::VectorALUOp<IR::OP_VCMPGT, 8>(OpcodeArgs);
template
void OpDispatchBuilder::VectorALUOp<IR::OP_VCMPEQ, 1>(OpcodeArgs);
template
void OpDispatchBuilder::VectorALUOp<IR::OP_VCMPEQ, 2>(OpcodeArgs);
@@ -295,6 +297,24 @@ void OpDispatchBuilder::VectorALUOp<IR::OP_VUQSUB, 1>(OpcodeArgs);
template
void OpDispatchBuilder::VectorALUOp<IR::OP_VUQSUB, 2>(OpcodeArgs);
template<FEXCore::IR::IROps IROp, size_t ElementSize>
void OpDispatchBuilder::VectorALUROp(OpcodeArgs) {
auto Size = GetSrcSize(Op);
OrderedNode *Src = LoadSource(FPRClass, Op, Op->Src[0], Op->Flags, -1);
OrderedNode *Dest = LoadSource(FPRClass, Op, Op->Dest, Op->Flags, -1);
auto ALUOp = _VAdd(Size, ElementSize, Src, Dest);
// Overwrite our IR's op type
ALUOp.first->Header.Op = IROp;
StoreResult(FPRClass, Op, ALUOp, -1);
}
template
void OpDispatchBuilder::VectorALUROp<IR::OP_VFSUB, 4>(OpcodeArgs);
template
void OpDispatchBuilder::VectorALUROp<IR::OP_VFSUB, 8>(OpcodeArgs);
template<FEXCore::IR::IROps IROp, size_t ElementSize>
void OpDispatchBuilder::VectorScalarALUOp(OpcodeArgs) {
auto Size = GetSrcSize(Op);
@@ -390,14 +410,35 @@ void OpDispatchBuilder::VectorUnaryOp<IR::OP_VABS, 2, false>(OpcodeArgs);
template
void OpDispatchBuilder::VectorUnaryOp<IR::OP_VABS, 4, false>(OpcodeArgs);
template<FEXCore::IR::IROps IROp, size_t ElementSize>
void OpDispatchBuilder::VectorUnaryDuplicateOp(OpcodeArgs) {
auto Size = GetSrcSize(Op);
OrderedNode *Src = LoadSource(FPRClass, Op, Op->Src[0], Op->Flags, -1);
auto ALUOp = _VFSqrt(ElementSize, ElementSize, Src);
// Overwrite our IR's op type
ALUOp.first->Header.Op = IROp;
// Duplicate the lower bits
auto Result = _VDupElement(Size, ElementSize, ALUOp, 0);
StoreResult(FPRClass, Op, Result, -1);
}
template
void OpDispatchBuilder::VectorUnaryDuplicateOp<IR::OP_VFRSQRT, 4>(OpcodeArgs);
template
void OpDispatchBuilder::VectorUnaryDuplicateOp<IR::OP_VFRECP, 4>(OpcodeArgs);
void OpDispatchBuilder::MOVQOp(OpcodeArgs) {
OrderedNode *Src = LoadSource(FPRClass, Op, Op->Src[0], Op->Flags, -1);
// This instruction is a bit special that if the destination is a register then it'll ZEXT the 64bit source to 128bit
if (Op->Dest.IsGPR()) {
const auto gpr = Op->Dest.Data.GPR.GPR;
_StoreContext(FPRClass, 8, offsetof(FEXCore::Core::CPUState, xmm[gpr - FEXCore::X86State::REG_XMM_0][0]), Src);
_StoreContext(8, FPRClass, Src, offsetof(FEXCore::Core::CPUState, xmm[gpr - FEXCore::X86State::REG_XMM_0][0]));
auto Const = _Constant(0);
_StoreContext(GPRClass, 8, offsetof(FEXCore::Core::CPUState, xmm[gpr - FEXCore::X86State::REG_XMM_0][1]), Const);
_StoreContext(8, GPRClass, Const, offsetof(FEXCore::Core::CPUState, xmm[gpr - FEXCore::X86State::REG_XMM_0][1]));
}
else {
// This is simple, just store the result
@@ -438,14 +479,15 @@ void OpDispatchBuilder::MOVMSKOpOne(OpcodeArgs) {
//TODO: We could remove this VCastFromGOR + VInsGPR pair if we had a VDUPFromGPR instruction that maps directly to AArch64.
auto M = _Constant(0x80'40'20'10'08'04'02'01ULL);
OrderedNode *VMask = _VCastFromGPR(16, 8, M);
VMask = _VInsGPR(16, 8, VMask, M, 1);
auto VCMP = _VCMPLTZ(Src, 16, 1);
auto VAnd = _VAnd(VCMP, VMask, 16, 1);
VMask = _VInsGPR(16, 8, 1, VMask, M);
auto VAdd1 = _VAddP(VAnd, VAnd, 16, 1);
auto VAdd2 = _VAddP(VAdd1, VAdd1, 8, 1);
auto VAdd3 = _VAddP(VAdd2, VAdd2, 8, 1);
auto VCMP = _VCMPLTZ(16, 1, Src);
auto VAnd = _VAnd(16, 1, VCMP, VMask);
auto VAdd1 = _VAddP(16, 1, VAnd, VAnd);
auto VAdd2 = _VAddP(8, 1, VAdd1, VAdd1);
auto VAdd3 = _VAddP(8, 1, VAdd2, VAdd2);
StoreResult(GPRClass, Op, _VExtractToGPR(16, 2, VAdd3, 0), -1);
}
@@ -502,11 +544,11 @@ void OpDispatchBuilder::PSHUFBOp(OpcodeArgs) {
// Bits [6:4] is reserved for 128bit
// Bits [6:3] is reserved for 64bit
if (Size == 8) {
auto MaskVector = _VectorImm(0b1000'0111, Size, 1);
auto MaskVector = _VectorImm(Size, 1, 0b1000'0111);
Src = _VAnd(Size, Size, Src, MaskVector);
}
else {
auto MaskVector = _VectorImm(0b1000'1111, Size, 1);
auto MaskVector = _VectorImm(Size, 1, 0b1000'1111);
Src = _VAnd(Size, Size, Src, MaskVector);
}
auto Res = _VTBL1(Size, Dest, Src);
@@ -627,7 +669,7 @@ void OpDispatchBuilder::PINSROp(OpcodeArgs) {
Index &= NumElements - 1;
// This maps 1:1 to an AArch64 NEON Op
auto ALUOp = _VInsGPR(Size, ElementSize, Dest, Src, Index);
auto ALUOp = _VInsGPR(Size, ElementSize, Index, Dest, Src);
StoreResult(FPRClass, Op, ALUOp, -1);
}
@@ -670,10 +712,10 @@ void OpDispatchBuilder::InsertPSOp(OpcodeArgs) {
// ZMask happens after insert
if (ZMask == 0xF) {
Dest = _VectorImm(0, 16, 4);
Dest = _VectorImm(16, 4, 0);
}
else if (ZMask) {
auto Zero = _VectorImm(0, 16, 4);
auto Zero = _VectorImm(16, 4, 0);
for (size_t i = 0; i < 4; ++i) {
if (ZMask & (1 << i)) {
Dest = _VInsElement(GetDstSize(Op), 4, i, 0, Dest, Zero);
@@ -766,7 +808,7 @@ void OpDispatchBuilder::PSRLDOp(OpcodeArgs) {
OrderedNode *Result{};
// Incoming element size for the shift source is always 8
auto MaxShift = _VectorImm(ElementSize * 8, 8, 8);
auto MaxShift = _VectorImm(8, 8, ElementSize * 8);
Src = _VUMin(8, 8, MaxShift, Src);
Result = _VUShrS(Size, ElementSize, Dest, Src);
@@ -830,7 +872,7 @@ void OpDispatchBuilder::PSLL(OpcodeArgs) {
OrderedNode *Result{};
// Incoming element size for the shift source is always 8
auto MaxShift = _VectorImm(ElementSize * 8, 8, 8);
auto MaxShift = _VectorImm(8, 8, ElementSize * 8);
Src = _VUMin(8, 8, MaxShift, Src);
Result = _VUShlS(Size, ElementSize, Dest, Src);
@@ -854,7 +896,7 @@ void OpDispatchBuilder::PSRAOp(OpcodeArgs) {
OrderedNode *Result{};
// Incoming element size for the shift source is always 8
auto MaxShift = _VectorImm(ElementSize * 8, 8, 8);
auto MaxShift = _VectorImm(8, 8, ElementSize * 8);
Src = _VUMin(8, 8, MaxShift, Src);
Result = _VSShrS(Size, ElementSize, Dest, Src);
@@ -957,12 +999,11 @@ void OpDispatchBuilder::CVTFPR_To_GPR(OpcodeArgs) {
// Source Element size is determined by instruction
size_t GPRSize = GetDstSize(Op);
size_t ElementSize = SrcElementSize;
if constexpr (HostRoundingMode) {
Src = _Float_ToGPR_S(Src, ElementSize, GPRSize);
Src = _Float_ToGPR_S(GPRSize, SrcElementSize, Src);
}
else {
Src = _Float_ToGPR_ZS(Src, ElementSize, GPRSize);
Src = _Float_ToGPR_ZS(GPRSize, SrcElementSize, Src);
}
StoreResult_WithOpSize(GPRClass, Op, Op->Dest, Src, GPRSize, -1);
@@ -985,11 +1026,11 @@ void OpDispatchBuilder::Vector_CVT_Int_To_Float(OpcodeArgs) {
size_t ElementSize = SrcElementSize;
size_t Size = GetDstSize(Op);
if constexpr (Widen) {
Src = _VSXTL(Src, Size, ElementSize);
Src = _VSXTL(Size, ElementSize, Src);
ElementSize <<= 1;
}
Src = _Vector_SToF(Src, Size, ElementSize);
Src = _Vector_SToF(Size, ElementSize, Src);
StoreResult(FPRClass, Op, Src, -1);
}
@@ -1007,15 +1048,15 @@ void OpDispatchBuilder::Vector_CVT_Float_To_Int(OpcodeArgs) {
size_t Size = GetDstSize(Op);
if constexpr (Narrow) {
Src = _Vector_FToF(Size, SrcElementSize >> 1, SrcElementSize, Src);
Src = _Vector_FToF(Size, SrcElementSize >> 1, Src, SrcElementSize);
ElementSize >>= 1;
}
if constexpr (HostRoundingMode) {
Src = _Vector_FToS(Src, Size, ElementSize);
Src = _Vector_FToS(Size, ElementSize, Src);
}
else {
Src = _Vector_FToZS(Src, Size, ElementSize);
Src = _Vector_FToZS(Size, ElementSize, Src);
}
StoreResult_WithOpSize(FPRClass, Op, Op->Dest, Src, Size, -1);
@@ -1025,6 +1066,8 @@ template
void OpDispatchBuilder::Vector_CVT_Float_To_Int<4, false, false>(OpcodeArgs);
template
void OpDispatchBuilder::Vector_CVT_Float_To_Int<4, false, true>(OpcodeArgs);
template
void OpDispatchBuilder::Vector_CVT_Float_To_Int<4, true, false>(OpcodeArgs);
template
void OpDispatchBuilder::Vector_CVT_Float_To_Int<8, true, true>(OpcodeArgs);
@@ -1053,10 +1096,10 @@ void OpDispatchBuilder::Vector_CVT_Float_To_Float(OpcodeArgs) {
size_t Size = GetDstSize(Op);
if constexpr (DstElementSize > SrcElementSize) {
Src = _Vector_FToF(Size, SrcElementSize << 1, SrcElementSize, Src);
Src = _Vector_FToF(Size, SrcElementSize << 1, Src, SrcElementSize);
}
else {
Src = _Vector_FToF(Size, SrcElementSize >> 1, SrcElementSize, Src);
Src = _Vector_FToF(Size, SrcElementSize >> 1, Src, SrcElementSize);
}
StoreResult(FPRClass, Op, Src, -1);
@@ -1074,12 +1117,12 @@ void OpDispatchBuilder::MMX_To_XMM_Vector_CVT_Int_To_Float(OpcodeArgs) {
size_t ElementSize = SrcElementSize;
size_t DstSize = GetDstSize(Op);
if constexpr (Widen) {
Src = _VSXTL(Src, DstSize, ElementSize);
Src = _VSXTL(DstSize, ElementSize, Src);
ElementSize <<= 1;
}
// Always signed
Src = _Vector_SToF(Src, DstSize, ElementSize);
Src = _Vector_SToF(DstSize, ElementSize, Src);
OrderedNode *Dest{};
if constexpr (Widen) {
@@ -1107,14 +1150,14 @@ void OpDispatchBuilder::XMM_To_MMX_Vector_CVT_Float_To_Int(OpcodeArgs) {
size_t Size = GetDstSize(Op);
// Always narrows
Src = _Vector_FToF(Size, SrcElementSize >> 1, SrcElementSize, Src);
Src = _Vector_FToF(Size, SrcElementSize >> 1, Src, SrcElementSize);
ElementSize >>= 1;
if constexpr (HostRoundingMode) {
Src = _Vector_FToS(Src, Size, ElementSize);
Src = _Vector_FToS(Size, ElementSize, Src);
}
else {
Src = _Vector_FToZS(Src, Size, ElementSize);
Src = _Vector_FToZS(Size, ElementSize, Src);
}
StoreResult_WithOpSize(FPRClass, Op, Op->Dest, Src, Size, -1);
@@ -1133,7 +1176,7 @@ void OpDispatchBuilder::MASKMOVOp(OpcodeArgs) {
OrderedNode *Src = LoadSource(GPRClass, Op, Op->Src[0], Op->Flags, -1);
OrderedNode *Dest = LoadSource(GPRClass, Op, Op->Dest, Op->Flags, -1);
OrderedNode *MemDest = _LoadContext(GPRSize, GPROffset(X86State::REG_RDI), GPRClass);
OrderedNode *MemDest = _LoadContext(GPRSize, GPRClass, GPROffset(X86State::REG_RDI));
const size_t NumElements = Size / 64;
for (size_t Element = 0; Element < NumElements; ++Element) {
@@ -1187,13 +1230,6 @@ void OpDispatchBuilder::VFCMPOp(OpcodeArgs) {
auto Size = GetSrcSize(Op);
OrderedNode *Src = LoadSource(FPRClass, Op, Op->Src[0], Op->Flags, -1);
OrderedNode *Dest = LoadSource_WithOpSize(FPRClass, Op, Op->Dest, GetDstSize(Op), Op->Flags, -1);
OrderedNode *Src2{};
if constexpr (Scalar) {
Src2 = _VExtractElement(GetDstSize(Op), Size, Dest, 0);
}
else {
Src2 = Dest;
}
uint8_t CompType = Op->Src[1].Data.Literal.Value;
OrderedNode *Result{};
@@ -1201,30 +1237,30 @@ void OpDispatchBuilder::VFCMPOp(OpcodeArgs) {
//auto ALUOp = _VCMPGT(Size, ElementSize, Dest, Src);
switch (CompType) {
case 0x00: case 0x08: case 0x10: case 0x18: // EQ
Result = _VFCMPEQ(Size, ElementSize, Src2, Src);
Result = _VFCMPEQ(Size, ElementSize, Dest, Src);
break;
case 0x01: case 0x09: case 0x11: case 0x19: // LT, GT(Swapped operand)
Result = _VFCMPLT(Size, ElementSize, Src2, Src);
Result = _VFCMPLT(Size, ElementSize, Dest, Src);
break;
case 0x02: case 0x0A: case 0x12: case 0x1A: // LE, GE(Swapped operand)
Result = _VFCMPLE(Size, ElementSize, Src2, Src);
Result = _VFCMPLE(Size, ElementSize, Dest, Src);
break;
case 0x03: case 0x0B: case 0x13: case 0x1B: // Unordered
Result = _VFCMPUNO(Size, ElementSize, Src2, Src);
Result = _VFCMPUNO(Size, ElementSize, Dest, Src);
break;
case 0x04: case 0x0C: case 0x14: case 0x1C: // NEQ
Result = _VFCMPNEQ(Size, ElementSize, Src2, Src);
Result = _VFCMPNEQ(Size, ElementSize, Dest, Src);
break;
case 0x05: case 0x0D: case 0x15: case 0x1D: // NLT, NGT(Swapped operand)
Result = _VFCMPLT(Size, ElementSize, Src2, Src);
Result = _VFCMPLT(Size, ElementSize, Dest, Src);
Result = _VNot(Size, ElementSize, Result);
break;
case 0x06: case 0x0E: case 0x16: case 0x1E: // NLE, NGE(Swapped operand)
Result = _VFCMPLE(Size, ElementSize, Src2, Src);
Result = _VFCMPLE(Size, ElementSize, Dest, Src);
Result = _VNot(Size, ElementSize, Result);
break;
case 0x07: case 0x0F: case 0x17: case 0x1F: // Ordered
Result = _VFCMPORD(Size, ElementSize, Src2, Src);
Result = _VFCMPORD(Size, ElementSize, Dest, Src);
break;
default:
LOGMAN_MSG_A_FMT("Unknown Comparison type: {}", CompType);
@@ -1268,7 +1304,7 @@ void OpDispatchBuilder::FXSaveOp(OpcodeArgs) {
}
{
auto FCW = _LoadContext(2, offsetof(FEXCore::Core::CPUState, FCW), GPRClass);
auto FCW = _LoadContext(2, GPRClass, offsetof(FEXCore::Core::CPUState, FCW));
_StoreMem(GPRClass, 2, Mem, FCW, 2);
}
@@ -1294,7 +1330,7 @@ void OpDispatchBuilder::FXSaveOp(OpcodeArgs) {
{
// FTW
OrderedNode *MemLocation = _Add(Mem, _Constant(4));
auto FTW = _LoadContext(2, offsetof(FEXCore::Core::CPUState, FTW), GPRClass);
auto FTW = _LoadContext(2, GPRClass, offsetof(FEXCore::Core::CPUState, FTW));
_StoreMem(GPRClass, 2, MemLocation, FTW, 2);
}
@@ -1343,7 +1379,7 @@ void OpDispatchBuilder::FXSaveOp(OpcodeArgs) {
// If OSFXSR bit in CR4 is not set than FXSAVE /may/ not save the XMM registers
// This is implementation dependent
for (unsigned i = 0; i < 8; ++i) {
OrderedNode *MMReg = _LoadContext(16, offsetof(FEXCore::Core::CPUState, mm[i]), FPRClass);
OrderedNode *MMReg = _LoadContext(16, FPRClass, offsetof(FEXCore::Core::CPUState, mm[i]));
OrderedNode *MemLocation = _Add(Mem, _Constant(i * 16 + 32));
_StoreMem(FPRClass, 16, MemLocation, MMReg, 16);
@@ -1351,7 +1387,7 @@ void OpDispatchBuilder::FXSaveOp(OpcodeArgs) {
unsigned NumRegs = CTX->Config.Is64BitMode ? 16 : 8;
for (unsigned i = 0; i < NumRegs; ++i) {
OrderedNode *XMMReg = _LoadContext(16, offsetof(FEXCore::Core::CPUState, xmm[i]), FPRClass);
OrderedNode *XMMReg = _LoadContext(16, FPRClass, offsetof(FEXCore::Core::CPUState, xmm[i]));
OrderedNode *MemLocation = _Add(Mem, _Constant(i * 16 + 160));
_StoreMem(FPRClass, 16, MemLocation, XMMReg, 16);
@@ -1364,7 +1400,7 @@ void OpDispatchBuilder::FXRStoreOp(OpcodeArgs) {
auto NewFCW = _LoadMem(GPRClass, 2, Mem, 2);
_F80LoadFCW(NewFCW);
_StoreContext(GPRClass, 2, offsetof(FEXCore::Core::CPUState, FCW), NewFCW);
_StoreContext(2, GPRClass, NewFCW, offsetof(FEXCore::Core::CPUState, FCW));
{
OrderedNode *MemLocation = _Add(Mem, _Constant(2));
@@ -1389,20 +1425,20 @@ void OpDispatchBuilder::FXRStoreOp(OpcodeArgs) {
// FTW
OrderedNode *MemLocation = _Add(Mem, _Constant(4));
auto NewFTW = _LoadMem(GPRClass, 2, MemLocation, 2);
_StoreContext(GPRClass, 2, offsetof(FEXCore::Core::CPUState, FTW), NewFTW);
_StoreContext(2, GPRClass, NewFTW, offsetof(FEXCore::Core::CPUState, FTW));
}
for (unsigned i = 0; i < 8; ++i) {
OrderedNode *MemLocation = _Add(Mem, _Constant(i * 16 + 32));
auto MMReg = _LoadMem(FPRClass, 16, MemLocation, 16);
_StoreContext(FPRClass, 16, offsetof(FEXCore::Core::CPUState, mm[i]), MMReg);
_StoreContext(16, FPRClass, MMReg, offsetof(FEXCore::Core::CPUState, mm[i]));
}
unsigned NumRegs = CTX->Config.Is64BitMode ? 16 : 8;
for (unsigned i = 0; i < NumRegs; ++i) {
OrderedNode *MemLocation = _Add(Mem, _Constant(i * 16 + 160));
auto XMMReg = _LoadMem(FPRClass, 16, MemLocation, 16);
_StoreContext(FPRClass, 16, offsetof(FEXCore::Core::CPUState, xmm[i]), XMMReg);
_StoreContext(16, FPRClass, XMMReg, offsetof(FEXCore::Core::CPUState, xmm[i]));
}
}
@@ -1427,23 +1463,12 @@ template<size_t ElementSize>
void OpDispatchBuilder::UCOMISxOp(OpcodeArgs) {
OrderedNode *Src1 = LoadSource(FPRClass, Op, Op->Dest, Op->Flags, -1);
OrderedNode *Src2 = LoadSource(FPRClass, Op, Op->Src[0], Op->Flags, -1);
OrderedNode *Res = _FCmp(Src1, Src2, ElementSize,
OrderedNode *Res = _FCmp(ElementSize, Src1, Src2,
(1 << FCMP_FLAG_EQ) |
(1 << FCMP_FLAG_LT) |
(1 << FCMP_FLAG_UNORDERED));
OrderedNode *HostFlag_CF = _GetHostFlag(Res, FCMP_FLAG_LT);
OrderedNode *HostFlag_ZF = _GetHostFlag(Res, FCMP_FLAG_EQ);
OrderedNode *HostFlag_Unordered = _GetHostFlag(Res, FCMP_FLAG_UNORDERED);
SetRFLAG<FEXCore::X86State::RFLAG_CF_LOC>(HostFlag_CF);
SetRFLAG<FEXCore::X86State::RFLAG_ZF_LOC>(HostFlag_ZF);
SetRFLAG<FEXCore::X86State::RFLAG_PF_LOC>(HostFlag_Unordered);
auto ZeroConst = _Constant(0);
SetRFLAG<FEXCore::X86State::RFLAG_AF_LOC>(ZeroConst);
SetRFLAG<FEXCore::X86State::RFLAG_SF_LOC>(ZeroConst);
SetRFLAG<FEXCore::X86State::RFLAG_OF_LOC>(ZeroConst);
GenerateFlags_FCMP(Op, Res, Src1, Src2);
flagsOp = SelectionFlag::FCMP;
flagsOpDest = Src1;
@@ -1559,8 +1584,8 @@ void OpDispatchBuilder::MOVQ2DQ(OpcodeArgs) {
// This instruction is a bit special in that if the source is MMX then it zexts to 128bit
if constexpr (ToXMM) {
Src = _VMov(Src, 16);
_StoreContext(FPRClass, 16, offsetof(FEXCore::Core::CPUState, xmm[Op->Dest.Data.GPR.GPR - FEXCore::X86State::REG_XMM_0][0]), Src);
Src = _VMov(16, Src);
_StoreContext(16, FPRClass, Src, offsetof(FEXCore::Core::CPUState, xmm[Op->Dest.Data.GPR.GPR - FEXCore::X86State::REG_XMM_0][0]));
}
else {
// This is simple, just store the result
@@ -1649,11 +1674,151 @@ void OpDispatchBuilder::ADDSUBPOp(OpcodeArgs) {
StoreResult(FPRClass, Op, ResAdd, -1);
}
void OpDispatchBuilder::PFNACCOp(OpcodeArgs) {
auto Size = GetSrcSize(Op);
OrderedNode *Dest = LoadSource(FPRClass, Op, Op->Dest, Op->Flags, -1);
OrderedNode *Src = LoadSource(FPRClass, Op, Op->Src[0], Op->Flags, -1);
OrderedNode *ResSubSrc{};
OrderedNode *ResSubDest{};
auto UpperSubDest = _VExtractElement(Size, 4, Dest, 1);
auto UpperSubSrc = _VExtractElement(Size, 4, Src, 1);
ResSubDest = _VFSub(4, 4, Dest, UpperSubDest);
ResSubSrc = _VFSub(4, 4, Src, UpperSubSrc);
auto Result = _VInsElement(8, 4, 1, 0, ResSubDest, ResSubSrc);
StoreResult(FPRClass, Op, Result, -1);
}
void OpDispatchBuilder::PFPNACCOp(OpcodeArgs) {
auto Size = GetSrcSize(Op);
OrderedNode *Dest = LoadSource(FPRClass, Op, Op->Dest, Op->Flags, -1);
OrderedNode *Src = LoadSource(FPRClass, Op, Op->Src[0], Op->Flags, -1);
OrderedNode *ResAdd{};
OrderedNode *ResSub{};
auto UpperSubDest = _VExtractElement(Size, 4, Dest, 1);
ResSub = _VFSub(4, 4, Dest, UpperSubDest);
ResAdd = _VFAddP(Size, 4, Src, Src);
auto Result = _VInsElement(8, 4, 1, 0, ResSub, ResAdd);
StoreResult(FPRClass, Op, Result, -1);
}
void OpDispatchBuilder::PSWAPDOp(OpcodeArgs) {
auto Size = GetSrcSize(Op);
OrderedNode *Src = LoadSource(FPRClass, Op, Op->Src[0], Op->Flags, -1);
auto Result = _VRev64(Size, 4, Src);
StoreResult(FPRClass, Op, Result, -1);
}
template
void OpDispatchBuilder::ADDSUBPOp<4>(OpcodeArgs);
template
void OpDispatchBuilder::ADDSUBPOp<8>(OpcodeArgs);
void OpDispatchBuilder::PI2FWOp(OpcodeArgs) {
OrderedNode *Src = LoadSource(FPRClass, Op, Op->Src[0], Op->Flags, -1);
size_t Size = GetDstSize(Op);
// We now need to transpose the lower 16-bits of each element together
// Only needing to move the upper element down in this case
Src = _VInsElement(Size, 2, 1, 2, Src, Src);
// Now we need to sign extend the 16bit value to 32-bit
Src = _VSXTL(Size, 2, Src);
// int32_t to float
Src = _Vector_SToF(Size, 4, Src);
StoreResult_WithOpSize(FPRClass, Op, Op->Dest, Src, Size, -1);
}
void OpDispatchBuilder::PF2IWOp(OpcodeArgs) {
OrderedNode *Src = LoadSource(FPRClass, Op, Op->Src[0], Op->Flags, -1);
size_t Size = GetDstSize(Op);
// Float to int32_t
Src = _Vector_FToZS(Size, 4, Src);
// We now need to transpose the lower 16-bits of each element together
// Only needing to move the upper element down in this case
Src = _VInsElement(Size, 2, 1, 2, Src, Src);
// Now we need to sign extend the 16bit value to 32-bit
Src = _VSXTL(Size, 2, Src);
StoreResult_WithOpSize(FPRClass, Op, Op->Dest, Src, Size, -1);
}
void OpDispatchBuilder::PMULHRWOp(OpcodeArgs) {
auto Size = GetSrcSize(Op);
OrderedNode *Dest = LoadSource(FPRClass, Op, Op->Dest, Op->Flags, -1);
OrderedNode *Src = LoadSource(FPRClass, Op, Op->Src[0], Op->Flags, -1);
OrderedNode *Res{};
// Implementation is more efficient for 8byte registers
// Multiplies 4 16bit values in to 4 32bit values
Res = _VSMull(Size * 2, 2, Dest, Src);
//TODO: We could remove this VCastFromGOR + VInsGPR pair if we had a VDUPFromGPR instruction that maps directly to AArch64.
auto M = _Constant(0x0000'8000'0000'8000ULL);
OrderedNode *VConstant = _VCastFromGPR(16, 8, M);
VConstant = _VInsGPR(16, 8, 1, VConstant, M);
Res = _VAdd(Size * 2, 4, Res, VConstant);
// Now shift and narrow to convert 32-bit values to 16bit, storing the top 16bits
Res = _VUShrNI(Size * 2, 4, Res, 16);
StoreResult(FPRClass, Op, Res, -1);
}
template<uint8_t CompType>
void OpDispatchBuilder::VPFCMPOp(OpcodeArgs) {
auto Size = GetSrcSize(Op);
OrderedNode *Src = LoadSource(FPRClass, Op, Op->Src[0], Op->Flags, -1);
OrderedNode *Dest = LoadSource_WithOpSize(FPRClass, Op, Op->Dest, GetDstSize(Op), Op->Flags, -1);
OrderedNode *Result{};
// This maps 1:1 to an AArch64 NEON Op
//auto ALUOp = _VCMPGT(Size, 4, Dest, Src);
LogMan::Msg::DFmt("CompType: {} Size: {}", CompType, Size);
switch (CompType) {
case 0x00: // EQ
Result = _VFCMPEQ(Size, 4, Dest, Src);
break;
case 0x01: // GE(Swapped operand)
Result = _VFCMPLE(Size, 4, Src, Dest);
break;
case 0x02: // GT
Result = _VFCMPGT(Size, 4, Dest, Src);
break;
default:
LOGMAN_MSG_A_FMT("Unknown Comparison type: {}", CompType);
break;
}
StoreResult(FPRClass, Op, Result, -1);
ShouldDump = true;
}
template
void OpDispatchBuilder::VPFCMPOp<0>(OpcodeArgs);
template
void OpDispatchBuilder::VPFCMPOp<1>(OpcodeArgs);
template
void OpDispatchBuilder::VPFCMPOp<2>(OpcodeArgs);
void OpDispatchBuilder::PMADDWD(OpcodeArgs) {
// This is a pretty curious operation
// Does two MADD operations across 4 16bit signed integers and accumulates to 32bit integers in the destination
@@ -1806,7 +1971,7 @@ void OpDispatchBuilder::PMULHRSW(OpcodeArgs) {
// Implementation is more efficient for 8byte registers
Res = _VSMull(Size * 2, 2, Dest, Src);
Res = _VSShrI(Size * 2, 4, Res, 14);
auto OneVector = _VectorImm(1, Size * 2, 4);
auto OneVector = _VectorImm(Size * 2, 4, 1);
Res = _VAdd(Size * 2, 4, Res, OneVector);
Res = _VUShrNI(Size * 2, 4, Res, 1);
}
@@ -1820,7 +1985,7 @@ void OpDispatchBuilder::PMULHRSW(OpcodeArgs) {
ResultLow = _VSShrI(Size, 4, ResultLow, 14);
ResultHigh = _VSShrI(Size, 4, ResultHigh, 14);
auto OneVector = _VectorImm(1, Size, 4);
auto OneVector = _VectorImm(Size, 4, 1);
ResultLow = _VAdd(Size, 4, ResultLow, OneVector);
ResultHigh = _VAdd(Size, 4, ResultHigh, OneVector);
@@ -2075,10 +2240,10 @@ void OpDispatchBuilder::ExtendVectorElements(OpcodeArgs) {
CurrentElementSize != DstElementSize;
CurrentElementSize <<= 1) {
if constexpr (Signed) {
Result = _VSXTL(Result, Size, CurrentElementSize);
Result = _VSXTL(Size, CurrentElementSize, Result);
}
else {
Result = _VUXTL(Result, Size, CurrentElementSize);
Result = _VUXTL(Size, CurrentElementSize, Result);
}
}
StoreResult(FPRClass, Op, Result, -1);
@@ -2132,7 +2297,7 @@ void OpDispatchBuilder::VectorRound(OpcodeArgs) {
FEXCore::IR::Round_Host,
};
Src = _Vector_FToI(Src, SourceModes[(RoundControlSource << 2) | RoundControl], Size, ElementSize);
Src = _Vector_FToI(Size, ElementSize, Src, SourceModes[(RoundControlSource << 2) | RoundControl]);
if constexpr (Scalar) {
// Insert the lower bits
@@ -2186,13 +2351,14 @@ void OpDispatchBuilder::VectorVariableBlend(OpcodeArgs) {
OrderedNode *Src = LoadSource(FPRClass, Op, Op->Src[0], Op->Flags, -1);
// The mask is hardcoded to be xmm0 in this instruction
OrderedNode *Mask = _LoadContext(16, offsetof(FEXCore::Core::CPUState, xmm[0]), FPRClass);
OrderedNode *Mask = _LoadContext(16, FPRClass, offsetof(FEXCore::Core::CPUState, xmm[0]));
// Each element is selected by the high bit of that element size
// Dest[ElementIdx] = Xmm0[ElementIndex][HighBit] ? Src : Dest;
//
// To emulate this on AArch64
// Arithmetic shift right by the element size, then use BSL to select the registers
Mask = _VSShrI(Size, ElementSize, Mask, (ElementSize * 8) - 1);
auto Result = _VBSL(Mask, Src, Dest);
StoreResult(FPRClass, Op, Result, -1);
@@ -2205,13 +2371,16 @@ template
void OpDispatchBuilder::VectorVariableBlend<8>(OpcodeArgs);
void OpDispatchBuilder::PTestOp(OpcodeArgs) {
// Invalidate deferred flags early
InvalidateDeferredFlags();
auto Size = GetSrcSize(Op);
OrderedNode *Dest = LoadSource(FPRClass, Op, Op->Dest, Op->Flags, -1);
OrderedNode *Src = LoadSource(FPRClass, Op, Op->Src[0], Op->Flags, -1);
OrderedNode *Test1 = _VAnd(Dest, Src, Size, 1);
OrderedNode *Test2 = _VBic(Src, Dest, Size, 1);
OrderedNode *Test1 = _VAnd(Size, 1, Dest, Src);
OrderedNode *Test2 = _VBic(Size, 1, Src, Dest);
Test1 = _VPopcount(Size, 1, Test1);
Test2 = _VPopcount(Size, 1, Test2);
@@ -2233,8 +2402,14 @@ void OpDispatchBuilder::PTestOp(OpcodeArgs) {
Test2 = _Select(FEXCore::IR::COND_EQ,
Test2, ZeroConst, OneConst, ZeroConst);
// Careful, these flags are different between {V,}PTEST and VTESTP{S,D}
SetRFLAG<FEXCore::X86State::RFLAG_ZF_LOC>(Test1);
SetRFLAG<FEXCore::X86State::RFLAG_CF_LOC>(Test2);
SetRFLAG<FEXCore::X86State::RFLAG_AF_LOC>(ZeroConst);
SetRFLAG<FEXCore::X86State::RFLAG_SF_LOC>(ZeroConst);
SetRFLAG<FEXCore::X86State::RFLAG_OF_LOC>(ZeroConst);
SetRFLAG<FEXCore::X86State::RFLAG_PF_LOC>(ZeroConst);
}
void OpDispatchBuilder::PHMINPOSUWOp(OpcodeArgs) {
@@ -2267,10 +2442,10 @@ void OpDispatchBuilder::PHMINPOSUWOp(OpcodeArgs) {
}
// Insert the minimum in to bits [15:0]
OrderedNode *Result = _VMov(Min, 2);
OrderedNode *Result = _VMov(2, Min);
// Insert position in to bits [18:16]
Result = _VInsGPR(16, 2, Result, Pos, 1);
Result = _VInsGPR(16, 2, 1, Result, Pos);
StoreResult(FPRClass, Op, Result, -1);
}
@@ -25,12 +25,12 @@ class OrderedNode;
OrderedNode *OpDispatchBuilder::GetX87Top() {
// Yes, we are storing 3 bits in a single flag register.
// Deal with it
return _LoadContext(1, offsetof(FEXCore::Core::CPUState, flags) + FEXCore::X86State::X87FLAG_TOP_LOC, GPRClass);
return _LoadContext(1, GPRClass, offsetof(FEXCore::Core::CPUState, flags) + FEXCore::X86State::X87FLAG_TOP_LOC);
}
void OpDispatchBuilder::SetX87TopTag(OrderedNode *Value, X87Tag Tag) {
// if we are popping then we must first mark this location as empty
auto FTW = _LoadContext(2, offsetof(FEXCore::Core::CPUState, FTW), GPRClass);
auto FTW = _LoadContext(2, GPRClass, offsetof(FEXCore::Core::CPUState, FTW));
OrderedNode *Mask = _Constant(0b11);
auto TopOffset = _Lshl(Value, _Constant(1));
Mask = _Lshl(Mask, TopOffset);
@@ -40,11 +40,11 @@ void OpDispatchBuilder::SetX87TopTag(OrderedNode *Value, X87Tag Tag) {
NewFTW = _Or(NewFTW, TagVal);
}
_StoreContext(GPRClass, 2, offsetof(FEXCore::Core::CPUState, FTW), NewFTW);
_StoreContext(2, GPRClass, NewFTW, offsetof(FEXCore::Core::CPUState, FTW));
}
OrderedNode *OpDispatchBuilder::GetX87FTW(OrderedNode *Value) {
auto FTW = _LoadContext(2, offsetof(FEXCore::Core::CPUState, FTW), GPRClass);
auto FTW = _LoadContext(2, GPRClass, offsetof(FEXCore::Core::CPUState, FTW));
OrderedNode *Mask = _Constant(0b11);
auto TopOffset = _Lshl(Value, _Constant(1));
auto NewFTW = _Lshr(FTW, TopOffset);
@@ -52,7 +52,7 @@ OrderedNode *OpDispatchBuilder::GetX87FTW(OrderedNode *Value) {
}
void OpDispatchBuilder::SetX87Top(OrderedNode *Value) {
_StoreContext(GPRClass, 1, offsetof(FEXCore::Core::CPUState, flags) + FEXCore::X86State::X87FLAG_TOP_LOC, Value);
_StoreContext(1, GPRClass, Value, offsetof(FEXCore::Core::CPUState, flags) + FEXCore::X86State::X87FLAG_TOP_LOC);
}
template<size_t width>
@@ -136,7 +136,7 @@ void OpDispatchBuilder::FLD_Const(OpcodeArgs) {
auto low = _Constant(Lower);
auto high = _Constant(Upper);
OrderedNode *data = _VCastFromGPR(16, 8, low);
data = _VInsGPR(16, 8, data, high, 1);
data = _VInsGPR(16, 8, 1, data, high);
// Write to ST[TOP]
_StoreContextIndexed(data, top, 16, MMBaseOffset(), 16, FPRClass);
}
@@ -202,7 +202,7 @@ void OpDispatchBuilder::FST(OpcodeArgs) {
StoreResult_WithOpSize(FPRClass, Op, Op->Dest, data, 10, 1);
}
else if constexpr (width == 32 || width == 64) {
auto result = _F80CVT(data, width / 8);
auto result = _F80CVT(width / 8, data);
StoreResult_WithOpSize(FPRClass, Op, Op->Dest, result, width / 8, 1);
}
@@ -228,7 +228,7 @@ void OpDispatchBuilder::FIST(OpcodeArgs) {
auto orig_top = GetX87Top();
OrderedNode *data = _LoadContextIndexed(orig_top, 16, MMBaseOffset(), 16, FPRClass);
data = _F80CVTInt(data, Truncate, Size);
data = _F80CVTInt(Size, data, Truncate);
StoreResult_WithOpSize(GPRClass, Op, Op->Dest, data, Size, 1);
@@ -547,9 +547,9 @@ void OpDispatchBuilder::FCHS(OpcodeArgs) {
auto low = _Constant(0);
auto high = _Constant(0b1'000'0000'0000'0000ULL);
OrderedNode *data = _VCastFromGPR(16, 8, low);
data = _VInsGPR(16, 8, data, high, 1);
data = _VInsGPR(16, 8, 1, data, high);
auto result = _VXor(a, data, 16, 1);
auto result = _VXor(16, 1, a, data);
// Write to ST[TOP]
_StoreContextIndexed(result, top, 16, MMBaseOffset(), 16, FPRClass);
@@ -562,9 +562,9 @@ void OpDispatchBuilder::FABS(OpcodeArgs) {
auto low = _Constant(~0ULL);
auto high = _Constant(0b0'111'1111'1111'1111ULL);
OrderedNode *data = _VCastFromGPR(16, 8, low);
data = _VInsGPR(16, 8, data, high, 1);
data = _VInsGPR(16, 8, 1, data, high);
auto result = _VAnd(a, data, 16, 1);
auto result = _VAnd(16, 1, a, data);
// Write to ST[TOP]
_StoreContextIndexed(result, top, 16, MMBaseOffset(), 16, FPRClass);
@@ -624,7 +624,7 @@ void OpDispatchBuilder::FNINIT(OpcodeArgs) {
// Init FCW to 0x037
auto NewFCW = _Constant(16, 0x037);
_F80LoadFCW(NewFCW);
_StoreContext(GPRClass, 2, offsetof(FEXCore::Core::CPUState, FCW), NewFCW);
_StoreContext(2, GPRClass, NewFCW, offsetof(FEXCore::Core::CPUState, FCW));
// Init FSW to 0
SetX87Top(_Constant(0));
@@ -635,7 +635,7 @@ void OpDispatchBuilder::FNINIT(OpcodeArgs) {
SetRFLAG<FEXCore::X86State::X87FLAG_C3_LOC>(_Constant(0));
// Tags all get set to 0b11
_StoreContext(GPRClass, 2, offsetof(FEXCore::Core::CPUState, FTW), _Constant(0xFFFF));
_StoreContext(2, GPRClass, _Constant(0xFFFF), offsetof(FEXCore::Core::CPUState, FTW));
}
template<size_t width, bool Integer, OpDispatchBuilder::FCOMIFlags whichflags, bool poptwice>
@@ -685,12 +685,15 @@ void OpDispatchBuilder::FCOMI(OpcodeArgs) {
SetRFLAG<FEXCore::X86State::X87FLAG_C3_LOC>(HostFlag_ZF);
}
else {
// Invalidate deferred flags early
// OF, SF, AF, PF all undefined
InvalidateDeferredFlags();
SetRFLAG<FEXCore::X86State::RFLAG_CF_LOC>(HostFlag_CF);
SetRFLAG<FEXCore::X86State::RFLAG_ZF_LOC>(HostFlag_ZF);
SetRFLAG<FEXCore::X86State::RFLAG_PF_LOC>(HostFlag_Unordered);
}
if constexpr (poptwice) {
// if we are popping then we must first mark this location as empty
SetX87TopTag(top, X87Tag::Empty);
@@ -872,7 +875,7 @@ void OpDispatchBuilder::X87FYL2X(OpcodeArgs) {
auto low = _Constant(0x8000'0000'0000'0000ULL);
auto high = _Constant(0b0'011'1111'1111'1111);
OrderedNode *data = _VCastFromGPR(16, 8, low);
data = _VInsGPR(16, 8, data, high, 1);
data = _VInsGPR(16, 8, 1, data, high);
st0 = _F80Add(st0, data);
}
@@ -895,7 +898,7 @@ void OpDispatchBuilder::X87TAN(OpcodeArgs) {
auto low = _Constant(0x8000'0000'0000'0000ULL);
auto high = _Constant(0b0'011'1111'1111'1111ULL);
OrderedNode *data = _VCastFromGPR(16, 8, low);
data = _VInsGPR(16, 8, data, high, 1);
data = _VInsGPR(16, 8, 1, data, high);
// Write to ST[TOP]
_StoreContextIndexed(result, orig_top, 16, MMBaseOffset(), 16, FPRClass);
@@ -925,7 +928,7 @@ void OpDispatchBuilder::X87LDENV(OpcodeArgs) {
auto NewFCW = _LoadMem(GPRClass, 2, Mem, 2);
_F80LoadFCW(NewFCW);
_StoreContext(GPRClass, 2, offsetof(FEXCore::Core::CPUState, FCW), NewFCW);
_StoreContext(2, GPRClass, NewFCW, offsetof(FEXCore::Core::CPUState, FCW));
OrderedNode *MemLocation = _Add(Mem, _Constant(Size * 1));
auto NewFSW = _LoadMem(GPRClass, Size, MemLocation, Size);
@@ -948,7 +951,7 @@ void OpDispatchBuilder::X87LDENV(OpcodeArgs) {
// FTW
OrderedNode *MemLocation = _Add(Mem, _Constant(Size * 2));
auto NewFTW = _LoadMem(GPRClass, Size, MemLocation, Size);
_StoreContext(GPRClass, 2, offsetof(FEXCore::Core::CPUState, FTW), NewFTW);
_StoreContext(2, GPRClass, NewFTW, offsetof(FEXCore::Core::CPUState, FTW));
}
}
@@ -977,7 +980,7 @@ void OpDispatchBuilder::X87FNSTENV(OpcodeArgs) {
Mem = AppendSegmentOffset(Mem, Op->Flags);
{
auto FCW = _LoadContext(2, offsetof(FEXCore::Core::CPUState, FCW), GPRClass);
auto FCW = _LoadContext(2, GPRClass, offsetof(FEXCore::Core::CPUState, FCW));
_StoreMem(GPRClass, Size, Mem, FCW, Size);
}
@@ -1005,7 +1008,7 @@ void OpDispatchBuilder::X87FNSTENV(OpcodeArgs) {
{
// FTW
OrderedNode *MemLocation = _Add(Mem, _Constant(Size * 2));
auto FTW = _LoadContext(2, offsetof(FEXCore::Core::CPUState, FTW), GPRClass);
auto FTW = _LoadContext(2, GPRClass, offsetof(FEXCore::Core::CPUState, FTW));
_StoreMem(GPRClass, Size, MemLocation, FTW, Size);
}
@@ -1037,11 +1040,11 @@ void OpDispatchBuilder::X87FNSTENV(OpcodeArgs) {
void OpDispatchBuilder::X87FLDCW(OpcodeArgs) {
OrderedNode *NewFCW = LoadSource(GPRClass, Op, Op->Src[0], Op->Flags, -1);
_F80LoadFCW(NewFCW);
_StoreContext(GPRClass, 2, offsetof(FEXCore::Core::CPUState, FCW), NewFCW);
_StoreContext(2, GPRClass, NewFCW, offsetof(FEXCore::Core::CPUState, FCW));
}
void OpDispatchBuilder::X87FSTCW(OpcodeArgs) {
auto FCW = _LoadContext(2, offsetof(FEXCore::Core::CPUState, FCW), GPRClass);
auto FCW = _LoadContext(2, GPRClass, offsetof(FEXCore::Core::CPUState, FCW));
StoreResult(GPRClass, Op, FCW, -1);
}
@@ -1108,7 +1111,7 @@ void OpDispatchBuilder::X87FNSAVE(OpcodeArgs) {
OrderedNode *Top = GetX87Top();
{
auto FCW = _LoadContext(2, offsetof(FEXCore::Core::CPUState, FCW), GPRClass);
auto FCW = _LoadContext(2, GPRClass, offsetof(FEXCore::Core::CPUState, FCW));
_StoreMem(GPRClass, Size, Mem, FCW, Size);
}
@@ -1135,7 +1138,7 @@ void OpDispatchBuilder::X87FNSAVE(OpcodeArgs) {
{
// FTW
OrderedNode *MemLocation = _Add(Mem, _Constant(Size * 2));
auto FTW = _LoadContext(2, offsetof(FEXCore::Core::CPUState, FTW), GPRClass);
auto FTW = _LoadContext(2, GPRClass, offsetof(FEXCore::Core::CPUState, FTW));
_StoreMem(GPRClass, Size, MemLocation, FTW, Size);
}
@@ -1196,7 +1199,7 @@ void OpDispatchBuilder::X87FRSTOR(OpcodeArgs) {
auto NewFCW = _LoadMem(GPRClass, 2, Mem, 2);
_F80LoadFCW(NewFCW);
_StoreContext(GPRClass, 2, offsetof(FEXCore::Core::CPUState, FCW), NewFCW);
_StoreContext(2, GPRClass, NewFCW, offsetof(FEXCore::Core::CPUState, FCW));
OrderedNode *MemLocation = _Add(Mem, _Constant(Size * 1));
auto NewFSW = _LoadMem(GPRClass, Size, MemLocation, Size);
@@ -1219,7 +1222,7 @@ void OpDispatchBuilder::X87FRSTOR(OpcodeArgs) {
// FTW
OrderedNode *MemLocation = _Add(Mem, _Constant(Size * 2));
auto NewFTW = _LoadMem(GPRClass, Size, MemLocation, Size);
_StoreContext(GPRClass, 2, offsetof(FEXCore::Core::CPUState, FTW), NewFTW);
_StoreContext(2, GPRClass, NewFTW, offsetof(FEXCore::Core::CPUState, FTW));
}
OrderedNode *ST0Location = _Add(Mem, _Constant(Size * 7));
@@ -1231,7 +1234,7 @@ void OpDispatchBuilder::X87FRSTOR(OpcodeArgs) {
auto low = _Constant(~0ULL);
auto high = _Constant(0xFFFF);
OrderedNode *Mask = _VCastFromGPR(16, 8, low);
Mask = _VInsGPR(16, 8, Mask, high, 1);
Mask = _VInsGPR(16, 8, 1, Mask, high);
for (int i = 0; i < 7; ++i) {
OrderedNode *Reg = _LoadMem(FPRClass, 16, ST0Location, 1);
@@ -1359,7 +1362,7 @@ void OpDispatchBuilder::X87FCMOV(OpcodeArgs) {
SrcCond = _Sbfe(1, 0, SrcCond);
OrderedNode *VecCond = _VCastFromGPR(16, 8, SrcCond);
VecCond = _VInsGPR(16, 8, VecCond, SrcCond, 1);
VecCond = _VInsGPR(16, 8, 1, VecCond, SrcCond);
auto top = GetX87Top();
OrderedNode* arg;
@@ -1380,7 +1383,7 @@ void OpDispatchBuilder::X87FCMOV(OpcodeArgs) {
void OpDispatchBuilder::X87EMMS(OpcodeArgs) {
// Tags all get set to 0b11
_StoreContext(GPRClass, 2, offsetof(FEXCore::Core::CPUState, FTW), _Constant(0xFFFF));
_StoreContext(2, GPRClass, _Constant(0xFFFF), offsetof(FEXCore::Core::CPUState, FTW));
}
void OpDispatchBuilder::X87FFREE(OpcodeArgs) {
+2 -1
View File
@@ -1,5 +1,6 @@
#include <FEXCore/Core/SignalDelegator.h>
#include <FEXCore/Utils/LogManager.h>
#include <FEXHeaderUtils/Syscalls.h>
#include <unistd.h>
#include <signal.h>
@@ -91,7 +92,7 @@ namespace FEXCore {
HostSignalHandler &Handler = HostHandlers[Signal];
if (!Thread) {
LogMan::Msg::EFmt("[{}] Thread has received a signal and hasn't registered itself with the delegate! Programming error!", ::gettid());
LogMan::Msg::EFmt("[{}] Thread has received a signal and hasn't registered itself with the delegate! Programming error!", FHU::Syscalls::gettid());
}
else {
if (Handler.Handler &&
+4 -41
View File
@@ -6,44 +6,7 @@
namespace FEXCore::X86Tables::X86InstDebugInfo {
void InstallDebugInfo() {
using namespace FEXCore::X86Tables;
auto NoFlags = Flags {0};
for (auto &BaseOp : BaseOps)
BaseOp.DebugInfo = NoFlags;
for (auto &BaseOp : SecondBaseOps)
BaseOp.DebugInfo = NoFlags;
for (auto &BaseOp : RepModOps)
BaseOp.DebugInfo = NoFlags;
for (auto &BaseOp : RepNEModOps)
BaseOp.DebugInfo = NoFlags;
for (auto &BaseOp : OpSizeModOps)
BaseOp.DebugInfo = NoFlags;
for (auto &BaseOp : PrimaryInstGroupOps)
BaseOp.DebugInfo = NoFlags;
for (auto &BaseOp : SecondInstGroupOps)
BaseOp.DebugInfo = NoFlags;
for (auto &BaseOp : SecondModRMTableOps)
BaseOp.DebugInfo = NoFlags;
for (auto &BaseOp : X87Ops)
BaseOp.DebugInfo = NoFlags;
for (auto &BaseOp : DDDNowOps)
BaseOp.DebugInfo = NoFlags;
for (auto &BaseOp : H0F38TableOps)
BaseOp.DebugInfo = NoFlags;
for (auto &BaseOp : H0F3ATableOps)
BaseOp.DebugInfo = NoFlags;
for (auto &BaseOp : VEXTableOps)
BaseOp.DebugInfo = NoFlags;
for (auto &BaseOp : VEXTableGroupOps)
BaseOp.DebugInfo = NoFlags;
for (auto &BaseOp : XOPTableOps)
BaseOp.DebugInfo = NoFlags;
for (auto &BaseOp : XOPTableGroupOps)
BaseOp.DebugInfo = NoFlags;
const std::vector<std::tuple<uint8_t, uint8_t, Flags>> BaseOpTable = {
const std::tuple<uint8_t, uint8_t, Flags> BaseOpTable[] = {
{0x50, 8, {FLAGS_MEM_ACCESS}},
{0x58, 8, {FLAGS_MEM_ACCESS}},
@@ -62,7 +25,7 @@ void InstallDebugInfo() {
{0xF4, 1, {FLAGS_DEBUG}},
};
const std::vector<std::tuple<uint8_t, uint8_t, Flags>> TwoByteOpTable = {
const std::tuple<uint8_t, uint8_t, Flags> TwoByteOpTable[] = {
{0x0B, 1, {FLAGS_DEBUG}},
{0x19, 7, {FLAGS_DEBUG}},
{0x28, 2, {FLAGS_MEM_ALIGN_16}},
@@ -78,14 +41,14 @@ void InstallDebugInfo() {
{0xFF, 1, {FLAGS_DEBUG}},
};
const std::vector<std::tuple<uint8_t, uint8_t, Flags>> PrimaryGroupOpTable = {
const std::tuple<uint8_t, uint8_t, Flags> PrimaryGroupOpTable[] = {
#define OPD(group, prefix, Reg) (((group - FEXCore::X86Tables::TYPE_GROUP_1) << 6) | (prefix) << 3 | (Reg))
{OPD(TYPE_GROUP_3, OpToIndex(0xF6), 6), 2, {FLAGS_DIVIDE}},
{OPD(TYPE_GROUP_3, OpToIndex(0xF7), 6), 2, {FLAGS_DIVIDE}},
#undef OPD
};
const std::vector<std::tuple<uint16_t, uint8_t, Flags>> SecondaryExtensionOpTable = {
const std::tuple<uint16_t, uint8_t, Flags> SecondaryExtensionOpTable[] = {
#define PF_NONE 0
#define PF_F3 1
#define PF_66 2
+1 -1
View File
@@ -9,12 +9,12 @@ $end_info$
#include <FEXCore/Config/Config.h>
#include <FEXCore/Utils/Allocator.h>
#include <FEXHeaderUtils/Syscalls.h>
#include <cstdint>
#include <cstring>
#include <vector>
#include <sys/mman.h>
#include <bits/mman-map-flags-generic.h>
namespace FEXCore {
constexpr size_t CODE_SIZE = 0x1000;
+17 -56
View File
@@ -10,25 +10,24 @@ $end_info$
namespace FEXCore::X86Tables {
X86InstInfo BaseOps[MAX_PRIMARY_TABLE_SIZE];
std::array<X86InstInfo, MAX_PRIMARY_TABLE_SIZE> BaseOps{};
std::array<X86InstInfo, MAX_SECOND_TABLE_SIZE> SecondBaseOps{};
std::array<X86InstInfo, MAX_REP_MOD_TABLE_SIZE> RepModOps{};
std::array<X86InstInfo, MAX_REPNE_MOD_TABLE_SIZE> RepNEModOps{};
std::array<X86InstInfo, MAX_OPSIZE_MOD_TABLE_SIZE> OpSizeModOps{};
X86InstInfo SecondBaseOps[MAX_SECOND_TABLE_SIZE];
X86InstInfo RepModOps[MAX_REP_MOD_TABLE_SIZE];
X86InstInfo RepNEModOps[MAX_REPNE_MOD_TABLE_SIZE];
X86InstInfo OpSizeModOps[MAX_OPSIZE_MOD_TABLE_SIZE];
X86InstInfo PrimaryInstGroupOps[MAX_INST_GROUP_TABLE_SIZE];
X86InstInfo SecondInstGroupOps[MAX_INST_SECOND_GROUP_TABLE_SIZE];
X86InstInfo SecondModRMTableOps[MAX_SECOND_MODRM_TABLE_SIZE];
X86InstInfo X87Ops[MAX_X87_TABLE_SIZE];
X86InstInfo DDDNowOps[MAX_3DNOW_TABLE_SIZE];
X86InstInfo H0F38TableOps[MAX_0F_38_TABLE_SIZE];
X86InstInfo H0F3ATableOps[MAX_0F_3A_TABLE_SIZE];
X86InstInfo VEXTableOps[MAX_VEX_TABLE_SIZE];
X86InstInfo VEXTableGroupOps[MAX_VEX_GROUP_TABLE_SIZE];
X86InstInfo XOPTableOps[MAX_XOP_TABLE_SIZE];
X86InstInfo XOPTableGroupOps[MAX_XOP_GROUP_TABLE_SIZE];
X86InstInfo EVEXTableOps[MAX_EVEX_TABLE_SIZE];
std::array<X86InstInfo, MAX_INST_GROUP_TABLE_SIZE> PrimaryInstGroupOps{};
std::array<X86InstInfo, MAX_INST_SECOND_GROUP_TABLE_SIZE> SecondInstGroupOps{};
std::array<X86InstInfo, MAX_SECOND_MODRM_TABLE_SIZE> SecondModRMTableOps{};
std::array<X86InstInfo, MAX_X87_TABLE_SIZE> X87Ops{};
std::array<X86InstInfo, MAX_3DNOW_TABLE_SIZE> DDDNowOps{};
std::array<X86InstInfo, MAX_0F_38_TABLE_SIZE> H0F38TableOps{};
std::array<X86InstInfo, MAX_0F_3A_TABLE_SIZE> H0F3ATableOps{};
std::array<X86InstInfo, MAX_VEX_TABLE_SIZE> VEXTableOps{};
std::array<X86InstInfo, MAX_VEX_GROUP_TABLE_SIZE> VEXTableGroupOps{};
std::array<X86InstInfo, MAX_XOP_TABLE_SIZE> XOPTableOps{};
std::array<X86InstInfo, MAX_XOP_GROUP_TABLE_SIZE> XOPTableGroupOps{};
std::array<X86InstInfo, MAX_EVEX_TABLE_SIZE> EVEXTableOps{};
void InitializeBaseTables(Context::OperatingMode Mode);
void InitializeSecondaryTables(Context::OperatingMode Mode);
@@ -49,44 +48,6 @@ uint64_t NumInsts{};
#endif
void InitializeInfoTables(Context::OperatingMode Mode) {
using namespace FEXCore::X86Tables::InstFlags;
auto UnknownOp = X86InstInfo{"UND", TYPE_UNKNOWN, FLAGS_NONE, 0, nullptr};
for (auto &BaseOp : BaseOps)
BaseOp = UnknownOp;
for (auto &BaseOp : SecondBaseOps)
BaseOp = UnknownOp;
for (auto &BaseOp : RepModOps)
BaseOp = UnknownOp;
for (auto &BaseOp : RepNEModOps)
BaseOp = UnknownOp;
for (auto &BaseOp : OpSizeModOps)
BaseOp = UnknownOp;
for (auto &BaseOp : PrimaryInstGroupOps)
BaseOp = UnknownOp;
for (auto &BaseOp : SecondInstGroupOps)
BaseOp = UnknownOp;
for (auto &BaseOp : SecondModRMTableOps)
BaseOp = UnknownOp;
for (auto &BaseOp : X87Ops)
BaseOp = UnknownOp;
for (auto &BaseOp : DDDNowOps)
BaseOp = UnknownOp;
for (auto &BaseOp : H0F38TableOps)
BaseOp = UnknownOp;
for (auto &BaseOp : H0F3ATableOps)
BaseOp = UnknownOp;
for (auto &BaseOp : VEXTableOps)
BaseOp = UnknownOp;
for (auto &BaseOp : VEXTableGroupOps)
BaseOp = UnknownOp;
for (auto &BaseOp : XOPTableOps)
BaseOp = UnknownOp;
for (auto &BaseOp : XOPTableGroupOps)
BaseOp = UnknownOp;
for (auto &BaseOp : EVEXTableOps)
BaseOp = UnknownOp;
InitializeBaseTables(Mode);
InitializeSecondaryTables(Mode);
InitializePrimaryGroupTables(Mode);
@@ -288,13 +288,13 @@ void InitializeBaseTables(Context::OperatingMode Mode) {
{0xEA, 1, X86InstInfo{"JMPF", TYPE_INST, FLAGS_NONE, 0, nullptr}},
};
GenerateTable(BaseOps, BaseOpTable, std::size(BaseOpTable));
GenerateTable(&BaseOps.at(0), BaseOpTable, std::size(BaseOpTable));
if (Mode == Context::MODE_64BIT) {
GenerateTable(BaseOps, BaseOpTable_64, std::size(BaseOpTable_64));
GenerateTable(&BaseOps.at(0), BaseOpTable_64, std::size(BaseOpTable_64));
}
else {
GenerateTable(BaseOps, BaseOpTable_32, std::size(BaseOpTable_32));
GenerateTable(&BaseOps.at(0), BaseOpTable_32, std::size(BaseOpTable_32));
}
}
}
@@ -15,39 +15,44 @@ using namespace InstFlags;
void InitializeDDDTables() {
static constexpr U8U8InfoStruct DDDNowOpTable[] = {
{0x0C, 1, X86InstInfo{"PI2FW", TYPE_3DNOW_INST, FLAGS_MODRM, 0, nullptr}},
{0x0D, 1, X86InstInfo{"PI2FD", TYPE_3DNOW_INST, FLAGS_MODRM, 0, nullptr}},
{0x1C, 1, X86InstInfo{"PF2IW", TYPE_3DNOW_INST, FLAGS_MODRM, 0, nullptr}},
{0x1D, 1, X86InstInfo{"PF2ID", TYPE_3DNOW_INST, FLAGS_MODRM, 0, nullptr}},
{0x0C, 1, X86InstInfo{"PI2FW", TYPE_INST, GenFlagsSameSize(SIZE_64BIT) | FLAGS_MODRM | FLAGS_XMM_FLAGS | FLAGS_SF_MMX, 0, nullptr}},
{0x0D, 1, X86InstInfo{"PI2FD", TYPE_INST, GenFlagsSameSize(SIZE_64BIT) | FLAGS_MODRM | FLAGS_XMM_FLAGS | FLAGS_SF_MMX, 0, nullptr}},
{0x1C, 1, X86InstInfo{"PF2IW", TYPE_INST, GenFlagsSameSize(SIZE_64BIT) | FLAGS_MODRM | FLAGS_XMM_FLAGS | FLAGS_SF_MMX, 0, nullptr}},
{0x1D, 1, X86InstInfo{"PF2ID", TYPE_INST, GenFlagsSameSize(SIZE_64BIT) | FLAGS_MODRM | FLAGS_XMM_FLAGS | FLAGS_SF_MMX, 0, nullptr}},
{0x8A, 1, X86InstInfo{"PFNACC", TYPE_3DNOW_INST, FLAGS_MODRM, 0, nullptr}},
{0x8E, 1, X86InstInfo{"PFPNACC", TYPE_3DNOW_INST, FLAGS_MODRM, 0, nullptr}},
// Inverse 3DNow! These two instructions are Geode product line specific
// No CPUID for these, you're expected to read ID_CONFIG_MSR (1250h) bit 1
{0x86, 1, X86InstInfo{"PFRCPV", TYPE_INST, GenFlagsSameSize(SIZE_64BIT) | FLAGS_MODRM | FLAGS_XMM_FLAGS | FLAGS_SF_MMX, 0, nullptr}},
{0x87, 1, X86InstInfo{"PFRSQRTV", TYPE_INST, GenFlagsSameSize(SIZE_64BIT) | FLAGS_MODRM | FLAGS_XMM_FLAGS | FLAGS_SF_MMX, 0, nullptr}},
{0x9A, 1, X86InstInfo{"PFSUB", TYPE_3DNOW_INST, FLAGS_MODRM, 0, nullptr}},
{0x9E, 1, X86InstInfo{"PFADD", TYPE_3DNOW_INST, FLAGS_MODRM, 0, nullptr}},
{0x8A, 1, X86InstInfo{"PFNACC", TYPE_INST, GenFlagsSameSize(SIZE_64BIT) | FLAGS_MODRM | FLAGS_XMM_FLAGS | FLAGS_SF_MMX, 0, nullptr}},
{0x8E, 1, X86InstInfo{"PFPNACC", TYPE_INST, GenFlagsSameSize(SIZE_64BIT) | FLAGS_MODRM | FLAGS_XMM_FLAGS | FLAGS_SF_MMX, 0, nullptr}},
{0xAA, 1, X86InstInfo{"PFSUBR", TYPE_3DNOW_INST, FLAGS_MODRM, 0, nullptr}},
{0xAE, 1, X86InstInfo{"PFACC", TYPE_3DNOW_INST, FLAGS_MODRM, 0, nullptr}},
{0x90, 1, X86InstInfo{"PFCMPGE", TYPE_INST, GenFlagsSameSize(SIZE_64BIT) | FLAGS_MODRM | FLAGS_XMM_FLAGS | FLAGS_SF_MMX, 0, nullptr}},
{0x94, 1, X86InstInfo{"PFMIN", TYPE_INST, GenFlagsSameSize(SIZE_64BIT) | FLAGS_MODRM | FLAGS_XMM_FLAGS | FLAGS_SF_MMX, 0, nullptr}},
{0x96, 1, X86InstInfo{"PFRCP", TYPE_INST, GenFlagsSameSize(SIZE_64BIT) | FLAGS_MODRM | FLAGS_XMM_FLAGS | FLAGS_SF_MMX, 0, nullptr}},
{0x97, 1, X86InstInfo{"PFRSQRT", TYPE_INST, GenFlagsSameSize(SIZE_64BIT) | FLAGS_MODRM | FLAGS_XMM_FLAGS | FLAGS_SF_MMX, 0, nullptr}},
{0xBB, 1, X86InstInfo{"PSWAPD", TYPE_3DNOW_INST, FLAGS_MODRM, 0, nullptr}},
{0xBF, 1, X86InstInfo{"PAVGUSB", TYPE_3DNOW_INST, FLAGS_MODRM, 0, nullptr}},
{0x9A, 1, X86InstInfo{"PFSUB", TYPE_INST, GenFlagsSameSize(SIZE_64BIT) | FLAGS_MODRM | FLAGS_XMM_FLAGS | FLAGS_SF_MMX, 0, nullptr}},
{0x9E, 1, X86InstInfo{"PFADD", TYPE_INST, GenFlagsSameSize(SIZE_64BIT) | FLAGS_MODRM | FLAGS_XMM_FLAGS | FLAGS_SF_MMX, 0, nullptr}},
{0x90, 1, X86InstInfo{"PFCMPGE", TYPE_3DNOW_INST, FLAGS_MODRM, 0, nullptr}},
{0x94, 1, X86InstInfo{"PFMIN", TYPE_3DNOW_INST, FLAGS_MODRM, 0, nullptr}},
{0x96, 1, X86InstInfo{"PFRCP", TYPE_3DNOW_INST, FLAGS_MODRM, 0, nullptr}},
{0x97, 1, X86InstInfo{"PFRSQRT", TYPE_3DNOW_INST, FLAGS_MODRM, 0, nullptr}},
{0xA0, 1, X86InstInfo{"PFCMPGT", TYPE_INST, GenFlagsSameSize(SIZE_64BIT) | FLAGS_MODRM | FLAGS_XMM_FLAGS | FLAGS_SF_MMX, 0, nullptr}},
{0xA4, 1, X86InstInfo{"PFMAX", TYPE_INST, GenFlagsSameSize(SIZE_64BIT) | FLAGS_MODRM | FLAGS_XMM_FLAGS | FLAGS_SF_MMX, 0, nullptr}},
{0xA6, 1, X86InstInfo{"PFRCPIT1", TYPE_INST, GenFlagsSameSize(SIZE_64BIT) | FLAGS_MODRM | FLAGS_XMM_FLAGS | FLAGS_SF_MMX, 0, nullptr}},
{0xA7, 1, X86InstInfo{"PFRSQIT1", TYPE_INST, GenFlagsSameSize(SIZE_64BIT) | FLAGS_MODRM | FLAGS_XMM_FLAGS | FLAGS_SF_MMX, 0, nullptr}},
{0xA0, 1, X86InstInfo{"PFCMPGT", TYPE_3DNOW_INST, FLAGS_MODRM, 0, nullptr}},
{0xA4, 1, X86InstInfo{"PFMAX", TYPE_3DNOW_INST, FLAGS_MODRM, 0, nullptr}},
{0xA6, 1, X86InstInfo{"PFRCPIT1", TYPE_3DNOW_INST, FLAGS_MODRM, 0, nullptr}},
{0xA7, 1, X86InstInfo{"PFRSQIT1", TYPE_3DNOW_INST, FLAGS_MODRM, 0, nullptr}},
{0xAA, 1, X86InstInfo{"PFSUBR", TYPE_INST, GenFlagsSameSize(SIZE_64BIT) | FLAGS_MODRM | FLAGS_XMM_FLAGS | FLAGS_SF_MMX, 0, nullptr}},
{0xAE, 1, X86InstInfo{"PFACC", TYPE_INST, GenFlagsSameSize(SIZE_64BIT) | FLAGS_MODRM | FLAGS_XMM_FLAGS | FLAGS_SF_MMX, 0, nullptr}},
{0xB0, 1, X86InstInfo{"PFCMPEQ", TYPE_3DNOW_INST, FLAGS_MODRM, 0, nullptr}},
{0xB4, 1, X86InstInfo{"PFMUL", TYPE_3DNOW_INST, FLAGS_MODRM, 0, nullptr}},
{0xB6, 1, X86InstInfo{"PFRCPIT2", TYPE_3DNOW_INST, FLAGS_MODRM, 0, nullptr}},
{0xB7, 1, X86InstInfo{"PMULHRW", TYPE_3DNOW_INST, FLAGS_MODRM, 0, nullptr}},
{0xB0, 1, X86InstInfo{"PFCMPEQ", TYPE_INST, GenFlagsSameSize(SIZE_64BIT) | FLAGS_MODRM | FLAGS_XMM_FLAGS | FLAGS_SF_MMX, 0, nullptr}},
{0xB4, 1, X86InstInfo{"PFMUL", TYPE_INST, GenFlagsSameSize(SIZE_64BIT) | FLAGS_MODRM | FLAGS_XMM_FLAGS | FLAGS_SF_MMX, 0, nullptr}},
{0xB6, 1, X86InstInfo{"PFRCPIT2", TYPE_INST, GenFlagsSameSize(SIZE_64BIT) | FLAGS_MODRM | FLAGS_XMM_FLAGS | FLAGS_SF_MMX, 0, nullptr}},
{0xB7, 1, X86InstInfo{"PMULHRW", TYPE_INST, GenFlagsSameSize(SIZE_64BIT) | FLAGS_MODRM | FLAGS_XMM_FLAGS | FLAGS_SF_MMX, 0, nullptr}},
{0xBB, 1, X86InstInfo{"PSWAPD", TYPE_INST, GenFlagsSameSize(SIZE_64BIT) | FLAGS_MODRM | FLAGS_XMM_FLAGS | FLAGS_SF_MMX, 0, nullptr}},
{0xBF, 1, X86InstInfo{"PAVGUSB", TYPE_INST, GenFlagsSameSize(SIZE_64BIT) | FLAGS_MODRM | FLAGS_XMM_FLAGS | FLAGS_SF_MMX, 0, nullptr}},
};
GenerateTable(DDDNowOps, DDDNowOpTable, std::size(DDDNowOpTable));
GenerateTable(&DDDNowOps.at(0), DDDNowOpTable, std::size(DDDNowOpTable));
}
}
@@ -30,6 +30,6 @@ void InitializeEVEXTables() {
{0xE7, 1, X86InstInfo{"VMOVNTDQ", TYPE_INST, FLAGS_MODRM | FLAGS_SF_MOD_DST | FLAGS_XMM_FLAGS, 0, nullptr}},
};
GenerateTable(EVEXTableOps, EVEXTable, std::size(EVEXTable));
GenerateTable(&EVEXTableOps.at(0), EVEXTable, std::size(EVEXTable));
}
}
@@ -14,11 +14,11 @@ namespace FEXCore::X86Tables {
using namespace InstFlags;
void InitializeH0F38Tables() {
#define OPD(prefix, opcode) ((prefix << 8) | opcode)
#define OPD(prefix, opcode) (((prefix) << 8) | opcode)
constexpr uint16_t PF_38_NONE = 0;
constexpr uint16_t PF_38_66 = 1;
constexpr uint16_t PF_38_F2 = 2;
constexpr uint16_t PF_38_F3 = 3;
constexpr uint16_t PF_38_66 = (1U << 0);
constexpr uint16_t PF_38_F2 = (1U << 1);
constexpr uint16_t PF_38_F3 = (1U << 2);
static constexpr U16U8InfoStruct H0F38Table[] = {
{OPD(PF_38_NONE, 0x00), 1, X86InstInfo{"PSHUFB", TYPE_INST, GenFlagsSameSize(SIZE_64BIT) | FLAGS_MODRM | FLAGS_XMM_FLAGS | FLAGS_SF_MMX, 0, nullptr}},
@@ -74,6 +74,7 @@ void InitializeH0F38Tables() {
{OPD(PF_38_66, 0x33), 1, X86InstInfo{"PMOVZXWD", TYPE_INST, GenFlagsSizes(SIZE_128BIT, SIZE_64BIT) | FLAGS_MODRM | FLAGS_XMM_FLAGS, 0, nullptr}},
{OPD(PF_38_66, 0x34), 1, X86InstInfo{"PMOVZXWQ", TYPE_INST, GenFlagsSizes(SIZE_128BIT, SIZE_32BIT) | FLAGS_MODRM | FLAGS_XMM_FLAGS, 0, nullptr}},
{OPD(PF_38_66, 0x35), 1, X86InstInfo{"PMOVZXDQ", TYPE_INST, GenFlagsSizes(SIZE_128BIT, SIZE_64BIT) | FLAGS_MODRM | FLAGS_XMM_FLAGS, 0, nullptr}},
{OPD(PF_38_66, 0x37), 1, X86InstInfo{"PCMPGTQ", TYPE_INST, GenFlagsSameSize(SIZE_128BIT) | FLAGS_MODRM | FLAGS_XMM_FLAGS, 0, nullptr}},
{OPD(PF_38_66, 0x38), 1, X86InstInfo{"PMINSB", TYPE_INST, GenFlagsSameSize(SIZE_128BIT) | FLAGS_MODRM | FLAGS_XMM_FLAGS, 0, nullptr}},
{OPD(PF_38_66, 0x39), 1, X86InstInfo{"PMINSD", TYPE_INST, GenFlagsSameSize(SIZE_128BIT) | FLAGS_MODRM | FLAGS_XMM_FLAGS, 0, nullptr}},
{OPD(PF_38_66, 0x3A), 1, X86InstInfo{"PMINUW", TYPE_INST, GenFlagsSameSize(SIZE_128BIT) | FLAGS_MODRM | FLAGS_XMM_FLAGS, 0, nullptr}},
@@ -98,14 +99,16 @@ void InitializeH0F38Tables() {
{OPD(PF_38_66, 0xF0), 1, X86InstInfo{"MOVBE", TYPE_INST, FLAGS_MODRM | FLAGS_SF_MOD_MEM_ONLY, 0, nullptr}},
{OPD(PF_38_66, 0xF1), 1, X86InstInfo{"MOVBE", TYPE_INST, FLAGS_MODRM | FLAGS_SF_MOD_DST | FLAGS_SF_MOD_MEM_ONLY, 0, nullptr}},
{OPD(PF_38_F2, 0xF0), 1, X86InstInfo{"CRC32", TYPE_UNDEC, FLAGS_NONE, 0, nullptr}},
{OPD(PF_38_F2, 0xF1), 1, X86InstInfo{"CRC32", TYPE_UNDEC, FLAGS_NONE, 0, nullptr}},
{OPD(PF_38_F2, 0xF0), 1, X86InstInfo{"CRC32", TYPE_INST, GenFlagsSizes(SIZE_DEF, SIZE_8BIT) | FLAGS_MODRM, 0, nullptr}},
{OPD(PF_38_F2, 0xF1), 1, X86InstInfo{"CRC32", TYPE_INST, FLAGS_MODRM, 0, nullptr}},
{OPD(PF_38_66 | PF_38_F2, 0xF0), 1, X86InstInfo{"CRC32", TYPE_INST, GenFlagsSizes(SIZE_DEF, SIZE_8BIT) | FLAGS_MODRM, 0, nullptr}},
{OPD(PF_38_66 | PF_38_F2, 0xF1), 1, X86InstInfo{"CRC32", TYPE_INST, FLAGS_MODRM, 0, nullptr}},
{OPD(PF_38_66, 0xF6), 1, X86InstInfo{"ADCX", TYPE_INST, FLAGS_MODRM, 0, nullptr}},
{OPD(PF_38_F3, 0xF6), 1, X86InstInfo{"ADOX", TYPE_INST, FLAGS_MODRM, 0, nullptr}},
};
#undef OPD
GenerateTable(H0F38TableOps, H0F38Table, std::size(H0F38Table));
GenerateTable(&H0F38TableOps.at(0), H0F38Table, std::size(H0F38Table));
}
}
@@ -60,10 +60,10 @@ void InitializeH0F3ATables(Context::OperatingMode Mode) {
#undef OPD
GenerateTable(H0F3ATableOps, H0F3ATable, std::size(H0F3ATable));
GenerateTable(&H0F3ATableOps.at(0), H0F3ATable, std::size(H0F3ATable));
if (Mode == Context::MODE_64BIT) {
GenerateTable(H0F3ATableOps, H0F3ATable_64, std::size(H0F3ATable_64));
GenerateTable(&H0F3ATableOps.at(0), H0F3ATable_64, std::size(H0F3ATable_64));
}
}
}
@@ -163,14 +163,13 @@ void InitializePrimaryGroupTables(Context::OperatingMode Mode) {
#undef OPD
GenerateTable(PrimaryInstGroupOps, PrimaryGroupOpTable, std::size(PrimaryGroupOpTable));
GenerateTable(&PrimaryInstGroupOps.at(0), PrimaryGroupOpTable, std::size(PrimaryGroupOpTable));
if (Mode == Context::MODE_64BIT) {
GenerateTable(PrimaryInstGroupOps, PrimaryGroupOpTable_64, std::size(PrimaryGroupOpTable_64));
GenerateTable(&PrimaryInstGroupOps.at(0), PrimaryGroupOpTable_64, std::size(PrimaryGroupOpTable_64));
}
else {
GenerateTable(PrimaryInstGroupOps, PrimaryGroupOpTable_32, std::size(PrimaryGroupOpTable_32));
GenerateTable(&PrimaryInstGroupOps.at(0), PrimaryGroupOpTable_32, std::size(PrimaryGroupOpTable_32));
}
}
}
@@ -144,38 +144,38 @@ void InitializeSecondaryGroupTables() {
// AMD documentation is a bit broken for Group 9
// Claims the entire group has n/a applied for the prefix (Implies that the prefix is ignored)
// RDRAND/RDSEED only work with no prefix
// RDRAND/RDSEED only work with no prefix (Other than 66h)
// CMPXCHG8B/16B works with all prefixes
// Tooling fails to decode CMPXCHG with prefix
{OPD(TYPE_GROUP_9, PF_NONE, 0), 1, X86InstInfo{"", TYPE_INVALID, FLAGS_NONE, 0, nullptr}},
{OPD(TYPE_GROUP_9, PF_NONE, 1), 1, X86InstInfo{"CMPXCHG16B", TYPE_INST, FLAGS_MODRM | FLAGS_SF_MOD_DST | FLAGS_SF_MOD_MEM_ONLY, 0, nullptr}},
{OPD(TYPE_GROUP_9, PF_NONE, 1), 1, X86InstInfo{"CMPXCHG8B/16B", TYPE_INST, FLAGS_MODRM | FLAGS_SF_MOD_DST | FLAGS_SF_MOD_MEM_ONLY, 0, nullptr}},
{OPD(TYPE_GROUP_9, PF_NONE, 2), 1, X86InstInfo{"", TYPE_INVALID, FLAGS_NONE, 0, nullptr}},
{OPD(TYPE_GROUP_9, PF_NONE, 3), 1, X86InstInfo{"", TYPE_INVALID, FLAGS_NONE, 0, nullptr}},
{OPD(TYPE_GROUP_9, PF_NONE, 4), 1, X86InstInfo{"", TYPE_INVALID, FLAGS_NONE, 0, nullptr}},
{OPD(TYPE_GROUP_9, PF_NONE, 5), 1, X86InstInfo{"", TYPE_INVALID, FLAGS_NONE, 0, nullptr}},
{OPD(TYPE_GROUP_9, PF_NONE, 6), 1, X86InstInfo{"RDRAND", TYPE_UNDEC, FLAGS_MODRM | FLAGS_SF_MOD_DST | FLAGS_SF_MOD_REG_ONLY, 0, nullptr}},
{OPD(TYPE_GROUP_9, PF_NONE, 7), 1, X86InstInfo{"RDSEED", TYPE_UNDEC, FLAGS_MODRM | FLAGS_SF_MOD_DST | FLAGS_SF_MOD_REG_ONLY, 0, nullptr}},
{OPD(TYPE_GROUP_9, PF_NONE, 6), 1, X86InstInfo{"RDRAND", TYPE_INST, FLAGS_MODRM | FLAGS_SF_MOD_DST | FLAGS_SF_MOD_REG_ONLY, 0, nullptr}},
{OPD(TYPE_GROUP_9, PF_NONE, 7), 1, X86InstInfo{"RDSEED", TYPE_INST, FLAGS_MODRM | FLAGS_SF_MOD_DST | FLAGS_SF_MOD_REG_ONLY, 0, nullptr}},
{OPD(TYPE_GROUP_9, PF_F3, 0), 1, X86InstInfo{"", TYPE_INVALID, FLAGS_NONE, 0, nullptr}},
{OPD(TYPE_GROUP_9, PF_F3, 1), 1, X86InstInfo{"CMPXCHG16B", TYPE_INVALID, FLAGS_MODRM | FLAGS_SF_MOD_DST | FLAGS_SF_MOD_MEM_ONLY, 0, nullptr}},
{OPD(TYPE_GROUP_9, PF_F3, 1), 1, X86InstInfo{"CMPXCHG8B/16B", TYPE_INST, FLAGS_MODRM | FLAGS_SF_MOD_DST | FLAGS_SF_MOD_MEM_ONLY, 0, nullptr}},
{OPD(TYPE_GROUP_9, PF_F3, 2), 1, X86InstInfo{"", TYPE_INVALID, FLAGS_NONE, 0, nullptr}},
{OPD(TYPE_GROUP_9, PF_F3, 3), 1, X86InstInfo{"", TYPE_INVALID, FLAGS_NONE, 0, nullptr}},
{OPD(TYPE_GROUP_9, PF_F3, 4), 1, X86InstInfo{"", TYPE_INVALID, FLAGS_NONE, 0, nullptr}},
{OPD(TYPE_GROUP_9, PF_F3, 5), 1, X86InstInfo{"", TYPE_INVALID, FLAGS_NONE, 0, nullptr}},
{OPD(TYPE_GROUP_9, PF_F3, 6), 1, X86InstInfo{"", TYPE_INVALID, FLAGS_NONE, 0, nullptr}},
{OPD(TYPE_GROUP_9, PF_F3, 7), 1, X86InstInfo{"", TYPE_INVALID, FLAGS_NONE, 0, nullptr}},
{OPD(TYPE_GROUP_9, PF_F3, 7), 1, X86InstInfo{"RDPID", TYPE_INVALID, FLAGS_NONE, 0, nullptr}},
{OPD(TYPE_GROUP_9, PF_66, 0), 1, X86InstInfo{"", TYPE_INVALID, FLAGS_NONE, 0, nullptr}},
{OPD(TYPE_GROUP_9, PF_66, 1), 1, X86InstInfo{"CMPXCHG16B", TYPE_INVALID, FLAGS_MODRM | FLAGS_SF_MOD_DST | FLAGS_SF_MOD_MEM_ONLY, 0, nullptr}},
{OPD(TYPE_GROUP_9, PF_66, 1), 1, X86InstInfo{"CMPXCHG8B/16B", TYPE_INST, FLAGS_MODRM | FLAGS_SF_MOD_DST | FLAGS_SF_MOD_MEM_ONLY, 0, nullptr}},
{OPD(TYPE_GROUP_9, PF_66, 2), 1, X86InstInfo{"", TYPE_INVALID, FLAGS_NONE, 0, nullptr}},
{OPD(TYPE_GROUP_9, PF_66, 3), 1, X86InstInfo{"", TYPE_INVALID, FLAGS_NONE, 0, nullptr}},
{OPD(TYPE_GROUP_9, PF_66, 4), 1, X86InstInfo{"", TYPE_INVALID, FLAGS_NONE, 0, nullptr}},
{OPD(TYPE_GROUP_9, PF_66, 5), 1, X86InstInfo{"", TYPE_INVALID, FLAGS_NONE, 0, nullptr}},
{OPD(TYPE_GROUP_9, PF_66, 6), 1, X86InstInfo{"", TYPE_INVALID, FLAGS_NONE, 0, nullptr}},
{OPD(TYPE_GROUP_9, PF_66, 7), 1, X86InstInfo{"", TYPE_INVALID, FLAGS_NONE, 0, nullptr}},
{OPD(TYPE_GROUP_9, PF_66, 6), 1, X86InstInfo{"RDRAND", TYPE_INST, FLAGS_MODRM | FLAGS_SF_MOD_DST | FLAGS_SF_MOD_REG_ONLY, 0, nullptr}},
{OPD(TYPE_GROUP_9, PF_66, 7), 1, X86InstInfo{"RDSEED", TYPE_INST, FLAGS_MODRM | FLAGS_SF_MOD_DST | FLAGS_SF_MOD_REG_ONLY, 0, nullptr}},
{OPD(TYPE_GROUP_9, PF_F2, 0), 1, X86InstInfo{"", TYPE_INVALID, FLAGS_NONE, 0, nullptr}},
{OPD(TYPE_GROUP_9, PF_F2, 1), 1, X86InstInfo{"CMPXCHG16B", TYPE_INVALID, FLAGS_MODRM | FLAGS_SF_MOD_DST | FLAGS_SF_MOD_MEM_ONLY, 0, nullptr}},
{OPD(TYPE_GROUP_9, PF_F2, 1), 1, X86InstInfo{"CMPXCHG8B/16B", TYPE_INST, FLAGS_MODRM | FLAGS_SF_MOD_DST | FLAGS_SF_MOD_MEM_ONLY, 0, nullptr}},
{OPD(TYPE_GROUP_9, PF_F2, 2), 1, X86InstInfo{"", TYPE_INVALID, FLAGS_NONE, 0, nullptr}},
{OPD(TYPE_GROUP_9, PF_F2, 3), 1, X86InstInfo{"", TYPE_INVALID, FLAGS_NONE, 0, nullptr}},
{OPD(TYPE_GROUP_9, PF_F2, 4), 1, X86InstInfo{"", TYPE_INVALID, FLAGS_NONE, 0, nullptr}},
@@ -488,7 +488,7 @@ void InitializeSecondaryGroupTables() {
};
#undef OPD
GenerateTable(SecondInstGroupOps, SecondaryExtensionOpTable, std::size(SecondaryExtensionOpTable));
GenerateTable(&SecondInstGroupOps.at(0), SecondaryExtensionOpTable, std::size(SecondaryExtensionOpTable));
}
}
@@ -47,15 +47,15 @@ void InitializeSecondaryModRMTables() {
// REG /7
{((3 << 3) | 0), 1, X86InstInfo{"SWAPGS", TYPE_PRIV, FLAGS_NONE, 0, nullptr}},
{((3 << 3) | 1), 1, X86InstInfo{"RDTSCP", TYPE_PRIV, FLAGS_NONE, 0, nullptr}},
{((3 << 3) | 1), 1, X86InstInfo{"RDTSCP", TYPE_INST, FLAGS_NONE, 0, nullptr}},
{((3 << 3) | 2), 1, X86InstInfo{"MONITORX", TYPE_PRIV, FLAGS_NONE, 0, nullptr}},
{((3 << 3) | 3), 1, X86InstInfo{"MWAITX", TYPE_PRIV, FLAGS_NONE, 0, nullptr}},
{((3 << 3) | 4), 1, X86InstInfo{"", TYPE_INVALID, FLAGS_NONE, 0, nullptr}},
{((3 << 3) | 4), 1, X86InstInfo{"CLZERO", TYPE_INST, FLAGS_SF_SRC_RAX, 0, nullptr}},
{((3 << 3) | 5), 1, X86InstInfo{"", TYPE_INVALID, FLAGS_NONE, 0, nullptr}},
{((3 << 3) | 6), 1, X86InstInfo{"", TYPE_INVALID, FLAGS_NONE, 0, nullptr}},
{((3 << 3) | 7), 1, X86InstInfo{"", TYPE_INVALID, FLAGS_NONE, 0, nullptr}},
};
GenerateTable(SecondModRMTableOps, SecondaryModRMExtensionOpTable, std::size(SecondaryModRMExtensionOpTable));
GenerateTable(&SecondModRMTableOps.at(0), SecondaryModRMExtensionOpTable, std::size(SecondaryModRMExtensionOpTable));
}
}
@@ -120,9 +120,9 @@ void InitializeSecondaryTables(Context::OperatingMode Mode) {
{0x6F, 1, X86InstInfo{"MOVQ", TYPE_INST, GenFlagsSameSize(SIZE_64BIT) | FLAGS_MODRM | FLAGS_XMM_FLAGS | FLAGS_SF_MMX, 0, nullptr}},
{0x70, 1, X86InstInfo{"PSHUFW", TYPE_INST, GenFlagsSameSize(SIZE_64BIT) | FLAGS_MODRM | FLAGS_XMM_FLAGS | FLAGS_SF_MMX, 1, nullptr}},
{0x71, 1, X86InstInfo{"", TYPE_GROUP_12, FLAGS_NONE, 0, nullptr}},
{0x72, 1, X86InstInfo{"", TYPE_GROUP_13, FLAGS_NONE, 0, nullptr}},
{0x73, 1, X86InstInfo{"", TYPE_GROUP_14, FLAGS_NONE, 0, nullptr}},
{0x71, 1, X86InstInfo{"", TYPE_GROUP_12, FLAGS_NO_OVERLAY, 0, nullptr}},
{0x72, 1, X86InstInfo{"", TYPE_GROUP_13, FLAGS_NO_OVERLAY, 0, nullptr}},
{0x73, 1, X86InstInfo{"", TYPE_GROUP_14, FLAGS_NO_OVERLAY, 0, nullptr}},
{0x74, 1, X86InstInfo{"PCMPEQB", TYPE_INST, GenFlagsSameSize(SIZE_64BIT) | FLAGS_MODRM | FLAGS_XMM_FLAGS | FLAGS_SF_MMX, 0, nullptr}},
{0x75, 1, X86InstInfo{"PCMPEQW", TYPE_INST, GenFlagsSameSize(SIZE_64BIT) | FLAGS_MODRM | FLAGS_XMM_FLAGS | FLAGS_SF_MMX, 0, nullptr}},
{0x76, 1, X86InstInfo{"PCMPEQD", TYPE_INST, GenFlagsSameSize(SIZE_64BIT) | FLAGS_MODRM | FLAGS_XMM_FLAGS | FLAGS_SF_MMX, 0, nullptr}},
@@ -186,8 +186,8 @@ void InitializeSecondaryTables(Context::OperatingMode Mode) {
{0xB6, 1, X86InstInfo{"MOVZX", TYPE_INST, GenFlagsSrcSize(SIZE_8BIT) | FLAGS_MODRM | FLAGS_NO_OVERLAY, 0, nullptr}},
{0xB7, 1, X86InstInfo{"MOVZX", TYPE_INST, GenFlagsSrcSize(SIZE_16BIT) | FLAGS_MODRM | FLAGS_NO_OVERLAY, 0, nullptr}},
{0xB8, 1, X86InstInfo{"", TYPE_INVALID, FLAGS_NONE, 0, nullptr}},
{0xB9, 1, X86InstInfo{"", TYPE_GROUP_10, FLAGS_NONE, 0, nullptr}},
{0xBA, 1, X86InstInfo{"", TYPE_GROUP_8, FLAGS_NONE, 0, nullptr}},
{0xB9, 1, X86InstInfo{"", TYPE_GROUP_10, FLAGS_NO_OVERLAY, 0, nullptr}},
{0xBA, 1, X86InstInfo{"", TYPE_GROUP_8, FLAGS_NO_OVERLAY, 0, nullptr}},
{0xBB, 1, X86InstInfo{"BTC", TYPE_INST, FLAGS_DEBUG_MEM_ACCESS | FLAGS_MODRM | FLAGS_SF_MOD_DST | FLAGS_NO_OVERLAY, 0, nullptr}},
{0xBC, 1, X86InstInfo{"BSF", TYPE_INST, FLAGS_MODRM | FLAGS_NO_OVERLAY66, 0, nullptr}},
{0xBD, 1, X86InstInfo{"BSR", TYPE_INST, FLAGS_MODRM | FLAGS_NO_OVERLAY66, 0, nullptr}},
@@ -201,7 +201,7 @@ void InitializeSecondaryTables(Context::OperatingMode Mode) {
{0xC4, 1, X86InstInfo{"PINSRW", TYPE_INST, GenFlagsSizes(SIZE_64BIT, SIZE_16BIT) | FLAGS_MODRM | FLAGS_XMM_FLAGS | FLAGS_SF_MMX | FLAGS_SF_SRC_GPR, 1, nullptr}},
{0xC5, 1, X86InstInfo{"PEXTRW", TYPE_INST, GenFlagsSizes(SIZE_32BIT, SIZE_64BIT) | FLAGS_MODRM | FLAGS_SF_MOD_REG_ONLY | FLAGS_SF_DST_GPR | FLAGS_XMM_FLAGS | FLAGS_SF_MMX, 1, nullptr}},
{0xC6, 1, X86InstInfo{"SHUFPS", TYPE_INST, GenFlagsSameSize(SIZE_128BIT) | FLAGS_MODRM | FLAGS_XMM_FLAGS, 1, nullptr}},
{0xC7, 1, X86InstInfo{"", TYPE_GROUP_9, FLAGS_NONE, 0, nullptr}},
{0xC7, 1, X86InstInfo{"", TYPE_GROUP_9, FLAGS_NO_OVERLAY, 0, nullptr}},
{0xC8, 8, X86InstInfo{"BSWAP", TYPE_INST, FLAGS_SF_REX_IN_BYTE | FLAGS_NO_OVERLAY, 0, nullptr}},
{0xD0, 1, X86InstInfo{"", TYPE_INVALID, FLAGS_NONE, 0, nullptr}},
@@ -582,18 +582,18 @@ void InitializeSecondaryTables(Context::OperatingMode Mode) {
{0xFF, 1, X86InstInfo{"", TYPE_COPY_OTHER, FLAGS_NONE, 0, nullptr}},
};
GenerateTable(SecondBaseOps, TwoByteOpTable, std::size(TwoByteOpTable));
GenerateTable(&SecondBaseOps.at(0), TwoByteOpTable, std::size(TwoByteOpTable));
if (Mode == Context::MODE_64BIT) {
GenerateTable(SecondBaseOps, TwoByteOpTable_64, std::size(TwoByteOpTable_64));
if (Mode == Context::MODE_64BIT) {
GenerateTable(&SecondBaseOps.at(0), TwoByteOpTable_64, std::size(TwoByteOpTable_64));
}
else {
GenerateTable(SecondBaseOps, TwoByteOpTable_32, std::size(TwoByteOpTable_32));
GenerateTable(&SecondBaseOps.at(0), TwoByteOpTable_32, std::size(TwoByteOpTable_32));
}
GenerateTableWithCopy(RepModOps, RepModOpTable, std::size(RepModOpTable), SecondBaseOps);
GenerateTableWithCopy(RepNEModOps, RepNEModOpTable, std::size(RepNEModOpTable), SecondBaseOps);
GenerateTableWithCopy(OpSizeModOps, OpSizeModOpTable, std::size(OpSizeModOpTable), SecondBaseOps);
GenerateTableWithCopy(&RepModOps.at(0), RepModOpTable, std::size(RepModOpTable), &SecondBaseOps.at(0));
GenerateTableWithCopy(&RepNEModOps.at(0), RepNEModOpTable, std::size(RepNEModOpTable), &SecondBaseOps.at(0));
GenerateTableWithCopy(&OpSizeModOps.at(0), OpSizeModOpTable, std::size(OpSizeModOpTable), &SecondBaseOps.at(0));
}
@@ -394,8 +394,9 @@ void InitializeVEXTables() {
{OPD(2, 0b11, 0xF3), 1, X86InstInfo{"", TYPE_VEX_GROUP_17, FLAGS_NONE, 0, nullptr}}, // VEX Group 17
{OPD(2, 0b00, 0xF5), 1, X86InstInfo{"BZHI", TYPE_INST, FLAGS_MODRM | FLAGS_VEX_2ND_SRC, 0, nullptr}},
{OPD(2, 0b01, 0xF5), 1, X86InstInfo{"PEXT", TYPE_UNDEC, FLAGS_NONE, 0, nullptr}},
{OPD(2, 0b11, 0xF5), 1, X86InstInfo{"PDEP", TYPE_UNDEC, FLAGS_NONE, 0, nullptr}},
// AMD reference manual is incorrect. PEXT actually maps to 0b10, not 0b01.
{OPD(2, 0b10, 0xF5), 1, X86InstInfo{"PEXT", TYPE_INST, FLAGS_MODRM | FLAGS_VEX_1ST_SRC, 0, nullptr}},
{OPD(2, 0b11, 0xF5), 1, X86InstInfo{"PDEP", TYPE_INST, FLAGS_MODRM | FLAGS_VEX_1ST_SRC, 0, nullptr}},
{OPD(2, 0b11, 0xF6), 1, X86InstInfo{"MULX", TYPE_INST, FLAGS_MODRM | FLAGS_VEX_1ST_SRC, 0, nullptr}},
@@ -509,7 +510,7 @@ void InitializeVEXTables() {
};
#undef OPD
GenerateTable(VEXTableOps, VEXTable, std::size(VEXTable));
GenerateTable(VEXTableGroupOps, VEXGroupTable, std::size(VEXGroupTable));
GenerateTable(&VEXTableOps.at(0), VEXTable, std::size(VEXTable));
GenerateTable(&VEXTableGroupOps.at(0), VEXGroupTable, std::size(VEXGroupTable));
}
}
+14 -30
View File
@@ -17,20 +17,19 @@ extern uint64_t Total;
extern uint64_t NumInsts;
#endif
struct U8U8InfoStruct {
uint8_t first, second;
X86InstInfo Info;
};
struct U16U8InfoStruct {
uint16_t first;
template <typename OpcodeType>
struct X86TablesInfoStruct {
OpcodeType first;
uint8_t second;
X86InstInfo Info;
};
using U8U8InfoStruct = X86TablesInfoStruct<uint8_t>;
using U16U8InfoStruct = X86TablesInfoStruct<uint16_t>;
static inline void GenerateTable(X86InstInfo *FinalTable, U8U8InfoStruct const *LocalTable, size_t TableSize) {
template<typename OpcodeType>
static inline void GenerateTable(X86InstInfo *FinalTable, X86TablesInfoStruct<OpcodeType> const *LocalTable, size_t TableSize) {
for (size_t j = 0; j < TableSize; ++j) {
U8U8InfoStruct const &Op = LocalTable[j];
X86TablesInfoStruct<OpcodeType> const &Op = LocalTable[j];
auto OpNum = Op.first;
X86InstInfo const &Info = Op.Info;
for (uint32_t i = 0; i < Op.second; ++i) {
@@ -45,26 +44,10 @@ static inline void GenerateTable(X86InstInfo *FinalTable, U8U8InfoStruct const *
}
};
static inline void GenerateTable(X86InstInfo *FinalTable, U16U8InfoStruct const *LocalTable, size_t TableSize) {
template<typename OpcodeType>
static inline void GenerateTableWithCopy(X86InstInfo *FinalTable, X86TablesInfoStruct<OpcodeType> const *LocalTable, size_t TableSize, X86InstInfo *OtherLocal) {
for (size_t j = 0; j < TableSize; ++j) {
U16U8InfoStruct const &Op = LocalTable[j];
auto OpNum = Op.first;
X86InstInfo const &Info = Op.Info;
for (uint32_t i = 0; i < Op.second; ++i) {
LOGMAN_THROW_A_FMT(FinalTable[OpNum + i].Type == TYPE_UNKNOWN, "Duplicate Entry {}->{}", FinalTable[OpNum + i].Name, Info.Name);
FinalTable[OpNum + i] = Info;
#ifndef NDEBUG
++Total;
if (Info.Type == TYPE_INST)
NumInsts++;
#endif
}
}
};
static inline void GenerateTableWithCopy(X86InstInfo *FinalTable, U8U8InfoStruct const *LocalTable, size_t TableSize, X86InstInfo *OtherLocal) {
for (size_t j = 0; j < TableSize; ++j) {
U8U8InfoStruct const &Op = LocalTable[j];
X86TablesInfoStruct<OpcodeType> const &Op = LocalTable[j];
auto OpNum = Op.first;
X86InstInfo const &Info = Op.Info;
for (uint32_t i = 0; i < Op.second; ++i) {
@@ -84,9 +67,10 @@ static inline void GenerateTableWithCopy(X86InstInfo *FinalTable, U8U8InfoStruct
}
};
static inline void GenerateX87Table(X86InstInfo *FinalTable, U16U8InfoStruct const *LocalTable, size_t TableSize) {
template<typename OpcodeType>
static inline void GenerateX87Table(X86InstInfo *FinalTable, X86TablesInfoStruct<OpcodeType> const *LocalTable, size_t TableSize) {
for (size_t j = 0; j < TableSize; ++j) {
U16U8InfoStruct const &Op = LocalTable[j];
X86TablesInfoStruct<OpcodeType> const &Op = LocalTable[j];
auto OpNum = Op.first;
X86InstInfo const &Info = Op.Info;
for (uint32_t i = 0; i < Op.second; ++i) {
@@ -263,6 +263,6 @@ void InitializeX87Tables() {
#undef OPD
#undef OPDReg
GenerateX87Table(X87Ops, X87OpTable, std::size(X87OpTable));
GenerateX87Table(&X87Ops.at(0), X87OpTable, std::size(X87OpTable));
}
}
@@ -130,7 +130,7 @@ void InitializeXOPTables() {
};
#undef OPD
GenerateTable(XOPTableOps, XOPTable, std::size(XOPTable));
GenerateTable(XOPTableGroupOps, XOPGroupTable, std::size(XOPGroupTable));
GenerateTable(&XOPTableOps.at(0), XOPTable, std::size(XOPTable));
GenerateTable(&XOPTableGroupOps.at(0), XOPGroupTable, std::size(XOPGroupTable));
}
}
+408
View File
@@ -0,0 +1,408 @@
#include "Interface/Context/Context.h"
#include "Interface/IR/AOTIR.h"
#include <FEXCore/IR/IntrusiveIRList.h>
#include <FEXCore/IR/RegisterAllocationData.h>
#include <FEXCore/Utils/Allocator.h>
#include <cstddef>
#include <cstdint>
#include <filesystem>
#include <fstream>
#include <mutex>
#include <sys/mman.h>
#include <sys/stat.h>
#include <unistd.h>
#include <xxhash.h>
namespace FEXCore::IR {
AOTIRInlineEntry *AOTIRInlineIndex::GetInlineEntry(uint64_t DataOffset) {
uintptr_t This = (uintptr_t)this;
return (AOTIRInlineEntry*)(This + DataBase + DataOffset);
}
AOTIRInlineEntry *AOTIRInlineIndex::Find(uint64_t GuestStart) {
ssize_t l = 0;
ssize_t r = Count - 1;
while (l <= r) {
size_t m = l + (r - l) / 2;
if (Entries[m].GuestStart == GuestStart)
return GetInlineEntry(Entries[m].DataOffset);
else if (Entries[m].GuestStart < GuestStart)
l = m + 1;
else
r = m - 1;
}
return nullptr;
}
IR::RegisterAllocationData *AOTIRInlineEntry::GetRAData() {
return (IR::RegisterAllocationData *)InlineData;
}
IR::IRListView *AOTIRInlineEntry::GetIRData() {
auto RAData = GetRAData();
auto Offset = RAData->Size(RAData->MapCount);
return (IR::IRListView *)&InlineData[Offset];
}
void AOTIRCaptureCacheEntry::AppendAOTIRCaptureCache(uint64_t GuestRIP, uint64_t Start, uint64_t Length, uint64_t Hash, FEXCore::IR::IRListView *IRList, FEXCore::IR::RegisterAllocationData *RAData) {
auto Inserted = Index.emplace(GuestRIP, Stream->tellp());
if (Inserted.second) {
//GuestHash
Stream->write((const char*)&Hash, sizeof(Hash));
//GuestLength
Stream->write((const char*)&Length, sizeof(Length));
RAData->Serialize(*Stream);
// IRData (inline)
IRList->Serialize(*Stream);
}
}
static bool readAll(int fd, void *data, size_t size) {
int rv = read(fd, data, size);
if (rv != size)
return false;
else
return true;
}
bool LoadAOTIRCache(AOTCacheType *AOTIRCache, int streamfd) {
uint64_t tag;
if (!readAll(streamfd, (char*)&tag, sizeof(tag)) || tag != FEXCore::IR::AOTIR_COOKIE)
return false;
std::string Module;
uint64_t ModSize;
uint64_t IndexSize;
lseek(streamfd, -sizeof(ModSize), SEEK_END);
if (!readAll(streamfd, (char*)&ModSize, sizeof(ModSize)))
return false;
Module.resize(ModSize);
lseek(streamfd, -sizeof(ModSize) - ModSize, SEEK_END);
if (!readAll(streamfd, (char*)&Module[0], Module.size()))
return false;
lseek(streamfd, -sizeof(ModSize) - ModSize - sizeof(IndexSize), SEEK_END);
if (!readAll(streamfd, (char*)&IndexSize, sizeof(IndexSize)))
return false;
struct stat fileinfo;
if (fstat(streamfd, &fileinfo) < 0)
return false;
size_t Size = (fileinfo.st_size + 4095) & ~4095;
size_t IndexOffset = fileinfo.st_size - IndexSize -sizeof(ModSize) - ModSize - sizeof(IndexSize);
void *FilePtr = FEXCore::Allocator::mmap(nullptr, Size, PROT_READ, MAP_SHARED, streamfd, 0);
if (FilePtr == MAP_FAILED) {
return false;
}
auto Array = (AOTIRInlineIndex *)((char*)FilePtr + IndexOffset);
AOTIRCache->insert({Module, {Array, FilePtr, Size}});
LogMan::Msg::DFmt("AOTIR: Module {} has {} functions", Module, Array->Count);
return true;
}
AOTIRCaptureCache::~AOTIRCaptureCache() {
for (auto &Mod: AOTIRCache) {
FEXCore::Allocator::munmap(Mod.second.mapping, Mod.second.size);
}
}
void AOTIRCaptureCache::FinalizeAOTIRCache() {
AOTIRCaptureCacheWriteoutQueue_Flush();
std::unique_lock lk(AOTIRCacheLock);
for (auto& [String, Entry] : AOTIRCaptureCacheMap) {
if (!Entry.Stream) {
continue;
}
const auto ModSize = String.size();
auto &stream = Entry.Stream;
// pad to 32 bytes
constexpr char Zero = 0;
while(stream->tellp() & 31)
stream->write(&Zero, 1);
// AOTIRInlineIndex
const auto FnCount = Entry.Index.size();
const size_t DataBase = -stream->tellp();
stream->write((const char*)&FnCount, sizeof(FnCount));
stream->write((const char*)&DataBase, sizeof(DataBase));
for (const auto& [GuestStart, DataOffset] : Entry.Index) {
//AOTIRInlineIndexEntry
// GuestStart
stream->write((const char*)&GuestStart, sizeof(GuestStart));
// DataOffset
stream->write((const char*)&DataOffset, sizeof(DataOffset));
}
// End of file header
const auto IndexSize = FnCount * sizeof(FEXCore::IR::AOTIRInlineIndexEntry) + sizeof(DataBase) + sizeof(FnCount);
stream->write((const char*)&IndexSize, sizeof(IndexSize));
stream->write(String.c_str(), ModSize);
stream->write((const char*)&ModSize, sizeof(ModSize));
// Close the stream
stream->close();
// Rename the file to atomically update the cache with the temporary file
AOTIRRenamer(String);
}
}
void AOTIRCaptureCache::AOTIRCaptureCacheWriteoutQueue_Flush() {
{
std::shared_lock lk{AOTIRCaptureCacheWriteoutLock};
if (AOTIRCaptureCacheWriteoutQueue.size() == 0) {
AOTIRCaptureCacheWriteoutFlusing.store(false);
return;
}
}
for (;;) {
AOTIRCaptureCacheWriteoutLock.lock();
std::function<void()> fn = std::move(AOTIRCaptureCacheWriteoutQueue.front());
bool MaybeEmpty = false;
AOTIRCaptureCacheWriteoutQueue.pop();
MaybeEmpty = AOTIRCaptureCacheWriteoutQueue.size() == 0;
AOTIRCaptureCacheWriteoutLock.unlock();
fn();
if (MaybeEmpty) {
std::shared_lock lk{AOTIRCaptureCacheWriteoutLock};
if (AOTIRCaptureCacheWriteoutQueue.size() == 0) {
AOTIRCaptureCacheWriteoutFlusing.store(false);
return;
}
}
}
LOGMAN_MSG_A_FMT("Must never get here");
}
void AOTIRCaptureCache::AOTIRCaptureCacheWriteoutQueue_Append(const std::function<void()> &fn) {
bool Flush = false;
{
std::unique_lock lk{AOTIRCaptureCacheWriteoutLock};
AOTIRCaptureCacheWriteoutQueue.push(fn);
if (AOTIRCaptureCacheWriteoutQueue.size() > 10000) {
Flush = true;
}
}
bool test_val = false;
if (Flush && AOTIRCaptureCacheWriteoutFlusing.compare_exchange_strong(test_val, true)) {
AOTIRCaptureCacheWriteoutQueue_Flush();
}
}
void AOTIRCaptureCache::WriteFilesWithCode(std::function<void(const std::string& fileid, const std::string& filename)> Writer) {
std::shared_lock lk(AOTIRCacheLock);
for( const auto &File: FilesWithCode) {
Writer(File.first, File.second);
}
}
AOTIRCaptureCache::PreGenerateIRFetchResult AOTIRCaptureCache::PreGenerateIRFetch(uint64_t GuestRIP, FEXCore::IR::IRListView *IRList) {
{
std::shared_lock lk(AOTIRCacheLock);
auto file = AddrToFile.lower_bound(GuestRIP);
if (file != AddrToFile.begin()) {
--file;
if (!file->second.ContainsCode) {
file->second.ContainsCode = true;
FilesWithCode[file->second.fileid] = file->second.filename;
}
}
}
PreGenerateIRFetchResult Result{};
if (IRList == nullptr && CTX->Config.AOTIRLoad()) {
std::shared_lock lk(AOTIRCacheLock);
auto file = AddrToFile.lower_bound(GuestRIP);
if (file != AddrToFile.begin()) {
--file;
auto Mod = (FEXCore::IR::AOTIRInlineIndex*)file->second.CachedFileEntry;
if (Mod == nullptr) {
file->second.CachedFileEntry = Mod = AOTIRCache[file->second.fileid].Array;
}
if (Mod != nullptr)
{
auto AOTEntry = Mod->Find(GuestRIP - file->second.Start + file->second.Offset);
if (AOTEntry) {
// verify hash
auto MappedStart = GuestRIP;
auto hash = XXH3_64bits((void*)MappedStart, AOTEntry->GuestLength);
if (hash == AOTEntry->GuestHash) {
Result.IRList = AOTEntry->GetIRData();
//LogMan::Msg::DFmt("using {} + {:x} -> {:x}\n", file->second.fileid, AOTEntry->first, GuestRIP);
Result.RAData = AOTEntry->GetRAData();;
Result.DebugData = new FEXCore::Core::DebugData();
Result.StartAddr = MappedStart;
Result.Length = AOTEntry->GuestLength;
Result.GeneratedIR = true;
} else {
LogMan::Msg::IFmt("AOTIR: hash check failed {:x}\n", MappedStart);
}
} else {
//LogMan::Msg::IFmt("AOTIR: Failed to find {:x}, {:x}, {}\n", GuestRIP, GuestRIP - file->second.Start + file->second.Offset, file->second.fileid);
}
}
}
}
return Result;
}
bool AOTIRCaptureCache::PostCompileCode(
FEXCore::Core::InternalThreadState *Thread,
void* CodePtr,
uint64_t GuestRIP,
uint64_t StartAddr,
uint64_t Length,
FEXCore::IR::RegisterAllocationData *RAData,
FEXCore::IR::IRListView *IRList,
FEXCore::Core::DebugData *DebugData,
bool GeneratedIR,
bool DecrementRefCount) {
// Both generated ir and LibraryJITName need a named region lookup
if (GeneratedIR || CTX->Config.LibraryJITNaming()) {
std::shared_lock lk(AOTIRCacheLock);
auto file = FindAddrForFile(StartAddr, Length);
// Only go down this path if we actually found a library region
if (file != AddrToFile.end()) {
if (DebugData && CTX->Config.LibraryJITNaming()) {
CTX->Symbols.RegisterNamedRegion(CodePtr, DebugData->HostCodeSize, file->second.filename);
}
// Add to AOT cache if aot generation is enabled
if (GeneratedIR && RAData &&
(CTX->Config.AOTIRCapture() || CTX->Config.AOTIRGenerate())) {
auto hash = XXH3_64bits((void*)StartAddr, Length);
auto LocalRIP = GuestRIP - file->second.Start + file->second.Offset;
auto LocalStartAddr = StartAddr - file->second.Start + file->second.Offset;
auto fileid = file->second.fileid;
AOTIRCaptureCacheWriteoutQueue_Append([this, LocalRIP, LocalStartAddr, Length, hash, IRList, RAData, fileid]() {
auto *AotFile = &AOTIRCaptureCacheMap[fileid];
if (!AotFile->Stream) {
AotFile->Stream = AOTIRWriter(fileid);
uint64_t tag = FEXCore::IR::AOTIR_COOKIE;
AotFile->Stream->write((char*)&tag, sizeof(tag));
}
AotFile->AppendAOTIRCaptureCache(LocalRIP, LocalStartAddr, Length, hash, IRList, RAData);
});
if (CTX->Config.AOTIRGenerate()) {
// cleanup memory and early exit here -- we're not running the application
if (DecrementRefCount) {
--Thread->CompileBlockReentrantRefCount;
}
Thread->CPUBackend->ClearCache();
return true;
}
}
}
// Insert to caches if we generated IR
if (GeneratedIR) {
// Add to thread local ir cache
Core::LocalIREntry Entry = {StartAddr, Length, decltype(Entry.IR)(IRList), decltype(Entry.RAData)(RAData), decltype(Entry.DebugData)(DebugData)};
Thread->LocalIRCache.insert({GuestRIP, std::move(Entry)});
}
}
return false;
}
AOTIRCaptureCache::AddrToFileMapType::iterator AOTIRCaptureCache::FindAddrForFile(uint64_t Entry, uint64_t Length) {
// Thread safety here! We are returning an iterator to the map object
// This needs the AOTIRCacheLock locked prior to coming in to the function
auto file = AddrToFile.lower_bound(Entry);
if (file != AddrToFile.begin()) {
--file;
if (file->second.Start <= Entry && (file->second.Start + file->second.Len) >= (Entry + Length)) {
return file;
}
}
return AddrToFile.end();
}
void AOTIRCaptureCache::AddNamedRegion(uintptr_t Base, uintptr_t Size, uintptr_t Offset, const std::string &filename) {
// TODO: Support overlapping maps and region splitting
auto base_filename = std::filesystem::path(filename).filename().string();
if (!base_filename.empty()) {
auto filename_hash = XXH3_64bits(filename.c_str(), filename.size());
auto fileid = base_filename + "-" + std::to_string(filename_hash) + "-";
// append optimization flags to the fileid
fileid += (CTX->Config.SMCChecks == FEXCore::Config::CONFIG_SMC_FULL) ? "S" : "s";
fileid += CTX->Config.TSOEnabled ? "T" : "t";
fileid += CTX->Config.ABILocalFlags ? "L" : "l";
fileid += CTX->Config.ABINoPF ? "p" : "P";
std::unique_lock lk(AOTIRCacheLock);
AddrToFile.insert({ Base, { Base, Size, Offset, fileid, filename, nullptr, false} });
if (CTX->Config.AOTIRLoad && !AOTIRCache.contains(fileid) && AOTIRLoader) {
auto streamfd = AOTIRLoader(fileid);
if (streamfd != -1) {
FEXCore::IR::LoadAOTIRCache(&AOTIRCache, streamfd);
close(streamfd);
}
}
}
}
void AOTIRCaptureCache::RemoveNamedRegion(uintptr_t Base, uintptr_t Size) {
std::unique_lock lk(AOTIRCacheLock);
// TODO: Support partial removing
AddrToFile.erase(Base);
}
}
+162
View File
@@ -0,0 +1,162 @@
#pragma once
#include <FEXCore/Config/Config.h>
#include <atomic>
#include <cstdint>
#include <functional>
#include <fstream>
#include <memory>
#include <map>
#include <unordered_map>
#include <shared_mutex>
#include <queue>
namespace FEXCore::Core {
struct DebugData;
}
namespace FEXCore::IR {
class RegisterAllocationData;
class IRListView;
constexpr auto COOKIE_VERSION = [](const char CookieText[4], uint32_t Version) {
uint64_t Cookie = Version;
Cookie <<= 32;
// Make the cookie text be the lower bits
Cookie |= CookieText[3];
Cookie <<= 8;
Cookie |= CookieText[2];
Cookie <<= 8;
Cookie |= CookieText[1];
Cookie <<= 8;
Cookie |= CookieText[0];
return Cookie;
};
constexpr static uint32_t AOTIR_VERSION = 0x0000'00004;
constexpr static uint64_t AOTIR_COOKIE = COOKIE_VERSION("FEXI", AOTIR_VERSION);
struct AOTIRInlineEntry {
uint64_t GuestHash;
uint64_t GuestLength;
/* RAData followed by IRData */
uint8_t InlineData[0];
IR::RegisterAllocationData *GetRAData();
IR::IRListView *GetIRData();
};
struct AOTIRInlineIndexEntry {
uint64_t GuestStart;
uint64_t DataOffset;
};
struct AOTIRInlineIndex {
uint64_t Count;
uint64_t DataBase;
AOTIRInlineIndexEntry Entries[0];
AOTIRInlineEntry *Find(uint64_t GuestStart);
AOTIRInlineEntry *GetInlineEntry(uint64_t DataOffset);
};
struct AOTIRCaptureCacheEntry {
std::unique_ptr<std::ofstream> Stream;
std::map<uint64_t, uint64_t> Index;
void AppendAOTIRCaptureCache(uint64_t GuestRIP, uint64_t Start, uint64_t Length, uint64_t Hash, FEXCore::IR::IRListView *IRList, FEXCore::IR::RegisterAllocationData *RAData);
};
struct AOTIRCacheEntry {
AOTIRInlineIndex *Array;
void *mapping;
size_t size;
};
using AOTCacheType = std::unordered_map<std::string, FEXCore::IR::AOTIRCacheEntry>;
bool LoadAOTIRCache(AOTCacheType *AOTIRCache, int streamfd);
class AOTIRCaptureCache final {
public:
AOTIRCaptureCache(FEXCore::Context::Context *ctx) : CTX {ctx} {}
~AOTIRCaptureCache();
void FinalizeAOTIRCache();
void AOTIRCaptureCacheWriteoutQueue_Flush();
void AOTIRCaptureCacheWriteoutQueue_Append(const std::function<void()> &fn);
void WriteFilesWithCode(std::function<void(const std::string& fileid, const std::string& filename)> Writer);
struct PreGenerateIRFetchResult {
FEXCore::IR::IRListView *IRList {};
FEXCore::IR::RegisterAllocationData *RAData {};
FEXCore::Core::DebugData *DebugData {};
uint64_t StartAddr {};
uint64_t Length {};
bool GeneratedIR {};
};
[[nodiscard]] PreGenerateIRFetchResult PreGenerateIRFetch(uint64_t GuestRIP, FEXCore::IR::IRListView *IRList);
bool PostCompileCode(FEXCore::Core::InternalThreadState *Thread,
void* CodePtr,
uint64_t GuestRIP,
uint64_t StartAddr,
uint64_t Length,
FEXCore::IR::RegisterAllocationData *RAData,
FEXCore::IR::IRListView *IRList,
FEXCore::Core::DebugData *DebugData,
bool GeneratedIR,
bool DecrementRefCount);
void AddNamedRegion(uintptr_t Base, uintptr_t Size, uintptr_t Offset, const std::string &filename);
void RemoveNamedRegion(uintptr_t Base, uintptr_t Size);
// Callbacks
void SetAOTIRLoader(std::function<int(const std::string&)> CacheReader) {
AOTIRLoader = CacheReader;
}
void SetAOTIRWriter(std::function<std::unique_ptr<std::ofstream>(const std::string&)> CacheWriter) {
AOTIRWriter = CacheWriter;
}
void SetAOTIRRenamer(std::function<void(const std::string&)> CacheRenamer) {
AOTIRRenamer = CacheRenamer;
}
private:
FEXCore::Context::Context *CTX;
std::shared_mutex AOTIRCacheLock;
std::shared_mutex AOTIRCaptureCacheWriteoutLock;
std::atomic<bool> AOTIRCaptureCacheWriteoutFlusing;
std::queue<std::function<void()>> AOTIRCaptureCacheWriteoutQueue;
std::map<std::string, std::string> FilesWithCode;
struct AddrToFileEntry {
uint64_t Start;
uint64_t Len;
uint64_t Offset;
std::string fileid;
std::string filename;
void *CachedFileEntry;
bool ContainsCode;
};
using AddrToFileMapType = std::map<uint64_t, AddrToFileEntry>;
AddrToFileMapType AddrToFile;
FEXCore::IR::AOTCacheType AOTIRCache;
std::function<int(const std::string&)> AOTIRLoader;
std::function<std::unique_ptr<std::ofstream>(const std::string&)> AOTIRWriter;
std::function<void(const std::string&)> AOTIRRenamer;
std::unordered_map<std::string, FEXCore::IR::AOTIRCaptureCacheEntry> AOTIRCaptureCacheMap;
AddrToFileMapType::iterator FindAddrForFile(uint64_t Entry, uint64_t Length);
};
}
File diff suppressed because it is too large. Load diff
+15 -2
View File
@@ -171,6 +171,19 @@ static void PrintArg(std::stringstream *out, [[maybe_unused]] IRListView const*
}
}
static void PrintArg(std::stringstream *out, [[maybe_unused]] IRListView const* IR, FEXCore::IR::SyscallFlags Arg) {
switch (Arg) {
case FEXCore::IR::SyscallFlags::DEFAULT: *out << "Default"; break;
case FEXCore::IR::SyscallFlags::OPTIMIZETHROUGH: *out << "Optimize Through"; break;
case FEXCore::IR::SyscallFlags::NOSYNCSTATEONENTRY: *out << "No Sync State on Entry"; break;
case FEXCore::IR::SyscallFlags::NORETURN: *out << "No Return"; break;
case FEXCore::IR::SyscallFlags::NOSIDEEFFECTS: *out << "No Side Effects"; break;
default: *out << "<Unknown Round Type>"; break;
}
}
void Dump(std::stringstream *out, IRListView const* IR, IR::RegisterAllocationData *RAData) {
auto HeaderOp = IR->GetHeader();
@@ -225,7 +238,7 @@ void Dump(std::stringstream *out, IRListView const* IR, IR::RegisterAllocationDa
NumElements /= ElementSize;
}
*out << "%ssa" << ID;
*out << "%ssa" << std::dec << ID;
if (RAData) {
auto PhyReg = RAData->GetNodeRegister(ID);
@@ -265,7 +278,7 @@ void Dump(std::stringstream *out, IRListView const* IR, IR::RegisterAllocationDa
NumElements = IROp->Size / ElementSize;
}
*out << "(%ssa" << ID << ' ';
*out << "(%ssa" << std::dec << ID << ' ';
*out << 'i' << std::dec << (ElementSize * 8);
if (NumElements > 1) {
*out << 'v' << std::dec << NumElements;
+56
View File
@@ -16,6 +16,62 @@ $end_info$
#include <vector>
namespace FEXCore::IR {
FEXCore::IR::RegisterClassType IREmitter::WalkFindRegClass(OrderedNode *Node) {
auto Class = GetOpRegClass(Node);
switch (Class) {
case GPRClass:
case GPRPairClass:
case FPRClass:
case GPRFixedClass:
case FPRFixedClass:
case InvalidClass:
return Class;
default: break;
}
// Complex case, needs to be handled on an op by op basis
uintptr_t DataBegin = DualListData.DataBegin();
FEXCore::IR::IROp_Header *IROp = Node->Op(DataBegin);
switch (IROp->Op) {
case IROps::OP_LOADREGISTER: {
auto Op = IROp->C<IROp_LoadRegister>();
return Op->Class;
break;
}
case IROps::OP_LOADCONTEXT: {
auto Op = IROp->C<IROp_LoadContext>();
return Op->Class;
break;
}
case IROps::OP_LOADCONTEXTINDEXED: {
auto Op = IROp->C<IROp_LoadContextIndexed>();
return Op->Class;
break;
}
case IROps::OP_FILLREGISTER: {
auto Op = IROp->C<IROp_FillRegister>();
return Op->Class;
break;
}
case IROps::OP_LOADMEM: {
auto Op = IROp->C<IROp_LoadMem>();
return Op->Class;
break;
}
case IROps::OP_LOADMEMTSO: {
auto Op = IROp->C<IROp_LoadMemTSO>();
return Op->Class;
break;
}
default:
LOGMAN_MSG_A_FMT("Unhandled op type: {} {} in argument class validation", IROp->Op, GetOpName(Node));
break;
}
return InvalidClass;
}
void IREmitter::ResetWorkingList() {
DualListData.Reset();
CodeBlocks.clear();
+13 -5
View File
@@ -170,7 +170,7 @@ class IRParser: public FEXCore::IR::IREmitter {
}
Result.data[i] = high * 16 + low;
}
return {DecodeFailure::DECODE_OKAY, Result};
}
@@ -351,10 +351,6 @@ class IRParser: public FEXCore::IR::IREmitter {
bool Loaded = false;
#define IROP_PARSER_ALLOCATE_HELPERS
#include <FEXCore/IR/IRDefines.inc>
bool Parse() {
const auto CheckPrintError = [&](const LineDefinition &Def, DecodeFailure Failure) -> bool {
if (Failure != DecodeFailure::DECODE_OKAY) {
@@ -367,6 +363,18 @@ class IRParser: public FEXCore::IR::IREmitter {
return true;
};
const auto CheckPrintErrorArg = [&](const LineDefinition &Def, DecodeFailure Failure, size_t Arg) -> bool {
if (Failure != DecodeFailure::DECODE_OKAY) {
LogMan::Msg::EFmt("Error on Line: {}", Def.LineNumber);
LogMan::Msg::EFmt("{}", Lines[Def.LineNumber]);
LogMan::Msg::EFmt("Argument Number {}: {}", Arg + 1, Def.Args[Arg]);
LogMan::Msg::EFmt("Value Couldn't be decoded due to {}", DecodeErrorToString(Failure));
return false;
}
return true;
};
// String parse every line for our definitions
for (size_t i = 0; i < Lines.size(); ++i) {
std::string Line = Lines[i];
+39 -28
View File
@@ -286,7 +286,7 @@ void ConstProp::FCMPOptimization(IREmitter *IREmit, const IRListView& CurrentIR)
if (IROp->Op == OP_GETHOSTFLAG) {
auto ghf = IROp->CW<IR::IROp_GetHostFlag>();
auto fcmp = IREmit->GetOpHeader(ghf->GPR)->CW<IR::IROp_FCmp>();
auto fcmp = IREmit->GetOpHeader(ghf->Value)->CW<IR::IROp_FCmp>();
LOGMAN_THROW_A_FMT(fcmp->Header.Op == OP_FCMP || fcmp->Header.Op == OP_F80CMP, "Unexpected OP_GETHOSTFLAG source");
if(fcmp->Header.Op == OP_FCMP) {
fcmp->Flags |= 1 << ghf->Flag;
@@ -301,18 +301,29 @@ void ConstProp::LoadMemStoreMemImmediatePooling(IREmitter *IREmit, const IRListV
for (auto [BlockNode, BlockIROp] : CurrentIR.GetBlocks()) {
for (auto [CodeNode, IROp] : CurrentIR.GetCode(BlockNode)) {
if (IROp->Op == OP_LOADMEM || IROp->Op == OP_STOREMEM) {
size_t AddrIndex = 0;
size_t OffsetIndex = 0;
if (IROp->Op == OP_LOADMEM) {
AddrIndex = IR::IROp_LoadMem::Addr_Index;
OffsetIndex = IR::IROp_LoadMem::Offset_Index;
}
else {
AddrIndex = IR::IROp_StoreMem::Addr_Index;
OffsetIndex = IR::IROp_StoreMem::Offset_Index;
}
uint64_t Addr;
if (IREmit->IsValueConstant(IROp->Args[0], &Addr) && IROp->Args[1].IsInvalid()) {
if (IREmit->IsValueConstant(IROp->Args[AddrIndex], &Addr) && IROp->Args[OffsetIndex].IsInvalid()) {
for (auto& Const: AddressgenConsts) {
if ((Addr - Const.second) < 65536) {
IREmit->ReplaceNodeArgument(CodeNode, 0, Const.first);
IREmit->ReplaceNodeArgument(CodeNode, 1, IREmit->_Constant(Addr - Const.second));
IREmit->ReplaceNodeArgument(CodeNode, AddrIndex, Const.first);
IREmit->ReplaceNodeArgument(CodeNode, OffsetIndex, IREmit->_Constant(Addr - Const.second));
goto doneOp;
}
}
AddressgenConsts[IREmit->UnwrapNode(IROp->Args[0])] = Addr;
AddressgenConsts[IREmit->UnwrapNode(IROp->Args[AddrIndex])] = Addr;
}
doneOp:
;
@@ -374,8 +385,8 @@ bool ConstProp::ZextAndMaskingElimination(IREmitter *IREmit, const IRListView& C
auto Op = IROp->C<IR::IROp_Bfe>();
// Is this value already BFE'd?
if (IsBfeAlreadyDone(IREmit, IROp->Args[0], Op->Width)) {
IREmit->ReplaceAllUsesWith(CodeNode, CurrentIR.GetNode(IROp->Args[0]));
if (IsBfeAlreadyDone(IREmit, Op->Src, Op->Width)) {
IREmit->ReplaceAllUsesWith(CodeNode, CurrentIR.GetNode(Op->Src));
//printf("Removed BFE once \n");
break;
}
@@ -383,7 +394,7 @@ bool ConstProp::ZextAndMaskingElimination(IREmitter *IREmit, const IRListView& C
// Is this value already ZEXT'd?
if (Op->lsb == 0) {
//LoadMem, LoadMemTSO & LoadContext ZExt
auto source = IROp->Args[0];
auto source = Op->Src;
auto sourceHeader = IREmit->GetOpHeader(source);
if (Op->Width >= (sourceHeader->Size*8) &&
@@ -401,10 +412,10 @@ bool ConstProp::ZextAndMaskingElimination(IREmitter *IREmit, const IRListView& C
imm = (imm-1) *2 + 1;
imm <<= Op->lsb;
auto newArg = RemoveUselessMasking(IREmit, IROp->Args[0], imm);
auto newArg = RemoveUselessMasking(IREmit, Op->Src, imm);
if (newArg.ID() != IROp->Args[0].ID()) {
IREmit->ReplaceNodeArgument(CodeNode, 0, IREmit->UnwrapNode(newArg));
if (newArg.ID() != Op->Src.ID()) {
IREmit->ReplaceNodeArgument(CodeNode, Op->Src_Index, IREmit->UnwrapNode(newArg));
Changed = true;
}
break;
@@ -418,10 +429,10 @@ bool ConstProp::ZextAndMaskingElimination(IREmitter *IREmit, const IRListView& C
imm = (imm-1) *2 + 1;
imm <<= Op->lsb;
auto newArg = RemoveUselessMasking(IREmit, IROp->Args[0], imm);
auto newArg = RemoveUselessMasking(IREmit, Op->Src, imm);
if (newArg.ID() != IROp->Args[0].ID()) {
IREmit->ReplaceNodeArgument(CodeNode, 0, IREmit->UnwrapNode(newArg));
if (newArg.ID() != Op->Src.ID()) {
IREmit->ReplaceNodeArgument(CodeNode, Op->Src_Index, IREmit->UnwrapNode(newArg));
Changed = true;
}
break;
@@ -528,15 +539,15 @@ bool ConstProp::ConstantPropagation(IREmitter *IREmit, const IRListView& Current
case OP_LOADMEM: {
auto Op = IROp->CW<IR::IROp_LoadMem>();
auto AddressHeader = IREmit->GetOpHeader(Op->Header.Args[0]);
auto AddressHeader = IREmit->GetOpHeader(Op->Addr);
if (AddressHeader->Op == OP_ADD && AddressHeader->Size == 8) {
auto [OffsetType, OffsetScale, Arg0, Arg1] = MemExtendedAddressing(IREmit, IROp->Size, AddressHeader);
Op->OffsetType = OffsetType;
Op->OffsetScale = OffsetScale;
IREmit->ReplaceNodeArgument(CodeNode, 0, Arg0);
IREmit->ReplaceNodeArgument(CodeNode, 1, Arg1);
IREmit->ReplaceNodeArgument(CodeNode, Op->Addr_Index, Arg0); // Addr
IREmit->ReplaceNodeArgument(CodeNode, Op->Offset_Index, Arg1); // Offset
Changed = true;
}
@@ -545,15 +556,15 @@ bool ConstProp::ConstantPropagation(IREmitter *IREmit, const IRListView& Current
case OP_STOREMEM: {
auto Op = IROp->CW<IR::IROp_StoreMem>();
auto AddressHeader = IREmit->GetOpHeader(Op->Header.Args[0]);
auto AddressHeader = IREmit->GetOpHeader(Op->Addr);
if (AddressHeader->Op == OP_ADD && AddressHeader->Size == 8) {
auto [OffsetType, OffsetScale, Arg0, Arg1] = MemExtendedAddressing(IREmit, IROp->Size, AddressHeader);
Op->OffsetType = OffsetType;
Op->OffsetScale = OffsetScale;
IREmit->ReplaceNodeArgument(CodeNode, 0, Arg0);
IREmit->ReplaceNodeArgument(CodeNode, 2, Arg1);
IREmit->ReplaceNodeArgument(CodeNode, Op->Addr_Index, Arg0); // Addr
IREmit->ReplaceNodeArgument(CodeNode, Op->Offset_Index, Arg1); // Offset
Changed = true;
}
@@ -707,7 +718,7 @@ bool ConstProp::ConstantPropagation(IREmitter *IREmit, const IRListView& Current
case OP_BFE: {
auto Op = IROp->C<IR::IROp_Bfe>();
uint64_t Constant;
if (IROp->Size <= 8 && IREmit->IsValueConstant(Op->Header.Args[0], &Constant)) {
if (IROp->Size <= 8 && IREmit->IsValueConstant(Op->Src, &Constant)) {
uint64_t SourceMask = (1ULL << Op->Width) - 1;
if (Op->Width == 64)
SourceMask = ~0ULL;
@@ -897,7 +908,7 @@ bool ConstProp::ConstantInlining(IREmitter *IREmit, const IRListView& CurrentIR)
uint64_t Constant{};
if (IREmit->IsValueConstant(Op->NewRIP, &Constant)) {
IREmit->SetWriteCursor(CurrentIR.GetNode(Op->NewRIP));
IREmit->ReplaceNodeArgument(CodeNode, 0, IREmit->_InlineConstant(Constant));
@@ -940,11 +951,11 @@ bool ConstProp::ConstantInlining(IREmitter *IREmit, const IRListView& CurrentIR)
auto Op = IROp->CW<IR::IROp_LoadMem>();
uint64_t Constant2{};
if (Op->OffsetType == MEM_OFFSET_SXTX && IREmit->IsValueConstant(Op->Header.Args[1], &Constant2)) {
if (Op->OffsetType == MEM_OFFSET_SXTX && IREmit->IsValueConstant(Op->Offset, &Constant2)) {
if (IsImmMemory(Constant2, IROp->Size)) {
IREmit->SetWriteCursor(CurrentIR.GetNode(Op->Header.Args[1]));
IREmit->SetWriteCursor(CurrentIR.GetNode(Op->Offset));
IREmit->ReplaceNodeArgument(CodeNode, 1, IREmit->_InlineConstant(Constant2));
IREmit->ReplaceNodeArgument(CodeNode, Op->Offset_Index, IREmit->_InlineConstant(Constant2));
Changed = true;
}
@@ -957,11 +968,11 @@ bool ConstProp::ConstantInlining(IREmitter *IREmit, const IRListView& CurrentIR)
auto Op = IROp->CW<IR::IROp_StoreMem>();
uint64_t Constant2{};
if (Op->OffsetType == MEM_OFFSET_SXTX && IREmit->IsValueConstant(Op->Header.Args[2], &Constant2)) {
if (Op->OffsetType == MEM_OFFSET_SXTX && IREmit->IsValueConstant(Op->Offset, &Constant2)) {
if (IsImmMemory(Constant2, IROp->Size)) {
IREmit->SetWriteCursor(CurrentIR.GetNode(Op->Header.Args[2]));
IREmit->SetWriteCursor(CurrentIR.GetNode(Op->Offset));
IREmit->ReplaceNodeArgument(CodeNode, 2, IREmit->_InlineConstant(Constant2));
IREmit->ReplaceNodeArgument(CodeNode, Op->Offset_Index, IREmit->_InlineConstant(Constant2));
Changed = true;
}
@@ -37,9 +37,27 @@ bool DeadCodeElimination::Run(IREmitter *IREmit) {
while (1) {
auto [CodeNode, IROp] = CodeLast();
bool HasSideEffects = IR::HasSideEffects(IROp->Op);
if (IROp->Op == OP_SYSCALL ||
IROp->Op == OP_INLINESYSCALL) {
FEXCore::IR::SyscallFlags Flags{};
if (IROp->Op == OP_SYSCALL) {
auto Op = IROp->C<IR::IROp_Syscall>();
Flags = Op->Flags;
}
else {
auto Op = IROp->C<IR::IROp_InlineSyscall>();
Flags = Op->Flags;
}
if ((Flags & FEXCore::IR::SyscallFlags::NOSIDEEFFECTS) == FEXCore::IR::SyscallFlags::NOSIDEEFFECTS) {
HasSideEffects = false;
}
}
// Skip over anything that has side effects
// Use count tracking can't safely remove anything with side effects
if (!IR::HasSideEffects(IROp->Op)) {
if (!HasSideEffects) {
if (CodeNode->GetUses() == 0) {
NumRemoved++;
IREmit->Remove(CodeNode);
@@ -47,10 +47,12 @@ namespace {
return (Type & ACCESS_TYPE_MASK) == ACCESS_READ;
}
[[maybe_unused]]
static bool IsInvalidAccess(LastAccessType Type) {
return (Type & ACCESS_TYPE_MASK) == ACCESS_INVALID;
}
[[maybe_unused]]
static bool IsPartialAccess(LastAccessType Type) {
return (Type & ACCESS_PARTIAL) == ACCESS_PARTIAL;
}
@@ -254,7 +256,7 @@ namespace {
});
size_t ClassifiedStructSize{};
[[maybe_unused]] size_t ClassifiedStructSize{};
ContextClassificationInfo->Lookup.reserve(sizeof(FEXCore::Core::CPUState));
for (auto &it : *ContextClassification) {
LOGMAN_THROW_A_FMT(it.Class.Offset == ContextClassificationInfo->Lookup.size(), "Offset mismatch (offset={})", it.Class.Offset);
@@ -559,7 +561,7 @@ bool RCLSE::RedundantStoreLoadElimination(FEXCore::IR::IREmitter *IREmit) {
// the vector element
IREmit->SetWriteCursor(CodeNode);
// zext to size
LastNode = IREmit->_VMov(LastNode, IROp->Size);
LastNode = IREmit->_VMov(IROp->Size, LastNode);
IREmit->ReplaceAllUsesWithRange(CodeNode, LastNode, IREmit->GetIterator(IREmit->WrapNode(CodeNode)), BlockEnd);
RecordAccess(Info, Op->Class, Op->Offset, IROp->Size, ACCESS_READ, LastNode);
@@ -575,7 +577,7 @@ bool RCLSE::RedundantStoreLoadElimination(FEXCore::IR::IREmitter *IREmit) {
IROp->Size < IREmit->GetOpSize(LastNode)) {
IREmit->SetWriteCursor(CodeNode);
// trucate to size
LastNode = IREmit->_VMov(LastNode, IROp->Size);
LastNode = IREmit->_VMov(IROp->Size, LastNode);
IREmit->ReplaceAllUsesWithRange(CodeNode, LastNode, IREmit->GetIterator(IREmit->WrapNode(CodeNode)), BlockEnd);
RecordAccess(Info, Op->Class, Op->Offset, IROp->Size, ACCESS_READ, LastNode);
Changed = true;
@@ -583,7 +585,7 @@ bool RCLSE::RedundantStoreLoadElimination(FEXCore::IR::IREmitter *IREmit) {
IROp->Size > IREmit->GetOpSize(LastNode)) {
IREmit->SetWriteCursor(CodeNode);
// zext to size
LastNode = IREmit->_VMov(LastNode, IROp->Size);
LastNode = IREmit->_VMov(IROp->Size, LastNode);
IREmit->ReplaceAllUsesWithRange(CodeNode, LastNode, IREmit->GetIterator(IREmit->WrapNode(CodeNode)), BlockEnd);
RecordAccess(Info, Op->Class, Op->Offset, IROp->Size, ACCESS_READ, LastNode);
@@ -659,10 +661,25 @@ bool RCLSE::RedundantStoreLoadElimination(FEXCore::IR::IREmitter *IREmit) {
Changed = true;
}
}
else if (IROp->Op == OP_SYSCALL ||
IROp->Op == OP_INLINESYSCALL) {
FEXCore::IR::SyscallFlags Flags{};
if (IROp->Op == OP_SYSCALL) {
auto Op = IROp->C<IR::IROp_Syscall>();
Flags = Op->Flags;
}
else {
auto Op = IROp->C<IR::IROp_InlineSyscall>();
Flags = Op->Flags;
}
if ((Flags & FEXCore::IR::SyscallFlags::OPTIMIZETHROUGH) != FEXCore::IR::SyscallFlags::OPTIMIZETHROUGH) {
// We can't track through these
ResetClassificationAccesses(&LocalInfo);
}
}
else if (IROp->Op == OP_STORECONTEXTINDEXED ||
IROp->Op == OP_LOADCONTEXTINDEXED ||
IROp->Op == OP_SYSCALL ||
IROp->Op == OP_INLINESYSCALL ||
IROp->Op == OP_BREAK) {
// We can't track through these
ResetClassificationAccesses(&LocalInfo);
@@ -77,7 +77,7 @@ struct FPRInfo {
bool IsFPR(uint32_t Offset) {
auto begin = offsetof(FEXCore::Core::CpuStateFrame, State.xmm[0][0]);
auto end = offsetof(FEXCore::Core::CpuStateFrame, State.xmm[17][0]);
auto end = offsetof(FEXCore::Core::CpuStateFrame, State.xmm[16][0]);
if (Offset < begin || Offset >= end)
return false;
Loaded 100 of 440 files, more files were not shown because too many files have changed in this diff. Show more