Ryan Houdek 2536bfe42c Correctly fixes prefix ordering in the frontend ...

This was sort of partially worked around.
The issue comes down to REX.W and Operand size prefix ordering on an
instruction.
When an instruction has both prefixes, then the one that has occured
later will overwrite the previous one.
Additionally in the case of 66h being reused as an escape prefix, we
needed to ignore it in this case.

The previous workaround was that if an instruction happened to have both
widening and narrowing, then widening would win(for MODRM args),
but then later on in the instruction decoding the literal constant
decoding would properly check which order the flags came in.
This resulted in literal constants working, but not modrm bytes

Instead replace the "last" flag handling with a two deep stack that we
can push and pop the flags correctly on to it, this ensures that the
flags stay around in the 66h ignoring mode (so widening doesn't get
obliterated).

This effects every instruction that supports REX.W and 66h Operand size
prefix, so it could be fixing some unknown bugs elsewhere.
Very specifically this one was found in OpenSSL, because it looks like
GCC is padding a mov instruction with 66h.

Instruction Examples:
Only difference is the prefix order being swapped
0x00000000   4                 66488b1f  mov rbx, qword [rdi]
0x00000004   4                 48668b1f  mov bx, word [rdi]

2020-07-15 00:10:20 -07:00

..

FEXCore

Correctly fixes prefix ordering in the frontend

2020-07-15 00:10:20 -07:00