SoftFloat: fix FPREM

fixes incorrect signs on FPREM. in turn should fix end-to-end failures logging
into Steam.

Thank you to Sergio Lopez for tracking down the JavaScript fail, and Ryan for
finding the bug.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>

FEXCore/Source/Common/SoftFloat.h

@@ -160,7 +160,25 @@ struct FEX_PACKED X80SoftFloat {
 
     return Result;
 #else
-    return extF80_rem(state, lhs, rhs);
+    /*
+     * FPREM is not an IEEE-754 remainder.  From the spec:
+     *
+     *    Computes the remainder obtained from dividing the value in the ST(0)
+     *    register (the dividend) by the value in the ST(1) register (the divisor
+     *    or modulus), and stores the result in ST(0). The remainder represents the
+     *    following value:
+     *
+     *    Remainder := ST(0) − (Q * ST(1))
+     *
+     *    Here, Q is an integer value that is obtained by truncating the
+     *    floating-point number quotient of [ST(0) / ST(1)] toward zero.
+     *
+     * We implement this sequence literally. softfloat_round_minMag means
+     * "truncate towards zero".
+     */
+    extFloat80_t quotient = extF80_div(state, lhs, rhs);
+    extFloat80_t Q = extF80_roundToInt(state, quotient, softfloat_round_minMag, true);
+    return extF80_sub(state, lhs, extF80_mul(state, Q, rhs));
 #endif
   }