CVTTSD2SI - a truncating instruction - uses rounding with "inexact" results?

Question

According to the Intel documentation the instruction CVTTSD2SI, which truncates a fp-value to a 64 bit signed integer, rounds according to the MXCSR rounding-mode if the conversion is inexact.

But how can a simple truncating conversion be inexact? I think that either the value fits into the range of the integer or not; if not, there's no rounding but the value is -1 << 63 and an exception may be raised.

Here's what the Intel doc says:

When a conversion is inexact, the value returned is rounded according to the rounding control bits in the MXCSR register.

Answer 1

An FP->integer conversion is inexact when the FP value was not already an integer. e.g. 1.25 converts to integer 1 with a rounding error of 0.25. But 1.0 converts exactly to 1. This is the same FP exception flag that would be raised by 1.0 / 3.0 with divsd, but not by 1.0 / 2.0, because 0.5 is exactly representable as a float or double.

That sentence is an error in the manual. CVTTSD2SI always uses truncation when the conversion is inexact, not the current rounding mode in MXCSR (which defaults to nearest, tie-break to even)

The exact same single-sentence paragraph is present in the CVTSD2SI manual entry, which does use MXCSR, so this is a copy/paste error by Intel.

---

Out-of-range is a different condition

if not there's no rounding but the value is -1 << 63 and an exception may be raised.

That's a separate kind of exception from inexact. The cvttsd2si manual covers this with

If a converted result exceeds the range limits of signed doubleword integer ([with a 32-bit destination]), the floating-point invalid exception is raised, and if this exception is masked, the indefinite integer value (80000000H) is returned.

Similarly for 64-bit operand-size, the indefinite value is MSB set, other bits zero. That same bit-pattern is used whether the FP value was positive or negative.

(It can of course also be a result of valid conversions, since the most-negative 2's complement number is a power of 2 and thus exactly representable as a double or even float. A better choice might have been some large-magnitude by odd number, like 80000001h, although that can still be the result of a valid double->int32 conversion. But not float->int32 or (for a wider 64-bit version) double->int64.)

Check Intel's manuals for the full set of flags in MXCSR. Or some article about that specifically, such as https://softpixel.com/~cwright/programming/simd/sse.php

IE (Invalid-operation Exception) is a separate flag from FP overflow (OE) or underflow (UE).

---

Experimental test

Just to make sure, I tested with GDB display $mxcsr to show (and decode) MXCSR after every single-step. (On an i7-6700k Skylake, but this behaviour is required so other CPUs should behave identically.)

section .rodata
frac: dq 1.75
whole: dq 2.0

section .text
global _start
_start:
  stmxcsr [rsp-8]                   ; in the red zone; x86-64 SysV
  cvtsd2si  eax, [frac]
  ldmxcsr [rsp-8]
  cvttsd2si edx, [frac]

  ldmxcsr [rsp-8]
  cvttsd2si ecx, [whole]
 ;... fall off the end because I only want to single-step this, and can exit GDB before letting it step past here

$ nasm -felf64 foo.asm && ld -o foo  foo.o 
$ gdb ./foo
(gdb) layout reg
 # and with recent GDB, layout n  to get regs + disassembly
(gdb) starti
(gdb) display $mxcsr
1: $mxcsr = [ IM DM ZM OM UM PM ]     
      # bunch of things masked, no raised exceptions
(gdb) si   a couple times
 # after <_start+5>     cvtsd2si eax,QWORD PTR [rip+0xff3]        # 0x402000
1: $mxcsr = [ PE IM DM ZM OM UM PM ]
         # PE = precision exception = inexact, plus the earlier mask flags
...

Each ldmxcsr reloads a no-exceptions-raised state so we can see what the next instruction did.

• cvtsd2si and cvttsd2si both set PE, the inexact exception flag, when converting 1.75 to integer.
• Both leave it clear when converting 2.0 to integer.

I didn't unmask any of the exceptions because I just wanted them recorded in the MXCSR sticky bits, not to trap and get a SIGFPE.