ASP Fix/Int functions not behaving as expected

92 Views Asked by At

While debugging I found this weird thing in a custom RoundMid() function :

    test = 148.325 * 100 + 0.5
    Response.Write test ' Returns 14833
    Response.Write Fix(test) ' Returns 14832 ???????

    test = 14832.5 + 0.5
    Response.Write test ' Returns 14833
    Response.Write Fix(test) ' Returns 14833

Same problem with Int() instead of Fix().
Any idea what can I do to obtain 14833 on the second line ?
Thank you.

1

There are 1 best solutions below

0
alex-dl On

In the second calculation, both the numbers 0.5 and 14832.5 can be represented as powers of 2. They can be stored exactly in binary without rounding errors.

0.5 = 2-1
14832.5 = 213 + 212 + 211 + 28 + 27 + 26 + 25 + 24 + 2-1

The sum 14832.5 + 0.5 can also be represented exactly:
14833 = 213 + 212 + 211 + 28 + 27 + 26 + 25 + 24 + 20

In the first calculation 100 can be represented exactly in binary:
100 = 26 + 25 + 22
But the number 148.325 cannot be represented exactly. The binary representation of this number is:
10010100.010 1001 1001 1001 ... with the 1001 repeating.

So the binary representation will always be slightly less than 148.325 no matter the number of digits used.

When the decimal part of the calculation 148.325 * 100 + 0.5 is truncated, the result is 14832.