Notifications
Clear all
[Closed] Numerical Comparison
The wonderful world of single precision floating point calculations.
You can read on Wikipedia about how 32bit computers represent floating point numbers internally to get an idea why 1.1-1.0 is not really precisely 0.1.
Here is an example:
a = 0.1
0.1
b = 1.1-1.0 --this causes an imprecision, the value in b is not perfectly 0.1
0.1
a==b
false
a-b
0.0
formattedPrint a --formatted print reveals the two values are not exactly the same
"1e+008"
formattedPrint b
"1.0000002e+008"
a = 0.1 * 10^9 --if you multiply by extremely large numbers, it becomes visible too
1e+008
b = (1.1 - 1.0)*10^9
1e+008
a==b
false
a-b
-24.0
close_enough a b 10 --this method should be used in such cases
true
a as string == b as string --or you can compare their printed representation
true
c = 0.1 --when both specified explicitly, no problem
0.1
d = 0.1
0.1
c==d
true
Oct 10, 2008 12:22 am
Thanks for the explanation. That makes sense, and I figured it was something going on internally like that.
“close_enough” seems like the best way to check for this, but I’ve never really understood how it works. So I’ll probably use the string comparison method.