Notifications
Clear all

[Closed] Numerical Comparison

Why does this return false?

(1.1 – 1.0) == 0.1

2 Replies
1 Reply
(@bobo)
Joined: 1 year ago

Posts: 0

The wonderful world of single precision floating point calculations.
You can read on Wikipedia about how 32bit computers represent floating point numbers internally to get an idea why 1.1-1.0 is not really precisely 0.1.

Here is an example:

a = 0.1 
 0.1
 b = 1.1-1.0 --this causes an imprecision, the value in b is not perfectly 0.1
 0.1
 a==b
 false
 a-b
 0.0
 formattedPrint a --formatted print reveals the two values are not exactly the same
 "1e+008"
 formattedPrint b
 "1.0000002e+008"
 
 a = 0.1 * 10^9 --if you multiply by extremely large numbers, it becomes visible too
 1e+008
 b = (1.1 - 1.0)*10^9
 1e+008
 a==b
 false
 a-b
 -24.0
 
 close_enough a b 10 --this method should be used in such cases
 true
 
 a as string == b as string --or you can compare their printed representation
 true
 
 
 c = 0.1 --when both specified explicitly, no problem
 0.1
 d = 0.1
 0.1
 c==d
 true

Thanks for the explanation. That makes sense, and I figured it was something going on internally like that.

“close_enough” seems like the best way to check for this, but I’ve never really understood how it works. So I’ll probably use the string comparison method.