I’m trying to write a script that converts UTM coordinates to geographic lat/long coordinates. The formula contains sections where small numbers (stored as double) are raised to the power of 5 or 6. These end up getting truncated to zero, even though Windows Calculater show results in the range ~ 2e-10. Just testing input directly in the Listener, it appears that anything smaller than 1d-6 is truncated to zero. I’ve tried formattedPrint as well, to see if it was only the display, but it makes no difference. Reading about double precision floats, I’m a bit confused about the different notations used, but if I understand correctly, double values should have a lower range limit around 2.225d-308? Am I missing something obvious, or are there issues with the maxscript number implementation?
Can you show an example of a number that is getting truncated in code?
Well, in my code I have a variable “D” that contains the following value:
0.0243115d0
If you then do DDDDD*D, the result is 0.0d0
Bump. Anyone? How could I circumvent the issue? Thinking about scaling all values by 1000 or so, but the formulas are kind of complex, so I’m a bit sceptical.