[Closed] Float: inner value vs display value????

Can I ask what it is you need such accuracy for?

You cannot cast a float to a double and expect it to magically increase its precision.
What happens here is that MAXScript parses the number first, makes it a float (which has the wrong value already), then it finds the AS DOUBLE and turns the “wrong” result into a Double.

d = ((1.23456789012345) as double) --the brackets show the order of evaluation
1.23457d0 --you get a double, but the content is from the initial float evaluation
formattedPrint d format:".20g" --so it is wrong already
"1.2345678806304932" 

When reading from Maya, you should read each number as string, add the “d0” part to the end to turn it to a double, then execute() the result to get the real value.

For example

a = "1.23456789012345" --let's assume this is a string you got from a file
"1.23456789012345"
b = a+"d0" --turn it into a double by adding D and exponent of 0
"1.23456789012345d0"
c = execute b --convert to a value - now MAXScript knows it is a double
1.23457d0
formattedPrint c format:".20g" --and it shows like one
"1.23456789012345"

uhm… well… I am writing a KML exporter for current project. And its algorithm require very very accurate data. Input longtitude and lattitude is some time 10 number after decimal point, then they will be canculated with current vertex number in so finally we get KML data, that at least 8 number accurate after decimal

a = 34.047413 as float
34.0474
(a *= 1000000) as integer
34047412

Does it magical enough to you? :applause:

Not really. The maxscript reference says:

When you display or print a Float, MAXScript prints the value to the 6th significant digit.

It maybe annoying, but it’s not magical. It’s just the difference between the Stored Value and the Displayed Value.

[left]

[/left]
[left]

[/left]
[left]

[/left]

I finish the exporter now, thanks alot to community, especially Bobo for helping me out

Now, for double value, when we want to declare it, It will be float before double as Bobo said so we need a function for these case

			fn makeDouble dnumber =
			(
				dnumber = dnumber as string
				dnumber2 = execute("\""+ dnumber + "d0\"")
				return dnumber2
			)

I find that out lacking of ” cause a very severe issue to the script, that lead to unstable and cause Max point out an unexplainable inform: unknown system exception. And it is more painful is script still run, but only for first time. And so second time it will break with that inform.

I’ve had trouble with double precision before that I never resolved. Upon reading this, I tried executing the following two lines of code:

real_long=0.0243113d0
formattedPrint real_long format:"0.20g"

Resulting output:

0.0d0
"0"

Hmm? This is clearly wrong. I’ve had problems before with MAX and Regional settings for desimal character (in Norway we use “,” for this), so I tried changing it to “.” New result:

 0.0243113d0
"0.024311300000000001"

Tada! Makes me want to tear my hair off, but at least I know what went wrong.

I spoke too soon. Nothing fixes what must be a serious shortcoming of the MAX implementation of double precision. This code:

  sm_a = 6378137.0d0
 sm_b = 6356752.314d0
 n=0d0
 n += (sm_a - sm_b) / (sm_a + sm_b)
 tmp=0d0
 tmp+=pow n 3d0
 formattedprint tmp format:"0.20g"

produces this output:

6.37814d+006
6.35675d+006
0.0d0
0.00167925d0
0.0d0
0.0d0
"0"

0.001679 to the power of 3 is certainly well within the precision “scope” of doubles?