Hello!
I’m looking for a fast way to write a lot of text lines in a file (several files actually).
Using maxscript’s Filestream takes around 40 seconds for 32 files with over 720k lines total. This is a too long and I’m looking to optimize it by writing the text through dotNET instead.
I’m a noob at .NET, but after scouring the internet I managed to get it working by appending all my text lines in an array and using WriteAllLines
contents = #()
for v in vertices do append contents ("v " + v.x as string + " " + v.y as string + " " + v.z as string)
for vn in normals do append contents ("vn " + vn.x as string + " " + vn.y as string + " " + vn.z as string)
for f in faces do append contents ("f " + (f.x as integer) as string + "//" + (f.x as integer) as string + " " + (f.y as integer) as string + "//" + (f.y as integer) as string + " " + (f.z as integer) as string + "//" + (f.z as integer) as string)
(dotNetClass "System.IO.File").WriteAllLines myfile contents
This takes 25 seconds, which is still too much.
Options:
-
write each line directly in the file
This should speed up the process by eliminating the need to append an array. I also learned from experience that maxscript can get slow when using huge arrays, and the ones I’m working with end up having over 100k text elements. -
Write floats and integers without converting to text
I’ve seen that the StreamWriter class has methods to write the text representation of values, which might also speed up the process.
I am clueless on how to implement these 2 ideas. Could anyone please help with examples on how to use this StreamWriter class?
If you’re looking for speed, and do not require human readability, storing binary values is the way to go. Not to mention your file sizes will be much much smaller.
Consider “v0.111111 0.222222 0.333333” – 27 bytes of ascii text
vs a binary representation which would be 12 bytes.
You may even get decent results with the BinStream type in MAXScript.
Thanks for your reply!
I’m well aware of the advantages of binary formats Maxscript’s BinStream gives more than decent results – I can write the same data mentioned before in a little over 1 second.
But for now I am forced to use a text format, and dotNET is my last resort (other than compiling my own plugin). If I can get it under 10-15 seconds it will be good enough.
what 25-40 seconds are talking about?
here is sample for 1,000,000 lines and as you can see it takes 7s:
num = 1000000
fn saveTXTFile num:num =
(
ss = createfile (getdir #temp + @"\longtext.txt")
for k=1 to num do format "% >> 0x%
" k (formattedprint k format:"08x") to:ss
flush ss
close ss
)
t1 = timestamp()
m1 = heapfree
saveTXTFile()
format "time:% memory:%
" (timestamp() - t1) (m1 - heapfree)
maxscript file stream is c++ written and it’s pretty fast. using c++ (.net) you can speed up only number to string converting and string formatting operations.
You would also be wise to cache the format strings:
local fmt = "% >> 0x%
"
local fmtNum = "08x"
for k=1 to num do format fmt k (formattedprint k format:fmtNum) to:ss
This will save only 5%-10% in time, but considerably less memory.
The reason your code might be taking so long is because you are using heavy string concatenation. This creates a new string for every operation. Use format as Denis demonstrated.
@denis 40 seconds for this, ran 32 times (32 files) for a total of 720000 lines
myfile = openFile (modelpath + meshname + ".oft") mode:"wt"
for v in vertices do format "v % % %
" v.x v.y v.z to:OBJfile
for vn in normals do format "vn % % %
" vn.x vn.y vn.z to:OBJfile
for f in faces do format "f %//% %//% %//%
" (f.x as integer) (f.x as integer) (f.y as integer) (f.y as integer) (f.z as integer) (f.z as integer) to:OBJfile
close OBJfile
I also ran your code, and here’s the result: time:35044 memory:20916104L
My PC is no slow poke: Core i5 2500K@4.2Ghz, 16GB DDR3@2133, intel SSD
But I believe what you say, so there must be something wrong on my side…
@lo as you can see I was already using format, but that’s not the issue. There must be somethign wrong with my max or PC
EDIT: ran denis’ code in max 2012, results are: time:3201 memory:10296616L :bounce:
My code that took 40 seconds in max 2013 now runs in 2.5 seconds
So either there’s something wrong with my 2013 installation, or this another 2013 fk-up.
Thanks again for all your input guys, and sorry for the trouble.
Could it be that your heap size is very small and you’re forcing max to allocate and collect lots of memory throughout the process?
Try setting a very large heapsize and see if it makes a difference.
Sorry, I edited. There’s something wrong with my max 2013. I switched to 2012 and it’s blazing fast.
I had already tried with a larger heapsize but results were the same.