[Closed] Point Colour Conversion Efficiency

your code is right. but if the target (origin) file is very big you might have a problem with performance and for sure with memory.
how many lines of data has the origin file?

how many lines of data has the origin file?

Anywhere from 10000 to 300 million plus. The big files take forever and sometimes crash due to memory. Task manager is only showing one thread being used as well.

Another problem with my script is that you can only do one file at a time…ideally i want to put some batch support in there…like picking a directory of files and letting it go through and convert each file. I’ve been attempting that one, without success, because there is a sample script in the help for loading maxfiles…doing something…then closing them. Figured that was a good place to start.

does it mean the average file has millions lines of data?

i don’t think you can do it with MXS. you method crashes because of running out of memory…

Another problem with my script is that you can only do one file at a time…ideally i want to put some batch support in there…

forget about this problem. as against the first one it’s not a problem at all.

i would try to do it using c# dll… but we will try to make an on-fly assembly.
wait a bit…

well… i have a result.
1,000,000 line file converted to right format for ~10 sec. with no memory leak.
so hopefully 100,000,000 should be processed for ~ 2 min… Slow? Yes. But why do you make so big files?

global LargeFileOps = 
(
	source = ""
	source += "using System;
"
	source += "using System.Text;
"
	source += "using System.IO;
"
	source += "public class LargeFileOps
"
	source += "{
"
	source += "	public int MakeLargeFile(string sourceFile, string line, int count)
"
	source += "	{
"
	source += "		StringBuilder sb = new StringBuilder(); 
"
	source += "		for (int k = 0; k < count; k++)
"
	source += "		{
"
	source += "			sb.AppendLine(line);
"
	source += "		}
"
	source += "		StreamWriter sw = new StreamWriter(sourceFile);
"
	source += "		sw.Write(sb);
"
	source += "		sw.Close();
"
	source += "		return count;
"
	source += "	}
"
	source += "	public int ConvertCloud(String sourceFile, String targetFile)
"
	source += "	{
"
	source += "		String[] data = File.ReadAllLines(targetFile);
"
	source += "		if (data != null)
"
	source += "		{
"
	source += "			StringBuilder sb = new StringBuilder();
"
	source += "			String format = \"{3} {4} {5}\
\";
"
	source += "			char[] sp = new char[] { ' ' };
"
	source += "			foreach (String d in data)
"
	source += "			{
"
	source += "				String[] str = d.Split(sp, StringSplitOptions.RemoveEmptyEntries);
"
	source += "				if (str.Length == 6)
"
	source += "				{
"
	source += "					sb.AppendFormat(format, Array.ConvertAll(str, new Converter<String, String>(CloudColor)));
"
	source += "				}
"
	source += "			}
"
	source += "			StreamWriter sw = new StreamWriter(sourceFile);
"
	source += "			sw.Write(sb);
"
	source += "			sw.Close();
"
	source += "			return data.Length;
"
	source += "		}
"
	source += "		else return -1;
"
	source += "	}
"
	source += "	public static String CloudColor(String st)
"
	source += "	{
"
	source += "		return (String.Format(\"{0:0.000}\", Single.Parse(st)/255.0).ToString());
"
	source += "	}
"
	source += "}
"


	csharpProvider = dotnetobject "Microsoft.CSharp.CSharpCodeProvider"
	compilerParams = dotnetobject "System.CodeDom.Compiler.CompilerParameters"

	compilerParams.ReferencedAssemblies.AddRange #("System.dll")

	compilerParams.GenerateInMemory = on
	compilerResults = csharpProvider.CompileAssemblyFromSource compilerParams #(source)
	
	assembly = compilerResults.CompiledAssembly
	assembly.CreateInstance "LargeFileOps"
)

(
	LargeFileOps.MakeLargeFile @"c:	emp\cloud.txt" "0 0 0 -20 100 255" 1000000

	gc()
	t1 = timestamp()
	m1 = heapfree
	d = LargeFileOps.convertCloud @"c:	emp\cloud_cc.txt" @"c:	emp\cloud.txt"
	format "data: %
" d
	format "C#  >> time:% memory:%
" (timestamp() - t1) (m1 - heapfree)
)

i’m lazy to copy/paste a line 1,000,000 times to make a test file, so i added MakeLargeFile to the class

what things can be improved? has anyone an idea?

#1 for really large files we need #progress event
#2 can we find anything better than string.split?
#3 is there anything faster than use array.convertall?

anything else?

C# should be the faster solution as denis has said but FYI your code could be more memory efficent by replacing

theText += theVal as string + " "

with

append theText (theVal as string + " ")

or alternatively removing theText variable altogether and formatting straight to your output file.

OR again – formatting to a stringstream and then writing the result to the file every N iterations where N = 1000 or 100000 or so to reduce the number of individual write calls.

everything is right what you said. but our case is very-very-very problematic. a simple mxs tricks can’t solve it.
try for example simply create > 1,000,000 strings in mxs and see the memory leak.