Hi there,
I’m writing a script which needs to download multiple images from the internet. I’m exploring the possibility of using multiple threads to download the images. There are several async methods available in C# which are easy to work with, and which work great.
One of the patterns with using async methods like these is you store its result in a task. Then you can wait on the task to finish before continuing with the rest of the program. This works great in C#, and downloading multiple files works great in maxscript as well. BUT, finally waiting on the task to finish makes 3ds Max hang.
If you run the example here, you’ll see it downloads 10 files just fine. If you uncomment the last line, the files are downloaded again but max hangs afterwards.
Is there anybody who has any insight, maybe experienced this before and has a solution or workaround?
(
clearListener()
--this .net class sets up async downloads
--you can specify the number of concurrent async download tasks
local strAssembly = "using System.Collections.Generic;
using System.IO;
using System.Net;
using System.Threading.Tasks;
namespace Downloader
{
public class Download
{
public static async Task DownloadFileAsync(string remoteUrl, string localUrl)
{
using (WebClient webClient = new WebClient())
{
await webClient.DownloadFileTaskAsync(remoteUrl, localUrl);
}
}
public static async Task DownloadMultipleFilesAsync(string remoteUrl, string localFolder, int downloadTimes)
{
List<Task> allTasks = new List<Task>();
for (int n = 0;n < downloadTimes; n++)
{
string localUrl = Path.Combine(localFolder, n.ToString());
localUrl = Path.ChangeExtension(localUrl, (Path.GetExtension(remoteUrl)));
allTasks.Add(DownloadFileAsync(remoteUrl, localUrl));
}
await Task.WhenAll(allTasks);
}
}
}
"
function fn_loadClass strAssembly =
(
/*<FUNCTION>
Description
Loads a .net class from a string
Arguments
Return
compilerResults which can be instanciated
<FUNCTION>*/
local csharpProvider = dotnetobject "Microsoft.CSharp.CSharpCodeProvider"
local compilerParams = dotnetobject "System.CodeDom.Compiler.CompilerParameters"
compilerParams.ReferencedAssemblies.AddRange #("System.dll", "System.Management.dll")
compilerParams.GenerateInMemory = on
local compilerResults = csharpProvider.CompileAssemblyFromSource compilerParams #(strAssembly)
)
local compilerResults = fn_loadClass strAssembly
local Download = compilerResults.CompiledAssembly.CreateInstance "Downloader.Download"
local remoteUrl = @"http://a.tile.openstreetmap.org/8/131/84.png"; --we'll download this file
local localFolder = (dotnetClass "system.IO.Path").Combine (pathConfig.removePathLeaf (getSourceFileName())) "output"
makeDir localFolder
local downloadTasks = 10 --this is the number of times we'll download the file concurrently
local theTask = Download.DownloadMultipleFilesAsync remoteUrl localFolder downloadTasks
--WATCH OUT!
--we need to wait for the async download task to finish, otherwise we might get in trouble with the rest of the script. If we don't
--wait, we're counting on the files to be downloaded while they might not be there yet.
--Calling the Wait method on the task makes 3dsMax hang though (tested in max 2016/7)
-- theTask.Wait()
)
Don’t know if it’s still running asynchronously but running this task in background worker doesn’t seem to hang max.
ps. And that’s because the rest of the script runs as if Wait() doesn’t do anything. My bad.
Some sort of timer could be used to check if RunWorkerCompleted event has raised to proceed with rest of the script. I would like to know less ugly ways to do so.
local theTask = undefined
fn work = (
theTask = Download.DownloadMultipleFilesAsync remoteUrl localFolder downloadTasks
theTask.Wait()
)
fn oncomplete = ( format "Completed succesfully
" )
local bw = dotNetObject "CSharpUtilities.SynchronizingBackgroundWorker"
dotnet.addEventHandler bw "DoWork" work
dotnet.addEventHandler bw "RunWorkerCompleted" oncomplete
bw.runWorkerAsync()
format "%
" "this should be printed after task completed"
Serejah,
I see what you mean. Overall, the script flow doesn’t wait, but the Wait() seems to work well within the work function. Code after Wait() is only executed once all files have been downloaded. Increase the download count to 100 to see this happen. At least max doesn’t hang anymore.
Now find an elegant way to halt the script flow until the wait() is over.
looks like i’ve found it. check this thread for details
rollout modalRollout "Test"
(
label test "TestLabel"
button btTest "Close Me"
on btTest pressed do
DestroyDialog modalRollout
)
fn work = (
theTask = Download.DownloadMultipleFilesAsync remoteUrl localFolder downloadTasks
theTask.Wait()
)
fn oncomplete = (
format "Completed succesfully
"
DestroyDialog modalRollout
)
local bw = dotNetObject "CSharpUtilities.SynchronizingBackgroundWorker"
dotnet.addEventHandler bw "DoWork" work
dotnet.addEventHandler bw "RunWorkerCompleted" oncomplete
bw.runWorkerAsync()
createdialog modalRollout pos:[-888,-888] modal:true
format "%
" "this should be printed after task completed"
That’s insane, and it works! So basically you just pop up a modal dialog off screen to stall further script execution and let the background worker finish.
May I ask, how did you come up with the idea to put the async method and the wait() method in another thread?
I think someone here on forum mentioned that it’s not a good idea to run multi-threaded tasks from max main thread
Got a couple of useful functions to delay or stop execution until condition is met
-- this function will delay further script execution by defined time in milliseconds
fn pauseScript msec = (
global msec = timestamp() + msec
if msec != undefined then (
if msec as integer > 0 then (
rollout pauseScriptModalRollout ""
(
timer delay interval:25 active:false
on pauseScriptModalRollout open do (
delay.active = true
)
on delay tick do (
if timestamp() >= ::msec then (
delay.active = false
globalvars.remove "msec"
DestroyDialog pauseScriptModalRollout
)
)
)
createdialog pauseScriptModalRollout modal:true pos:[-666,-666]
)
)
)
-- function will pause further script execution until given expression return true
fn pauseUntil expression = (
global pause_expr = expression
rollout pauseScriptUntilModalRollout ""
(
timer delay interval:50 active:false
on pauseScriptUntilModalRollout open do (
delay.active = true
)
on delay tick do (
if execute ::pause_expr == true then (
delay.active = false
globalvars.remove "pause_expr"
DestroyDialog pauseScriptUntilModalRollout
)
)
)
createdialog pauseScriptUntilModalRollout pos:[500,500] modal:true
)
-- this function will delay further script execution by defined time in milliseconds
fn pauseScriptDotNet msec = (
global msec = timestamp() + msec
if msec != undefined then (
if msec as integer > 0 then (
rollout pauseScriptModalRollout ""
(
local t = dotnetobject "System.Windows.Forms.Timer"
fn tick s e = (
if timestamp() >= ::msec then (
s.stop()
globalvars.remove "msec"
DestroyDialog pauseScriptModalRollout
)
)
on pauseScriptModalRollout open do (
dotnet.addEventHandler t "Tick" tick
t.interval = 25
t.start()
)
)
createdialog pauseScriptModalRollout modal:true pos:[200,200]
)
)
)
format "dotnet timer
"
(
global t1=timestamp()
pauseScriptDotNet 2000
format "Time: %sec.
" ((timestamp()-t1)/1000 as float)
)
format "max timer
"
(
global t1=timestamp()
pauseScript 2000
format "Time: %sec.
" ((timestamp()-t1)/1000 as float)
)
Fantastic thread! Thanks both for sharing. :bowdown:
It’s a shame for me to know I’ll never be at your programming level.
reading your threads i have exact same thoughts
Thanks Serejah for the extra research.
After digging more into this, I’ve found this excellent video about the async/await patterns in .net: https://channel9.msdn.com/Events/Build/2013/3-301 . In there Stephen Toub mentions a ui deadlock when calling the Wait() method in some cases (at around 52 minutes in). I also found a question on stackoverflow which mentions that WebClient doesn’t work as well as HttpClient with this async stuff (mentioned here http://stackoverflow.com/questions/18524609/configureawaitfalse-still-deadlocks ).
With the help of this SO question http://codereview.stackexchange.com/questions/18519/real-world-async-and-await-code-example I’ve put one and one together. I’m using HttpClient to download multiple images async and I can wait for them in maxscript without doing the (very clever) modal dialog hacks described above.
The following code will not crash on you like my example at the top did.
(
clearListener()
--this .net class sets up async downloads
local strAssembly = "using System.Collections.Generic;
using System.IO;
using System.Net.Http;
using System.Threading.Tasks;
namespace Downloader
{
public class Download
{
// http://codereview.stackexchange.com/questions/18519/real-world-async-and-await-code-example
public static async Task DownloadFileAsync(string remoteUrl, string localUrl)
{
HttpClient client = new HttpClient();
HttpResponseMessage responseMessage = await client.GetAsync(remoteUrl).ConfigureAwait(false);
var byteArray = await responseMessage.Content.ReadAsByteArrayAsync().ConfigureAwait(false);
using (FileStream filestream = new FileStream(localUrl, FileMode.Create, FileAccess.Write, FileShare.None, bufferSize:4096, useAsync:true))
{
await filestream.WriteAsync(byteArray, 0, byteArray.Length);
}
}
public static async Task DownloadMultipleFilesAsync(string remoteUrl, string localFolder, int downloadTimes)
{
List<Task> allTasks = new List<Task>();
for (int n = 0; n < downloadTimes; n++)
{
string localUrl = Path.Combine(localFolder, n.ToString());
localUrl = Path.ChangeExtension(localUrl, (Path.GetExtension(remoteUrl)));
allTasks.Add(DownloadFileAsync(remoteUrl, localUrl));
}
await Task.WhenAll(allTasks).ConfigureAwait(false);
}
}
}
"
function fn_loadClass strAssembly =
(
/*<FUNCTION>
Description
Loads a .net class from a string
Arguments
Return
compilerResults which can be instanciated
<FUNCTION>*/
local csharpProvider = dotnetobject "Microsoft.CSharp.CSharpCodeProvider"
local compilerParams = dotnetobject "System.CodeDom.Compiler.CompilerParameters"
compilerParams.ReferencedAssemblies.AddRange #("System.Net.Http.dll","System.dll", "System.Management.dll")
compilerParams.GenerateInMemory = on
local compilerResults = csharpProvider.CompileAssemblyFromSource compilerParams #(strAssembly)
)
local compilerResults = fn_loadClass strAssembly
local Download = compilerResults.CompiledAssembly.CreateInstance "Downloader.Download"
local remoteUrl = @"http://a.tile.openstreetmap.org/8/131/84.png"; --we'll download this file
local localFolder = (dotnetClass "system.IO.Path").Combine (pathConfig.removePathLeaf (getSourceFileName())) "output"
makeDir localFolder
local downloadTasks = 10 --this is the number of times we'll download the file concurrently
local theTask = Download.DownloadMultipleFilesAsync remoteUrl localFolder downloadTasks
--we need to wait for the async download task to finish, otherwise we might get in trouble with the rest of the script. If we don't
--wait, we're counting on the files to be downloaded while they might not be there yet.
theTask.Wait()
local arrFile = getFiles (localFolder + @"\*.png")
format "Downloaded % files
" arrFile.count
)