Running multiple scripts in parallel Julia sessions

[Ritchie Lee]

Let's say I have 10 julia scripts, scripts = ["script1.jl", "script2.jl", ...., "script10.jl"] and I would like to run them in parallel in separate Julia sessions, but 4 at a time (since I only have 4 cores on my machine).  Is there any way to do this programmatically?

I tried doing this:

addprocs(4)

pmap(include, scripts)

or

addprocs(4)

@parallel for s in scripts

include(s)

end

However, this seems to reuse the same session, so all the global consts in the script file are colliding.  I would like to make sure that the namespaces are completely separate.

Thanks!

[Stefan Karpinski]

Any reason not to run them all as separate processes?

[Ritchie Lee]

Do you mean using @spawn, success, or just from the command prompt?

The scripts are long-running experiments where I am also tracking things like CPU time.  I would like them queued and run 4 at a time, i.e., if 1 finishes ahead of others, then the next one will start running on the free processor.  Is there a way to schedule into separate processes?

[Tim Holy]

pmap?

--Tim

[Ritchie Lee]

The scripts contain a lot of global consts and other things.  pmap seems to mix the namespaces for all scripts executed on the same processor.

[Tim Holy]

You could write your own pmap-like function (see its source code) that starts
a new worker and then shuts it down again when done.

Or, just wrap all your scripts inside modules?

module Script1

# code that was in script1.jl

end

--Tim

[Kenta Sato]

I think GNU parallel is the best tool for that purpose. http://www.gnu.org/software/parallel/

You can pass -j option to control the number of maximum jobs at a time.

[michae...@gmail.com]

It would be easy to do this using the MPI.jl package. That's what I would do, but only because I'm familiar with how it works. The native parallel methods of Julia may work just as well, though.

[Ritchie Lee]

Thanks for the suggestions everyone!