help with parallel

[Ethan Anderes]

I have a very simple parallel problem but I'm having a hard time executing what I want. I'm basically just filling up a large array with @parallel as follows:

addprocs(10)
require("functions.jl")
biglist = @parallel (vcat) for k=1:procs
smalllist(args1, args2,...)
end

smalllist (defined in functions.jl) returns an Vector{Float64} and I'm just stacking them together to make biglist. My problem is that I can't seem to get args1 and args2 to the other processors. I would like something like put! but I don't have a RemoteRef to send it to. I've also thought about defining args1, args2 etc in functions.jl but (a) it's an unnatural place to put these definitions and (b) in the construction of args1, args2 I make calls to rand() and I'm worried args1, args2 will be different for each processor. I've tried @everywhere but there is a lot of code going into the construction of args1, args2 and I don't want 100 lines of code with @everywhere sending all the unnecessary temporary variables to each proc. I guess I'm looking for something like

@shareall args1, args2, ...
or
@sendall args1, args2, ...

Any help would me much appreciated. BTW: it would be especially nice if I didn't have to send args1, args2 at all, just make them visible with shared memory.

[James Porter]

Hi Ethan,

Hmmm this is odd, in general you shouldn't need to explicitly send variables you read in a parallel for loop to other processors. For example:

julia> addprocs(4)

4-element Array{Any,1}:

 2

 3

 4

 5

julia> args1 = 1

1

julia> @parallel (+) for i=1:10 args1 end

10

Can you provide a runnable example of where this doesn't work for you?

—James

[Ethan Anderes]

Now I feel really silly. I had a typo: args1 was erroneously written arg1...hence the error that the workers couldn't find arg1.

Sorry for the noise and thanks for the help (debugging as it were)!

[James Porter]

No problem! Rubber ducking can be an effective debugging technique, even over the internet :)

[Ethan Anderes]

Maybe I spoke too soon. I don't understand why @parallel can see `a` but not foo in the following (I hope this isn't another silly mistake on my part, else I be banished)?

julia> a=1
1

julia> addprocs(3)
3-element Array{Any,1}:
2
3
4

julia> x = @parallel (vcat) for k=1:3
sin(a)
end
3-element Array{Float64,1}:
0.841471
0.841471
0.841471

julia> foo(a) = sin(a)
foo (generic function with 1 method)

julia> x = @parallel (vcat) for k=1:3
foo(a)
end
exception on exception on exception on 324: : : ERROR: foo not defined
in anonymous at no file:2
in anonymous at multi.jl:1263
in anonymous at multi.jl:840

[Ethan Anderes]

There also seems to be a scope issue as well. The following throws an error that `a` is not defined. I guess it's because of the local scope but I would hope this wouldn't be a problem for @parallel:

runparl = true
if runparl
addprocs(3)
a = rand(5)

x = @parallel (vcat) for k=1:3
sin(a)
end

end

[James Porter]

The first one is easy to solve; function definitions aren't moved implicitly like data is, if you use:

@everywhere foo(a) = sin(a)

To make the definition happen on all worker processes then it works.

As to the second issue I'm not sure what's wrong. Based on the description of @parallel in the manual this is arguably a bug, I'd suggest filing an issue on github if no one else chimes in here.

[Ethan Anderes]

Now I'm having a hard time understanding how variables get passes to and from a local worker. In the following, I ask a local worker to to compute sin(x) and return the result. Notice that x is defined on the main worker, but @spawnat knows to send it's value to the local worker.

addprocs(1)
x=10
fetch(@spawnat 2 sin(x))

Why can't the same thing be done with a function?

foo(x) = sin(x)
fetch(@spawnat 2 foo(x)) #<--- gives an error

Of course, I could get this to work using @everywhere foo(x) = sin(x), but for my situation foo depends on a bunch of other functions, which are defined before I add the other workers and I don't want to redefine all these function with @everywhere once the workers are added.

Any help would be appreciated:)

[James Porter]

I have no authority to speak on this but my impression is this is because functions are a bit special. Data is pretty straightforward to serialize and send over the wire, but a generic function in Julia is not just an ordinary object. Defining functions has global side-effects such as registering them in the method table so they will be correctly dispatched on, etc. For this reason, the simplest thing to do if you want to get a definition over the wire is to just use @everywhere to execute the code that defines it on another machine.

As for your specific issue, why not just put all your code in a file and then `include` it on all your processors? For example:

addprocs(4) #<- do this first
@everywhere include("file_that_defines_all_your_functions.jl")

# now do work involving all processors here

[Ethan Anderes]

Thanks James:

That's basically the solution that I've got implemented now. Except I use require("functions.jl") which seems to make these function available to all workers. It feels a bit un-natural since my use of parallelism is a small part of my code, so I end up calling require("functions.jl") twice, first at the top of the script, then again just after addprocs(...).

I also tried looking at the source code for this stuff, in the hopes that I could figure out the finer points of passing/requesting variable from works and local worker namespaces but I just couldn't penetrate it. Maybe I need to wait till someone does a detailed blog post on a particular problem.

Cheers!

[Amit Murthy]

Some discussion regarding this on github:

https://github.com/JuliaLang/julia/issues/3674

https://github.com/JuliaLang/julia/issues/5232

[Amit Murthy]

Also, it will be great if you could file a bug regarding the scope issue mentioned previously. I think it is a bug too.

[Ethan Anderes]

Thanks Amit. First time filling an issue:

https://github.com/JuliaLang/julia/issues/6760

...hope I didn't mess it up.

[James Porter]

I would recommend just calling addprocs at the very beginning. If you know your code needs to use multiple processors, may as well say so right away.