Still confused about how to use pmap with sharedarrays in Julia

[Graydon Barz]

What is the best way to modify the following Julia code:

my_output = pmap(x -> my_function(x, my_shared_array), x_list)

so that 'output' is consistent with

my_output = map(x -> my_function(x, myarray), x_list)?

My naive attempt above resulted in a an error: 
RemoteException(pid#, CapturedException(UndefVarError: my_shared_array),...  which I understand from other posts is because the other processes do not know of my_shared_array.

However, attempts to use various permutations of @sync @async have resulted in output = Task (done) @x0007 etc., and not the desired 'output' list.

[Greg Plowman]

I have also found the combination of shared arrays, anonymous functions and parallel constructs confusing.

StackOverflow question helped me Shared array usage in Julia

In essence, "although the underlying data is shared to all workers, the declaration is not. You will still need to pass in the reference to [the shared array]"

S = SharedArray(Int,3,4)
S[:] = 1:length(S)
@everywhere f(S,i) = (println("S[$i] = $(S[i])"); S[i])

output1 = pmap(i -> f(S,i), 1:length(S))    # error
output2 = pmap((S,i) -> f(S,i), fill(S,length(S)), 1:length(S))
output3 = pmap(f, fill(S,length(S)), 1:length(S))

In seems then, in version1, the reference S is local to the worker, but S is not defined on the worker -> error.

In versions 2&3 the local S is passed as argument to worker and all works as expected.

[Graydon Barz]

Thank your for the suggestions as they enabled me to solve the issue - though for a different reason than I thought.

I had seen indeed seen the StackArray post you mentioned, but my initial attempt based on that post did not work. Specifically, following that post, I had tried:

output = pmap((x,y) -> f(x, y), ["strg1", "strg2", "strg_3"] , [S,S,S])

and received an error message for each of the three components of the output starting with: RemoteException(pid#,CapituredException(MethodError(getindex,(0.000115947... 

However, modifying it per your suggestion by defining SVector= fill(S, 3) and then running

output = pmap((x,y)->f(x,y), ["strg1", "strg2", "strg3"] , SVector)

worked.

The fact that meaning of  "," changes depending on what is in placed in [a,b,c] seems have been the source of the issue.

(As an aside, this inconsistency seems to me to be somewhat of a less-than-desireable feature in Julia.)

[Greg Plowman]

The fact that meaning of  "," changes depending on what is in placed in [a,b,c] seems have been the source of the issue.

(As an aside, this inconsistency seems to me to be somewhat of a less-than-desireable feature in Julia.)

I think this is changing in v0.5. See https://github.com/JuliaLang/julia/blob/master/NEWS.md#language-changes-1

Square brackets and commas (e.g. [x, y]) no longer concatenate arrays, and always simply construct a vector of the provided values. If x and y are arrays, [x, y] will be an array of arrays (#3737, #2488, #8599).

[Greg Plowman]

pmap inside a function also seems to work:

@everywhere f(A,i) = (println("A[$i] = $(A[i])+1"); A[i] += 1)
wrapped_pmap(A) = pmap(i -> f(A,i), 1:length(A))

S = SharedArray(Int,10)
S[:] = 1:length(S)

output1 = pmap(i -> f(S,i), 1:length(S)) #error

output2 = wrapped_pmap(S) #seems to work

I'm quite confused about why second version with pmap wrapped in a function works.

Perhaps someone can explain?

 

[Amit Murthy]

Since S is global, the closure i -> f(S,i) only captures a reference to S. And since this binding is not defined on the workers we see an error.

For example,

function foo()

    S = SharedArray(Int,10)

    S[:] = 1:length(S)

    output1 = pmap(i -> f(S,i), 1:length(S))

end

foo()

works as expected.

I agree that this is confusing.

[Amit Murthy]

This would work too

let S=S

    output1 = pmap(i -> f(S,i), 1:length(S))

end