Best way to make a module available to all pmap workers?

161 views
Skip to first unread message

Thomas Covert

unread,
Apr 5, 2016, 9:36:22 AM4/5/16
to julia-users
I have some code in a module (say, MyModule.jl) for which I would like each worker in a pmap() to have access.  Right now, I accomplish this by something like:

addprocs(CPU_CORES)
@everywhere using MyModule
function dostuff(x)
# code that uses functions exported from MyModule
end
results = pmap(dostuff, X)

where X is an array of things to pass to dostuff

This works.  However, the @everywhere macro called on the using statement results in this set of warnings:

WARNING: replacing module MyModule

WARNING: replacing module MyModule

WARNING: replacing module MyModule

WARNING: replacing module MyModule


Is this the idiomatic way to accomplish this task?  I would have thought there is a way to write this code so that warnings such as these don't appear.

Tim Holy

unread,
Apr 5, 2016, 12:23:58 PM4/5/16
to julia...@googlegroups.com
Do you need the @everywhere? using should be good enough on its own.

However, this isn't working perfectly: see
https://github.com/JuliaLang/julia/issues/9245
https://github.com/JuliaLang/julia/issues/3674

for some things you may need to work around (or better yet, submit a fix for!).
In particular, your experience and #9245 can both be worked around with the
following "amusing" sequence:

$ julia -p 2
_
_ _ _(_)_ | A fresh approach to technical computing
(_) | (_) (_) | Documentation: http://docs.julialang.org
_ _ _| |_ __ _ | Type "?help" for help.
| | | | | | |/ _` | |
| | |_| | | | (_| | | Version 0.4.4-pre+2 (2016-01-18 02:17 UTC)
_/ |\__'_|_|_|\__'_| | Commit a85c3a0* (78 days old release-0.4)
|__/ | x86_64-linux-gnu

julia> using ColorTypes

julia> @everywhere using ColorTypes


The first one loads the code on all workers; the only thing the second one does
is to bring the bindings into Main (if you need that).

Best,
--Tim
Reply all
Reply to author
Forward
0 new messages