"WARNING: replacing module" when invoking function with @everywhere - but module not available otherwise

[Adrian Salceanu]

Hi, 

I'm fumbling around with a little script with the end goal of running HttpServer handlers on multiple ports, in parallel, with each handler on a separate worker. 

The code looks like this: 

# parallel_http.jl
using HttpServer


function serve(port::Int)
  http = HttpHandler() do req::Request, res::Response
      Dates.now() |> string
  end


  server = Server(http)
  run(server, port)
end


function run_in_parallel()
  servers = Vector{RemoteRef}()
  for w in workers()
    println("About to start server on $(8000 + w)")
    push!(servers, @spawn serve(8000 + w))
  end


  servers
end

And in the REPL, running with julia -p 2: 

julia> @everywhere include("parallel_http.jl")
WARNING: replacing module HttpServer
WARNING: Method definition write(Base.IO, HttpCommon.Response) in module HttpServer at /Users/adrian/.julia/v0.4/HttpServer/src/HttpServer.jl:178 overwritten in module HttpServer at /Users/adrian/.julia/v0.4/HttpServer/src/HttpServer.jl:178.
WARNING: replacing module HttpServer
WARNING: Method definition write(Base.IO, HttpCommon.Response) in module HttpServer at /Users/adrian/.julia/v0.4/HttpServer/src/HttpServer.jl:178 overwritten in module HttpServer at /Users/adrian/.julia/v0.4/HttpServer/src/HttpServer.jl:178.


julia> servers = run_in_parallel()
About to start server on 8002
About to start server on 8003
2-element Array{RemoteRef{T<:AbstractChannel},1}:
  From worker 3: Listening on 0.0.0.0:8003...
 From worker 2: Listening on 0.0.0.0:8002...
RemoteRef{Channel{Any}}(2,1,17)
 RemoteRef{Channel{Any}}(3,1,18)

The WARNING seems to imply that I'm doing something wrong - but if I don't run "using" on each worker, to avoid the warning, the module is not available to the worker. Am i missing something? Is there a better way to do this? Thanks! 

[Patrick Belliveau]

Hi,

     Running your code effectively executes

@everywhere using HttpServer

This is known to generate those method redefinition warnings. The behaviour of using in a parallel environment is a known unresolved bug. It seems like the best syntax to use right now is

import HttpServer #Executed only on master process

@everywhere using HttpServer

For a more detailed discussion and links to the relevant issues on github see this thread.

[Adrian Salceanu]

Thanks for the info and for the link, it's a good read. I wonder if anybody else has run into the issues Tim Holy mentions, with @everywhere using X. 

Cheers!