Using JuMP module across multiple processes

[Kirsten Westeinde]

Hello,

I am trying to do some computations using JuMP and Cbc. Everything works as desired when I run it with one process, but the second I try to use multiple processes I am getting this:

WARNING: Module JuMP not defined on process 2

fatal error on 2: ERROR: UndefVarError: JuMP not defined

I have tried: @everywhere using JuMP but I still get the error. I thought it may be a problem with JuliaStudio so I have also tried running it from the Julia command line but am still getting the same error.

Thanks in advance for any guidance,

Kirsten

[Miles Lubin]

Hi Kirsten,

Do you have a small example script you can share that reproduces this error?

Make sure that you're running the most recent release of Julia (0.3.6). JuliaStudio is no longer supported and I'd highly recommend switching to a different IDE like Juno.

[Kirsten Westeinde]

The code that is failing can be found here

It fails when I add more than one process. I have tried on Julia 0.3.2, 0.3.6 and 0.4.0-dev. I also stopped using JuliaStudio and was just using a text editor and running it from Julia command line.

[Miles Lubin]

That code is a bit too complex for me to diagnose the issue. If you can reduce it down to ~20 standalone lines then it will be much easier to find someone willing to help out.

[Kirsten Westeinde]

Ok. I am having the same issue with the code below. Any instance of trying to use JuMP among multiple processes.

using JuMP

using Clp

using Cbc

function main()

    model = Model(solver=ClpSolver())

    @defVar(model, a[1:10] )

    @defVar(model, e[1:10] >= 0)

    @defConstrRef constraints[1:10]

    constraints = [@addConstraint(model, a[i] + e[i] >= 1) for i=1:10]

    @sync @parallel for j = 1:10

        chgConstrRHS(constraints[j], -999)

        solve(model)

    end

end

[Stefan Karpinski]

Yep, that's stated in the second half of Kristen's original email.

On Mon, Mar 23, 2015 at 1:26 PM, Martin Klemann <mkle...@mailbox.org> wrote:

Did you try the @everywhere macro?

@everywhere include("foo.jl")

--
You received this message because you are subscribed to the Google Groups "julia-opt" group.
To unsubscribe from this group and stop receiving emails from it, send an email to julia-opt+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Miles Lubin]

I get a segfault inside of Julia's serialization code when trying to run this.

What are you trying to do? The iterations inside of a parallel for loop have to be independent and can run in any order, but this code is trying to modify the same JuMP model object in parallel (never mind that parallel for doesn't use shared memory parallelism).

I would recommend not trying to combine parallelism and a single JuMP model object. Instead, you can write a function f that generates a new JuMP model with the problem which you want to solve and returns the solution data which you need. Then you can solve many problems in parallel by using pmap(f,args).

[Kirsten Westeinde]

Good point. I will try that and let you know how it goes.

Thanks for your help!

[Alex Dowling]

Hello,

I am experiencing a similar issue with using JuMP in a parallel loop. Here is a small code example (contents of basic_JuMP_ex3.jl):

using JuMP

##### This file looks at solving an optimization in parallel using pmap()

# More info: http://julia.readthedocs.org/en/latest/manual/parallel-computing/

# Use "julia -p n" to start Julia with n 

### Import optimization problem definition/solve in a function

# using DummyModule

@everywhere function solveOptProb2(p)

a = p[1]

b = p[2]

m = Model()

@defVar(m, 0 <= x <= 2 )

@defVar(m, 0 <= y <= 30 )

@setObjective(m, Max, a*x + 3*y )

@addConstraint(m, 1x + 5y <= b )

status = solve(m)

return getObjectiveValue(m), [getValue(x); getValue(y)], status

end

### Test function

(obj, z, status) = solveOptProb2((1,1))

### Iterate over possible values of a and b

aOpt = 0:5

bOpt = 1:3

aN = length(aOpt)

bN = length(bOpt)

# Assemble array of input parameters for the optimization problem

P = Array(Tuple,aN*bN)

for i in 1:length(aOpt)

for j in 1:length(bOpt)

P[(i-1)*bN + j] = (aOpt[i],bOpt[j])

end

end

# Solve the optimization problem in parallel

# Note: results is an array of tuples, with the same dimensions as P

results = pmap(solveOptProb2, P)

And here is the error:

ERROR: LoadError: On worker 2:

UndefVarError(symbol("@defVar"))

 in eval at ./sysimg.jl:14

 in anonymous at multi.jl:1350

 in anonymous at multi.jl:892

 in run_work_thunk at multi.jl:645

 [inlined code] from multi.jl:892

 in anonymous at task.jl:68

 in remotecall_fetch at multi.jl:731

 in remotecall_fetch at multi.jl:734

 in anonymous at multi.jl:1352

...and 2 other exceptions.

 in sync_end at ./task.jl:414

 in anonymous at multi.jl:1361

 in include at ./boot.jl:260

 in include_from_node1 at ./loading.jl:271

while loading /home/path_to_file/basic_JuMP_ex3.jl, in expression starting on line 11

 Version 0.4.0-rc1+7 (2015-09-10 20:01 UTC)

 Commit 39e2493 (6 days old release-0.4)

 x86_64-linux-gnu

[Alexandros Fakos]

Hi,

Did anyone manage to successfully parallelize code using JuMP? Is it possible? I face similar problems as above. 

Thanks,

Alex

[Alexandros Fakos]

Hi, I attempted what Miles suggested above but I still get an error in Julia 0.4.6

@everywhere using JuMP, NLopt

function main(max :: Int)

    argin = [i for i=1:max]

    result = Array(Float64,max,2)

    for i =1:max

        result[i,:] = testjump(argin[i])

    end 

    presult = pmap( testjump, argin )

    return result,presult

end 

function testjump(seed :: Int)

    N = 20

    srand(seed)

    Z=[ones(N) randn(N)]

    b=[2; 3]  

    y=Z*b+rand(N)

    m = Model( solver=NLoptSolver(algorithm=:LD_SLSQP))

    @variable(m, x[ i = 1 : size(Z)[2] ]  )

    @NLobjective(m, Min, sum{(y[i]- sum{ Z[i,j]*x[j], j=1:size(Z)[2]})^2, i=1:size(Z)[1]})

    print(m)

    solve(m)

    println("beta= ",getvalue(x))

    return getvalue(x)

end

I start julia

;cd /to/directory

addprocs(1)

include("thefile.jl")

main(3)

and I get

julia> main(3)

Min (nonlinear expression)

Subject to

 x[i] free ∀ i ∈ {1,2}

beta= [2.4562065996319573,2.9402595697321345]

Min (nonlinear expression)

Subject to

 x[i] free ∀ i ∈ {1,2}

beta= [2.387999953872386,3.069169773706371]

Min (nonlinear expression)

Subject to

 x[i] free ∀ i ∈ {1,2}

beta= [2.5433281110335435,3.059284556879825]

(

3x2 Array{Float64,2}:

 2.45621  2.94026

 2.388    3.06917

 2.54333  3.05928,

Any[RemoteException(2,CapturedException(ErrorException("function testjump not defined on process 2"),Any[(:error,symbol("/Applications/Julia-0.4.6.app/Contents/Resources/julia/lib/julia/sys.dylib"),-1,symbol(""),-1,1),(:anonymous,symbol("serialize.jl"),526,symbol(""),-1,1),(:anonymous,symbol("multi.jl"),923,symbol(""),-1,1),(:run_work_thunk,symbol("multi.jl"),661,symbol(""),-1,1),(:anonymous,symbol("multi.jl"),923,symbol("task.jl"),63,1)]))])

anyone knows why? 

Thanks a lot,

Alex

[Alexandros Fakos]

Ok. I should have written

@everywhere include("thefile.jl") instead of include("thefile.jl")

then it works.

sorry to bother you,

Alex