Run Julia job on several workers on a cluster

[Theodore Papamarkou]

I want to run a population MCMC simulation using power density estimators on 50 nodes of the departmental cluster, which uses SGE. Each of the 50 nodes realizes a separate MCMC chain. The question generalizes to any parallel job which needs to reserve several nodes. I have found two relevant posts, namely

https://groups.google.com/forum/?fromgroups=#!searchin/julia-users/cluster/julia-users/IATlfsu4VJU/yw1y7N_dPg0J

and

https://groups.google.com/forum/?fromgroups=#!searchin/julia-users/cluster/julia-users/IlPuQSwtTSQ/vpGCPA27uMYJ

but I haven't found a finalized set of instructions yet to achieve the required result.

1. I guess the first step would be to instruct qsub directly to reserve the required number of nodes. Typically, resource requirements are specified with the "-l" option. Is "qsub -l nodes=50" the appropriate invocation?
2. The next question is how the number of processes nprocs() relates to the number of reserved nodes (workers). Should I call the command addprocs(50)? I think this is not good enough because Julia would not know which nodes to select. addprocs({"hostname1", "hostname2",... "hostnam50"}) seems to be the right command. Nevertheless it is not handy to fetch the hostnames of the reserved nodes. So the natural question becomes whether addprocs_sge(50) does what I have in mind; does this command require step 1 or it acts as a scheduler?
3. Does a command such as remotecall(i, simulateMCMC, MCMCargs) run one of the chains on the i-th node while I am on the head node, i.e. on the node with myid() equal to 1? I mean, do I regulate communciation in an MPI fashion by being on a so called head node, is this possible, or I have to change the implementation?

Thank you for any help and feedback - if I manage to make the popMCMC simulation run on the cluster, I will share the code and "howto".

[samc...@gmail.com]

I haven't tried to do what you are describing yet, but I know a little. In SGE there should be a file named "machinefile" somewhere. It might be "$TMP/machinefile", but don't quote me. If you have this file, which contains the hostnames of the nodes, you should be able to pass it to julia on startup with the "--machinefile" option. An example of this is on the Julia blog http://julialang.org/blog/2013/04/distributed-numerical-optimization/

I hope that helps a little.

[Theodore Papamarkou]

The "--machinefile" option and the blogpost on distributed numerical optimization are potentially excellent sources to help me, thanks a lot. I will try to make use of them and will post here once I make some progress.

[Kevin Squire]

The relevant sge line in $JULIA_HOME/base/multi.jl has

qsub_cmd = `echo $home/julia-release-basic --worker` | `qsub -N JULIA -terse -cwd -j y -o $sgedir -t 1:$n`

So addprocs_sge() will do the qsub for you.  When I just tried it, the workers started okay, but I received an error:

julia> addprocs_sge(2)
ERROR: assertion failed: ?
 in error at error.jl:22
 in assert at error.jl:43
 in success at process.jl:392
 in map at abstractarray.jl:1478
 in success at process.jl:394
 in start_sge_workers at multi.jl:1009
 in addprocs_sge at multi.jl:1044

$ qstat -u "kmsquire"

job-ID  prior   name       user         state submit/start at     queue                          slots ja-task-ID 

-----------------------------------------------------------------------------------------------------------------

 358164 10.50000 JULIA      kmsquire     r     06/16/2013 14:01:52 al...@compute-4-60.local           1 1

 358164 10.50000 JULIA      kmsquire     r     06/16/2013 14:01:52 al...@compute-4-53.local           1 2

Kevin

[Theodore Papamarkou]

Thanks for trying this out Kevin. I tried the same after you and got the same error, although the job was queued:

julia> addprocs_sge(2)                                                                                                                                                  

ERROR: assertion failed: ?                                                                                                                                              

 in error at error.jl:22                                                                                                                                                

 in assert at error.jl:43                                                                                                                                               

 in success at process.jl:394                                                                                                                                           

 in all at reduce.jl:175                                                                                                                                                

 in success at process.jl:401                                                                                                                                           

 in start_sge_workers at multi.jl:941                                                                                                                                   

 in addprocs_sge at multi.jl:976

$ qstat -u "ucaktpa"

job-ID  prior   name       user         state submit/start at     queue                          slots ja-task-ID 

-----------------------------------------------------------------------------------------------------------------

9696992 0.50290 JULIA      ucaktpa      qw    06/16/2013 22:16:14                                    1 1,2

I checked the line in multi.jl you mentioned, and was thinking that I pass several other options to qsub, e.x. in order to allocate memory or set runtime thresholds (-l h_vmem=8G,vf=8G -l h_rt=0:3:0). It may be good to pass them as extra arguments to start_sge_workers(); alternatively, we could pass a single argument, which could be a configuration file, similar to the matlab sample code below:

sched = findResource('scheduler', 'configuration', configuration);

pjob = createParallelJob(sched);

set(pjob, 'MaximumNumberOfWorkers', minNumWorkers);

set(pjob, 'MinimumNumberOfWorkers', maxNumWorkers);

I will try to trace the addprocs_sge() error message...

[Kevin Squire]

I checked the line in multi.jl you mentioned, and was thinking that I pass several other options to qsub, e.x. in order to allocate memory or set runtime thresholds (-l h_vmem=8G,vf=8G -l h_rt=0:3:0). It may be good to pass them as extra arguments to start_sge_workers(); alternatively, we could pass a single argument, which could be a configuration file, similar to the matlab sample code below:

I think both of those would be quite useful!

 

sched = findResource('scheduler', 'configuration', configuration);

pjob = createParallelJob(sched);

set(pjob, 'MaximumNumberOfWorkers', minNumWorkers);

set(pjob, 'MinimumNumberOfWorkers', maxNumWorkers);

I will try to trace the addprocs_sge() error message...

 Good luck!  I probably won't be able to look at it anytime soon.

Kevin

[Ben Lauwens]

Hello

I did some debugging and it seems in my case that the environment variables are not set

/home/blauwens/julia/usr/bin/julia-release-basic: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.15' not found (required by /home/blauwens/julia/usr/bin/../lib/libjulia-release.so)

Adding the -V argument to qsub 

qsub_cmd = `echo $home/julia-release-basic --worker` |> `qsub -N JULIA -V -terse -cwd -j y -o $sgedir -t 1:$n`

and putting

sleep(0.5)

before the success check solves this problem but I get another one. The output stream file reads

bash: module: line 1: syntax error: unexpected end of file
bash: error importing function definition for `module'
julia_worker:9009#192.168.1.226

The connection info is the last line but will never be read by the function start_sge_workers. Here is a small hack that does the job. 

while !fexists
    try
        fl = open(fname)
        try
            while !fexists
                conninfo = readline(fl)
                hostname, port = parse_connection_info(conninfo)
                fexists = (hostname != "")
            end
        finally
            close(fl)
        end
    catch
        print(".");
        sleep(0.5)
    end
end
After these modifications, 
addprocs_sge()
works on a HP cluster running x86_64 GNU/Linux.
Some feedback from other SGE users should be useful and perhaps this hack can be merged in julia base.

Ben
 

[Theodore Papamarkou]

Many thanks Ben, I noted your hack down to try it out and will get back to you as soon as I do so (which will be rather soon). Anyone else's input is always welcome.

[Viral Shah]

Also do see this issue, which talks about abstracting various cluster schedulers into a separate package. This should make it easier to support more schedulers, and also make it easier to patch these as we go along.

https://github.com/JuliaLang/julia/issues/3549

-viral

[Theodore Papamarkou]

It took me a while to start using Julia on the departmental cluster, but I got there. I am aware that there are already some packages for cluster management, such as PTools, ClusterManagers and MATLABCluster. My preferred way of dealing with the matter is via the ${PE_HOSTFILE} SGE environment variable. ${PE_HOSTFILE} is created at qsub's runtime and holds the full path to the machine file. So, I wrote my own awk script to process this machine file, whose format may differ depending on the cluster-specific configuration, so as to get a Julia readable machine file, consisting of one column with the names of the nodes in it. Then I simply start julia via "julia --machinefile mymachinefile.txt" from the shell wrapper that I submit via qsub. I will provide an example on github in case someone likes this approach.

[Thibaut Lamadon]

Dear all, is there a working example of julia with SGE and MPI?

thank you, 

t.

[Theodore Papamarkou]

Hi Thibaut,

It is possible to run jobs on a cluster, but the integration between Julia and SGE (and in general with job schedulers) has not reached yet the desired state. There are a couple of packages around for this purpose, yet more work is needed towards this direction. In theory, it should be possible to integrate MPI with Julia, since the parallel computing paradigm of Julia has the necessary components that are needed for calling MPI (a volunteer must find the time to write a Julia wrapper for the MPI library). In the meantime, the parallel environment of Julia is excellent if you want it to use it as a stand-alone entity (and if you know the machine hostnames, in case you can avoid using SGE or in case you have good IT support with sge).

You may have a look at this post too:

https://groups.google.com/forum/#!searchin/julia-dev/papamarkou/julia-dev/fuATZYYAYK8/8_2rsvdyFakJ

I noticed you are based at UCL, is this correct? Unfortunately, my experience with the IT support provided by the computer science department of UCL has not been ideal with regards to the use of Julia (at least with regards to their cluster and the specific support team I had to deal with). I had opened a ticket, chased the IT team for 3 months by sending emails on a weekly basis, they had me on the waiting list constantly postponing the matter, and at the end they simply closed the ticket without bothering to look at it! I would say don't waste your time trying to run a Julia job submitted via SGE on their clusters, the support is hopeless (furthermore, there is not much you can do by yourself since you wouldn't have the admin privileges to access the SGE of course). You may try the Legion cluster though at UCL, you may have better support and luck with that, if you are willing to give it a shot.

Hope this helps,
Theo

[Thibaut Lamadon]

Thank you very much for this answer. Actually I was able to compile Julia on the economics cluster, and run a parallel command using 

ClusterManagers.addprocs_sge. This is already great, but now I want to be able to submit jobs to the main scheduler. 

I am going to look into extracting the machine file given by the scheduler and start from there. 

Concerning UCL I am leaving soon, but I think it would be nice for them to get it up and running on legion. 

Thank you for your response, and I will update this thread if I am able to write up a good submission script.

very best,

t.

[Isaac]

Hi All,
 
 I also tried to submit the julia jobs on the cluster but failed. I wrote the job script as follows:
for((i = 1; i < 10; i++))
do
echo "# cd /data
#PBS -l walltime=00:10:00
module add gcc/4.7.2
module add julia/0.2.0
module load julia
include("test.jl")
test($i)">test1job$i;
qsub test1job$i;

done
I got the errors:
julia/0.2.0(16):ERROR:151: Module 'julia/0.2.0' depends on one of the module(s) 'gcc/4.7.2'
julia/0.2.0(16):ERROR:102: Tcl command execution failed: prereq gcc/4.7.2

/cm/local/apps/torque/current/mom_priv/jobs/1053.cluster.SC: line 7: syntax error near unexpected token `a0d0.jl'
/cm/local/apps/torque/current/mom_priv/jobs/1053.cluster.SC: line 7: `include(a0d0.jl)'

Does anybody know how to write the job script to submit julia job on a cluster? Could you give an example?
Thanks in advance.

Isaac

 

[ken...@sdsc.edu]

I've run parallel julia on a Torque cluster with Infiniband.  I start an interactive session with qsub -I,
look for allocated nodes in $PBS_NODEFILE, convert to IB interface names, and addprocs.

filestream = open(ENV["PBS_NODEFILE"])
seekstart(filestream)
linearray = readlines(filestream)
strippedarray = similar(linearray)
for i in 1:length(linearray)
    strippedarray[i] = strip(linearray[i]) * "-ipoib.ipoib"
    end
for i in 1:length(strippedarray)
     singlearray = [strip(strippedarray[i])]
     addprocs(singlearray)
    end
print(workers())

To start an interactive job, depending on your node configuration and queue names:
qsub -I -l nodes=2:ppn=32,walltime=00:30:00 -q normal

When you get your nodes, start julia with the above setup file with:
julia --load setupfilename

This should addprocs then give you the julia prompt.

But it looks like something is wrong with your modules?

[Thibaut Lamadon]

looks good, I guess we should wrap this code into a ClusterManager. I am going to look into this.

[Florian Oswald]

Hi Ken,

sorry can i just ask you a question on this? Thibaut (post below) was actually able to use your function to set Julia up on our departmental SGE cluster and it works fine. That facility is down with a disk failure for the time being though, so I have been trying to get going on a different PBS cluster. I can do interactive on one node, but I am stuck going across nodes. 

I'm not sure I understand what you mean when you say "convert to IB interface names". The names on my PBS_NODEFILE are the same for each node, i.e. all CPUs on node 1 are called node 1. 

I wrote this up in more detail in this SO post. http://stackoverflow.com/questions/25089733/julia-on-pbs-cluster-what-to-give-to-addprocs

It ends up throwing an SSH error. Do you have any clue what might be going wrong here? Thanks!

Florian

On Wednesday, 7 May 2014 20:52:17 UTC+1, ken...@sdsc.edu wrote:

[Florian Oswald]

Hi Ben,

I'm getting exactly that error message on my cluster. I'm wondering where I am supposed to place the sleep(0.5) command? it seems the start_sge_worker function doesn't exist anymore (not in base julia anyway.) Would you know of any other way to make sure the environment vars are loaded? I always thought putting  source ~/.bashrc into my submit script would do that job?

best

florian