Configuring WESTPA to run in a queuing system (SLURM)

[rcre...@gmail.com]

Dear all,

I'd like to run WESTPA in a supercomputing facility that uses the SLURM queuing system (in particular the Barcelona Supercomputing Center). I also need to run the dynamics with Gromacs.

I've been searching the wiki and this group to find some information on how to setup WESPA, but I couldn't find anything. Maybe I missed something. Are there any general guidelines or tutorials?

Thanks in advance,

Ramon

[Alex DeGrave]

Hi Ramon,

My response below assumes that you wish to run WESTPA using multiple computing nodes. If you wish to run WESTPA on a single compute node of a supercomputing facility (less likely), you can continue to use the processes work manager.

To run a single WESTPA job that uses multiple computing nodes, I suggest using the ZeroMQ work manager (zmq).  The basic idea is that your Slurm submission script will start WESTPA as a master ZMQ server. Using srun, you will then launch a WESTPA process on each compute node; these WESTPA processes will be ZMQ clients ("--zmq-mode=node") which do the actual processing for the trajectory segments.  WESTPA ZMQ internals will handle the allocation of trajectory segments among the nodes and handle data collection.  

There are a few additional considerations when running WESTPA on a compute cluster/supercomputer.  First, since you will be running the job on multiple machines, there is additional room for errors with environment variable inheritance.  On newer systems I have not had much of an issue, but you may find it helpful to print environment variables (e.g., by running env) at the beginning of your runseg.sh or other scripts, at least until you have the simulation working.  Second, since you will be running many processes at the same time, it is possible to overwhelm your computing facility's global storage arrays and network.  For this reason, I suggest writing to local scratch space whenever performing a calculation (e.g., when running gmx mdrun), and then copying any desired files to global storage using rsync, after the calculations for a segment complete.  You may also find it helpful to set an environment variable to the absolute path of each executable you commonly use (e.g., GMX=$(which gmx)) when starting the WESTPA server on each compute node; this decreases the number of times that your shell needs to resolve paths to executables.

In the attached tarball, I adapted one of my Slurm/WESTPA configurations into the gmx_nacl tutorial.  This is intended to provide an example of how you might use the ZeroMQ work manager on a Slurm system.  While I have tested the Slurm+ZMQ configuration and Gromacs tutorial separately, note that I have not tested the two together, as Gromacs 2016 is not installed on the clusters I use. For this reason, and because computing clusters differ, it is quite possible that the attached example will not work "out of the box".  However it should provide a good starting place, and if you encounter errors, I would be happy to assist further.

Best,

Alex

Virus-free. www.avast.com

--
You received this message because you are subscribed to the Google Groups "westpa-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to westpa-users+unsubscribe@googlegroups.com.
To post to this group, send email to westpa...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Ramon Crehuet]

Wow! Great! Amazing!

I'll try everything and get back to you if I have unsolvable problems ;-)

Thanks!

Ramon

--
You received this message because you are subscribed to a topic in the Google Groups "westpa-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/westpa-users/xZU6LDfLblk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to westpa-users+unsubscribe@googlegroups.com.

[Sreyoshi Sur]

Hi Alex,

I tried to use your slurm example to run WESTPA on our Bluehive supercomputer. I am getting the following error:
Traceback (most recent call last):
  File "/home/ssur2/westpa/lib/cmds/w_run.py", line 37, in <module>
    work_manager = westpa.rc.work_manager = make_work_manager()
  File "/home/ssur2/westpa/lib/wwmgr/work_managers/environment.py", line 127, in make_work_manager
    return _available_work_managers[work_manager_name].from_environ(self)
  File "/home/ssur2/westpa/lib/wwmgr/work_managers/zeromq/work_manager.py", line 144, in from_environ
    raise ZMQWMEnvironmentError('at least one upstream endpoint unspecified')
work_managers.zeromq.core.ZMQWMEnvironmentError: at least one upstream endpoint unspecified

when it reaches the $WEST_ROOT/bin/w_run "$@" &> $LOG  part of node.sh. Let me know if you have faced similar issues with your script.

Thanks
Sreyoshi

To unsubscribe from this group and stop receiving emails from it, send an email to westpa-users...@googlegroups.com.

[Alex DeGrave]

Hi Sreyoshi,

Try adding the following line to the end of env.sh:

export SERVER_INFO=${WEST_SIM_ROOT}/server_info.json

I had inadvertently left this variable unset when writing the example.  Let me know if this helps!

-Alex

Virus-free. www.avast.com

To unsubscribe from this group and stop receiving emails from it, send an email to westpa-users+unsubscribe@googlegroups.com.

[rcre...@gmail.com]

Hi Alex,

I resuscitate this thread as I would like to submit my WESTPA run to a supercomputing facility.

I was re-reading your post and the example you gently provided. I have two questions.

1. The init.sh contains a line with --work-manager=threads. If run.sh uses ZMQ, shouldn't init.sh also specify --work-manager=zmq?
2. If I submit to slurm with sbatch asking for 24 nodes with -N=24 (equivalent to --nodes=24), and assuming that each gromacs is not run in parallel, i.e., each gromacs mdrun uses 1 node, does that translate to setting --nworkers=24 in the srun command of run.sh?

Thanks again!

Ramon

El divendres, 28 juliol de 2017 16:27:11 UTC+2, Alex DeGrave va escriure:

To unsubscribe from this group and stop receiving emails from it, send an email to westpa-users...@googlegroups.com.

[Alex DeGrave]

Hi Ramon,

1. The init.sh contains a line with --work-manager=threads. If run.sh uses ZMQ, shouldn't init.sh also specify --work-manager=zmq?

Even when using ZMQ to parallelize across multiple compute nodes in the main WESTPA run (handled by run.sh), I typically use the "threads" work manager in init.sh.  This is a good idea if the calculations in run.sh do not require a great enough amount of computing power to substantially benefit from parallelizing across multiple nodes.  If the calculations performed by init.sh would benefit substantially from parallelization across multiple compute nodes, then I would recommend using the ZMQ work manager in your init.sh. This would likely require some additional modifications--I have not done this before, but I think that a configuration similar to your run.sh + run_segment.sh/runseg.sh + node.sh could be used.

1. If I submit to slurm with sbatch asking for 24 nodes with -N=24 (equivalent to --nodes=24), and assuming that each gromacs is not run in parallel, i.e., each gromacs mdrun uses 1 node, does that translate to setting --nworkers=24 in the srun command of run.sh?

The "--nworkers=XX" flag is passed by "srun" to "wrun" on each of the nodes and determines the number of segments to run at once on each node. As you say, I would recommend using a single process for each instance of gromacs, so you should set "--nworkers" to the number of processors on each node.  For example, if you are using 24 nodes with 16 processors per node, you should set "--nworkers=16". The number of nodes is determined by your "srun" command. For example, "--ntasks=$SLURM_NNODES --ntasks-per-node=1" will tell slurm to launch a wrun process on each node allocated to the job, and then the flag "--nworkers=16" would tell that wrun process to run 16 segments (16 gromacs instances) on each node.

Hopefully this helps; let me know if you have questions.

Best,

Alex

[rcre...@gmail.com]

Thanks Alex,

That was very helpful.

Cheers,

Ramon

El divendres, 24 agost de 2018 17:44:35 UTC+2, Alex DeGrave va escriure: