recommendations for resources

[chris blair]

Hi all, 

I was hoping to solicit some recommendations regarding computational resources for RAxML. I plan on running the data through RAxML-NG, but would also like to run standard RAxML for comparison. The data contain 47 taxa with 2,039,295 alignment patterns. I am running the hybrid MPI-Pthreads version of RAxML 8.2.10. The cluster I am using contains 36 nodes with 16 cores each. Any ballpark suggestions to try would be welcome. I am currently running an analysis on two nodes with 10 cores each and two threads, and it is extremely slow. Thanks a bunch.

Chris

[Lucas Czech]

Hi Chris,

for starters, you can use the RAxML memory requirements calculator: http://sco.h-its.org/exelixis/web/software/raxml/index.html

Lucas

--
You received this message because you are subscribed to the Google Groups "raxml" group.
To unsubscribe from this group and stop receiving emails from it, send an email to raxml+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[chris blair]

Yeah I usually check this prior to analyses. About 12 GB RAM in this case.

I setup a new analysis using 10 MPI processes and 30 threads. I am running a combined rapid bootstrap (100 reps) + ML search. After about 10 hours only 1 bs rep has finished. Everything looks ok, so perhaps this is the speed of the analysis......

[Alexey Kozlov]

Hi Chris,

in general, you should try to assign between 5000 and 10000 DNA sites/thread for the best efficiency. Clearly, this
optimal distribution is not always achievable.

With RAxML8 in hybrid mode, you are limited to 16 threads only (since your server has 16 cores). If you are using 30,
you're oversubscribing the cores (>1 threads/core), which you should never do with RAxML as it results in a slowdown
(this is described in the manual Lucas linked above). Obviously, it will take a while to compute 1 BS with just 16
threads, but on the bright side you can run multiple BS reps in parallel (e.g. 10 BS if you use 10 MPI processes).

Alternatively, you can try ExaML (https://github.com/stamatak/ExaML) which employs across-sites parallelization with MPI
(i.e., you can use 300 MPI ranks for the single inference and thus be in the optimal range regarding sites/process).
However, ExaML doesn't support bootstrapping, so you would need some basic scripting to generate BS replicate alignments
and compute support (described in ExaML manual).

Finally, RAxML-NG supports both MPI-based across-site parallelization and bootstrapping. You can run RAxML-NG with plain
MPI (e.g. 320 processes), or in hybrid MPI/threads mode (e.g., 20 MPI processes x 16 threads, '--threads 16' option).
The latter option could be more efficient, but is often trickier to configure properly in the submission script.

Best,
Alexey

>> <javascript:>.
>> For more options, visit https://groups.google.com/d/optout <https://groups.google.com/d/optout>.

>
> --
> You received this message because you are subscribed to the Google Groups "raxml" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to raxml+un...@googlegroups.com

> <mailto:raxml+un...@googlegroups.com>.

[chris blair]

Hi Alexey, 

Thanks for the great suggestions. So the limitation for number of threads is simply the total number of cores per node? In other words, it would not be the number of cores x number of nodes? I am attaching an example job submission script. If I am understanding MPI correctly, the number of processes should equal the number of requested nodes? I'm just trying to wrap my head around setting values for nodes, ppn, np, and -T.

Chris

[Alexey Kozlov]

yes, exactly: you always specify the number of threads *per node* with "-T" / "--threads" option

On 30.03.2017 15:12, chris blair wrote:
> Hi Alexey,
>
> Thanks for the great suggestions. So the limitation for number of threads is simply the total number of cores per node?
> In other words, it would not be the number of cores x number of nodes? I am attaching an example job submission script.
> If I am understanding MPI correctly, the number of processes should equal the number of requested nodes? I'm just trying
> to wrap my head around setting values for nodes, ppn, np, and -T.
>
> Chris
>
> On Thursday, 30 March 2017 08:54:26 UTC-4, Alexey Kozlov wrote:
>
> Hi Chris,
>
> in general, you should try to assign between 5000 and 10000 DNA sites/thread for the best efficiency. Clearly, this
> optimal distribution is not always achievable.
>
> With RAxML8 in hybrid mode, you are limited to 16 threads only (since your server has 16 cores). If you are using 30,
> you're oversubscribing the cores (>1 threads/core), which you should never do with RAxML as it results in a slowdown
> (this is described in the manual Lucas linked above). Obviously, it will take a while to compute 1 BS with just 16
> threads, but on the bright side you can run multiple BS reps in parallel (e.g. 10 BS if you use 10 MPI processes).
>

> Alternatively, you can try ExaML (https://github.com/stamatak/ExaML <https://github.com/stamatak/ExaML>) which

> <javascript:>
> > <mailto:raxml+un...@googlegroups.com <javascript:>>.

[chris blair]

Hi Alexey, 

I have been running RAxML using 10 MPI processes and 15 threads, but keep receiving the attached termination error. Our sysadmin is not sure what could be causing this. I was hoping that you guys had some suggestions. The analysis always runs for a couple days prior to termination. Thanks.

Chris

[Alexandros Stamatakis]

do you have more information from the standard output and the RAxML
output files, please also indicate your command line and the queue
submission script

alexis

--
Alexandros (Alexis) Stamatakis

Research Group Leader, Heidelberg Institute for Theoretical Studies
Full Professor, Dept. of Informatics, Karlsruhe Institute of Technology
Adjunct Professor, Dept. of Ecology and Evolutionary Biology, University
of Arizona at Tucson

www.exelixis-lab.org

[chris blair]

Hi Alexis, 

Please see attached. Thanks.

Chris

[Alexey Kozlov]

Hi Chris,

the most likely reasons for the RAxML termination are:

1) your job exceeded the running time limit (48h or 72h are quite common, please check your cluster manual)

2) your job exceeded the memory limit (please check how much your nodes have and how/whether you can ask for more in the
PBS script)

Best,
Alexey

> Alexandros (Alexis) Stamatakis
>
> Research Group Leader, Heidelberg Institute for Theoretical Studies
> Full Professor, Dept. of Informatics, Karlsruhe Institute of Technology
> Adjunct Professor, Dept. of Ecology and Evolutionary Biology, University
> of Arizona at Tucson
>

> www.exelixis-lab.org <http://www.exelixis-lab.org>

[chris blair]

Hi guys, 

Thanks for the suggestions. There is no time limit on our cluster. I tested the memory requirements of the data set on the RAxML webpage (~12 GB). Interestingly, I setup two additional analyses, one with 1 node and 1 cpu/thread and another with 1 node and 2 threads and the analyses are still running. It will be interesting to see if I get the same termination error with these runs. 

[chris blair]

Our sysadmin asked me to ask you guys the following question:

"does your code try to expand the amount of memory during its work? Say, to allocate additional memory if it runs out of memory?"

[Alexandros Stamatakis]

On 07.04.2017 20:14, chris blair wrote:
> Our sysadmin asked me to ask you guys the following question:
>
> "does your code try to expand the amount of memory during its work? Say,
> to allocate additional memory if it runs out of memory?"

not really, have you tried running it with raxml-ng?

it could be a raxml-related issue but it's really hard to tell ....

alexis

> > Alexandros (Alexis) Stamatakis
> >
> > Research Group Leader, Heidelberg Institute for
> Theoretical Studies
> > Full Professor, Dept. of Informatics, Karlsruhe Institute
> of Technology
> > Adjunct Professor, Dept. of Ecology and Evolutionary
> Biology, University
> > of Arizona at Tucson
> >
> > www.exelixis-lab.org <http://www.exelixis-lab.org>
> <http://www.exelixis-lab.org>
> >
> > --
> > You received this message because you are subscribed to the
> Google Groups "raxml" group.
> > To unsubscribe from this group and stop receiving emails from
> it, send an email to raxml+un...@googlegroups.com
> > <mailto:raxml+un...@googlegroups.com>.

> > For more options, visit https://groups.google.com/d/optout

> <https://groups.google.com/d/optout>.
>
> --
> You received this message because you are subscribed to the Google
> Groups "raxml" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to raxml+un...@googlegroups.com

> <mailto:raxml+un...@googlegroups.com>.
> For more options, visit https://groups.google.com/d/optout.

[chris blair]

OK I'll give RAxML-NG a try and see what happens. However, I receive the attached error. I am assuming I just forgot to load something needed by the program. I am also attaching the PBS submission script.

Chris

[Alexey Kozlov]

Hi Chris,

1) raxml-ng: there is again something wrong with your modules/mpi configuration. why do you actually load both
"mpich-3.2" and "openmpi-1.6/gnu" in your pbs script? you should load exactly the same modules you used for raxml-ng
compilation.

2) raxml memory consumption: do you know how much RAM per node you have? is there any way you can monitor memory
consumption during the program run (e.g. by logging in to the node and running top/htop)? raxml can allocate /
re-allocate memory during the run and thus expand its memory footprint, but this additional amount should be rather
small and thus could only be a problem if it was barely enough memory in the beginning. Another thing I'd double-check
is that each MPI rank runs on its own node (and not all 10 ranks on the same node).

Best,
Alexey

> > > Alexandros (Alexis) Stamatakis
> > >
> > > Research Group Leader, Heidelberg Institute for
> > Theoretical Studies
> > > Full Professor, Dept. of Informatics, Karlsruhe Institute
> > of Technology
> > > Adjunct Professor, Dept. of Ecology and Evolutionary
> > Biology, University
> > > of Arizona at Tucson
> > >
> > > www.exelixis-lab.org <http://www.exelixis-lab.org> <http://www.exelixis-lab.org>
> > <http://www.exelixis-lab.org>
> > >
> > > --
> > > You received this message because you are subscribed to the
> > Google Groups "raxml" group.
> > > To unsubscribe from this group and stop receiving emails from
> > it, send an email to raxml+un...@googlegroups.com
> > > <mailto:raxml+un...@googlegroups.com>.
> > > For more options, visit https://groups.google.com/d/optout <https://groups.google.com/d/optout>

> > <https://groups.google.com/d/optout <https://groups.google.com/d/optout>>.
> >
> > --
> > You received this message because you are subscribed to the Google
> > Groups "raxml" group.
> > To unsubscribe from this group and stop receiving emails from it, send
> > an email to raxml+un...@googlegroups.com <javascript:>
> > <mailto:raxml+un...@googlegroups.com <javascript:>>.

> > For more options, visit https://groups.google.com/d/optout <https://groups.google.com/d/optout>.

>
> --
> Alexandros (Alexis) Stamatakis
>
> Research Group Leader, Heidelberg Institute for Theoretical Studies
> Full Professor, Dept. of Informatics, Karlsruhe Institute of Technology
> Adjunct Professor, Dept. of Ecology and Evolutionary Biology, University
> of Arizona at Tucson
>
> www.exelixis-lab.org <http://www.exelixis-lab.org>
>

[chris blair]

Hi Alexey, 

Responses below in bold. 

On Saturday, 8 April 2017 06:17:48 UTC-4, Alexey Kozlov wrote:

Hi Chris,

1) raxml-ng: there is again something wrong with your modules/mpi configuration. why do you actually load both
"mpich-3.2" and "openmpi-1.6/gnu" in your pbs script? you should load exactly the same modules you used for raxml-ng
compilation.

If I recall correctly, raxml-ng was compiled using mpich-3.2 and gcc-4.9.2. However, without loading openmpi there is no mpirun executable to run the program. There is probably an easy fix here, but I am not sure. 

 

2) raxml memory consumption: do you know how much RAM per node you have? is there any way you can monitor memory
consumption during the program run (e.g. by logging in to the node and running top/htop)? raxml can allocate /
re-allocate memory during the run and thus expand its memory footprint, but this additional amount should be rather
small and thus could only be a problem if it was barely enough memory in the beginning. Another thing I'd double-check
is that each MPI rank runs on its own node (and not all 10 ranks on the same node).

We have 128 GB RAM per node. How can I check that MPI ranks are running on different nodes? Interestingly, my trial analyses I set up with 1 node and 2 threads are still running and I am not sure why/how. I thought that I would receive some sort of memory error here. These analyses appear to be running fine, albeit very slowly. This is pretty much my first serious dive into parallel computing. I hope others have an easier time. :/

[chris blair]

I also setup a RAxML analysis using two MPI processes and 15 threads that has been running ok for about 5 days. Thus, it appears that the error is related to requesting a particular number of MPI processes. Not sure why though...

Chris

[Alexey Kozlov]

Hi Chris,

> I also setup a RAxML analysis using two MPI processes and 15 threads that has been running ok for about 5 days. Thus, it
> appears that the error is related to requesting a particular number of MPI processes. Not sure why though...

which indicates that there might be indeed multiple MPI processes running on the same node in your "big" job..


> *If I recall correctly, raxml-ng was compiled using mpich-3.2 and gcc-4.9.2. However, without loading openmpi there
> is no mpirun executable to run the program. There is probably an easy fix here, but I am not sure. *

this is weird, you should ask your admins for clarification. in your last error screenshot, raxml-ng was unable to load
the MPI dynamic library, this might be because wrong modules were loaded, or due to errors in the module definition
itself (e.g. LD_LIBRARY_PATH is not set properly). in either case, please ask you admins for advice, since this error
has nothing to do with raxml code, but with MPI configuration on your system.

> 2) raxml memory consumption: do you know how much RAM per node you have? is there any way you can monitor memory
> consumption during the program run (e.g. by logging in to the node and running top/htop)? raxml can allocate /
> re-allocate memory during the run and thus expand its memory footprint, but this additional amount should be rather
> small and thus could only be a problem if it was barely enough memory in the beginning. Another thing I'd
> double-check
> is that each MPI rank runs on its own node (and not all 10 ranks on the same node).
>

> *We have 128 GB RAM per node. How can I check that MPI ranks are running on different nodes?

if you can ssh to the cluster node(s) assigned to you job, you can check it visually with top command (there must be
only 1 raxml-ng-mpi process per node in your configuration). otherwise, please ask your admins to check this for you.

Best,
Alexey

[chris blair]

Hi Alexey and Alexis, 

I wanted to ask if you have a dummy PBS script and corresponding explanation about how to properly parallelize RAxML analyses. According to our sysadmin, I am oversubscribing in my script (see attached). Our cluster has 36 nodes and 16 cores per node. I understand that in my script I am requesting 2 nodes and 15 processors per node. However, how does this related to the 'np' command in mpirun and -T in RAxML? Any additional info would be great. Thanks a bunch!

Chris

[Alexey Kozlov]

Hi Chris,

> I wanted to ask if you have a dummy PBS script and corresponding explanation about how to properly parallelize RAxML
> analyses. According to our sysadmin, I am oversubscribing in my script (see attached). Our cluster has 36 nodes and 16
> cores per node. I understand that in my script I am requesting 2 nodes and 15 processors per node.

I'm not fluent in PBS, but as far as I can tell, this is correct.

> However, how does
> this related to the 'np' command in mpirun and -T in RAxML? Any additional info would be great. Thanks a bunch!

With '-np' you specify how many MPI ranks you start, and with '-T' - how many threads per rank.

So actually your script looks fine, but there are still some things which can go wrong, e.g.:

1) "ppn" might start 15 MPI processes (not sure about this). BTW why do you use 15 if your nodes have 16 cores?
2) both MPI ranks can be started on the same node
3) all threads might be pinned to the same core

So please ask your sysadmin how he figured out that you are oversubscribing? E.g. htop screenshot or any output where I
can see how threads were mapped to CPU cores...

Best,
Alexey