[maker-devel] A question regarding the parallelization of MAKER on a cluster

[Gerrit]

Hello fellow MAKER-Users!

I am working on a new non-model-organism genome for my diploma thesis,
for this purpose I am using the MAKER for genome annotation.

Our cluster consists of 14 nodes, each node being a dual quad core and
I want to use 8 out of the 14 nodes for my MAKER run.
Therefore I have chosen n= 8 for the mpiexec command

But unfortunately I am not sure how to configure the maker_opts.ctl
How should I define the value for „cpus“ under the headline
“#-----External Application Specific Options“?

Our In-House System Administrator has told me that 1 CPU can either
refer to 1 core (8 for each node for a dual quad core) or to 1 processor
(2 for each node for a dual quad core).
Therefore the correct value for „cpus“ for running MAKER with 8 nodes
should either be 64 (8*8) or 16 (8*2).

Which one is right?

I very much appreciate your help!

Cheers Gerrit

Gerrit Hartig - Diploma student
Center for Molecular Biodiversity
Zoological Research Museum Alexander Koenig
Adenauerallee 160-162
D-53113 Bonn
E-mail: ge...@uni-bonn.de

http://www.zfmk.de

_______________________________________________
maker-devel mailing list
maker...@box290.bluehost.com
http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org

[Carson Holt]

You want to use 1 cpu in the maker_opts.ctl file.  This refers to the number of cpus to give to blast as part of the blast command line rather than to MPI usage.  So if you were to run mpiexec -n 4 and also set cpus to 4, then BLAST would run 4 times with 4 cpus each, so 16 cpus total.  So set cpus to 1.

The cpu option lets users parallelize BLAST without using MPI, so this requires that all cpus for the BLAST instance be on the same machine.  MPI usage with MAKER on the other hand, actually parallelizes all steps and not just BLAST and can be split across multiple machines.  I know at first glance it seems a little confusing.  Just think of cpus as a less effective MAKER parallelization method for non-MPI machines.

There are cases though where both methods can be used simultaneously with certain advanced benefits such as memory management.  For example, each MAKER instance will use between 500Mb-1 Gb of RAM, so 8 nodes will use between 4Gb and 8Gb total (broken up across multiple nodes).  Increasing cpus has only a minor effect on memory usage.  So if you are memory limited you can mix the MPI –n option with the cpus option to increase cpu usage while keeping memory usage low.  In this case the max cpus value would be equal to the number of cores on a single node.  You would then have to ensure that each node starts only a single instance of MAKER using the --machinefile option for mpiexec.

Thanks,
Carson