Levels of Parallelism

[Maripaz Montero]

I am currently trying to run Exabayes in parallel in our cluster, but We have had some problems trying to understand how to adapt all different level of parallelism. 

I am working with an alignment (no partitions) that has 311 taxa with sequences of 1331 pair base long. Initially, I ran the sequential analysis and it took up to 200 hours which seems a lot compared to what is reported in your documentation. 

We have been running some test trying different combinations of the parameters implied in the parallel process including -R, -C, -np, numRuns, numCoupledChains, and numswapspergen, but unfortunately, we haven't found a logical way of combining the parameters to reach a decent time. The best Test until now lasted 20:29:01 (hh:mm:ss) and these are the configuration file and the command we are using:

1. Configuration File

#NEXUS

begin run;

numruns 2

numCoupledChains 2

numgen 4e6

numswapspergen 4

parsimonyStart  true

checkPointInterval 1e6

printFreq 1e6
end;

2. Command 

#!/bin/bash

#PBS -N exa_paral_10

#PBS -q phi-n1h72

#PBS -l nodes=1:ppn=256

#PBS -l walltime=70:00:00

cd $PBS_O_WORKDIR

module load exabayes/1.5

module load mpich/3.2.1

mpiexec -np 64 exabayes -f asco_uni.phy  -m DNA -n para10 -s 123 -c config.nex -R 2 -C 2

Do you have any suggestion about this? Especially in the config file and the -R and -C flags?. I could give you more details of our experiments if you need to understand better.

Also, we were trying to reduce the print frequency but we haven't seen any changes in results print at the output files, even with the printFreq 1e6. Is there any problem with my parameters at the config file?

Thank you in advance.

[Andre Aberer]

Hi Maripaz,

the given dataset is too small (particularly in terms of base pairs) to
use a lot of parallelism efficiently. Typically, a single process
requires around 1000 bp to have enough work. If the processes have too
little to do, then using more processes actually makes the overall
runtime increase.

* So for a single chain, you can try 1-4 processes.

* Definitely multiply that number with x (and use "numRuns x" and "-R x").

* Optionally also multiply with z (and use "numCoupledChains y"
and "-C z"), whereas z <= y.

* for parallel efficiency, reduce the swapping among coupled chains: for
numCoupledChains == 2, use "numSwapsPerGen 1" (potentially a fraction
of 1, I do not recall whether rational numebr are supported for this
option).

--
Best regards,
Andre



Maripaz Montero writes:

> I am currently trying to run Exabayes inÂparallel in our cluster, but We have
> had some problems trying toÂunderstand how to adapt all different level of
> parallelism.Â
>
> I am working with an alignment (no partitions) that has 311Âtaxa with
> sequences of 1331 pair base long. Initially, I ran the sequentialÂanalysis and
> it took up to 200_hours which seems a lot compared to whatÂis reported in your
> documentation.Â

>
> We have been running some test trying different combinations of the parameters
> implied in the parallel process including -R, -C, -np, numRuns,
> numCoupledChains, and numswapspergen, but unfortunately, we haven't found a
> logical way of combining the parameters to reach a decent time. The best Test

> until now lastedÂ20:29:01Â(hh:mm:ss) and these are the configuration file and

> the command we are using:
>
> 1. Configuration File
>
> #NEXUS
> begin run;
> numruns 2
> numCoupledChains 2
> numgen 4e6
> numswapspergen 4

> parsimonyStart Âtrue

> checkPointInterval 1e6
> printFreq 1e6
> end;
>
>

> 2. CommandÂ

>
> #!/bin/bash
> #PBS -N exa_paral_10
> #PBS -q phi-n1h72
> #PBS -l nodes=1:ppn=256
> #PBS -l walltime=70:00:00
> cd $PBS_O_WORKDIR
>
>
> module load exabayes/1.5
> module load mpich/3.2.1
>

> mpiexec -np 64 exabayes -f asco_uni.phy Â-m DNA -n para10 -s 123 -c config.nex

> -R 2 -C 2
>
>
>
> Do you have any suggestion about this? Especially in the config file and the -
> R and -C flags?. I could give you more details of our experiments if you need
> to understand better.
>
> Also, we were trying to reduce the print frequency but we haven't seen any

> changes in results print at the output files, even with theprintFreq 1e6. Is