differences in runtime for large trees using raxml, raxml-ng and raxml-ng-mpi

[martha.k...@yahoo.com]

Hello RAxML developers,

I am trying to calculate a large tree with more than 35.000 tips but a short alignment of just about 5000bp. First, I run raxml-ng (mpi), but discovered that raxml seems to be faster.

Interestingly, the runs using raxml-ng did not even finish with the estimation of the phylogeny based on the first starting tree using a cluster with 16 cores subscribed to the job after running for 150 hours. I then tried to use the mpi implementation, but the cluster I have access to its difficult to get all CPU's per node and if I use multiple workers and threads (max 24CPU out of up to 56) there also seems to be no speed-up.

I then tried to use the RAXML-NG version on CIPRES to see if it was related to compilation mistakes or such. The job does not finish the first tree during the maximum allowed runtime (48hours). I then decided to use the RAXML implementation on CIPRES (raxmlHPC-PTHREADS-AVX), which finished the job in less than 40hours.

My question is now, why could that be? I assumed RAXML-NG will be faster than raxml. I do think I set up the run wrongly on the cluster (with PBS scheduler). I assume that I am specifying settings wrongly on my home cluster, but as also the RAXML job is finished on CIPRES much faster, let's me wonder...

Any advice  and ideas why raxml seems to be faster and on how to set up a raxml-ng-mpi run would be appreciated - I attached the script I used.

Below I provided some of the commands I was using as well as how it was executed on CIPRES.

Thank you very much in advance for any insights.

Martha

---------------------------

Commands used:

For RAXML-NG I used:

 raxml-ng --msa concatenated_edited.out --model GTR+G --threads auto{MAX} --workers auto{MAX} --tree-constraint constraint_input_topology_edited.tre --prefix concat_addafro_raxmlv1_auto

While for raxml-ng-mpi, I used different combinations of thread and worker options: specifying them, use MAX{value} and not specifying workers at all, and including/excluding --extra thread-pin :

raxml-ng-mpi --tree rand{2} --msa concatenated_edited.out --model GTR+G --threads 16 --workers 2 --tree-constraint constraint_input_topology_edited.tre --prefix concat_addafro_raxmlmpi

On CIPRES the following commands were used:

1. raxmlHPC-PTHREADS-AVX -T 16 -F -n result -s infile.txt -c 25 -m GTRCAT -p 12345 -g constraint.tre

2. raxml-ng-mpi --force perf_threads --search --msa infile --prefix cipres --tree-constraint constraint.tre --model GTR+IO+FC

[Alexey Kozlov]

Hi Martha,

could you please attach log files from all runs?

In general, this situation has been discussed several times on this group, please see e.g.:

https://groups.google.com/g/raxml/search?q=raxml-ng%20slower%20raxml

https://groups.google.com/g/raxml/c/sbnYMV_pW1A/m/yj2XKTJOAgAJ

Best,
Alexey

> --
> You received this message because you are subscribed to the Google Groups "raxml" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to
> raxml+un...@googlegroups.com <mailto:raxml+un...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/raxml/05998a7b-bb63-42cf-a860-9d93d0a1cfb6n%40googlegroups.com
> <https://groups.google.com/d/msgid/raxml/05998a7b-bb63-42cf-a860-9d93d0a1cfb6n%40googlegroups.com?utm_medium=email&utm_source=footer>.

[Martha Kandziora]

Hi Alexey,

Thanks for your prompt reply.  Log files of the Cipres runs and two log
files from my cluster runs (one raxml-ng one raxml-ng-mpi) attached.

I have been reading quite a bit in the google groups, but could not come
up with an answer why raxml seems to be faster and finishes in a very
short time, while raxml-ng seems to run forever. As I said, I first
assumed it is because the way I try to call raxml-ng in mpi mode, but
that the same thing happened on CIPRES surprised me.

Best,

Martha

[Alexey Kozlov]

Hi Martha,

I will write a more detailed response tomorrow, but here are just a few points which I spotted after
looking at your "local" run of raxml-ng:

- you have ~4k duplicate sequences, please remove them

- you should put *total* number of threads in the --threads option, ie it should be
"--threads auto{32} --workers auto{4}"

- GTR+I is ~4x faster then GTR+G and thus more comparable to GTRCAT in terms of runtime

- RAxML uses a single parsimony starting tree by default, whereas your raxml-ng runs performed
multiple parallel inferences starting from random trees

- how resolved is your constraint tree (# internal nodes)?

Hope this helps,
Alexey

[Martha Kandziora]

Hi Alexey,

Answers to your suggestions and questions below:

1. I am trying to reproduce an existing study, hence I would like to
keep the duplicated sequences if possible

2. Ah, I need to do that because of the higher number of virtual CPU's I
assume. Thank you.

3. Ok, I will adjust my substitution model

4. I believe the constraint topology only has 471 internal nodes  with
36105 tips - it is used to keep the backbone resolution to what is
assumed to be correct; compared to an alignment with 36190 tips

Best,

Martha

[Alexey Kozlov]

Hi Martha,

thanks for your answers/clarifications!

I guess that after fixing technical issues and with "GTR+I" model, you should see ~similar runtimes
between raxml and raxml-ng. Although raxml can still be faster due to using a parsimony starting
tree or other factors. In general, depending on the dataset, raxml-ng can be slower but find a
better tree (there is a reason why we use 20 starting tree by default). However, this does not
really matter, because:

The underlying problem - which we discussed here a few times, most recently w.r.t. SARS-Cov-2 data -
is the lack of signal in the alignment. In your case, there are >36000 taxa and just under 5000
sites, of which ~50% are invariant and ~80% are gaps. We know from experience with both empirical
and simulated data, that it is virtually impossible to recover a stable, let alone the "true"
topology from such data. Sure, there will be probably some "good" branches with high support. But in
most cases, the branching order will be either completely random (duplicate seqs) or almost random
(seqs differ in gap pattern or in 1-2 positions -> could be a sequencing error).

So, if you really need to get a bifurcating tree with all 36000 taxa, and you accept that most of
its branches will be random - you can use FastTree and save lots of time/energy :) The additional
work that raxml/raxml-ng are investing into finding a better-scoring tree simply wouldn't pay off:
as can be seen in simulations, for such "short" alignments, the true tree often has *lower*
likelihood score than a "wrong" topology found by raxml...

Alternatively, you can try to cluster your sequences into fewer OTUs with reasonable variation, and
then run raxml-ng on this reduced dataset.

Hope this helps,
Alexey

[mahwa...@gmail.com]

Hi Alexey, 

I have been reading several posts in the raxml google group to try and find advice for dealing with my particular dataset (many taxa, short alignment). 

In your previous reply you stated "The additional work that raxml/raxml-ng are investing into finding a better-scoring tree simply wouldn't pay off:
as can be seen in simulations, for such "short" alignments, the true tree often has *lower* likelihood score than a "wrong" topology found by raxml...". 

I found this very interesting, and wonder if there is a particular reference you can point me to that discusses this problem. 

Many thanks! 

Mahwash

[Oleksiy Kozlov]

Hi Mahwash,

please have a look at the following references:

https://www.biorxiv.org/content/10.1101/2022.10.31.514545v1

https://academic.oup.com/mbe/article/39/12/msac254/6832260?login=false

Best,
Oleksiy

> <https://groups.google.com/g/raxml/search?q=raxml-ng%20slower%20raxml>
> >>>>
> >>>> https://groups.google.com/g/raxml/c/sbnYMV_pW1A/m/yj2XKTJOAgAJ

> <https://groups.google.com/d/msgid/raxml/05998a7b-bb63-42cf-a860-9d93d0a1cfb6n%40googlegroups.com?utm_medium=email&utm_source=footer <https://groups.google.com/d/msgid/raxml/05998a7b-bb63-42cf-a860-9d93d0a1cfb6n%40googlegroups.com?utm_medium=email&utm_source=footer>>.

> >>>>>
> >>>>>
> >>>>
> >>>
> >>
> >
>
> --
> You received this message because you are subscribed to the Google Groups "raxml" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to
> raxml+un...@googlegroups.com <mailto:raxml+un...@googlegroups.com>.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/raxml/c4d83ee9-0111-43ea-99cb-7c4bbcc40a4bn%40googlegroups.com
> <https://groups.google.com/d/msgid/raxml/c4d83ee9-0111-43ea-99cb-7c4bbcc40a4bn%40googlegroups.com?utm_medium=email&utm_source=footer>.

[mahwa...@gmail.com]

Fantastic, thank you so much!! 

Best regards,

Mahwash