raxml-ng compile issue

[Simey]

Hello Alexey Kozlov

I am testing raxml-ng 0.5.0 on an Ubuntu 16.04 with 96 threads and 1TB RAM system.

I am looking for a way to run 200 bootstrap reps of >600 unique alignments each with >250 taxa (UCE alignments).

in standard RAxML v8, I ran three independent threaded analyses of 2, 9 and 27 threads

For e.g.:

raxmlHPC-PTHREADS-SSE3 -m GTRGAMMA -N 200 -p 25258 -b 2045 -n uce-133.nexus.phylip_boot.tre -s uce-133.phylip -T 2

the times were:

-T 2 = Overall Time for 200 Bootstraps 5862.316698

-T 9 = Overall Time for 200 Bootstraps 4026.192554

-T 27 = Overall Time for 200 Bootstraps 4545.911414

I then tried combinations of 2 and 18 threads with raxml-ng (PTHREADS version - BTW, I had to install flex and bison before raxml-ng would compile):

raxml-ng --bootstrap --msa uce-133.nexus.phylip --model GTR+G --bs-trees 200 --threads 18

I can tell from the screen output that the raxml-ng commands I used are going much slower: In fact, the 18 threaded run is quite a bit slower than the 2 threaded run.

$ ./raxml-ng --bootstrap --msa  uce-133.phylip --model GTR+G --bs-trees 200 --threads 2

RAxML-NG v. 0.5.0 BETA released on 10.09.2017 by The Exelixis Lab.

Authors: Alexey Kozlov, Alexandros Stamatakis, Diego Darriba, Tomas Flouri, Benoit Morel.

Latest version: https://github.com/amkozlov/raxml-ng

Questions/problems/suggestions? Please visit: https://groups.google.com/forum/#!forum/raxml

WARNING: This is a BETA release, please use at your own risk!

RAxML-NG was called as follows:

./raxml-ng --bootstrap --msa uce-133.phylip --model GTR+G --bs-trees 200 --threads 2

Analysis options:

  run mode: Bootstrapping

  start tree(s): random

  random seed: 1508793531

  tip-inner: ON

  pattern compression: ON

  per-rate scalers: OFF

  site repeats: OFF

  branch lengths: ML estimate (linked)

  SIMD kernels: AVX2

  parallelization: PTHREADS (2 threads)

[00:00:00] Reading alignment from file: /home/ubuntu/ABySS-run/mafft-nexus-min75_101117/uce-133.nexus.phylipnew/uce-133.nexus.phylip

[00:00:00] Loaded alignment with 267 taxa and 148 sites

Alignment comprises 1 partitions and 135 patterns

Partition 0: noname

Model: GTR+FO+G4m

Alignment sites / patterns: 148 / 135

Gaps: 6.96 %

Invariant sites: 26.35 %

[00:00:00] Generating random starting tree(s) with 267 taxa

[00:00:00] Data distribution: partitions/thread: 1-1, patterns/thread: 67-68

[00:00:00] Starting bootstrapping analysis with 200 replicates.

[00:00:46] Bootstrap tree #1, logLikelihood: -2492.572653

[00:01:16] Bootstrap tree #2, logLikelihood: -2155.639042

[00:02:13] Bootstrap tree #3, logLikelihood: -2285.014841

[00:03:19] Bootstrap tree #4, logLikelihood: -2183.314685

[00:05:00] Bootstrap tree #5, logLikelihood: -2262.891542

[00:05:53] Bootstrap tree #6, logLikelihood: -2421.516161

[00:07:10] Bootstrap tree #7, logLikelihood: -2580.204480

[00:08:53] Bootstrap tree #8, logLikelihood: -2338.716213

[00:10:16] Bootstrap tree #9, logLikelihood: -2586.819640

[00:10:56] Bootstrap tree #10, logLikelihood: -2437.456459

[00:11:47] Bootstrap tree #11, logLikelihood: -2207.260018

[00:13:08] Bootstrap tree #12, logLikelihood: -2901.567688

[00:14:01] Bootstrap tree #13, logLikelihood: -2385.922956

[00:14:34] Bootstrap tree #14, logLikelihood: -2250.683590

[00:15:19] Bootstrap tree #15, logLikelihood: -2719.960573

[00:16:38] Bootstrap tree #16, logLikelihood: -2615.709718

[00:17:33] Bootstrap tree #17, logLikelihood: -2401.533426

[00:18:34] Bootstrap tree #18, logLikelihood: -2485.960925

[00:19:39] Bootstrap tree #19, logLikelihood: -2523.752384

[00:20:48] Bootstrap tree #20, logLikelihood: -2311.703863

[00:21:25] Bootstrap tree #21, logLikelihood: -2544.800858

[00:22:33] Bootstrap tree #22, logLikelihood: -2352.783373

[00:23:19] Bootstrap tree #23, logLikelihood: -2516.178745

[00:23:57] Bootstrap tree #24, logLikelihood: -2424.140548

[00:25:27] Bootstrap tree #25, logLikelihood: -2385.909791

[00:26:19] Bootstrap tree #26, logLikelihood: -2597.422304

[00:27:29] Bootstrap tree #27, logLikelihood: -2312.359387

[00:29:11] Bootstrap tree #28, logLikelihood: -2693.774128

[00:30:11] Bootstrap tree #29, logLikelihood: -2560.257694

[00:31:02] Bootstrap tree #30, logLikelihood: -2257.553248

[00:32:09] Bootstrap tree #31, logLikelihood: -2579.613811

[00:33:02] Bootstrap tree #32, logLikelihood: -2460.225493

[00:34:16] Bootstrap tree #33, logLikelihood: -2491.777318

[00:36:06] Bootstrap tree #34, logLikelihood: -2491.504897

[00:36:39] Bootstrap tree #35, logLikelihood: -2542.115038

[00:37:44] Bootstrap tree #36, logLikelihood: -2740.781229

[00:38:41] Bootstrap tree #37, logLikelihood: -2518.216137

[00:39:57] Bootstrap tree #38, logLikelihood: -2390.058095

[00:40:36] Bootstrap tree #39, logLikelihood: -2700.039225

[00:41:39] Bootstrap tree #40, logLikelihood: -2896.804410

[00:42:15] Bootstrap tree #41, logLikelihood: -2325.960740

[00:44:20] Bootstrap tree #42, logLikelihood: -2560.804338

$ ./raxml-ng --bootstrap --msa uce-133.phylip --model GTR+G --bs-trees 200 --threads 18

RAxML-NG v. 0.5.0 BETA released on 10.09.2017 by The Exelixis Lab.

Authors: Alexey Kozlov, Alexandros Stamatakis, Diego Darriba, Tomas Flouri, Benoit Morel.

Latest version: https://github.com/amkozlov/raxml-ng

Questions/problems/suggestions? Please visit: https://groups.google.com/forum/#!forum/raxml

WARNING: This is a BETA release, please use at your own risk!

RAxML-NG was called as follows:

./raxml-ng --bootstrap --msa uce-133.phylip --model GTR+G --bs-trees 200 --threads 18

Analysis options:

  run mode: Bootstrapping

  start tree(s): random

  random seed: 1508794173

  tip-inner: ON

  pattern compression: ON

  per-rate scalers: OFF

  site repeats: OFF

  branch lengths: ML estimate (linked)

  SIMD kernels: AVX2

  parallelization: PTHREADS (18 threads)

[00:00:00] Reading alignment from file: /home/ubuntu/ABySS-run/mafft-nexus-min75_101117/uce-133.nexus.phylipnew/uce-133.nexus.phylip

[00:00:00] Loaded alignment with 267 taxa and 148 sites

Alignment comprises 1 partitions and 135 patterns

Partition 0: noname

Model: GTR+FO+G4m

Alignment sites / patterns: 148 / 135

Gaps: 6.96 %

Invariant sites: 26.35 %

[00:00:00] NOTE: Resuming execution from checkpoint (logLH: -2438.17, ML trees: 0, bootstraps: 9)

[00:00:00] Generating random starting tree(s) with 267 taxa

[00:00:00] Data distribution: partitions/thread: 1-1, patterns/thread: 7-8

[00:00:00] Starting bootstrapping analysis with 200 replicates.

[00:04:11] Bootstrap tree #10, logLikelihood: -2519.487203

[00:06:02] Bootstrap tree #11, logLikelihood: -2262.312945

[00:07:18] Bootstrap tree #12, logLikelihood: -2255.822415

[00:08:30] Bootstrap tree #13, logLikelihood: -2563.356058

[00:09:24] Bootstrap tree #14, logLikelihood: -2264.489331

[00:10:34] Bootstrap tree #15, logLikelihood: -2375.369165

[00:12:44] Bootstrap tree #16, logLikelihood: -2310.770906

[00:14:41] Bootstrap tree #17, logLikelihood: -2181.449061

[00:17:14] Bootstrap tree #18, logLikelihood: -2310.273271

[00:18:43] Bootstrap tree #19, logLikelihood: -2282.335232

[00:20:44] Bootstrap tree #20, logLikelihood: -2542.262080

[00:22:04] Bootstrap tree #21, logLikelihood: -2227.117891

[00:23:41] Bootstrap tree #22, logLikelihood: -2468.203492

[00:25:32] Bootstrap tree #23, logLikelihood: -2283.919004

[00:27:06] Bootstrap tree #24, logLikelihood: -2475.734104

[00:28:36] Bootstrap tree #25, logLikelihood: -2899.481659

[00:29:59] Bootstrap tree #26, logLikelihood: -2510.022625

[00:31:26] Bootstrap tree #27, logLikelihood: -2537.418850

[00:33:07] Bootstrap tree #28, logLikelihood: -2622.239538

I must not be running equivalent analyses.

Can you see what I am doing wrong here?

[Simey]

Sorry about the misleading post title. I could not find a way to edit that after posting. :P

[Alexey Kozlov]

Hi Simey,

a couple of things:

- you do use slightly different models, namely:

GTRGAMMA = GTR+G+F = empirical base freqs (RAxML8 default)
GTRGAMMAX = GTR+G = GTR+G+FO = ML estimate of base freqs (raxml-ng default)

- more importantly, you shouldn't use multiple threads here since your alignment is very short (~100 AAs). Especially
since you have 600 alignments to analyze and you server has lots of memory, it would be MUCH more efficient to run up to
96 individual single-threaded (--threads 1) raxml-ng instances in parallel, with different alignments.

this is a very common mistake unfortunately, so I'll add a corresponding warning in the next raxml-ng release

- regarding the compilation, I believe flex and bison dependencies are mentioned in the github README, so that's OK

Hope this helps,
Alexey

> --
> You received this message because you are subscribed to the Google Groups "raxml" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to raxml+un...@googlegroups.com
> <mailto:raxml+un...@googlegroups.com>.
> For more options, visit https://groups.google.com/d/optout.

[Simey]

Thank you Alexey!

my raxml-ng bootstrap completed using the different model and they were significantly slower than raxmlHPC-PTHREADS-SSE3. I am testing raxml-ng again with the appropriate model using --thread 1. I do not know what the "restart" message is about on the 18 thread run (see below).

Many of the UCE alignments have many taxa with identical sequences and I do have the "reduced" version of the phylip file. Obviously using the reduced version will speed things up. But, I am running these bootstrap analyses for downstream Astral analyses. I am not sure if using the reduced files will impact the results of the Astral analyses. Something I will test.

the results were:

raxmlHPC-PTHREADS-SSE3 (GTRGAMMA)

-T 2 = Overall Time for 200 Bootstraps 5862.316698

-T 9 = Overall Time for 200 Bootstraps 4026.192554

-T 27 = Overall Time for 200 Bootstraps 4545.911414

raxml-ng (GTRGAMMAX)

--threads 2 = Elapsed time: 12073.097 seconds

--threads 18 = Elapsed time: 17998.946 seconds (this run) / 18641.102 seconds (total with restarts)

[Alexandros Stamatakis]

On 24.10.2017 20:04, Simey wrote:
> Thank you Alexey!
>
> my raxml-ng bootstrap completed using the different model and they were
> significantly slower than raxmlHPC-PTHREADS-SSE3. I am testing raxml-ng
> again with the appropriate model using --thread 1. I do not know what
> the "restart" message is about on the 18 thread run (see below).
> Many of the UCE alignments have many taxa with identical sequences and I
> do have the "reduced" version of the phylip file. Obviously using the
> reduced version will speed things up. But, I am running these bootstrap
> analyses for downstream Astral analyses. I am not sure if using the
> reduced files will impact the results of the Astral analyses. Something
> I will test.

If you have identical sequences in your alignments you should absolutely
remove them as they may perturb the results if you want to input them
into Astral.

Alexis

> send an email to raxml+un...@googlegroups.com <javascript:>
> > <mailto:raxml+un...@googlegroups.com <javascript:>>.

> > For more options, visit https://groups.google.com/d/optout

> <https://groups.google.com/d/optout>.

>
> --
> You received this message because you are subscribed to the Google
> Groups "raxml" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to raxml+un...@googlegroups.com
> <mailto:raxml+un...@googlegroups.com>.
> For more options, visit https://groups.google.com/d/optout.

--
Alexandros (Alexis) Stamatakis

Research Group Leader, Heidelberg Institute for Theoretical Studies
Full Professor, Dept. of Informatics, Karlsruhe Institute of Technology

www.exelixis-lab.org

[Simey]

I ran a couple of bootstrap tests comparing AVX, SSE3, and raxml-ng (AVX2 throws an instruction error on my system (four Xeon E5-4657LV2)):

 

1 thread:

raxmlHPC-AVX -m GTRGAMMA -N 100 -p 25258 -b 2045 -n uce-133.AVX-1thread -s uce-133.phylip

Overall Time for 100 Bootstraps 2983.729318

 

raxmlHPC-SSE3 -m GTRGAMMA -N 100 -p 25258 -b 2045 -n uce-133.SSE3-1thread -s uce-133.phylip

Overall Time for 100 Bootstraps 3704.440428

 

raxml-ng --bootstrap --msa uce-133.phylip --model GTR+G+F --bs-trees 100 --threads 1

Elapsed time: 6429.665 second

 

2 threads:

raxmlHPC-PTHREADS-AVX -m GTRGAMMA -N 100 -p 25258 -b 2045 -n uce-133.pthreadAVX-2thread -s uce-133.phylip -T 2 

Overall Time for 100 Bootstraps 2365.280369

 

raxmlHPC-PTHREADS-SSE3 -m GTRGAMMA -N 100 -p 25258 -b 2045 -n uce-133.pthreadSSE3-2thread -s uce-133.phylip -T 2 

Overall Time for 100 Bootstraps 2554.114177

 

raxml-ng --bootstrap --msa uce-133.phylip --model GTR+G+F --bs-trees 100 --redo --threads 2

Elapsed time: 5660.265 seconds

 

On my system, AVX performs the best and since two threads is not twice as fast as 1 thread, it makes sense, as Alexey suggested, to run twice as many single thread runs than half as many two thread runs.

I don't know why raxml-ng was so much slower. The commands must not call for equivalent analyses.

[Alexey Kozlov]

Hi Simey,

thanks for testing and reporting back the results!

This is indeed unexpected that raxml-ng is slower in this case, could you please send me this alignment of yours?

Thanks!

Best,
Alexey

On 26.10.2017 01:24, Simey wrote:
> I ran a couple of bootstrap tests comparing AVX, SSE3, and raxml-ng (AVX2 throws an instruction error on my system (four
> Xeon E5-4657LV2)):
>

> _*1 thread:*_

>
> raxmlHPC-AVX -m GTRGAMMA -N 100 -p 25258 -b 2045 -n uce-133.AVX-1thread -s uce-133.phylip
>
> Overall Time for 100 Bootstraps 2983.729318
>
> raxmlHPC-SSE3 -m GTRGAMMA -N 100 -p 25258 -b 2045 -n uce-133.SSE3-1thread -s uce-133.phylip
>
> Overall Time for 100 Bootstraps 3704.440428
>
> raxml-ng --bootstrap --msa uce-133.phylip --model GTR+G+F --bs-trees 100 --threads 1
>
> Elapsed time: 6429.665 second
>

> _*2 threads:*_

> Alexandros (Alexis) Stamatakis
>
> Research Group Leader, Heidelberg Institute for Theoretical Studies
> Full Professor, Dept. of Informatics, Karlsruhe Institute of Technology
>

> www.exelixis-lab.org <http://www.exelixis-lab.org>

[Simey]

No problem Alkexey, where shall I send the file? I will send you the reduced file as well if you like.

[Alexey Kozlov]

thanks Simey, just send it to my e-mail.

although I have an idea what the difference might be - by default, RAxML8 uses parsimony starting trees for bootstraps,
whereas RAxML-NG uses the random ones.

so in order to make both analyses equivalent you'd need to add "-d" option to the raxmlHPC-AVX command line

> > www.exelixis-lab.org <http://www.exelixis-lab.org> <http://www.exelixis-lab.org>

> >
> > --
> > You received this message because you are subscribed to the Google Groups "raxml" group.
> > To unsubscribe from this group and stop receiving emails from it, send an email to raxml+un...@googlegroups.com

> <javascript:>
> > <mailto:raxml+un...@googlegroups.com <javascript:>>.
> > For more options, visit https://groups.google.com/d/optout <https://groups.google.com/d/optout>.
>
> --