RAxML-NG v0.9.0 | Estimated Running Time.

[George Pacheco]

Dear all, 

I am running RAxML-NG v0.9.0 as below on a dataset with 596 samples and 1,331,267. 

raxml-ng --threads 25 --all --model GTR+G --site-repeats on --redo --msa ./FPGP--GoodSamples_SuperPhylogeny--Article--Ultra.fasta --tree ./FPGP--GoodSamples_SuperPhylogeny--Article--Ultra.nwk --prefix ./FPGP--GoodSamples_SuperPhylogeny--Article--Ultra.ngsDist

Currently, the log file ends like this:

Alignment comprises 1 partitions and 381863 patterns

Partition 0: noname

Model: GTR+FO+G4m

Alignment sites / patterns: 1331267 / 381863

Gaps: 0.53 %

Invariant sites: 69.41 %

NOTE: Binary MSA file created: /groups/hologenomics/pacheco/data/Pigeons/FPGP/FPGP--Analyses/FPGP--Phylogenies/SuperPhy/FPGP--GoodSamples_SuperPhylogeny--Article--Ultra.ngsDist.raxml.rba

[00:00:31] Loading user starting tree(s) from: /groups/hologenomics/pacheco/data/Pigeons/FPGP/FPGP--Analyses/FPGP--Phylogenies/SuperPhy/FPGP--GoodSamples_SuperPhylogeny--Article--Ultra.nwk

[00:00:31] Data distribution: max. partitions/sites/weight per thread: 1 / 15275 / 244400

Starting ML tree search with 1 distinct starting trees

It has been running for more than 48h and I would like to know if there would be any way to estimate the expected running time in this case.

Many thanks in advance, George.

[George Pacheco]

and 1,331,267 SITES ***

[Alexey Kozlov]

Hi George,

that's not an easy question to answer...

You can restart your job with "--log progress", this way you will at least see which tree search
state you are currently at.

Another thing to (always) double-check is thread/core mapping (i.e that multiple threads don't share
a singe CPU core). This could be done with "htop" or "ps", eg:

$ watch -tdn0.5 ps -mo pid,tid,%cpu,psr -p \`pgrep raxml-ng\`

Although your dataset is pretty large, so >48h runtime could well be genuine. In this case, I would
recommend allocating more CPU cores (if available).

Best,
Alexey

On 19.11.19 15:46, George Pacheco wrote:
> and 1,331,267 SITES ***
>
> On Tuesday, 19 November 2019 14:45:21 UTC, George Pacheco wrote:
>
> Dear all,
>
> I am running RAxML-NG v0.9.0 as below on a dataset with 596 samples and 1,331,267.
>

> *raxml-ng --threads 25 --all --model GTR+G --site-repeats on --redo --msa
> ./FPGP--GoodSamples_SuperPhylogeny--Article--Ultra.fasta --tree
> ./FPGP--GoodSamples_SuperPhylogeny--Article--Ultra.nwk --prefix
> ./FPGP--GoodSamples_SuperPhylogeny--Article--Ultra.ngsDist*
>
> Currently, the log file ends like this:*

>
> Alignment comprises 1 partitions and 381863 patterns
>
> Partition 0: noname
>
> Model: GTR+FO+G4m
>
> Alignment sites / patterns: 1331267 / 381863
>
> Gaps: 0.53 %
>
> Invariant sites: 69.41 %
>
> NOTE: Binary MSA file created:
> /groups/hologenomics/pacheco/data/Pigeons/FPGP/FPGP--Analyses/FPGP--Phylogenies/SuperPhy/FPGP--GoodSamples_SuperPhylogeny--Article--Ultra.ngsDist.raxml.rba
>
>
> [00:00:31] Loading user starting tree(s) from:
> /groups/hologenomics/pacheco/data/Pigeons/FPGP/FPGP--Analyses/FPGP--Phylogenies/SuperPhy/FPGP--GoodSamples_SuperPhylogeny--Article--Ultra.nwk
>
>
> [00:00:31] Data distribution: max. partitions/sites/weight per thread: 1 / 15275 / 244400
>
> Starting ML tree search with 1 distinct starting trees

> *

> It has been running for more than 48h and I would like to know if there would be any way to
> estimate the expected running time in this case.
>
> Many thanks in advance, George.
>

> --
> You received this message because you are subscribed to the Google Groups "raxml" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to
> raxml+un...@googlegroups.com <mailto:raxml+un...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/raxml/5ef7231e-2da9-4562-80b8-b4588e42a165%40googlegroups.com
> <https://groups.google.com/d/msgid/raxml/5ef7231e-2da9-4562-80b8-b4588e42a165%40googlegroups.com?utm_medium=email&utm_source=footer>.

[George Pacheco]

Hello Alexey, 

Thanks very much for your swift reply. 

I have initiated another run with more CPUs (37) the option you indicated -- now it says:

Starting ML tree search with 1 distinct starting trees

[00:01:12 -14624116.283151] Initial branch length optimization

And the watch -tdn0.5 ps -mo pid,tid,%cpu,psr -p \`pgrep raxml-ng\` command returns this:

   PID    TID %CPU PSR

 96491      - 3358   -

     -  96491 86.1   6

     -  96492 83.7  65

     -  96493 88.1  67

     -  96494 90.6  75

     -  96495 91.2  81

     -  96496 91.0   2

     -  96497 89.3  71

     -  96498 89.0   7

     -  96499 88.5  68

     -  96500 94.1  66

     -  96501 88.5  11

     -  96502 91.6  73

     -  96503 90.6  71

     -  96504 93.6  83

     -  96505 92.2  20

     -  96506 91.2  72

     -  96507 92.7  10

     -  96508 90.7  64

     -  96510 91.3  18

     -  96511 94.7  82

     -  96512 89.6   4

     -  96513 92.3  73

     -  96514 92.1   9

     -  96515 91.6   0

     -  96516 88.2   3

     -  96517 91.1  21

     -  96518 87.0   5

     -  96519 91.0   1

     -  96520 86.4  74

     -  96521 91.0   2

     -  96522 92.0  70

     -  96523 91.8   8

     -  96524 93.0  19

     -  96525 93.0  17

     -  96526 92.4  69

     -  96527 93.7  85

     -  96528 92.0  84

I am not quite sure how to read these values -- could you please be so kind as to tell me if this is what you would expect to see?

Many thanks in advance, George.

> ra...@googlegroups.com <mailto:ra...@googlegroups.com>.

[Alexey Kozlov]

Hi George,

> I have initiated another run with more CPUs (37) the option you indicated -- now it says:
>
> [00:01:12 -14624116.283151] Initial branch length optimization

OK, now you can monitor the progress :)


> And the *watch -tdn0.5 ps -mo pid,tid,%cpu,psr -p \`pgrep raxml-ng\` command returns this:

>
> I am not quite sure how to read these values -- could you please be so kind as to tell me if this is
> what you would expect to see?

this looks good - all threads are mapped to different CPU cores (last column).

in general, for planning your runs, you should consider how many cores you have in total (eg
"lscpu"), and whether it's a shared server where other users will run their programs alongside
raxml-ng...

Best,
Alexey

> > ra...@googlegroups.com <javascript:> <mailto:ra...@googlegroups.com <javascript:>>.

> > To view this discussion on the web visit
> >
> https://groups.google.com/d/msgid/raxml/5ef7231e-2da9-4562-80b8-b4588e42a165%40googlegroups.com
> <https://groups.google.com/d/msgid/raxml/5ef7231e-2da9-4562-80b8-b4588e42a165%40googlegroups.com>
> >

> <https://groups.google.com/d/msgid/raxml/5ef7231e-2da9-4562-80b8-b4588e42a165%40googlegroups.com?utm_medium=email&utm_source=footer
> <https://groups.google.com/d/msgid/raxml/5ef7231e-2da9-4562-80b8-b4588e42a165%40googlegroups.com?utm_medium=email&utm_source=footer>>.

>
>
> --
> You received this message because you are subscribed to the Google Groups "raxml" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to

> raxml+un...@googlegroups.com <mailto:raxml+un...@googlegroups.com>.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/raxml/2dc32481-8af7-4e8b-a6e7-be1e21f6b92e%40googlegroups.com
> <https://groups.google.com/d/msgid/raxml/2dc32481-8af7-4e8b-a6e7-be1e21f6b92e%40googlegroups.com?utm_medium=email&utm_source=footer>.

[George Pacheco]

Hello Alexey, 

Thanks very much for your help! 

It has been running for almost 21h now and this is the current status. Do you think that I could predict anything out of these passed times? I mean, is there a specific order of steps RAxML-NG will perform or it is data-dependent? I mean, 4 rounds of AUTODETECT spr, then 4 rounds of FAST spr, then ... Sorry for trying to get this estimation, but I am really in a rush as you can imagine. 

Starting ML tree search with 1 distinct starting trees

[00:01:12 -14624116.283151] Initial branch length optimization

[00:22:28 -13918622.840160] Model parameter optimization (eps = 10.000000)

[01:12:36 -11353818.669215] AUTODETECT spr round 1 (radius: 5)

[01:36:53 -11343900.673598] AUTODETECT spr round 2 (radius: 10)

[02:47:53 -11343450.037261] AUTODETECT spr round 3 (radius: 15)

[05:21:57 -11343398.796316] AUTODETECT spr round 4 (radius: 20)

[08:12:51 -11343398.796215] SPR radius for FAST iterations: 15 (autodetect)

[08:12:51 -11343398.796215] Model parameter optimization (eps = 3.000000)

[08:14:18 -11343393.609096] FAST spr round 1 (radius: 15)

[13:13:00 -11342126.386642] FAST spr round 2 (radius: 15)

[17:32:53 -11341762.803441] FAST spr round 3 (radius: 15)

in general, for planning your runs, you should consider how many cores you have in total (eg
"lscpu"), and whether it's a shared server where other users will run their programs alongside
raxml-ng...

I share this server (with a total of 128 CPUs) with loads of people, so unless I give my job ALL the 128 CPUs -- which will lead my colleagues to hate me forever -- I cannot be certain that my cores are not being shared :( Not sure how I could solve this -- I will ask out system admin.

Thanks a lot once again!

George.

[Alexey Kozlov]

Hi George,

> It has been running for almost 21h now and this is the current status.

Do you know what CPUs you have ("lscpu")?

>Do you think that I could
> predict anything out of these passed times? I mean, is there a specific order of steps RAxML-NG will
> perform or it is data-dependent? I mean, 4 rounds of AUTODETECT spr, then 4 rounds of FAST spr, then

The order of steps is fixed, but the number of iterations at each step is data-dependent. Please see
Section 4.2.3 (p.50) of my phd thesis for a brief description of raxml-ng search algorithm:

https://cme.h-its.org/exelixis/pubs/dissAlexey.pdf

> I share this server (with a total of 128 CPUs) with loads of people, so unless I give my job ALL the
> 128 CPUs -- which will lead my colleagues to hate me forever -- I cannot be certain that my cores
> are not being shared :( Not sure how I could solve this -- I will ask out system admin.

OK I see, unfortunately that's not much you can with this setup, and raxml-ng will become slower if
server is under high load.

Best,
Alexey