Why RAxML aborts before ML search?

[Liang LÜ]

Dear RAxML maintainers and users,

I am using RAxML to build a phylogeny with a 659 taxa * 7132 positions alignment (mixed with 3 protein partitions and 3 DNA partitions). My command is like:

   mpirun -n 10 raxmlHPC-HYBRID-AVX -s alignment.phy -n output.tre -q partitionmodel.dat -g constrain.phy -m GTRGAMMA -x 12345 -p 234560 -T 4 -# 1000 -f a -o og1, og2,og3,og4,og5

I first used version 8.1.2x, and performed several successful runs, but the results have some small problems, so I modified the alignment but very it was a very small modification and even the dimension did not change. But unluckily, the run aborted after the rapid bootstrapping and before the fast ML search, so I had got only the boot trees and a info file. I thought it would be out of memory, so I added 32GB DIMMs and enhanced the total memory up to 128G (126G avalable to RAxML), and upgraded to the 8.2.9 version and then tried again but failed at the same place.

So could anyone help me with the problem and tell me why it does not work (it did very good jobs but why does not now)?  And if I would like to execute the "-f a" function in separate steps, how to do it. For example, in my current situation, if I want to continue to find the best-scoring ML tree and estimate the nodal bipartition support values, how should I do?

Many thanks!

Best wishes,
Liang

[Alexey Kozlov]

Dear Liang,

could you please post your full console output or RAxML_info file, so that we can see the error message?
It would also help if you could send you dataset to my personal e-mail.

> I am using RAxML to build a phylogeny with a 659 taxa * 7132 positions alignment (mixed with 3 protein partitions and 3
> DNA partitions). My command is like:
>

> * mpirun -n 10 raxmlHPC-HYBRID-AVX -s alignment.phy -n output.tre -q partitionmodel.dat -g constrain.phy -m GTRGAMMA
> -x 12345 -p 234560 -T 4 -# 1000 -f a -o og1, og2,og3,og4,og5*

Your command line looks fine.

> I first used version 8.1.2x, and performed several successful runs, but the results have some small problems, so I
> modified the alignment but very it was a very small modification and even the dimension did not change. But unluckily,
> the run aborted after the rapid bootstrapping and before the fast ML search, so I had got only the boot trees and a info
> file. I thought it would be out of memory, so I added 32GB DIMMs and enhanced the total memory up to 128G (126G avalable
> to RAxML), and upgraded to the 8.2.9 version and then tried again but failed at the same place.

128GB should be more than enough for the dataset of these dimensions.

> So could anyone help me with the problem and tell me why it does not work (it did very good jobs but why does not now)?
> And if I would like to execute the "-f a" function in separate steps, how to do it. For example, in my current
> situation, if I want to continue to find the best-scoring ML tree and estimate the nodal bipartition support values, how
> should I do?

You can use the "-f b" option to compute node support values from a pre-existing set of bootstrap trees, e.g.

raxmlHPC -n support -z bootstrap_trees.newick -f b -m PROTGAMMALG -t best_ml_tree.newick

Best,
Alexey

[Liang LÜ]

Dear Alexey,

many thanks for the suggestions!

I used "screen" commond to run RAxML on a server through the SSH client (not always connected), the process terminated when it encountered the problem, so I do not have the last snapshot. Here I just post the last lines in the "info" file (see attachment):

Bootstrap[797]: Time 1401.281089 seconds, bootstrap likelihood -284373.956486, best rearrangement setting 14
Bootstrap[798]: Time 1178.393412 seconds, bootstrap likelihood -274234.344352, best rearrangement setting 12
Bootstrap[397]: Time 1539.476141 seconds, bootstrap likelihood -274023.496882, best rearrangement setting 12
Bootstrap[299]: Time 1834.713784 seconds, bootstrap likelihood -281992.845614, best rearrangement setting 14
Bootstrap[498]: Time 1884.913102 seconds, bootstrap likelihood -286344.892440, best rearrangement setting 14
Bootstrap[799]: Time 914.005443 seconds, bootstrap likelihood -271025.084679, best rearrangement setting 9
Bootstrap[499]: Time 1719.098331 seconds, bootstrap likelihood -278817.170155, best rearrangement setting 14
Bootstrap[398]: Time 1887.886370 seconds, bootstrap likelihood -286730.578284, best rearrangement setting 15
Bootstrap[399]: Time 1203.507701 seconds, bootstrap likelihood -280748.837155, best rearrangement setting 9


Overall Time for 1000 Rapid Bootstraps 141954.239616 seconds
Average Time per Rapid Bootstrap 141.954240 seconds

Starting ML Search ...

<the end>

And alternatively, now I use "-f d" to search the best ML tree and use "-f b" to get the support values as you suggested. I find they works well with the bootstrapping trees.

I also suppose the memory should be enough, because I monitored the usage of CPU and memory sometimes. CPU was always fully used but the memory was never over 50%. Comparing the successful and unsuccessful runs, I really fail to find any "fatal" differences: they had same dimentions, same persentage of missing data, same partition models and patterns, and were run by same commands (parameters, outgroups, monophyletic constrains...). Some previous reports of similar problems the I saw in the Google group were suggested using "­­--no­-bfgs" flag, I did not try this, because my data is partitioned.

Thanks and best wishes,

Liang

--
You received this message because you are subscribed to a topic in the Google Groups "raxml" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/raxml/4I3GcKh-Kuc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to raxml+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Alexey Kozlov]

Dear Liang,

> Overall Time for 1000 Rapid Bootstraps 141954.239616 seconds
> Average Time per Rapid Bootstrap 141.954240 seconds
>
> Starting ML Search ...
>
> <the end>

There is no error message in your output, so it's hard to say what happened there...
Do you have any job runtime limits on your cluster?

> And alternatively, now I use "-f d" to search the best ML tree and use "-f b" to get the support values as you
> suggested. I find they works well with the bootstrapping trees.

Great.

> I also suppose the memory should be enough, because I monitored the usage of CPU and memory sometimes. CPU was always
> fully used but the memory was never over 50%. Comparing the successful and unsuccessful runs, I really fail to find any
> "fatal" differences: they had same dimentions, same persentage of missing data, same partition models and patterns, and
> were run by same commands (parameters, outgroups, monophyletic constrains...). Some previous reports of similar problems
> the I saw in the Google group were suggested using "­­--no­-bfgs" flag, I did not try this, because my data is partitioned.

I guess you can use "--no-bfgs" flag with partitioned datasets as well. You could also try to reduce the number of BS
replicates to see if you can reproduce the problem faster.


Best,
Alexey

[Alexandros Stamatakis]

> I guess you can use "--no-bfgs" flag with partitioned datasets as well.
> You could also try to reduce the number of BS replicates to see if you
> can reproduce the problem faster.

just two comments from my side:

1. --no-bfgs is enabled by default for partitioned datasets
2. to adress your problem you can run RAxML the hard way based on the
description of the manual:
http://sco.h-its.org/exelixis/resource/download/NewManual.pdf

There is also an analogoius description of basic commands at:

http://sco.h-its.org/exelixis/web/software/raxml/hands_on.html

The most probable cause as Alexey mentioned, might be that you exceeded
a queue time limit if there is one.

Alexis

>
>
> Best,
> Alexey
>

--
Alexandros (Alexis) Stamatakis

Research Group Leader, Heidelberg Institute for Theoretical Studies
Full Professor, Dept. of Informatics, Karlsruhe Institute of Technology
Adjunct Professor, Dept. of Ecology and Evolutionary Biology, University
of Arizona at Tucson

www.exelixis-lab.org

[Liang LÜ]

Dear Alexis and Alexey,

Thanks for your reply!

I do not agree that it is would be the problem of the queue time limit. My server (not that big and public clusters) is exclusive, and I used all the cores in this RAxML computation, did not assign any other parallel jobs.

I will try a reduced rapid bootstrapping when the current job is finished and test whether it is caused by the 1000 gens of bootstrapping.

Best wishes,
Liang

[Alexandros Stamatakis]

okay, let us know if you need further help,

alexis

> www.exelixis-lab.org <http://www.exelixis-lab.org>
>
> --
> You received this message because you are subscribed to the Google
> Groups "raxml" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to raxml+un...@googlegroups.com
> <mailto:raxml+un...@googlegroups.com>.

> For more options, visit https://groups.google.com/d/optout.

[Liang LÜ]

Dear Alexis and Alexey,

I have tried the same codes as previous but with 1000 bootstrapping and 500 boot strapping with "-f a" setting (not using the "-no-bfgs" flag). Both failed, but this time I have got the screen log, I paste one of as follow, the other is almost the same:

...
Bootstrap[398]: Time 1717.899004 seconds, bootstrap likelihood -278471.539652, best rearrangement setting 14
Bootstrap[99]: Time 709.544277 seconds, bootstrap likelihood -286523.506046, best rearrangement setting 5
Bootstrap[399]: Time 822.369591 seconds, bootstrap likelihood -288798.486397, best rearrangement setting 8


Overall Time for 500 Rapid Bootstraps 70348.495620 seconds
Average Time per Rapid Bootstrap 140.696991 seconds

Starting ML Search ...

-300385.3949930144590325653553009033203125000000 -300332.6646089532878249883651733398437500000000
raxmlHPC-HYBRID-AVX: optimizeModel.c:1571: optParamGeneric: Assertion `tr->likelihood >= initialLH' failed.
-293927.1689932990120723843574523925781250000000 -293879.6456539278151467442512512207031250000000
raxmlHPC-HYBRID-AVX: optimizeModel.c:1571: optParamGeneric: Assertion `tr->likelihood >= initialLH' failed.

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   PID 49091 RUNNING AT rovebeetles
=   EXIT CODE: 134
=   CLEANING UP REMAINING PROCESSES
=   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
===================================================================================
YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Aborted (signal 6)
This typically refers to a problem with your application.
Please see the FAQ page for debugging suggestions

Hope it helps in finding the problems!

Best wishes,
Liang

[Alexandros Stamatakis]

could you let me know which RAxML version you are using?

if it's not v 8.2.9 please upgrade.

alexis

On 21.09.2016 08:10, Liang LÜ wrote:
> Dear Alexis and Alexey,
>

> I have tried the same codes as previous but with 1000 bootstrapping and
> 500 boot strapping with "-f a" setting (not using the "-no-bfgs" flag).
> Both failed, but this time I have got the screen log, I paste one of as
> follow, the other is almost the same:
>

> *...

> *

> > an email to raxml+un...@googlegroups.com <javascript:>
> > <mailto:raxml+un...@googlegroups.com <javascript:>>.

> > For more options, visit https://groups.google.com/d/optout

> <https://groups.google.com/d/optout>.

[Liang LÜ]

Dear Alexis,

the version is surely 8.2.9.
I have sent you the script and the log file in a separate and private email.

Cheers,

Liang

an email to raxml+unsubscribe@googlegroups.com
<mailto:raxml+unsubscribe@googlegroups.com>.

For more options, visit https://groups.google.com/d/optout.

--
Alexandros (Alexis) Stamatakis

Research Group Leader, Heidelberg Institute for Theoretical Studies
Full Professor, Dept. of Informatics, Karlsruhe Institute of Technology
Adjunct Professor, Dept. of Ecology and Evolutionary Biology, University
of Arizona at Tucson

www.exelixis-lab.org

--

You received this message because you are subscribed to a topic in the Google Groups "raxml" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/raxml/4I3GcKh-Kuc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to raxml+unsubscribe@googlegroups.com.

[Alexandra Meziti]

Dear Liang, Alexis and Alexey,
I have the exact same problem almost one year after using 8.2.10

I run
# raxml
ameziti3@iw-h43-30: ~/data3/E.lakelanier/06.phylogenetic/lld19 ❤ raxmlHPC-PTHREADS -f a -m GTRGAMMA -s alltimebin19_gene2066_reads.aln -# 100 -x 123 -n alltimebin19_gene2066_reads -p 1234 -T 16

and I get that

Bootstrap[99]: Time 102.096807 seconds, bootstrap likelihood -22691.187563, best rearrangement setting 12


Overall Time for 100 Rapid Bootstraps 11884.087152 seconds
Average Time per Rapid Bootstrap 118.840872 seconds

Starting ML Search ...

and it stops there...

Have you managed to solve this?

Thank you
Alexandra

an email to raxml+un...@googlegroups.com
<mailto:raxml+un...@googlegroups.com>.

For more options, visit https://groups.google.com/d/optout.

--
Alexandros (Alexis) Stamatakis

Research Group Leader, Heidelberg Institute for Theoretical Studies
Full Professor, Dept. of Informatics, Karlsruhe Institute of Technology
Adjunct Professor, Dept. of Ecology and Evolutionary Biology, University
of Arizona at Tucson

www.exelixis-lab.org

--
You received this message because you are subscribed to a topic in the Google Groups "raxml" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/raxml/4I3GcKh-Kuc/unsubscribe.

To unsubscribe from this group and all its topics, send an email to raxml+un...@googlegroups.com.

[Alexandros Stamatakis]

In your case I don't know what might be wrong is there not any error
message you obtain? How long has it stayed there? Typically this phase
is quite compute-intensive and you just have to wait a bit for RAxML to
print stuff again.

Alexis

> You received this message because you are subscribed to a topic
> in the Google Groups "raxml" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/raxml/4I3GcKh-Kuc/unsubscribe

> <https://groups.google.com/d/topic/raxml/4I3GcKh-Kuc/unsubscribe>.

> To unsubscribe from this group and all its topics, send an email

> to raxml+un...@googlegroups.com <javascript:>.

> For more options, visit https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>.
>
>
> --

[Alexandra Meziti]

Hello Alexis,

Thanks for your answer.

It was indeed a matter of my impatience.

It took more than 10h but at the end it worked.

Thanks

Alexandra

To unsubscribe from this group and stop receiving emails from it, send an email to raxml+unsubscribe@googlegroups.com <mailto:raxml+unsubscribe@googlegroups.com>.

For more options, visit https://groups.google.com/d/optout.

--
Alexandros (Alexis) Stamatakis

Research Group Leader, Heidelberg Institute for Theoretical Studies
Full Professor, Dept. of Informatics, Karlsruhe Institute of Technology
Adjunct Professor, Dept. of Ecology and Evolutionary Biology, University
of Arizona at Tucson

www.exelixis-lab.org

--
You received this message because you are subscribed to a topic in the Google Groups "raxml" group.

To unsubscribe from this topic, visit https://groups.google.com/d/topic/raxml/4I3GcKh-Kuc/unsubscribe.

To unsubscribe from this group and all its topics, send an email to raxml+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Meziti Alexandra

Biologist, M.Sc., Dr.

Konstantinoupoleos 21-23,
15124, Marousi,

Greece

+30-694-451-7530