Cluster aborting/dumping ML search

[Camille Delavaux]

Hi all,

Just wondering if anyone else has had issues with the cluster/core dumping or aborting the ML search. In the joint BS and ML search, the command stops after making the _bootstrap file, and when seperately running an ML search, the core dumps the command.

Command:

#!/bin/bash

#MSUB -l excludenodes=m028:r11r29n4

#MSUB -l nodes=1:ppn=16,mem=122gb,walltime=60:00:00:00

#MSUB -M camille....@ku.edu

#MSUB -m abe

#MSUB -N MLDODOOMYAVXraxml.output

#MSUB -j oe

module load legacy raxml

cd ~/RAxMLanalDODOOMY

raxmlHPC-PTHREADS-AVX -T 16 -m GTRGAMMA -s Oomy_outgroupAlignment.phy -f T -t RAxML_bootstrap.OOMYOUTTREEAVX -n PLEASEWORK2

Error/output (End):

RAxML was called as follows:

raxmlHPC-PTHREADS-AVX -T 16 -m GTRGAMMA -s Oomy_outgroupAlignment.phy -f T -t RAxML_bootstrap.OOMYOUTTREEAVX -n PLEASEWORK

Start likelihood: -424833.877294

/var/spool/torque/mom_priv/jobs/2941459.sched.SC: line 26: 271319 Aborted                 (core dumped) raxmlHPC-PTHREADS-AVX -T 16 -m GTRGAMM$

2941461

Any help would be greatly appreciated,

Camille

[Camille Delavaux]

We have concluded it is not the cluster, but an error sent from raxml..

Please share if you have any advice.

Camille

[Camille Delavaux]

FYI: Exit_status=134

On Thursday, June 14, 2018 at 9:38:29 AM UTC-5, Camille Delavaux wrote:

[Alexey Kozlov]

Hi Camille,

could you please post the full output file (MLDODOOMYAVXraxml.output)? The error message sometimes appears at the very top.

Best,
Alexey

On 14.06.2018 21:52, Camille Delavaux wrote:
> FYI: Exit_status=134
>
> On Thursday, June 14, 2018 at 9:38:29 AM UTC-5, Camille Delavaux wrote:
>
> Hi all,
>
> Just wondering if anyone else has had issues with the cluster/core dumping or aborting the ML search. In the joint
> BS and ML search, the command stops after making the _bootstrap file, and when seperately running an ML search, the
> core dumps the command.
>
> Command:
>
> #!/bin/bash
>
> #MSUB -l excludenodes=m028:r11r29n4
>
> #MSUB -l nodes=1:ppn=16,mem=122gb,walltime=60:00:00:00
>

> #MSUB -M camille....@ku.edu <mailto:camille....@ku.edu>

>
> #MSUB -m abe
>
> #MSUB -N MLDODOOMYAVXraxml.output
>
> #MSUB -j oe
>
>
> module load legacy raxml
>
> cd ~/RAxMLanalDODOOMY
>
>
> raxmlHPC-PTHREADS-AVX -T 16 -m GTRGAMMA -s Oomy_outgroupAlignment.phy -f T -t RAxML_bootstrap.OOMYOUTTREEAVX -n
> PLEASEWORK2
>
>
> Error/output (End):
>
> RAxML was called as follows:
>
>
> raxmlHPC-PTHREADS-AVX -T 16 -m GTRGAMMA -s Oomy_outgroupAlignment.phy -f T -t RAxML_bootstrap.OOMYOUTTREEAVX -n
> PLEASEWORK
>
>
>
>
> Start likelihood: -424833.877294
>
>

> /var/spool/torque/mom_priv/jobs/2941459.sched.SC <http://2941459.sched.SC>: line 26: 271319 Aborted (core dumped)

> raxmlHPC-PTHREADS-AVX -T 16 -m GTRGAMM$
>
>
> 2941461
>
>
> Any help would be greatly appreciated,
>
>
> Camille
>

> --
> You received this message because you are subscribed to the Google Groups "raxml" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to raxml+un...@googlegroups.com
> <mailto:raxml+un...@googlegroups.com>.
> For more options, visit https://groups.google.com/d/optout.

[Camille Delavaux]

Here is the full output :

raxmlHPC-PTHREADS-AVX: rapidBootstrap.c:432: treeOptimizeThorough: Assertion `res > 0' failed.

WARNING: The number of threads is currently set to 0

You can specify the number of threads to run via -T numberOfThreads

NumberOfThreads must be set to an integer value greater than 1

RAxML, will now set the number of threads automatically to 2 !

Using BFGS method to optimize GTR rate parameters, to disable this specify "--no-bfgs"

This is the RAxML Master Pthread

This is RAxML Worker Pthread Number: 1

This is RAxML version 8.1.20 released by Alexandros Stamatakis on April 18 2015.

With greatly appreciated code contributions by:

Andre Aberer   (HITS)

Simon Berger   (HITS)

Alexey Kozlov     (HITS)

Kassian Kobert    (HITS)

David Dao         (KIT and HITS)

Nick Pattengale   (Sandia)

Wayne Pfeiffer    (SDSC)

Akifumi S. Tanabe (NRIFS)

Alignment has 3243 distinct alignment patterns

Proportion of gaps and completely undetermined characters in this alignment: 93.20%

RAxML thorough tree optimization

Using 1 distinct models/data partitions with joint branch length optimization

All free model parameters will be estimated by RAxML

GAMMA model of rate heteorgeneity, ML estimate of alpha-parameter

GAMMA Model parameters will be estimated up to an accuracy of 0.1000000000 Log Likelihood units

Partition: 0

Alignment Patterns: 3243

Name: No Name Provided

DataType: DNA

Substitution Matrix: GTR

RAxML was called as follows:

raxmlHPC-PTHREADS-AVX -m GTRGAMMA -s Oomy_outgroupAlignment.phy -f T -t RAxML_bootstrap.OOMYOUTTREEAVX -n NOWWORK T 16

Start likelihood: -424833.877290

/var/spool/torque/mom_priv/jobs/2960047.sched.SC: line 25: 316605 Aborted                 (core dumped) raxmlHPC-PTHREADS-AVX T 16 -m GTRGAMMA -s Oomy_outgroupAlignme$

[Camille Delavaux]

This is the original error/output. I've rerun this in many variations, but this is the original:

raxmlHPC-PTHREADS-AVX: rapidBootstrap.c:432: treeOptimizeThorough: Assertion `res > 0' failed.

Using BFGS method to optimize GTR rate parameters, to disable this specify "--no-bfgs"

This is the RAxML Master Pthread

This is RAxML Worker Pthread Number: 1

This is RAxML Worker Pthread Number: 7

This is RAxML Worker Pthread Number: 3

This is RAxML Worker Pthread Number: 2

This is RAxML Worker Pthread Number: 4

This is RAxML Worker Pthread Number: 6

This is RAxML Worker Pthread Number: 5

This is RAxML Worker Pthread Number: 8

This is RAxML Worker Pthread Number: 9

This is RAxML Worker Pthread Number: 10

This is RAxML Worker Pthread Number: 13

This is RAxML Worker Pthread Number: 12

This is RAxML Worker Pthread Number: 11

This is RAxML Worker Pthread Number: 14

This is RAxML Worker Pthread Number: 15

This is RAxML version 8.1.20 released by Alexandros Stamatakis on April 18 2015.

With greatly appreciated code contributions by:

Andre Aberer   (HITS)

Simon Berger   (HITS)

Alexey Kozlov     (HITS)

Kassian Kobert    (HITS)

David Dao         (KIT and HITS)

Nick Pattengale   (Sandia)

Wayne Pfeiffer    (SDSC)

Akifumi S. Tanabe (NRIFS)

Alignment has 3243 distinct alignment patterns

Proportion of gaps and completely undetermined characters in this alignment: 93.20%

RAxML thorough tree optimization

Using 1 distinct models/data partitions with joint branch length optimization

All free model parameters will be estimated by RAxML

GAMMA model of rate heteorgeneity, ML estimate of alpha-parameter

GAMMA Model parameters will be estimated up to an accuracy of 0.1000000000 Log Likelihood units

Partition: 0

Alignment Patterns: 3243

Name: No Name Provided

DataType: DNA

Substitution Matrix: GTR

RAxML was called as follows:

raxmlHPC-PTHREADS-AVX -T 16 -m GTRGAMMA -s Oomy_outgroupAlignment.phy -f T -t RAxML_bootstrap.OOMYOUTTREEAVX -n PLEASEWORK

Start likelihood: -424833.877294

/var/spool/torque/mom_priv/jobs/2941459.sched.SC: line 26: 271319 Aborted                 (core dumped) raxmlHPC-PTHREADS-AVX -T 16 -m GTRGAMMA -s Oomy_outgroupAlignm$

2941461

On Thursday, June 14, 2018 at 9:38:29 AM UTC-5, Camille Delavaux wrote:

[Alexey Kozlov]

thanks, as expected, the error message is in the first line.

could you please double-check that "RAxML_bootstrap.OOMYOUTTREEAVX" file is not empty and contains valid Newick trees?

Alexey

> #MSUB -M camille....@ku.edu <mailto:camille....@ku.edu>

>
> #MSUB -m abe
>
> #MSUB -N MLDODOOMYAVXraxml.output
>
> #MSUB -j oe
>
>
> module load legacy raxml
>
> cd ~/RAxMLanalDODOOMY
>
>
> raxmlHPC-PTHREADS-AVX -T 16 -m GTRGAMMA -s Oomy_outgroupAlignment.phy -f T -t RAxML_bootstrap.OOMYOUTTREEAVX -n
> PLEASEWORK2
>
>
> Error/output (End):
>
> RAxML was called as follows:
>
>
> raxmlHPC-PTHREADS-AVX -T 16 -m GTRGAMMA -s Oomy_outgroupAlignment.phy -f T -t RAxML_bootstrap.OOMYOUTTREEAVX -n
> PLEASEWORK
>
>
>
>
> Start likelihood: -424833.877294
>
>

> /var/spool/torque/mom_priv/jobs/2941459.sched.SC <http://2941459.sched.SC>: line 26: 271319 Aborted (core dumped)

> raxmlHPC-PTHREADS-AVX -T 16 -m GTRGAMM$
>
>
> 2941461
>
>
> Any help would be greatly appreciated,
>
>
> Camille
>

[Camille Delavaux]

It is not empty and contains the following:

"

((New.CleanUp.ReferenceOTU5136,New.ReferenceOTU142),((New.ReferenceOTU1219,New.ReferenceOTU2781),(((((((New.ReferenceOTU153,New.Refere$

(((((((New.ReferenceOTU1214,(New.ReferenceOTU1538,New.ReferenceOTU1540)),(New.ReferenceOTU3472,New.CleanUp.ReferenceOTU70)),New.CleanU$

(((New.ReferenceOTU1538,(New.ReferenceOTU191,(New.CleanUp.ReferenceOTU137,New.ReferenceOTU783))),((New.ReferenceOTU1222,((New.CleanUp.$

(((New.ReferenceOTU191,(New.ReferenceOTU1538,(New.CleanUp.ReferenceOTU137,New.ReferenceOTU783))),((New.ReferenceOTU1222,(New.CleanUp.R$

((New.ReferenceOTU2781,(New.CleanUp.ReferenceOTU5136,New.ReferenceOTU142)),((New.ReferenceOTU153,(New.ReferenceOTU191,((New.CleanUp.Re$

((((New.ReferenceOTU161,New.CleanUp.ReferenceOTU70),New.ReferenceOTU191),((New.CleanUp.ReferenceOTU137,New.ReferenceOTU783),(((New.Ref$

(((New.ReferenceOTU191,(New.CleanUp.ReferenceOTU70,(((New.CleanUp.ReferenceOTU137,New.ReferenceOTU783),((New.ReferenceOTU153,New.Refer$

((((((New.CleanUp.ReferenceOTU137,New.ReferenceOTU783),((New.CleanUp.ReferenceOTU5136,New.ReferenceOTU1540),(New.ReferenceOTU283,(New.$

(((((((New.CleanUp.ReferenceOTU137,New.ReferenceOTU783),((New.CleanUp.ReferenceOTU124,(New.ReferenceOTU3475,New.CleanUp.ReferenceOTU33$

((New.ReferenceOTU1219,New.ReferenceOTU2781),((((New.CleanUp.ReferenceOTU137,New.ReferenceOTU783),New.ReferenceOTU1222),(((New.CleanUp$

(New.ReferenceOTU1219,(New.ReferenceOTU2781,(New.ReferenceOTU142,(((New.ReferenceOTU283,(((New.CleanUp.ReferenceOTU825,New.CleanUp.Ref$

((((New.CleanUp.ReferenceOTU825,New.ReferenceOTU191),((New.CleanUp.ReferenceOTU137,New.ReferenceOTU783),(((((New.ReferenceOTU283,New.R$

(((New.ReferenceOTU191,(New.ReferenceOTU1222,(New.CleanUp.ReferenceOTU137,New.ReferenceOTU783))),(((New.CleanUp.ReferenceOTU654,New.Re$

((New.ReferenceOTU161,(New.ReferenceOTU191,(((New.CleanUp.ReferenceOTU137,New.ReferenceOTU783),(((((((((((New.ReferenceOTU3494,New.Ref$

((New.ReferenceOTU191,(((New.CleanUp.ReferenceOTU137,New.ReferenceOTU783),(New.ReferenceOTU1222,(((((New.ReferenceOTU1011,New.CleanUp.$

((New.ReferenceOTU1219,(New.ReferenceOTU191,((New.CleanUp.ReferenceOTU137,New.ReferenceOTU783),((New.CleanUp.ReferenceOTU5136,((New.Cl$

((New.ReferenceOTU191,((New.CleanUp.ReferenceOTU137,New.ReferenceOTU783),((New.ReferenceOTU1222,(New.CleanUp.ReferenceOTU124,(New.Clea$

((New.ReferenceOTU191,((New.CleanUp.ReferenceOTU137,New.ReferenceOTU783),((New.ReferenceOTU3398,(New.CleanUp.ReferenceOTU226,(((((((Ne$

(New.ReferenceOTU1219,((New.ReferenceOTU2781,New.ReferenceOTU142),(New.ReferenceOTU191,((New.CleanUp.ReferenceOTU7605,(New.ReferenceOT$

(((((New.CleanUp.ReferenceOTU137,New.ReferenceOTU783),(New.ReferenceOTU1222,(New.CleanUp.ReferenceOTU124,(New.CleanUp.ReferenceOTU334,$

(((New.ReferenceOTU191,((((New.ReferenceOTU283,(((New.CleanUp.ReferenceOTU124,(New.ReferenceOTU3475,New.CleanUp.ReferenceOTU334)),(New$

(New.CleanUp.ReferenceOTU5136,(New.ReferenceOTU142,(((New.ReferenceOTU191,((New.ReferenceOTU1222,(((New.CleanUp.ReferenceOTU124,(New.R$

(New.ReferenceOTU142,((New.CleanUp.ReferenceOTU5136,(((((New.CleanUp.ReferenceOTU7788,New.ReferenceOTU1809),(New.ReferenceOTU2684,((Ne$

(((New.ReferenceOTU191,((((New.CleanUp.ReferenceOTU3617,New.ReferenceOTU2530),New.CleanUp.ReferenceOTU3620),(New.ReferenceOTU260,((((N$

(((New.ReferenceOTU153,New.CleanUp.ReferenceOTU5136),New.ReferenceOTU142),((((((((New.CleanUp.ReferenceOTU610,New.ReferenceOTU73),((Ne$

((((((New.ReferenceOTU191,(New.CleanUp.ReferenceOTU70,(New.ReferenceOTU3472,New.ReferenceOTU1214))),((New.ReferenceOTU783,New.CleanUp.$

(New.ReferenceOTU161,(((New.CleanUp.ReferenceOTU5136,New.ReferenceOTU142),(New.ReferenceOTU1219,New.ReferenceOTU2781)),((New.Reference$

((New.ReferenceOTU1219,New.ReferenceOTU2781),(((New.ReferenceOTU191,(New.CleanUp.ReferenceOTU70,New.ReferenceOTU3472)),((New.Reference$

((((New.ReferenceOTU191,(New.CleanUp.ReferenceOTU70,New.ReferenceOTU3472)),((New.ReferenceOTU783,New.CleanUp.ReferenceOTU137),((New.Cl$

(((((New.CleanUp.ReferenceOTU70,(((New.ReferenceOTU1538,New.ReferenceOTU1540),New.ReferenceOTU1214),New.ReferenceOTU3472)),New.Referen$

(New.ReferenceOTU142,((New.ReferenceOTU1219,New.ReferenceOTU2781),(New.ReferenceOTU191,((New.ReferenceOTU783,New.CleanUp.ReferenceOTU1$

((New.ReferenceOTU142,(New.ReferenceOTU2781,New.ReferenceOTU1219)),(New.ReferenceOTU191,((New.ReferenceOTU783,New.CleanUp.ReferenceOTU$

(((New.CleanUp.ReferenceOTU5136,New.ReferenceOTU142),(New.ReferenceOTU2781,New.ReferenceOTU1219)),(((((New.ReferenceOTU1871,((((New.Re$

((New.ReferenceOTU142,(New.ReferenceOTU2781,New.ReferenceOTU1219)),((New.ReferenceOTU191,(New.ReferenceOTU1538,(New.ReferenceOTU783,Ne$

(((New.ReferenceOTU18,New.ReferenceOTU1219),(New.ReferenceOTU142,New.ReferenceOTU2781)),((New.ReferenceOTU191,(New.ReferenceOTU1222,(N$

(New.ReferenceOTU142,(New.ReferenceOTU2781,(((New.ReferenceOTU783,New.CleanUp.ReferenceOTU137),((New.ReferenceOTU153,(New.ReferenceOTU$

(((New.ReferenceOTU191,((New.ReferenceOTU783,New.CleanUp.ReferenceOTU137),(((((New.CleanUp.ReferenceOTU3261,New.ReferenceOTU2762),.........."

[Alexandros Stamatakis]

Hi Camille,

What are the dimensions of your dataset (#taxa, #sites)?

I think this is a numerical issue. Did you do the bootstraps under
GTRCAT or GTRGAMMA?

The preferred solution is that you switch to using RAxML-NG:

https://github.com/amkozlov/raxml-ng

which should be numerically more stable.

Alexis

> --
> You received this message because you are subscribed to the Google
> Groups "raxml" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to raxml+un...@googlegroups.com
> <mailto:raxml+un...@googlegroups.com>.
> For more options, visit https://groups.google.com/d/optout.

--
Alexandros (Alexis) Stamatakis

Research Group Leader, Heidelberg Institute for Theoretical Studies
Full Professor, Dept. of Informatics, Karlsruhe Institute of Technology

www.exelixis-lab.org

[Camille Delavaux]

Hi Alexis,

I really appreciate your help! I am doing about 5000 taxa with around 1000 sites each.

I used GTRGAMMA for bootstraps.

I saw an older post suggesting the NG version in a similar situation, but was not very clear on where all the options/ flags are specified. It looks like there is a subset of the full flags. Should I try combining old and new flags (I see the -f a, but what about -x, -p, outgroups)? My full run looks like this:

raxmlHPC-PTHREADS-AVX -T 16-f a -x 12345 -p 12345 -# autoMRE -m GTRGAMMA -s Oomy_outgroupAlignment.phy -o Oomy.OutKP201655.1,Oomy.OutDQ85803.1 -n OOMYOUTTREE.AVX

Let me know what you think,

Camille 

[Camille Delavaux]

Update: I am testing the NG version with a very small amount of iterations (2), but have two questions:

1. How do I specify an outgroup?

2. What is the GTR+ XX needed to be the GTRGAMMA equivalent?

Thank you,

Camille

[Alexandros Stamatakis]

Hi Camille,

> I really appreciate your help! I am doing about 5000 taxa with around
> 1000 sites each.

Okay, that is the size where standard RAxML can run into numerical
problems.

> I used GTRGAMMA for bootstraps.

Okay, but as far as I understand the bootstrap computation did complete
successfully, correct?

> I saw an older post suggesting the NG version in a similar situation,
> but was not very clear on where all the options/ flags are specified. It
> looks like there is a subset of the full flags.

Yes, it's a subset.

> Should I try combining
> old and new flags (I see the -f a, but what about -x, -p, outgroups)? My
> full run looks like this:
>

> raxmlHPC-PTHREADS-AVX -T 16-f a -x 12345 -p 12345 -# autoMRE-m GTRGAMMA

> -s Oomy_outgroupAlignment.phy -o Oomy.OutKP201655.1,Oomy.OutDQ85803.1 -n
> OOMYOUTTREE.AVX

No, if you just need to infer a couple of ML trees as you already do
have the bootestraps you can just run 10-20 ML tree searches.

Documentation on the models and command lines is available here:

https://github.com/amkozlov/raxml-ng/wiki

The on-line help you get in the terminal is also much more helpful than
that in standard RAxML.

Hope this helps,

Alexis

>
>
> Let me know what you think,
>
> Camille
>
> On Friday, June 15, 2018 at 4:03:33 AM UTC-5, Alexis wrote:
>
> Hi Camille,
>
> What are the dimensions of your dataset (#taxa, #sites)?
>
> I think this is a numerical issue. Did you do the bootstraps under
> GTRCAT or GTRGAMMA?
>
> The preferred solution is that you switch to using RAxML-NG:
>
> https://github.com/amkozlov/raxml-ng
> <https://github.com/amkozlov/raxml-ng>
>
> which should be numerically more stable.
>
> Alexis
>
> On 14.06.2018 16:38, Camille Delavaux wrote:
> > Hi all,
> >
> > Just wondering if anyone else has had issues with the cluster/core
> > dumping or aborting the ML search. In the joint BS and ML search,
> the
> > command stops after making the _bootstrap file, and when seperately
> > running an ML search, the core dumps the command.
> >
> > Command:
> >
> > #!/bin/bash
> >
> > #MSUB -l excludenodes=m028:r11r29n4
> >
> > #MSUB -l nodes=1:ppn=16,mem=122gb,walltime=60:00:00:00
> >

> > #MSUB -M camille....@ku.edu <javascript:>

> >
> > #MSUB -m abe
> >
> > #MSUB -N MLDODOOMYAVXraxml.output
> >
> > #MSUB -j oe
> >
> >
> > module load legacy raxml
> >
> > cd ~/RAxMLanalDODOOMY
> >
> >
> > raxmlHPC-PTHREADS-AVX -T 16 -m GTRGAMMA -s
> Oomy_outgroupAlignment.phy -f
> > T -t RAxML_bootstrap.OOMYOUTTREEAVX -n PLEASEWORK2
> >
> >
> > Error/output (End):
> >
> > RAxML was called as follows:
> >
> >
> > raxmlHPC-PTHREADS-AVX -T 16 -m GTRGAMMA -s
> Oomy_outgroupAlignment.phy -f
> > T -t RAxML_bootstrap.OOMYOUTTREEAVX -n PLEASEWORK
> >
> >
> >
> >
> > Start likelihood: -424833.877294
> >
> >
> > /var/spool/torque/mom_priv/jobs/2941459.sched.SC

> <http://2941459.sched.SC>: line 26: 271319

> > Aborted (core dumped) raxmlHPC-PTHREADS-AVX -T 16 -m GTRGAMM$
> >
> >
> > 2941461
> >
> >
> > Any help would be greatly appreciated,
> >
> >
> > Camille
> >
> > --
> > You received this message because you are subscribed to the Google
> > Groups "raxml" group.
> > To unsubscribe from this group and stop receiving emails from it,
> send

> > an email to raxml+un...@googlegroups.com <javascript:>
> > <mailto:raxml+un...@googlegroups.com <javascript:>>.

> > For more options, visit https://groups.google.com/d/optout

> <https://groups.google.com/d/optout>.

>
> --
> Alexandros (Alexis) Stamatakis
>
> Research Group Leader, Heidelberg Institute for Theoretical Studies
> Full Professor, Dept. of Informatics, Karlsruhe Institute of Technology
>

> www.exelixis-lab.org <http://www.exelixis-lab.org>

[Camille Delavaux]

Hi Alexis,

Thank you!

What is the flag for the bootstrap file?

Camille

[Camille Delavaux]

In addition what is the GTRGAMMA equivalent in raxmlNG?

[Alexandros Stamatakis]

Please be so kind to read:

usage examples at: https://github.com/amkozlov/raxml-ng

and

https://github.com/amkozlov/raxml-ng/wiki

If that does not help, let us know.

Alexis

On 15.06.2018 16:13, Camille Delavaux wrote:
> In addition what is the GTRGAMMA equivalent in raxmlNG?
>
> On Friday, June 15, 2018 at 9:11:40 AM UTC-5, Camille Delavaux wrote:
>
> Hi Alexis,
>
> Thank you!
>
> What is the flag for the bootstrap file?
>
> Camille
>
> On Thursday, June 14, 2018 at 9:38:29 AM UTC-5, Camille Delavaux wrote:
>
> Hi all,
>
> Just wondering if anyone else has had issues with the
> cluster/core dumping or aborting the ML search. In the joint BS
> and ML search, the command stops after making the _bootstrap
> file, and when seperately running an ML search, the core dumps
> the command.
>
> Command:
>
> #!/bin/bash
>
> #MSUB -l excludenodes=m028:r11r29n4
>
> #MSUB -l nodes=1:ppn=16,mem=122gb,walltime=60:00:00:00
>

> #MSUB -M camille....@ku.edu <mailto:camille....@ku.edu>

>
> #MSUB -m abe
>
> #MSUB -N MLDODOOMYAVXraxml.output
>
> #MSUB -j oe
>
>
> module load legacy raxml
>
> cd ~/RAxMLanalDODOOMY
>
>
> raxmlHPC-PTHREADS-AVX -T 16 -m GTRGAMMA -s
> Oomy_outgroupAlignment.phy -f T -t
> RAxML_bootstrap.OOMYOUTTREEAVX -n PLEASEWORK2
>
>
> Error/output (End):
>
> RAxML was called as follows:
>
>
> raxmlHPC-PTHREADS-AVX -T 16 -m GTRGAMMA -s
> Oomy_outgroupAlignment.phy -f T -t
> RAxML_bootstrap.OOMYOUTTREEAVX -n PLEASEWORK
>
>
>
>
> Start likelihood: -424833.877294
>
>
> /var/spool/torque/mom_priv/jobs/2941459.sched.SC

> <http://2941459.sched.SC>: line 26: 271319 Aborted (core dumped)

> raxmlHPC-PTHREADS-AVX -T 16 -m GTRGAMM$
>
>
> 2941461
>
>
> Any help would be greatly appreciated,
>
>
> Camille
>

[Camille Delavaux]

Thanks for the resource! Is there a --search example for just the ML search? I still can't find the _bootstrap file flag.

Camille

[Camille Delavaux]

I am using --bs-trees "_bootstrap file". One final question is how to specify ML #s.

Much appreciated,

Camille

[Camille Delavaux]

This is how far I've gotten going through both of those resources and some additional threads on this google group:

mpirun /panfs/pfs.local/work/kbs/c383d893/sw/raxml-ng/bin/raxml-ng-mpi --redo -—force -—search --threads 11 --msa Oomy_outgroupAlignment.phy --model GTR+G --bs-trees RAxML_bootstrap.OOMYOUTTREEAVX --prefix ML-NG-OomyDOD

The process has been stuck at the following for about 6 hours:

00:00:06 -950288.554078] Initial branch length optimization

Does my command seem ok? Is something causing this pause to happen?

Thank you,

Camille

The full log file:

RAxML-NG v. 0.5.1b BETA released on 01.12.2017 by The Exelixis Lab.

Authors: Alexey Kozlov, Alexandros Stamatakis, Diego Darriba, Tomas Flouri, Benoit Morel.

Latest version: https://github.com/amkozlov/raxml-ng

Questions/problems/suggestions? Please visit: https://groups.google.com/forum/#!forum/raxml

WARNING: This is a BETA release, please use at your own risk!

RAxML-NG was called as follows:

/panfs/pfs.local/work/kbs/c383d893/sw/raxml-ng/bin/raxml-ng-mpi --redo --force --search --threads 11 --msa Oomy_outgroupAlignment.phy --$

Analysis options:

  run mode: ML tree search

  start tree(s): random

  random seed: 1529076537

  tip-inner: ON

  pattern compression: ON

  per-rate scalers: OFF

  site repeats: OFF

  fast spr radius: AUTO

  spr subtree cutoff: 1.000000

  branch lengths: ML estimate (linked)

  SIMD kernels: AVX2

  parallelization: hybrid MPI+PTHREADS (16 ranks x 11 threads)

WARNING: Running in REDO mode: existing checkpoints are ignored, and all result files will be overwritten!

WARNING: Running in FORCE mode: all safety checks are disabled!

[00:00:00] Reading alignment from file: Oomy_outgroupAlignment.phy

[00:00:00] Loaded alignment with 3596 taxa and 3868 sites

Alignment comprises 1 partitions and 3243 patterns

Partition 0: noname

Model: GTR+FO+G4m

Alignment sites / patterns: 3868 / 3243

Gaps: 93.20 %

Invariant sites: 17.53 %

[00:00:03] Generating random starting tree(s) with 3596 taxa

[00:00:04] Data distribution: partitions/thread: 1-1, patterns/thread: 18-19

WARNING: You are using too many threads (11) for your alignment with 3243 unique patterns.

NOTE:    Please consider using 4 threads ('--threads 4' option) for the optimal performance.

NOTE:    As a general rule-of-thumb, please assign at least 200-1000 alignment patterns per thread.

Starting ML tree search with 1 distinct starting trees

[00:00:06 -950288.554078] Initial branch length optimization

[Alexandros Stamatakis]

Are you trying to compute bootstraps again or ML trees?

also, why are you using --force?

why do you set --threads to 11? This is a MPI parallel run, so you need
to specify the number of processes you want to immediately after mpirun.

Alexis

> <https://groups.google.com/d/optout>.

>
> --
> Alexandros (Alexis) Stamatakis
>
> Research Group Leader, Heidelberg Institute for Theoretical
> Studies
> Full Professor, Dept. of Informatics, Karlsruhe Institute of
> Technology
>

> www.exelixis-lab.org <http://www.exelixis-lab.org>

[Camille Delavaux]

I am just trying to do ML trees from the _bootstrap file that completed in the initial run.

I am using --force and --threads 11 because the process kept stopping with incorrect threads (either too high or to low), so I was forcing it to complete.

Can you suggest a method for selecting correct threads and how to specify ML numbers? Is this what you mean when you say I need to "specify the number of processes you want to immediately after mpirun"? For example:  

mpirun /panfs/pfs.local/work/kbs/c383d893/sw/raxml-ng/bin/raxml-ng-mpi --redo -—force -—search 20 --threads 11 --msa Oomy_outgroupAlignment.phy --model GTR+G --bs-trees RAxML_bootstrap.OOMYOUTTREEAVX --prefix ML-NG-OomyDOD

Thanks again,

Camille

[Alexey Kozlov]

Hi Camille,

I'll try to answer all you questions at once, starting with the last one :)

1) your alignment is very short (3243 patterns) and trying to use that many threads (16 x 11 = 176)
will only make it slower. so raxml-ng complained for a good reason, and you shouldn't overwrite this check - which was
introduced exactly for such cases - with "--force".
In fact, it makes no sense to use an MPI version on this dataset, if you have 16 cores/node, just use a normal/pthreads
version with "--threads 16". Just make sure to check you cluster documentation such that you don't start 16 copies thereof!

2) GTRGAMMA in RAxML8 corresponds to GTR+G+F in raxml-ng, although using GTR+G (ML estimate of base frequencies) is also
fine (and arguably "better")

3) if you want to run a simple ML search with 20 starting tree, you should use "--search --tree rand{20}" or better
"--search --tree rand{10},pars{10}" (please *do read* the wiki and check the examples in "raxml-ng -h"). you do not need
to specify bootstrap trees for this command, but you can map support values from these bootstraps later on using the
"--support" command (again, please see: https://github.com/amkozlov/raxml-ng/wiki/Input-data#analysis-type)

4) finally, please use raxml-ng v0.6.0 which I released just now:

https://github.com/amkozlov/raxml-ng/releases/tag/0.6.0

among other things, it also has outgroup rooting feature (--outgroup).

Hope this helps,
Alexey

> > Latest version: https://github.com/amkozlov/raxml-ng <https://github.com/amkozlov/raxml-ng>
> >
> > Questions/problems/suggestions? Please visit:
> > https://groups.google.com/forum/#!forum/raxml <https://groups.google.com/forum/#!forum/raxml>

> > www.exelixis-lab.org <http://www.exelixis-lab.org> <http://www.exelixis-lab.org>

> >
> > --
> > You received this message because you are subscribed to the Google
> > Groups "raxml" group.
> > To unsubscribe from this group and stop receiving emails from it, send

> > an email to raxml+un...@googlegroups.com <javascript:>
> > <mailto:raxml+un...@googlegroups.com <javascript:>>.

[Camille Delavaux]

Thank you!

I am running the following:

/panfs/pfs.local/work/kbs/c383d893/sw/raxml-ng/bin/raxml-ng-mpi --redo --search --tree rand{20} --msa Oomy_outgroupAlignment.phy --model GTR+G+F --bs-trees RAxML_bootstrap.OOMYOUTTREEAVX --prefix ML-NG-OomyDOD

Is it normal that the command stops for several hours at "Starting ML tree search with 20 distinct starting trees"?

Camille

[Alexey Kozlov]

hard to tell, could you please post you output?

you can add "--log progress" to the command to see how the tree search goes...

On 17.06.2018 16:48, Camille Delavaux wrote:
> Thank you!
>
> I am running the following:
>
> /panfs/pfs.local/work/kbs/c383d893/sw/raxml-ng/bin/raxml-ng-mpi --redo --search --tree rand{20} --msa
> Oomy_outgroupAlignment.phy --model GTR+G+F --bs-trees RAxML_bootstrap.OOMYOUTTREEAVX --prefix ML-NG-OomyDOD
>
>
> Is it normal that the command stops for several hours at "Starting ML tree search with 20 distinct starting trees"?
>
>
> Camille
>
>
>
> On Saturday, June 16, 2018 at 4:28:34 PM UTC-5, Alexey Kozlov wrote:
>
> Hi Camille,
>
> I'll try to answer all you questions at once, starting with the last one :)
>
> 1) your alignment is very short (3243 patterns) and trying to use that many threads (16 x 11 = 176)
> will only make it slower. so raxml-ng complained for a good reason, and you shouldn't overwrite this check - which was
> introduced exactly for such cases - with "--force".
> In fact, it makes no sense to use an MPI version on this dataset, if you have 16 cores/node, just use a normal/pthreads
> version with "--threads 16". Just make sure to check you cluster documentation such that you don't start 16 copies
> thereof!
>
> 2) GTRGAMMA in RAxML8 corresponds to GTR+G+F in raxml-ng, although using GTR+G (ML estimate of base frequencies) is
> also
> fine (and arguably "better")
>
> 3) if you want to run a simple ML search with 20 starting tree, you should use "--search --tree rand{20}" or better
> "--search --tree rand{10},pars{10}" (please *do read* the wiki and check the examples in "raxml-ng -h"). you do not
> need
> to specify bootstrap trees for this command, but you can map support values from these bootstraps later on using the
> "--support" command (again, please see: https://github.com/amkozlov/raxml-ng/wiki/Input-data#analysis-type

> <https://github.com/amkozlov/raxml-ng/wiki/Input-data#analysis-type>)

>
> 4) finally, please use raxml-ng v0.6.0 which I released just now:
>

> https://github.com/amkozlov/raxml-ng/releases/tag/0.6.0 <https://github.com/amkozlov/raxml-ng/releases/tag/0.6.0>

[Camille Delavaux]

Hi Alexey,

Here is the log file output so far:

RAxML-NG v. 0.5.1b BETA released on 01.12.2017 by The Exelixis Lab.

Authors: Alexey Kozlov, Alexandros Stamatakis, Diego Darriba, Tomas Flouri, Benoit Morel.

Latest version: https://github.com/amkozlov/raxml-ng

Questions/problems/suggestions? Please visit: https://groups.google.com/forum/#!forum/raxml

WARNING: This is a BETA release, please use at your own risk!

RAxML-NG was called as follows:

/panfs/pfs.local/work/kbs/c383d893/sw/raxml-ng/bin/raxml-ng-mpi --redo --search --tree rand{20} --msa Oomy_outgroupAlignment.phy --model GTR+G+F --bs-trees RAxML_bootstrap.OOMYOUTTREEA$

Analysis options:

  run mode: ML tree search

  start tree(s): random (20)

  random seed: 1529242989

  tip-inner: ON

  pattern compression: ON

  per-rate scalers: OFF

  site repeats: OFF

  fast spr radius: AUTO

  spr subtree cutoff: 1.000000

  branch lengths: ML estimate (linked)

  SIMD kernels: AVX2

  parallelization: MPI (16 ranks)

WARNING: Running in REDO mode: existing checkpoints are ignored, and all result files will be overwritten!

[00:00:00] Reading alignment from file: Oomy_outgroupAlignment.phy

[00:00:00] Loaded alignment with 3596 taxa and 3868 sites

NOTE: Per-rate scalers were automatically enabled to prevent numerical issues on taxa-rich alignments.

NOTE: You can use --force switch to skip this check and fall back to per-site scalers.

Alignment comprises 1 partitions and 3243 patterns

Partition 0: noname

Model: GTR+FC+G4m

Alignment sites / patterns: 3868 / 3243

Gaps: 93.20 %

Invariant sites: 17.53 %

[00:00:01] Generating random starting tree(s) with 3596 taxa

[00:00:01] Data distribution: partitions/thread: 1-1, patterns/thread: 202-203

Starting ML tree search with 20 distinct starting trees

[Alexey Kozlov]

Hi Camille,

this looks better, but please upgrade to v0.6 and re-run it with "--log progress".

Best, Alexey

[Camille Delavaux]

Hi all,

There has been a very confusing development here. The ultimate goal for our work is to run both A. Blasted oomycetes and B. unblasted oomycetes to build trees. The blasted (smaller) version is the one causing problems, and the unblasted (much larger) data has successfully completed tree building on the 'old' raxml. To me, this makes no sense since we determined it was likely the data size of the smaller dataset that was keeping it from completing the ML step.

Just thought I'd bring it up, because it is a little puzzling to me.

Camille

[Alexey Kozlov]

Hi Camille,

as I mentioned before, you have a very high missing data rate which could cause poor ML search convergence.

If you would send me your alignment, I can test it on our server to make sure it's not a bug or configuration issue.

Best,
Alexey

On 19.06.2018 16:25, Camille Delavaux wrote:
> Hi all,
>

> There has been a very confusing development here. The ultimate goal for our work is to run both A. Blasted oomycetes and
> B. unblasted oomycetes to build trees. The blasted (smaller) version is the one causing problems, and the unblasted
> (much larger) data has successfully completed tree building on the 'old' raxml. To me, this makes no sense since we
> determined it was likely the data size of the smaller dataset that was keeping it from completing the ML step.
>
> Just thought I'd bring it up, because it is a little puzzling to me.
>
> Camille
>
> On Sunday, June 17, 2018 at 7:10:07 PM UTC-5, Alexey Kozlov wrote:
>
> Hi Camille,
>
> this looks better, but please upgrade to v0.6 and re-run it with "--log progress".
>
> Best, Alexey
>
> On 17.06.2018 21:05, Camille Delavaux wrote:
> > Hi Alexey,
> >
> > Here is the log file output so far:
> >
> > RAxML-NG v. 0.5.1b BETA released on 01.12.2017 by The Exelixis Lab.
> >
> > Authors: Alexey Kozlov, Alexandros Stamatakis, Diego Darriba, Tomas Flouri, Benoit Morel.
> >

> > Latest version: https://github.com/amkozlov/raxml-ng <https://github.com/amkozlov/raxml-ng>

> >
> > Questions/problems/suggestions? Please visit: https://groups.google.com/forum/#!forum/raxml
> <https://groups.google.com/forum/#!forum/raxml>
> >
> >

[Camille Delavaux]

Alexey,

I tried to send you my alignment directly, but am not seeing it on the google group. Let me know if i should resend or if you've received it.

Best,

Camille

[Alexey Kozlov]

that's fine, I got it, will check and get back to you soon.

Alexey

> <https://github.com/amkozlov/raxml-ng <https://github.com/amkozlov/raxml-ng>>
> >      >
> >      > Questions/problems/suggestions? Please visit: https://groups.google.com/forum/#!forum/raxml
> <https://groups.google.com/forum/#!forum/raxml>
> >     <https://groups.google.com/forum/#!forum/raxml <https://groups.google.com/forum/#!forum/raxml>>
> >      >
> >      >

[Camille Delavaux]

I have an idea to get around all the gaps. I am thinking of building the smaller tree adding the reference database sequences to help in the tree building. Then, I can remove tips in R, and use the sample data only to calculate metrics I need.

Does this seem terrible? I know it's not ideal, but if anything, adding the small reference database will allow raxml to complete and improve tree quality?

Any success on the tree building with my phylogeny?

Camille

[Alexandros Stamatakis]

Dear Camille,

If you want to build a tree from short read sequences and analyze them,
then maybe you should consider using phylogenetic placement methods as
an alternative:

https://academic.oup.com/sysbio/article/60/3/291/1667010


Alexis

> >     <https://github.com/amkozlov/raxml-ng
> <https://github.com/amkozlov/raxml-ng>
> <https://github.com/amkozlov/raxml-ng
> <https://github.com/amkozlov/raxml-ng>>>
> >      >      >
> >      >      > Questions/problems/suggestions? Please visit:
> https://groups.google.com/forum/#!forum/raxml
> <https://groups.google.com/forum/#!forum/raxml>
> >     <https://groups.google.com/forum/#!forum/raxml
> <https://groups.google.com/forum/#!forum/raxml>>
> >      >     <https://groups.google.com/forum/#!forum/raxml
> <https://groups.google.com/forum/#!forum/raxml>
> <https://groups.google.com/forum/#!forum/raxml
> <https://groups.google.com/forum/#!forum/raxml>>>
> >      >      >
> >      >      >

> --
> You received this message because you are subscribed to the Google
> Groups "raxml" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to raxml+un...@googlegroups.com
> <mailto:raxml+un...@googlegroups.com>.
> For more options, visit https://groups.google.com/d/optout.

[Camille Delavaux]

Hi Alexis,

Thank you!

Did you have any luck running my phylogeny?

Camille

[Alexandros Stamatakis]

I assume this question is for Alexey, not for me,

alexis

[Alexey Kozlov]

Hi Camille,

sorry for the delay, I had a look at your dataset, and now I can confirm that this unusually long RAxML-NG runtime is
not due a technical problem.

Rather, RAxML-NG needs too many iterations to converge - and I believe this happens due to a problematic alignment.

It seems like you are working with ITS, which is known to be hypervariable and thus notoriously hard to align among
distantly related organisms. Moreover, even in conserved regions I can see few very obvious cases of misalignment (s.
attachment). If MUSCLE failed in those straightforward cases, it's hard to believe it did a better job in more difficult
regions.

So I'd focus on improving the alignment first. I'm not an expert in this topic, but here are a couple of things that
might be worth a try:

1) use a different alignment program (MAFFT, ClustalOmega, maybe there are better tools specifically optimized for ITS?)

2) add more reference sequences as you suggested

3) Alexis' idea of using placement workflow: build a reference alignment+tree first -> align sample seqs with
HMMER/PAPARA/etc. -> use phylogenetic placement algorithm (EPA) to find best attachment points for your samples in the
reference tree (alternatively: run regular tree inference in hope that alignment will be improved).

Hope this helps,
Alexey

On 25.06.2018 22:37, Camille Delavaux wrote:

> >     <https://github.com/amkozlov/raxml-ng <https://github.com/amkozlov/raxml-ng>
> <https://github.com/amkozlov/raxml-ng <https://github.com/amkozlov/raxml-ng>>>
> >      >      >
> >      >      > Questions/problems/suggestions? Please visit: https://groups.google.com/forum/#!forum/raxml
> <https://groups.google.com/forum/#!forum/raxml>
> >     <https://groups.google.com/forum/#!forum/raxml <https://groups.google.com/forum/#!forum/raxml>>
> >      >     <https://groups.google.com/forum/#!forum/raxml <https://groups.google.com/forum/#!forum/raxml>
> <https://groups.google.com/forum/#!forum/raxml <https://groups.google.com/forum/#!forum/raxml>>>
> >      >      >
> >      >      >

[Camille Delavaux]

Good morning Alexey,

Thank you very much for looking into this. You are right, this is a pretty bad alignment. I am working on improving it and hopefully rerunning RAxML.

Appreciate all of the help,

Camille

[Camille Delavaux]

I am happy to report that adding my reference database helped the alignment and the analysis! Thank you.

Camille

>     <<a href="https://github.com/amkozlov/raxml-ng/wiki/Input-data#analysis-type" target="_blank" rel="nofollow" onmousedown="this.href='https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Famkozlov%2Fraxml-ng%2Fwiki%2FInput-data%23analysis-type\x26sa\x3dD\x26sntz\x3d1\x

[Alexey Kozlov]

Great news, I'm glad to hear it worked out! :)

Alexey

> <https://www.google.com/url?q%5Cx3dhttps%3A%2F%2Fgithub.com%2Famkozlov%2Fraxml-ng%2Fwiki%2FInput-data%23analysis-type%5Cx26sa%5Cx3dD%5Cx26sntz%5Cx3d1%5Cx>