ExaML test version offering quartet mapping

56 views
Skip to first unread message

Alexandros Stamatakis

unread,
Mar 17, 2016, 10:46:21 AM3/17/16
to ra...@googlegroups.com
Dear All,

There is now an ExaML test version implementing quartet mapping in the
same way as RAxML.

It's not an official release yet, but available for testing under:

https://github.com/stamatak/ExaML

Things to note:

1. It is parallelized in a different way than in RAxML, hence will only
scale on long alignments

2. It is not yet documented in the PDF manual, but it is documented in
the on-line terminal help

3. checkpointing does not work yet, but I will implement this soon

4. if you have gappy datasets it will be worth checking if the -S option
yields better performance

5. Unlike RAxML the MPI version of ExaML will do the model param
optimization and quartet calculation in one single parallel run

Alexis




--
Alexandros (Alexis) Stamatakis

Research Group Leader, Heidelberg Institute for Theoretical Studies
Full Professor, Dept. of Informatics, Karlsruhe Institute of Technology
Adjunct Professor, Dept. of Ecology and Evolutionary Biology, University
of Arizona at Tucson

www.exelixis-lab.org

Alexis

unread,
Mar 18, 2016, 8:33:44 AM3/18/16
to raxml
Dear All,

I just released ExaML v3.0.17 that now contains an updated PDF manual documenting the quartet options and also offers checkpointing.

Alexis

Karen

unread,
Apr 12, 2016, 5:41:36 AM4/12/16
to raxml, ralph_...@hotmail.com, bmi...@uni-bonn.de, a.do...@zfmk.de, oliver....@gmail.com, daniela...@univie.ac.at, harald...@univie.at.at, in...@sabrina.simon.com
Dear all,

first many many thanks for implementing this feature in Examl.

We tried various runs with ExaML 3.0.17 for quartet mapping analyses from predifined groups (-f q -Y....). Unfortunately, analyses seem to crash for datasets, where the partitions have different models (incl LG4X) - anyway the parser did not give error messages when creating the binary file. Error messages are listed below (we tried 3 different versions). Either it crahses after several hours or it crashes immediatesly. The problem seems only to occur when the models for each partition are different. For a modification of this dataset where models are identical (e.g. WAG) for all partitions (and then created a binary with the parser), it works and does not crash which is great! :)
(These are tests at the moment also for other analyses / datasets / groups applying FcLM)

Is it more useful to sentd you the respective datasets that you can use them for testing?
If yes, I will send you a seperate link (they are quite big).

To me the error message is not clear (thy system, the program?), or is it an MPI problem?
We have enough RAM we think on the cluster 96 GB and datasets (binaries) with partitions with the same
model run well and finish successfully.

Would be great if you can help!!

Best, Karen & Ralph

#####
example for error messages from the *.o file
[....]

ExaML was called as follows:

/share/scientific_bin/ExaML/3.0.17/examl -f q -Y Tax_groups_FCLM.txt -m GAMMA -s dataset_permutationI.phy.binary -t RAxML_parsimonyTree.FCLM_PMI -n FCLM_TR_PMI -p 322099

Memory Saving Option: DISABLED
[compute-0-4.local][[27749,1],9][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.112.247 failed: Connection refused (111)
[compute-0-4.local][[27749,1],11][btl_tcp_frag.c:215:mca_btl_tcp_frag_recv] mca_btl_tcp_frag_recv: readv failed: Connection reset by peer (104)
[compute-0-4.local][[27749,1],16][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.112.247 failed: Connection refused (111)
[compute-0-4.local][[27749,1],17][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.112.249 failed: Connection refused (111)
[compute-0-4.local][[27749,1],18][btl_tcp_frag.c:215:mca_btl_tcp_frag_recv] mca_btl_tcp_frag_recv: readv failed: Connection reset by peer (104)
--------------------------------------------------------------------------
mpirun noticed that process rank 0 with PID 25937 on node compute-0-4.local exited on signal 9 (Killed).
--------------------------------------------------------------------------

###
ExaML was called as follows:
[...]

Memory Saving Option: DISABLED
--------------------------------------------------------------------------
mpirun noticed that process rank 2 with PID 11129 on node compute-0-3.local exited on signal 9 (Killed).
--------------------------------------------------------------------------
###






Am Donnerstag, 17. März 2016 15:46:21 UTC+1 schrieb Alexis:

Alexandros Stamatakis

unread,
Apr 12, 2016, 5:43:59 AM4/12/16
to ra...@googlegroups.com
hi karen,

can you send me a small test dataset with say 2 different partitions and
2 distinct models that fails?

alexis
> https://github.com/stamatak/ExaML <https://github.com/stamatak/ExaML>
>
> Things to note:
>
> 1. It is parallelized in a different way than in RAxML, hence will only
> scale on long alignments
>
> 2. It is not yet documented in the PDF manual, but it is documented in
> the on-line terminal help
>
> 3. checkpointing does not work yet, but I will implement this soon
>
> 4. if you have gappy datasets it will be worth checking if the -S
> option
> yields better performance
>
> 5. Unlike RAxML the MPI version of ExaML will do the model param
> optimization and quartet calculation in one single parallel run
>
> Alexis
>
>
>
>
> --
> Alexandros (Alexis) Stamatakis
>
> Research Group Leader, Heidelberg Institute for Theoretical Studies
> Full Professor, Dept. of Informatics, Karlsruhe Institute of Technology
> Adjunct Professor, Dept. of Ecology and Evolutionary Biology,
> University
> of Arizona at Tucson
>
> www.exelixis-lab.org <http://www.exelixis-lab.org>
>
> --
> You received this message because you are subscribed to the Google
> Groups "raxml" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to raxml+un...@googlegroups.com
> <mailto:raxml+un...@googlegroups.com>.
> For more options, visit https://groups.google.com/d/optout.

Karen

unread,
Apr 12, 2016, 5:49:43 AM4/12/16
to raxml
Hi,

its quite difficult to make an appropriate small testdataset, but I will try.
I will nevertheless send you also the datasets where it crashed because they are ready to go for testing

Thanks Karen

Alexey Kozlov

unread,
Apr 12, 2016, 6:09:17 AM4/12/16
to ra...@googlegroups.com
Dear Karen,

are you sure 96GB is enough for your dataset? What are the dimensions of it?

"Signal 9 (Killed)" usually means out-of-memory error...

Also, do you have any error messages printed to stderr? (*.e file? not sure how it's called on you system).

Cheers,
Alexey

On 12.04.2016 11:41, 'Karen' via raxml wrote:
> https://github.com/stamatak/ExaML <https://github.com/stamatak/ExaML>
>
> Things to note:
>
> 1. It is parallelized in a different way than in RAxML, hence will only
> scale on long alignments
>
> 2. It is not yet documented in the PDF manual, but it is documented in
> the on-line terminal help
>
> 3. checkpointing does not work yet, but I will implement this soon
>
> 4. if you have gappy datasets it will be worth checking if the -S option
> yields better performance
>
> 5. Unlike RAxML the MPI version of ExaML will do the model param
> optimization and quartet calculation in one single parallel run
>
> Alexis
>
>
>
>
> --
> Alexandros (Alexis) Stamatakis
>
> Research Group Leader, Heidelberg Institute for Theoretical Studies
> Full Professor, Dept. of Informatics, Karlsruhe Institute of Technology
> Adjunct Professor, Dept. of Ecology and Evolutionary Biology, University
> of Arizona at Tucson
>

Karen

unread,
Apr 12, 2016, 7:20:42 AM4/12/16
to raxml, ralph_...@hotmail.com, bmi...@uni-bonn.de, a.do...@zfmk.de
Dear Alexis & Alexey,

1) I will send you a seperate mail with full & small test datasets in ca. 1h (need a bit time for upload)

2) Alexey: we have sterror printed, I will attach them in the email

3) Alexey: We thought that is enough because the datasets with the same model run through successfully - unless when different models are assigned to different partitions
(incl LG4X) it neds heaps more RAM than only using one (non-LG4X) model? Is there a way to estimate how much more it would need then?

I put A. Donath here in cc, responsible for our Cluster - maybe he can say more about this issue.

I think (if possible) the best is if you could test these issues on your cluster as well (with maybe more RAM) - I just remember that we last year had a lot of problems when
LG4X was included and similar error messages (BS and treesearch) but definitely having enough RAM).

(Please let me know if its better to proceed this thread via emails instead the google group)

Best Karen

Karen

unread,
Oct 18, 2016, 8:49:17 AM10/18/16
to raxml, ralph....@zfmk.de
Dear all,

sorry I am slightly confused with ExaML and Raxml versions and options for quartet mapping from predefined groups:

Following the link below, I get to a version (still Examl 3.0.17) but with the latest commit and change from MAY 2016:
Latest commit d9e75f0 on 9 May stamatak fix quartet output for more than one MPI process
So this is not released in the master branch but still the test version?
Going to the ExaML normal releases page, I can get an older version: v3.0.17, 6749f7a

stamatak tagged this on 18 Mar · 4 commits to master since this tag ExaML version 3.0.17


Is it correct to use the TESTversion d9e75f0 on 9 May?


Moreover, I must use -f q -Y etc. for a proper run, correct,

(in contrast if I use so no need for -f e as in the RAXML MPI Quartet mapping version I first must optimize parameters of a dataset by a given tree (with -f e) to receive a site logLH filke

before I can run the Quartet mapping with RAxML MPI, correct?)



Many thanks
Karen



Am Donnerstag, 17. März 2016 15:46:21 UTC+1 schrieb Alexis:

Alexandros Stamatakis

unread,
Oct 19, 2016, 3:56:41 AM10/19/16
to ra...@googlegroups.com
Dear Karen,

The most up to date ExaML version is always the most recent one on github.

> Moreover, I must use -f q -Y etc. for a proper run, correct,

I guess so, I am pretty sure it is properly documented in the manual, if
not let me know and I will update it.

> (in contrast if I use so no need for -f e as in the RAXML MPI Quartet
> mapping version I first must optimize parameters of a dataset by a given
> tree (with -f e) to receive a site logLH filke

to generate a file containing the model parameter estimates.

Alexis

> before I can run the Quartet mapping with RAxML MPI, correct?)



>
>
>
> Many thanks
> Karen
>
>
> Am Donnerstag, 17. März 2016 15:46:21 UTC+1 schrieb Alexis:
>
> Dear All,
>
> There is now an ExaML test version implementing quartet mapping in the
> same way as RAxML.
>
> It's not an official release yet, but available for testing under:
>
> https://github.com/stamatak/ExaML <https://github.com/stamatak/ExaML>
>
> Things to note:
>
> 1. It is parallelized in a different way than in RAxML, hence will only
> scale on long alignments
>
> 2. It is not yet documented in the PDF manual, but it is documented in
> the on-line terminal help
>
> 3. checkpointing does not work yet, but I will implement this soon
>
> 4. if you have gappy datasets it will be worth checking if the -S
> option
> yields better performance
>
> 5. Unlike RAxML the MPI version of ExaML will do the model param
> optimization and quartet calculation in one single parallel run
>
> Alexis
>
>
>
>
> --
> Alexandros (Alexis) Stamatakis
>
> Research Group Leader, Heidelberg Institute for Theoretical Studies
> Full Professor, Dept. of Informatics, Karlsruhe Institute of Technology
> Adjunct Professor, Dept. of Ecology and Evolutionary Biology,
> University
> of Arizona at Tucson
>
> www.exelixis-lab.org <http://www.exelixis-lab.org>
>
> --
> You received this message because you are subscribed to the Google
> Groups "raxml" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to raxml+un...@googlegroups.com
> <mailto:raxml+un...@googlegroups.com>.
> For more options, visit https://groups.google.com/d/optout.

Karen

unread,
Oct 19, 2016, 4:56:19 AM10/19/16
to raxml, ralph....@zfmk.de, Bernhard Misof
Dear Alexis and all,

sorry for some misunderstandings - I still don't know what to use:

1) which ExaML version:

There is more than 1 version on github:
* master branch (v. 3.0.17) latest releases: https://github.com/stamatak/ExaML/releases/tag/v3.0.17
I assume this to be the most recent release.

Anyway: the one I can find here
https://github.com/stamatak/ExaML (looking at the code page)
is more recent (May) but a test version and still not released officially
( modified concerning the file examl/quartets.c)
https://github.com/stamatak/ExaML/commit/d9e75f0065c8d3133124240e5f82dce0697b1f30

seems more updated and bugs are fixed.

According to my understanding they differ and I should use this one (May)
but it seems not to be officialy released and different from the version from March

https://github.com/stamatak/ExaML/releases/tag/v3.0.17 which is officially the "most recent" one.

If the versions are similar, please send me a short note.

We forward what should be used to a lot of colleagues - so if some of you tell me what to use (bugs fixed)
would be highly appreciated. Sorry I might be too unfamiliar with the github structure.

You / Alexey might maybe answer via private email communication  since
I am not sure if it's helpful  on google group.

2) quartet mapping with RAXML-MPI_Quartet version:

You are of course right, Alexis:  -f e generates a file containing the model parameter estimates.
My question was:  Using the Quartet mapping with EXAML and the "normal" Raxml version
I don't need such a file, but using  Makefile.SSE3.QuartetMPI.gcc I do need such a file?
(This was a least in info when we tried to run the QuartetMPI axml version, however its not documented in the
Manual neitrher in the help caling RAxML - we just found out upon an error message)


Thanks, & best wishes, sorry again for asking
Karen

Alexandros Stamatakis

unread,
Oct 26, 2016, 8:15:32 AM10/26/16
to ra...@googlegroups.com
Dear Karen,

> 1) which ExaML version:
>
> There is more than 1 version on github:
> * master branch (v. 3.0.17) latest releases:
> https://github.com/stamatak/ExaML/releases/tag/v3.0.17
> I assume this to be the most recent release.
>
> Anyway: the one I can find here
> https://github.com/stamatak/ExaML (looking at the code page)
> is more recent (May) but a test version and still not released officially
> ( modified concerning the file examl/quartets.c)
> https://github.com/stamatak/ExaML/commit/d9e75f0065c8d3133124240e5f82dce0697b1f30
>
> seems more updated and bugs are fixed.

The one from May, that's what I meant with most recent one.

> According to my understanding they differ and I should use this one (May)
> but it seems not to be officialy released and different from the version
> from March

Just use the May version, releases are only done when a new option is
added usually or for major bug fixes only.

Has Alexey answered the other question already?

Alexis
> <https://github.com/stamatak/ExaML
> <https://www.google.com/url?q=https%3A%2F%2Fgithub.com%2Fstamatak%2FExaML&sa=D&sntz=1&usg=AFQjCNGAVcGQ9FC38MAf2duVjl1Y4hTlqw>>
> > an email to raxml+un...@googlegroups.com <javascript:>
> > <mailto:raxml+un...@googlegroups.com <javascript:>>.
> > For more options, visit https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>.

Karen

unread,
Oct 27, 2016, 1:46:03 PM10/27/16
to raxml
Dear Alexis,

yes  - Alexeay has answered the above and the other quetions last week already,
(the version question is of importance for a couple of analyses we currently (plan to) run.

Thanks a lot :)

Best Karen
Reply all
Reply to author
Forward
0 new messages