A free performance lunch for ML inference

[Alexandros Stamatakis]

Dear Users of RAxML,

There is some free lunch here :-)

The worst part about implementing phylogenetic Maximum Likelihood
methods is to devise stable numerical optimization routines.

But, then, how do we set appropriate convergence thresholds (the famous
epsilon in numerics) for all these ML optimizers?

In the first systematic study on the impact of these threshold settings
on reconstruction accuracy we find that IQ-Tree and RAxML-NG can be
accelerated by a factor of 1.3 and 1.9 respectively by setting some of
these thresholds to larger values while still obtaining equally good trees.

https://www.biorxiv.org/content/10.1101/2022.07.13.499893v1

Alexis

--
Alexandros (Alexis) Stamatakis

Research Group Leader, Heidelberg Institute for Theoretical Studies
Full Professor, Dept. of Informatics, Karlsruhe Institute of Technology
Affiliated Scientist, Evolutionary Genetics and Paleogenomics (EGP) lab,
Institute of Molecular Biology and Biotechnology, Foundation for
Research and Technology Hellas

www.exelixis-lab.org

[Pfeiffer, Wayne]

Hi Alexis,

This is great news!

As suggested in your paper, I changed two lines in constants.hpp of RAxML-NG 1.1.0 to relax two numerical thresholds as follows:

  #define DEF_LH_EPSILON            10.

  #define RAXML_BRLEN_DEFAULT       1000.

I then ran benchmarks on single 128-core nodes of our Expanse supercomputer for eight representative data sets and obtained the results in the attached spreadsheet. Four of the data sets are ones considered in your 2008 paper on rapid bootstrapping, while the other four are from users of the CIPRES gateway.

Columns J and L show the run times measured with the original code (affall) and changed code (fast), respectively, where the time listed is the median of three repeat runs starting from the same seed. The threads used for the runs (or jobs) are those specified by our rules for efficient execution. Jobs using 128 threads have exclusive use of the node, while jobs using fewer threads share the node with other jobs. Thus the run time for 128-thread jobs varies 10% or less from run to run, while the run time for jobs using fewer threads often varies more because of memory contention from other jobs on the same node. This can be seen from the ratio of the max time to the min time for the changed code in Column M.

Column Q shows the speedups obtained going from the original code to the changed code. There are several really impressive speedups, but most are more modest than reported in your paper. For the 140.AA data set the times for the changed code are actually a little slower than those for the original code but within the variation expected for shared-node runs. The median and average speedups across all the runs are 1.13 and 1.28, respectively, as noted in Cells P48 and P49. Such speedups will be welcomed by CIPRES users, so I modified our run script for RAxML-NG to use the changed code :)

Columns N and O show that the final maximum-likelihood score differs very little if at all between the original and changed codes, consistent with the findings in your paper.

* Based upon your findings and the results presented here, it seems like it would be good for Alexey to relax the numerical thresholds in the next release of RAxML-NG.

Thread-to-core affinity is not guaranteed by default for codes that use Pthreads on the AMD nodes of Expanse, which are scheduled by Slurm. Thus we run an SDSC-written bash script called slurm-affall with RAxML-NG jobs (as well as RAxML8 jobs) to ensure affinity. The results for the original and changed codes in Columns J and L used slurm-affall.

Since RAxML-NG has a built-in affinity option, --extra thread-pin, I also made some runs with that option instead of slurm-affall. Unfortunately, that option does not produce reliable results on our AMD nodes, as can be seen from the entries in Column K. Runs with that option using fewer than 128 threads generate the following error message for each thread:

  Error calling pthread_setaffinity_np: 22

* Someone at HITS might want to check whether these errors occur on your AMD nodes too, though maybe the Slurm scheduler is also required to produce the errors.

Best regards,  Wayne

On Jul 15, 2022, at 4:10 AM, Alexandros Stamatakis <alexandros...@gmail.com> wrote:

Dear Users of RAxML,

There is some free lunch here :-)

The worst part about implementing phylogenetic Maximum Likelihood methods is to devise stable numerical optimization routines.

But, then, how do we set appropriate convergence thresholds (the famous epsilon in numerics) for all these ML optimizers?

In the first systematic study on the impact of these threshold settings on reconstruction accuracy  we find that IQ-Tree and RAxML-NG can be accelerated by a factor of 1.3 and 1.9 respectively by setting some of these thresholds to larger values while still obtaining equally good trees.

https://urldefense.com/v3/__https://www.biorxiv.org/content/10.1101/2022.07.13.499893v1__;!!Mih3wA!HsKI560bu08iL24_UZnDEag71ybNtTfeVKPMh8xojsKWmbe779ckAr88YPa_ce3EUcmjZaxaDnLDW-8z-lnroAh_d20$

Alexis

--
Alexandros (Alexis) Stamatakis

Research Group Leader, Heidelberg Institute for Theoretical Studies
Full Professor, Dept. of Informatics, Karlsruhe Institute of Technology
Affiliated Scientist, Evolutionary Genetics and Paleogenomics (EGP) lab, Institute of Molecular Biology and Biotechnology, Foundation for Research and Technology Hellas

https://urldefense.com/v3/__http://www.exelixis-lab.org__;!!Mih3wA!HsKI560bu08iL24_UZnDEag71ybNtTfeVKPMh8xojsKWmbe779ckAr88YPa_ce3EUcmjZaxaDnLDW-8z-lnrZhPqvvc$
--
You received this message because you are subscribed to the Google Groups "raxml" group.
To unsubscribe from this group and stop receiving emails from it, send an email to raxml+un...@googlegroups.com.
To view this discussion on the web visit https://urldefense.com/v3/__https://groups.google.com/d/msgid/raxml/757401e5-d91e-5c52-ae02-330d13308f69*40gmail.com__;JQ!!Mih3wA!HsKI560bu08iL24_UZnDEag71ybNtTfeVKPMh8xojsKWmbe779ckAr88YPa_ce3EUcmjZaxaDnLDW-8z-lnrUNLWdxA$ .

[Alexey Kozlov]

Hello Wayne,

many thanks for your tests! Really appreciate that you always keep a sharp eye on our work :)

> As suggested in your paper, I changed two lines in constants.hpp of RAxML-NG 1.1.0 to relax two
> numerical thresholds as follows:
>
>   #define DEF_LH_EPSILON            10.

that's correct

>   #define RAXML_BRLEN_DEFAULT       1000.

This is unfortunately the wrong constant, since it defines the default branch length, i.e. starting
point and not the likelihood epsilon for branch length optimization. I'm a bit surprised that you
nevertheless got reasonable results :)

This epsilon was actually hardcoded, and we made it configurable in Julia's branch of raxml-ng only.
I guess it's this one:

https://github.com/amkozlov/raxml-ng/commit/f8bc1985c4d5a0fe972633d437bca66b0d55c5d9

but Julia should know better.

> Column Q shows the speedups obtained going from the original code to the changed code. There are
> several really impressive speedups, but most are more modest than reported in your paper. For the
> 140.AA data set the times for the changed code are actually a little slower than those for the
> original code but within the variation expected for shared-node runs. The median and average
> speedups across all the runs are 1.13 and 1.28, respectively, as noted in Cells P48 and P49. Such
> speedups will be welcomed by CIPRES users, so I modified our run script for RAxML-NG to use the
> changed code :)

You might want to revert RAXML_BRLEN_DEFAULT setting, see above.

> * Based upon your findings and the results presented here, it seems like it would be good for Alexey
> to relax the numerical thresholds in the next release of RAxML-NG.

Yes, will definitely do this.

> * Someone at HITS might want to check whether these errors occur on your AMD nodes too, though maybe
> the Slurm scheduler is also required to produce the errors.

Thread pinning option will try to pin N threads to the first N cores, so it will often fail in
shared-node scenario. If you have ideas how to make it more generic/bulletproof, feedback is welcome!

Best,
Alexey

>
> Best regards,  Wayne
>
>
>> On Jul 15, 2022, at 4:10 AM, Alexandros Stamatakis <alexandros...@gmail.com

>> <mailto:alexandros...@gmail.com>> wrote:
>>
>> Dear Users of RAxML,
>>
>> There is some free lunch here :-)
>>
>> The worst part about implementing phylogenetic Maximum Likelihood methods is to devise stable
>> numerical optimization routines.
>>
>> But, then, how do we set appropriate convergence thresholds (the famous epsilon in numerics) for
>> all these ML optimizers?
>>
>> In the first systematic study on the impact of these threshold settings on reconstruction accuracy
>>  we find that IQ-Tree and RAxML-NG can be accelerated by a factor of 1.3 and 1.9 respectively by
>> setting some of these thresholds to larger values while still obtaining equally good trees.
>>
>> https://urldefense.com/v3/__https://www.biorxiv.org/content/10.1101/2022.07.13.499893v1__;!!Mih3wA!HsKI560bu08iL24_UZnDEag71ybNtTfeVKPMh8xojsKWmbe779ckAr88YPa_ce3EUcmjZaxaDnLDW-8z-lnroAh_d20$
>> <https://urldefense.com/v3/__https://www.biorxiv.org/content/10.1101/2022.07.13.499893v1__;!!Mih3wA!HsKI560bu08iL24_UZnDEag71ybNtTfeVKPMh8xojsKWmbe779ckAr88YPa_ce3EUcmjZaxaDnLDW-8z-lnroAh_d20$>
>>
>> Alexis
>>
>> --
>> Alexandros (Alexis) Stamatakis
>>
>> Research Group Leader, Heidelberg Institute for Theoretical Studies
>> Full Professor, Dept. of Informatics, Karlsruhe Institute of Technology
>> Affiliated Scientist, Evolutionary Genetics and Paleogenomics (EGP) lab, Institute of Molecular
>> Biology and Biotechnology, Foundation for Research and Technology Hellas
>>
>> https://urldefense.com/v3/__http://www.exelixis-lab.org__;!!Mih3wA!HsKI560bu08iL24_UZnDEag71ybNtTfeVKPMh8xojsKWmbe779ckAr88YPa_ce3EUcmjZaxaDnLDW-8z-lnrZhPqvvc$
>> <https://urldefense.com/v3/__http://www.exelixis-lab.org__;!!Mih3wA!HsKI560bu08iL24_UZnDEag71ybNtTfeVKPMh8xojsKWmbe779ckAr88YPa_ce3EUcmjZaxaDnLDW-8z-lnrZhPqvvc$>
>>
>> --
>> You received this message because you are subscribed to the Google Groups "raxml" group.
>> To unsubscribe from this group and stop receiving emails from it, send an email to

>> raxml+un...@googlegroups.com <mailto:raxml+un...@googlegroups.com>.

>> To view this discussion on the web visit
>> https://urldefense.com/v3/__https://groups.google.com/d/msgid/raxml/757401e5-d91e-5c52-ae02-330d13308f69*40gmail.com__;JQ!!Mih3wA!HsKI560bu08iL24_UZnDEag71ybNtTfeVKPMh8xojsKWmbe779ckAr88YPa_ce3EUcmjZaxaDnLDW-8z-lnrUNLWdxA$
>> <https://urldefense.com/v3/__https://groups.google.com/d/msgid/raxml/757401e5-d91e-5c52-ae02-330d13308f69*40gmail.com__;JQ!!Mih3wA!HsKI560bu08iL24_UZnDEag71ybNtTfeVKPMh8xojsKWmbe779ckAr88YPa_ce3EUcmjZaxaDnLDW-8z-lnrUNLWdxA$>
>> .
>

> --
> You received this message because you are subscribed to the Google Groups "raxml" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to

> raxml+un...@googlegroups.com <mailto:raxml+un...@googlegroups.com>.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/raxml/7446509B-9DD0-4608-B0A1-BE3AD58607BC%40ucsd.edu
> <https://groups.google.com/d/msgid/raxml/7446509B-9DD0-4608-B0A1-BE3AD58607BC%40ucsd.edu?utm_medium=email&utm_source=footer>.

[Pfeiffer, Wayne]

Hi Alexey and Julia,

Thanks for your prompt replies and clarification that RAXML_BRLEN_DEFAULT is not the correct threshold to change for the second epsilon.

I would be happy to re-run my benchmarks with the correct threshold, but I want to do that with RAxML-NG 1.1.0 so that we can use the resulting code in production for CIPRES. The link that both of you gave me is to a modified 1.0.1 version.

* Please let me know if you can give me a suitable 1.1.0 version in which I can change both thresholds myself or in which they have already been changed.

As regards thread pinning, attached is the slurm-affall script that we use on Expanse. Perhaps this will give you an idea how to improve the --extra thread-pin option in RAxML-NG.

The slurm-aff script was originally written to not do anything for jobs in our compute partition, which have exclusive access to a node, because affinity was ensured for such jobs. However, that is no longer the case after recent upgrades of the OS and Slurm on Expanse. Fortunately, the slurm-affall script, with the compute partition test commented out, works for the case of 128 threads on a full node. On the other hand, it does not work for the case of 10 MPI processes and 12 Pthreads per process on a full node, which we sometimes use for RAxML8 jobs. For that case, I use the original slurm-aff script, which relies on the system to take care of affinity.

Best regards,  Wayne

On Jul 25, 2022, at 4:43 AM, Haag, Julia <julia...@h-its.org> wrote:

Hello Wayne,

Thanks for sharing your results, they are very interesting! :-) 

First of all what Alexey wrote about the second threshold is unfortunately correct, but the threshold he linked is yet a different threshold, you should best use this RAxML-NG version https://github.com/lukashuebner/param-raxml-ng and set --spr_lheps to 1000 (or change DEF_SPR_LH_EPSILON in the constants.hpp file) for your experiments.

Your experimental setup is a bit different than ours: Instead of re-running RAxML-NG with the same seed repeatedly, we used 50 different seeds (= 50 tree inferences per dataset, 25 using --tree rand{1} and 25 using --tree pars{1}; we only compared runtimes for tree inferences with the same random seed). But that is just a minor difference, so the results should not be influenced by this different setup that much.

What is interesting is that your results for the amino acid dataset 140 are very different from our results: we observe an impressive speedup with the new settings (see the attached plot). We observe a similar speedup if we only change the DEF_LH_EPSILON, so the issue with the different second parameter is not the cause of these differences. According to your spreadsheet you used a different substitution model than we did, so I'm guessing the model parameter optimization causes these difference between our analyses and yours. 

If you have the time to re-run your analyses with the second threshold changed as well, I'd be really interested to see the updated results :-) 

Greetings,
Julia

<raxml_new_default_values_runtime_box_with_outliers.pdf>

[Pfeiffer, Wayne]

Hi Julia,

I downloaded the version of RAxML-NG 1.1.0 from https://github.com/tschuelia/raxml-ng to my Mac, uploaded it to Expanse, and made your suggest changes to the thresholds. I discovered that pll-modules and terraphast were missing in lib, so I copied them from the production version of 1.1.0. When I then tried to build, make compiled both libraries successfully but failed with the appended errors. For some reason “Mac OS X” appears in the output even though the compilation is being done on a login node of Expanse.

When the code was downloaded to my Mac, the files were automatically unzipped. I tarred them before uploading to Expanse, but I wonder whether some Mac feature might have been added.

* Please advise how I should proceed.

Thanks,  Wayne

---

[cipres@login02| /expanse/projects/ngbt/opt/expanse/raxml-ng/1.1.0.fast/build]> make

[  1%] [FLEX][lex_rtree_t] Building scanner with flex 2.6.1

[  1%] [BISON][parse_utree_t] Building parser with bison 3.0.4

make[2]: Warning: File 'libs/pll-modules/libs/libpll/src/parse_utree.h' has modification time 0.0024 s in the future

[  2%] [FLEX][lex_utree_t] Building scanner with flex 2.6.1

[  2%] [BISON][parse_rtree_t] Building parser with bison 3.0.4

Scanning dependencies of target pll_obj

make[2]: warning:  Clock skew detected.  Your build may be incomplete.

[  2%] Building C object libs/pll-modules/libs/libpll/src/CMakeFiles/pll_obj.dir/compress.c.o

...

[ 56%] Building CXX object libs/terraphast/CMakeFiles/terraces.dir/lib/validation.cpp.o

[ 57%] Linking CXX static library ../../localdeps/lib/libterraces.a

[ 57%] Built target terraces

Scanning dependencies of target raxml_module

[ 57%] Building CXX object src/CMakeFiles/raxml_module.dir/._AncestralStates.cpp.o

/expanse/projects/ngbt/opt/expanse/raxml-ng/1.1.0.fast/src/._AncestralStates.cpp:1:1: warning: null character(s) ignored

    1 |     Mac OS X                2   ?      ?                                      ATTR       ?   ?   <                  ?   <  com.apple.quarantine q/0083;62dea199;Safari;4821FB9C-5CEA-4DD2-A5E1-2627186FBFEA 

      | ^

/expanse/projects/ngbt/opt/expanse/raxml-ng/1.1.0.fast/src/._AncestralStates.cpp:1:2: error: stray '\5' in program

    1 |     Mac OS X                2   ?      ?                                      ATTR       ?   ?   <                  ?   <  com.apple.quarantine q/0083;62dea199;Safari;4821FB9C-5CEA-4DD2-A5E1-2627186FBFEA 

      |  ^

...

make[2]: *** [src/CMakeFiles/raxml_module.dir/build.make:82: src/CMakeFiles/raxml_module.dir/._AncestralStates.cpp.o] Error 1

make[1]: *** [CMakeFiles/Makefile2:806: src/CMakeFiles/raxml_module.dir/all] Error 2

make: *** [Makefile:160: all] Error 2

On Jul 25, 2022, at 6:41 AM, Haag, Julia <julia...@h-its.org> wrote:

Hi Wayne,

You could use my version of RAxML-NG that is based on 1.1.0: https://github.com/tschuelia/raxml-ng

Note that there are more than two thresholds settable, so in order to get the correct threshold settings you need to change all of the following in the constants.hpp from 0.1 to 10 or 1000:

Line 7: DEF_LH_EPSILON 10

Line 8: DEF_LH_EPSILON_AUTO 10

Line 9: DEF_LH_EPSILON_FAST 10

Line 10: DEF_LH_EPSILON_SLOW 10

Line 15: DEF_LH_EPSILON_BRLEN_FULL 10

Line 16: DEF_LH_EPSILON_BRLEN_TRIPLET 1000

Greetings,

Julia 

<slurm-affall>

[Pfeiffer, Wayne]

Hi Julia,

Thanks for the revised instructions. I downloaded the code directly to Expanse, built the code after changing the thresholds, and repeated all of the benchmarks. An updated spreadsheet with new results is attached.

The speedups in Column P are much better now! Across all eight data sets, the median and average speedups are 1.51 and 1.72, respectively. That is really impressive and will be welcomed by our users!

I also highlighted two final scores from the fast code in Column O.

- The pink score in Cell O40 is the worst compared to the original code across all the data sets. Not surprisingly, the speedup in the next column is also the largest.

- The blue score in Cell O30 is actually better than that from the original code and is the best such case.

Best regards,  Wayne

On Jul 25, 2022, at 8:46 AM, Haag, Julia <julia...@h-its.org> wrote:

Hi Wayne,

 I discovered that pll-modules and terraphast were missing in lib, so I copied them from the production version of 1.1.0

For terraphast that should be fine, but in order for the brlen epsilon setting to work, you need a special version of the pll-modules which you can find here: https://github.com/tschuelia/pll-modules (pll-modules in the unmodified version has this threshold hardcoded to 0.1).

Alternatively you can get all required modules at once by cloning the repo recursive (git clone --recursive https://github.com/tschuelia/raxml-ng.git)

 I tarred them before uploading to Expanse

The OS X error is very odd, I have never seen this one before, but I have build my RAxML-NG version on different machines (Mac and Linux). I’m guessing that your Mac adds something to the files during the tar process. Can you try to download the files directly to Expanse (e.g. using git clone) without using the Mac and try building it again? 

Greetings,

Julia

[Alexandros Stamatakis]

Thank you so much Wayne,

Just a comment, not sure if it matters.

I assume you took the LnL values directly from the RAxML output produced
by the different runs.

However, in our experiments we took the final trees and then executed
the tree evaluation option on these with more conservative epsilon
parameter settings.

Maybe if you do this, some final LnL values obtained under the more
liberal epsilon settings for the tree searches might further improve in
this case?

Alexis

On 25.07.22 19:30, Pfeiffer, Wayne wrote:
> Hi Julia,
>
> Thanks for the revised instructions. I downloaded the code directly to
> Expanse, built the code after changing the thresholds, and repeated all
> of the benchmarks. An updated spreadsheet with new results is attached.
>
> The speedups in Column P are much better now! Across all eight data
> sets, the median and average speedups are 1.51 and 1.72, respectively.
> That is really impressive and will be welcomed by our users!
>
> I also highlighted two final scores from the fast code in Column O.
>
> - The pink score in Cell O40 is the worst compared to the original code
> across all the data sets. Not surprisingly, the speedup in the next
> column is also the largest.
>
> - The blue score in Cell O30 is actually better than that from the
> original code and is the best such case.
>
> Best regards,  Wayne
>
>
>> On Jul 25, 2022, at 8:46 AM, Haag, Julia <julia...@h-its.org

>> <mailto:julia...@h-its.org>> wrote:
>>
>> Hi Wayne,
>>
>>>  I discovered that pll-modules and terraphast were missing in lib, so
>>> I copied them from the production version of 1.1.0
>> For terraphast that should be fine, but in order for the brlen epsilon
>> setting to work, you need a special version of the pll-modules which
>> you can find here: https://github.com/tschuelia/pll-modules

>> <https://urldefense.com/v3/__https://github.com/tschuelia/pll-modules__;!!Mih3wA!DslakEwgDjrXMwk9XgWRjPVJFUsfOeSHHkkxKsED729Lj8Tew7a6APkoAPKcUQ5-ZPjEiP9qt9Le9BfDjVjPtA$> (pll-modules

>> in the unmodified version has this threshold hardcoded to 0.1).
>> Alternatively you can get all required modules at once by cloning the
>> repo recursive (git clone --recursive
>> https://github.com/tschuelia/raxml-ng.git

>> <https://urldefense.com/v3/__https://github.com/tschuelia/raxml-ng.git__;!!Mih3wA!DslakEwgDjrXMwk9XgWRjPVJFUsfOeSHHkkxKsED729Lj8Tew7a6APkoAPKcUQ5-ZPjEiP9qt9Le9BfI9-17TA$>)

>>
>>>  I tarred them before uploading to Expanse
>> The OS X error is very odd, I have never seen this one before, but I
>> have build my RAxML-NG version on different machines (Mac and Linux).
>> I’m guessing that your Mac adds something to the files during the tar
>> process. Can you try to download the files directly to Expanse (e.g.
>> using git clone) without using the Mac and try building it again?
>>
>> Greetings,
>> Julia
>>
>>> Am 25.07.2022 um 17:17 schrieb Pfeiffer, Wayne <pfei...@sdsc.edu

>>> <mailto:pfei...@sdsc.edu>>:

>>>
>>> Hi Julia,
>>>
>>> I downloaded the version of RAxML-NG 1.1.0 from
>>> https://github.com/tschuelia/raxml-ng

>>> <https://urldefense.com/v3/__https://github.com/tschuelia/raxml-ng__;!!Mih3wA!EBO5DPxGttMbeYRi65pgPGk-xamd33R6Vkwff_XXknUx3AgOhjzJKyrNBkc5Y9x9E85ynPKI4KBMHbiDs2KPKw$> to

>>>> <mailto:julia...@h-its.org>> wrote:
>>>>
>>>> Hi Wayne,
>>>>
>>>> You could use my version of RAxML-NG that is based on 1.1.0:
>>>> https://github.com/tschuelia/raxml-ng

>>>> <https://urldefense.com/v3/__https://github.com/tschuelia/raxml-ng__;!!Mih3wA!EBO5DPxGttMbeYRi65pgPGk-xamd33R6Vkwff_XXknUx3AgOhjzJKyrNBkc5Y9x9E85ynPKI4KBMHbiDs2KPKw$>

>>>> Note that there are more than two thresholds settable, so in order
>>>> to get the correct threshold settings you need to change all of the
>>>> following in the constants.hpp from 0.1 to 10 or 1000:
>>>>
>>>> Line 7: DEF_LH_EPSILON 10
>>>> Line 8: DEF_LH_EPSILON_AUTO 10
>>>> Line 9: DEF_LH_EPSILON_FAST 10
>>>> Line 10: DEF_LH_EPSILON_SLOW 10
>>>>
>>>> Line 15: DEF_LH_EPSILON_BRLEN_FULL 10
>>>> Line 16: DEF_LH_EPSILON_BRLEN_TRIPLET 1000
>>>>
>>>> Greetings,
>>>> Julia
>>>>
>>>>
>>>>
>>>>> Am 25.07.2022 um 15:18 schrieb Pfeiffer, Wayne <pfei...@sdsc.edu

>>>>> <mailto:pfei...@sdsc.edu>>:

>>>>>> <mailto:julia...@h-its.org>> wrote:
>>>>>>
>>>>>> Hello Wayne,
>>>>>>
>>>>>> Thanks for sharing your results, they are very interesting! :-)
>>>>>>
>>>>>> First of all what Alexey wrote about the second threshold is
>>>>>> unfortunately correct, but the threshold he linked is yet a
>>>>>> different threshold, you should best use this RAxML-NG
>>>>>> versionhttps://github.com/lukashuebner/param-raxml-ng

>>>>>> <https://urldefense.com/v3/__https://github.com/lukashuebner/param-raxml-ng__;!!Mih3wA!C4ZheRdD0R_Wo5DL1TrFInj9FJilUZN4CsUPgB7ABg2Yi5yPt6aqYva_Zj_vIcFj0Ftg0seDe-HBUGD75fFPkQ$> and

>>>>>>> <alexei...@gmail.com <mailto:alexei...@gmail.com>>:

>>>>>>>
>>>>>>> Hello Wayne,
>>>>>>>
>>>>>>> many thanks for your tests! Really appreciate that you always
>>>>>>> keep a sharp eye on our work :)
>>>>>>>
>>>>>>>> As suggested in your paper, I changed two lines in constants.hpp
>>>>>>>> of RAxML-NG 1.1.0 to relax two numerical thresholds as follows:
>>>>>>>>   #define DEF_LH_EPSILON            10.
>>>>>>>
>>>>>>> that's correct
>>>>>>>
>>>>>>>>   #define RAXML_BRLEN_DEFAULT       1000.
>>>>>>>
>>>>>>> This is unfortunately the wrong constant, since it defines the
>>>>>>> default branch length, i.e. starting point and not the likelihood
>>>>>>> epsilon for branch length optimization. I'm a bit surprised that
>>>>>>> you nevertheless got reasonable results :)
>>>>>>>
>>>>>>> This epsilon was actually hardcoded, and we made it configurable
>>>>>>> in Julia's branch of raxml-ng only. I guess it's this one:
>>>>>>>
>>>>>>> https://github.com/amkozlov/raxml-ng/commit/f8bc1985c4d5a0fe972633d437bca66b0d55c5d9

>>>>>>> <https://urldefense.com/v3/__https://github.com/amkozlov/raxml-ng/commit/f8bc1985c4d5a0fe972633d437bca66b0d55c5d9__;!!Mih3wA!EBO5DPxGttMbeYRi65pgPGk-xamd33R6Vkwff_XXknUx3AgOhjzJKyrNBkc5Y9x9E85ynPKI4KBMHbho8k5A8Q$>

>>>>>>>>> <mailto:alexandros...@gmail.com><mailto:alexandros...@gmail.com

>>>>>>>>> <mailto:alexandros...@gmail.com>>> wrote:
>>>>>>>>>
>>>>>>>>> Dear Users of RAxML,
>>>>>>>>>
>>>>>>>>> There is some free lunch here :-)
>>>>>>>>>
>>>>>>>>> The worst part about implementing phylogenetic Maximum
>>>>>>>>> Likelihood methods is to devise stable numerical optimization
>>>>>>>>> routines.
>>>>>>>>>
>>>>>>>>> But, then, how do we set appropriate convergence thresholds
>>>>>>>>> (the famous epsilon in numerics) for all these ML optimizers?
>>>>>>>>>
>>>>>>>>> In the first systematic study on the impact of these threshold
>>>>>>>>> settings on reconstruction accuracy  we find that IQ-Tree and
>>>>>>>>> RAxML-NG can be accelerated by a factor of 1.3 and 1.9
>>>>>>>>> respectively by setting some of these thresholds to larger
>>>>>>>>> values while still obtaining equally good trees.
>>>>>>>>>
>>>>>>>>> https://urldefense.com/v3/__https://www.biorxiv.org/content/10.1101/2022.07.13.499893v1__;!!Mih3wA!HsKI560bu08iL24_UZnDEag71ybNtTfeVKPMh8xojsKWmbe779ckAr88YPa_ce3EUcmjZaxaDnLDW-8z-lnroAh_d20$

>>>>>>>>> <https://urldefense.com/v3/__https://www.biorxiv.org/content/10.1101/2022.07.13.499893v1__;!!Mih3wA!HsKI560bu08iL24_UZnDEag71ybNtTfeVKPMh8xojsKWmbe779ckAr88YPa_ce3EUcmjZaxaDnLDW-8z-lnroAh_d20$><https://urldefense.com/v3/__https://www.biorxiv.org/content/10.1101/2022.07.13.499893v1__;!!Mih3wA!HsKI560bu08iL24_UZnDEag71ybNtTfeVKPMh8xojsKWmbe779ckAr88YPa_ce3EUcmjZaxaDnLDW-8z-lnroAh_d20$

>>>>>>>>> <https://urldefense.com/v3/__https://www.biorxiv.org/content/10.1101/2022.07.13.499893v1__;!!Mih3wA!HsKI560bu08iL24_UZnDEag71ybNtTfeVKPMh8xojsKWmbe779ckAr88YPa_ce3EUcmjZaxaDnLDW-8z-lnroAh_d20$>>
>>>>>>>>> Alexis
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Alexandros (Alexis) Stamatakis
>>>>>>>>>
>>>>>>>>> Research Group Leader, Heidelberg Institute for Theoretical Studies
>>>>>>>>> Full Professor, Dept. of Informatics, Karlsruhe Institute of
>>>>>>>>> Technology
>>>>>>>>> Affiliated Scientist, Evolutionary Genetics and Paleogenomics
>>>>>>>>> (EGP) lab, Institute of Molecular Biology and Biotechnology,
>>>>>>>>> Foundation for Research and Technology Hellas
>>>>>>>>>
>>>>>>>>> https://urldefense.com/v3/__http://www.exelixis-lab.org__;!!Mih3wA!HsKI560bu08iL24_UZnDEag71ybNtTfeVKPMh8xojsKWmbe779ckAr88YPa_ce3EUcmjZaxaDnLDW-8z-lnrZhPqvvc$

>>>>>>>>> <https://urldefense.com/v3/__http://www.exelixis-lab.org__;!!Mih3wA!HsKI560bu08iL24_UZnDEag71ybNtTfeVKPMh8xojsKWmbe779ckAr88YPa_ce3EUcmjZaxaDnLDW-8z-lnrZhPqvvc$><https://urldefense.com/v3/__http://www.exelixis-lab.org__;!!Mih3wA!HsKI560bu08iL24_UZnDEag71ybNtTfeVKPMh8xojsKWmbe779ckAr88YPa_ce3EUcmjZaxaDnLDW-8z-lnrZhPqvvc$

>>>>>>>>> <https://urldefense.com/v3/__http://www.exelixis-lab.org__;!!Mih3wA!HsKI560bu08iL24_UZnDEag71ybNtTfeVKPMh8xojsKWmbe779ckAr88YPa_ce3EUcmjZaxaDnLDW-8z-lnrZhPqvvc$>>
>>>>>>>>> --
>>>>>>>>> You received this message because you are subscribed to the
>>>>>>>>> Google Groups "raxml" group.
>>>>>>>>> To unsubscribe from this group and stop receiving emails from

>>>>>>>>> it, send an email toraxml+u...@googlegroups.com
>>>>>>>>> <mailto:raxml+un...@googlegroups.com><mailto:raxml+un...@googlegroups.com

>>>>>>>>> <mailto:raxml+un...@googlegroups.com>>.
>>>>>>>>> To view this discussion on the web

>>>>>>>>> visithttps://urldefense.com/v3/__https://groups.google.com/d/msgid/raxml/757401e5-d91e-5c52-ae02-330d13308f69*40gmail.com__;JQ!!Mih3wA!HsKI560bu08iL24_UZnDEag71ybNtTfeVKPMh8xojsKWmbe779ckAr88YPa_ce3EUcmjZaxaDnLDW-8z-lnrUNLWdxA$
>>>>>>>>> <https://urldefense.com/v3/__https://groups.google.com/d/msgid/raxml/757401e5-d91e-5c52-ae02-330d13308f69*40gmail.com__;JQ!!Mih3wA!HsKI560bu08iL24_UZnDEag71ybNtTfeVKPMh8xojsKWmbe779ckAr88YPa_ce3EUcmjZaxaDnLDW-8z-lnrUNLWdxA$><https://urldefense.com/v3/__https://groups.google.com/d/msgid/raxml/757401e5-d91e-5c52-ae02-330d13308f69*40gmail.com__;JQ!!Mih3wA!HsKI560bu08iL24_UZnDEag71ybNtTfeVKPMh8xojsKWmbe779ckAr88YPa_ce3EUcmjZaxaDnLDW-8z-lnrUNLWdxA$

>>>>>>>>> <https://urldefense.com/v3/__https://groups.google.com/d/msgid/raxml/757401e5-d91e-5c52-ae02-330d13308f69*40gmail.com__;JQ!!Mih3wA!HsKI560bu08iL24_UZnDEag71ybNtTfeVKPMh8xojsKWmbe779ckAr88YPa_ce3EUcmjZaxaDnLDW-8z-lnrUNLWdxA$>>
>>>>>>>>> .
>>>>>>>> --
>>>>>>>> You received this message because you are subscribed to the
>>>>>>>> Google Groups "raxml" group.
>>>>>>>> To unsubscribe from this group and stop receiving emails from

>>>>>>>> it, send an email toraxml+u...@googlegroups.com
>>>>>>>> <mailto:raxml+un...@googlegroups.com><mailto:raxml+un...@googlegroups.com

>>>>>>>> <mailto:raxml+un...@googlegroups.com>>.
>>>>>>>> To view this discussion on the web

>>>>>>>> visithttps://groups.google.com/d/msgid/raxml/7446509B-9DD0-4608-B0A1-BE3AD58607BC%40ucsd.edu
>>>>>>>> <https://urldefense.com/v3/__https://groups.google.com/d/msgid/raxml/7446509B-9DD0-4608-B0A1-BE3AD58607BC*40ucsd.edu__;JQ!!Mih3wA!EBO5DPxGttMbeYRi65pgPGk-xamd33R6Vkwff_XXknUx3AgOhjzJKyrNBkc5Y9x9E85ynPKI4KBMHbhP-ABc8w$><https://groups.google.com/d/msgid/raxml/7446509B-9DD0-4608-B0A1-BE3AD58607BC%40ucsd.edu?utm_medium=email&utm_source=footer
>>>>>>>> <https://urldefense.com/v3/__https://groups.google.com/d/msgid/raxml/7446509B-9DD0-4608-B0A1-BE3AD58607BC*40ucsd.edu?utm_medium=email&utm_source=footer__;JQ!!Mih3wA!EBO5DPxGttMbeYRi65pgPGk-xamd33R6Vkwff_XXknUx3AgOhjzJKyrNBkc5Y9x9E85ynPKI4KBMHbht5pl7Qw$>>.
>>>>>
>>>>> <slurm-affall>

>>>>
>>>
>>
>

--
Alexandros (Alexis) Stamatakis

Research Group Leader, Heidelberg Institute for Theoretical Studies
Full Professor, Dept. of Informatics, Karlsruhe Institute of Technology
Affiliated Scientist, Evolutionary Genetics and Paleogenomics (EGP) lab,
Institute of Molecular Biology and Biotechnology, Foundation for
Research and Technology Hellas

www.exelixis-lab.org

[Pfeiffer, Wayne]

Hi Alexis,

In response to your comment, I executed the tree evaluation option with the original code for the two cases where the fast score differed the most from the original score: 404 and 83F06.AA. The scores did not improve.

However, most CIPRES users do all-in-one analyses, in which 20 regular searches are done followed by bootstrap searches for RAxML-NG or vice versa for RAxML8. Thus I made all-in-one analyses using RAxML8 and the original and fast versions of RAxML-NG for the same eight data sets. The results are summarized in the attached spreadsheet.

The best scores are listed in Column S. For five data sets there is no difference in the scores between the original and fast versions of RAxML-NG. For two data sets — 404 and 83F06.AA — the fast score is slightly worse, while for one data set – 218 — the fast score is slightly better! These results provide confidence that the fast version of RAxML-NG will provide reliable results. Moreover, the scores from RAxML-NG are as good or better than those from RAxML8. For 125.34part the RAxML-NG score is much better.

On the other hand, the fast version of RAxML-NG is still slower than RAxML8 for all but the last data set, as can be seen by the slowdowns in Column V. That is because the bootstrap searches in RAxML-NG are much slower than those in RAxML8, as can be seen by the slowdowns in Column U.

* Perhaps the numerical thresholds in RAxML-NG could be further reduced for bootstrap searches.

I could help investigate this, if you like.

Best regards,  Wayne

> On Jul 26, 2022, at 12:00 AM, Alexandros Stamatakis <alexandros...@gmail.com> wrote:
>
> Thank you so much Wayne,
>
> Just a comment, not sure if it matters.
>
> I assume you took the LnL values directly from the RAxML output produced by the different runs.
>
> However, in our experiments we took the final trees and then executed the tree evaluation option on these with more conservative epsilon parameter settings.
>
> Maybe if you do this, some final LnL values obtained under the more liberal epsilon settings for the tree searches might further improve in this case?
>
> Alexis
>
> On 25.07.22 19:30, Pfeiffer, Wayne wrote:
>> Hi Julia,
>> Thanks for the revised instructions. I downloaded the code directly to Expanse, built the code after changing the thresholds, and repeated all of the benchmarks. An updated spreadsheet with new results is attached.
>> The speedups in Column P are much better now! Across all eight data sets, the median and average speedups are 1.51 and 1.72, respectively. That is really impressive and will be welcomed by our users!
>> I also highlighted two final scores from the fast code in Column O.
>> - The pink score in Cell O40 is the worst compared to the original code across all the data sets. Not surprisingly, the speedup in the next column is also the largest.
>> - The blue score in Cell O30 is actually better than that from the original code and is the best such case.
>> Best regards,  Wayne
>>> On Jul 25, 2022, at 8:46 AM, Haag, Julia <julia...@h-its.org <mailto:julia...@h-its.org>> wrote:
>>>
>>> Hi Wayne,
>>>
>>>>  I discovered that pll-modules and terraphast were missing in lib, so I copied them from the production version of 1.1.0

>>> For terraphast that should be fine, but in order for the brlen epsilon setting to work, you need a special version of the pll-modules which you can find here: https://urldefense.com/v3/__https://github.com/tschuelia/pll-modules__;!!Mih3wA!D30yUFXPbdOeWJlW4fAiDSrpL81FNE-OzUsFTmqXl3P1APkWAcSPmFjgE422o_ehkAQlVlEiffZ8XjsP0RJHj8q8yj4$  <https://urldefense.com/v3/__https://github.com/tschuelia/pll-modules__;!!Mih3wA!DslakEwgDjrXMwk9XgWRjPVJFUsfOeSHHkkxKsED729Lj8Tew7a6APkoAPKcUQ5-ZPjEiP9qt9Le9BfDjVjPtA$> (pll-modules in the unmodified version has this threshold hardcoded to 0.1).
>>> Alternatively you can get all required modules at once by cloning the repo recursive (git clone --recursive https://urldefense.com/v3/__https://github.com/tschuelia/raxml-ng.git__;!!Mih3wA!D30yUFXPbdOeWJlW4fAiDSrpL81FNE-OzUsFTmqXl3P1APkWAcSPmFjgE422o_ehkAQlVlEiffZ8XjsP0RJHLkGcfXU$  <https://urldefense.com/v3/__https://github.com/tschuelia/raxml-ng.git__;!!Mih3wA!DslakEwgDjrXMwk9XgWRjPVJFUsfOeSHHkkxKsED729Lj8Tew7a6APkoAPKcUQ5-ZPjEiP9qt9Le9BfI9-17TA$>)

>>>
>>>>  I tarred them before uploading to Expanse
>>> The OS X error is very odd, I have never seen this one before, but I have build my RAxML-NG version on different machines (Mac and Linux). I’m guessing that your Mac adds something to the files during the tar process. Can you try to download the files directly to Expanse (e.g. using git clone) without using the Mac and try building it again?
>>>
>>> Greetings,
>>> Julia
>>>
>>>> Am 25.07.2022 um 17:17 schrieb Pfeiffer, Wayne <pfei...@sdsc.edu <mailto:pfei...@sdsc.edu>>:
>>>>
>>>> Hi Julia,
>>>>

>>>> I downloaded the version of RAxML-NG 1.1.0 from https://urldefense.com/v3/__https://github.com/tschuelia/raxml-ng__;!!Mih3wA!D30yUFXPbdOeWJlW4fAiDSrpL81FNE-OzUsFTmqXl3P1APkWAcSPmFjgE422o_ehkAQlVlEiffZ8XjsP0RJHjlJdp6Y$  <https://urldefense.com/v3/__https://github.com/tschuelia/raxml-ng__;!!Mih3wA!EBO5DPxGttMbeYRi65pgPGk-xamd33R6Vkwff_XXknUx3AgOhjzJKyrNBkc5Y9x9E85ynPKI4KBMHbiDs2KPKw$> to my Mac, uploaded it to Expanse, and made your suggest changes to the thresholds. I discovered that pll-modules and terraphast were missing in lib, so I copied them from the production version of 1.1.0. When I then tried to build, make compiled both libraries successfully but failed with the appended errors. For some reason “Mac OS X” appears in the output even though the compilation is being done on a login node of Expanse.

>>>>> You could use my version of RAxML-NG that is based on 1.1.0: https://urldefense.com/v3/__https://github.com/tschuelia/raxml-ng__;!!Mih3wA!D30yUFXPbdOeWJlW4fAiDSrpL81FNE-OzUsFTmqXl3P1APkWAcSPmFjgE422o_ehkAQlVlEiffZ8XjsP0RJHjlJdp6Y$  <https://urldefense.com/v3/__https://github.com/tschuelia/raxml-ng__;!!Mih3wA!EBO5DPxGttMbeYRi65pgPGk-xamd33R6Vkwff_XXknUx3AgOhjzJKyrNBkc5Y9x9E85ynPKI4KBMHbiDs2KPKw$>

>>>>>>>> https://urldefense.com/v3/__https://github.com/amkozlov/raxml-ng/commit/f8bc1985c4d5a0fe972633d437bca66b0d55c5d9__;!!Mih3wA!D30yUFXPbdOeWJlW4fAiDSrpL81FNE-OzUsFTmqXl3P1APkWAcSPmFjgE422o_ehkAQlVlEiffZ8XjsP0RJHUeq-izU$  <https://urldefense.com/v3/__https://github.com/amkozlov/raxml-ng/commit/f8bc1985c4d5a0fe972633d437bca66b0d55c5d9__;!!Mih3wA!EBO5DPxGttMbeYRi65pgPGk-xamd33R6Vkwff_XXknUx3AgOhjzJKyrNBkc5Y9x9E85ynPKI4KBMHbho8k5A8Q$>

>>>>>>>>> To view this discussion on the web visithttps://groups.google.com/d/msgid/raxml/7446509B-9DD0-4608-B0A1-BE3AD58607BC%40ucsd.edu <https://urldefense.com/v3/__https://groups.google.com/d/msgid/raxml/7446509B-9DD0-4608-B0A1-BE3AD58607BC*40ucsd.edu__;JQ!!Mih3wA!EBO5DPxGttMbeYRi65pgPGk-xamd33R6Vkwff_XXknUx3AgOhjzJKyrNBkc5Y9x9E85ynPKI4KBMHbhP-ABc8w$><https://urldefense.com/v3/__https://groups.google.com/d/msgid/raxml/7446509B-9DD0-4608-B0A1-BE3AD58607BC*40ucsd.edu?utm_medium=email&utm_source=footer__;JQ!!Mih3wA!D30yUFXPbdOeWJlW4fAiDSrpL81FNE-OzUsFTmqXl3P1APkWAcSPmFjgE422o_ehkAQlVlEiffZ8XjsP0RJHJoCWyN8$  <https://urldefense.com/v3/__https://groups.google.com/d/msgid/raxml/7446509B-9DD0-4608-B0A1-BE3AD58607BC*40ucsd.edu?utm_medium=email&utm_source=footer__;JQ!!Mih3wA!EBO5DPxGttMbeYRi65pgPGk-xamd33R6Vkwff_XXknUx3AgOhjzJKyrNBkc5Y9x9E85ynPKI4KBMHbht5pl7Qw$>>.

>>>>>>
>>>>>> <slurm-affall>
>>>>>
>>>>
>>>
>
> --
> Alexandros (Alexis) Stamatakis
>
> Research Group Leader, Heidelberg Institute for Theoretical Studies
> Full Professor, Dept. of Informatics, Karlsruhe Institute of Technology
> Affiliated Scientist, Evolutionary Genetics and Paleogenomics (EGP) lab, Institute of Molecular Biology and Biotechnology, Foundation for Research and Technology Hellas
>

> https://urldefense.com/v3/__http://www.exelixis-lab.org__;!!Mih3wA!D30yUFXPbdOeWJlW4fAiDSrpL81FNE-OzUsFTmqXl3P1APkWAcSPmFjgE422o_ehkAQlVlEiffZ8XjsP0RJHczfajqg$