Can't get raxml to run on multiple cores

[Katya Kringle]

Greetings all, 

I did search extensively through other issues with specifying cores and time estimations, but I still can't find why running this seems to only use a single core. I have 24 cores available for use, and no job manager/scheduler. 

It's a large dataset, around 4 million characters across 100 taxa. I've tried understanding how to divvy the resources to get this to run at a reasonable rate, but only seem to be able to get it to run on 1 core. 

raxmlHPC -PTHREADS-AVX \
-f a \
-x $RANDOM \
-p $RANDOM \
-# autoMRE \
-m GTRGAMMA \
-T 10 \
-n mafft-edge-trimmed_coniidae_taxa.result \
-s mafft-edge-trimmed_all-coniidae_taxa_master_jan3_incomplete_clean_concat.phylip \
-o drilliidae \
-k \
-N 100 \
--asc-corr lewis

After looking through this problem and trying different variations of raxmlHPC -PTHREADS-AVX, raxml-ng, raxmlHPC-PTHREADS, I can't get this to run at a reasonable rate on more than one core! Please let me know if you see anything obvious here. 

Thank you! 

KK

[Alexey Kozlov]

Hello Katya,

the command line looks fine, and raxml-ng should use all 24 cores by default even if you do not
specify the number of threads. Please post your log files, then maybe we can help.

Best,
Alexey

> --
> You received this message because you are subscribed to the Google Groups "raxml" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to
> raxml+un...@googlegroups.com <mailto:raxml+un...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/raxml/fff21bd7-ca19-4ee4-b2f9-f294f222ca9an%40googlegroups.com
> <https://groups.google.com/d/msgid/raxml/fff21bd7-ca19-4ee4-b2f9-f294f222ca9an%40googlegroups.com?utm_medium=email&utm_source=footer>.

[Katya Kringle]

Thanks! As far as I can tell with this version of the command, the only log output I am getting is the file specified by -n , correct? 

(Other attempts produced different output, so let me know if I need to do something for the outfile)

It still looks to be only running on 1 core when I look at top reports. 

(phyluce-1.7.1) [lab@genax Trees]$ more RAxML_info.mafft-no-trim_all-coniidae_taxa_master_jan3_incomplete_fasta4_raxmlhpc_pthreadsavx3

Using BFGS method to optimize GTR rate parameters, to disable this specify "--no-bfgs" 

This is RAxML version 8.2.12 released by Alexandros Stamatakis on May 2018.

Alignment has 2988835 distinct alignment patterns

Proportion of gaps and completely undetermined characters in this alignment: 56.89%

RAxML rapid bootstrapping and subsequent ML search

Using 1 distinct models/data partitions with joint branch length optimization

Executing 100 rapid bootstrap inferences and thereafter a thorough ML search 

All free model parameters will be estimated by RAxML

GAMMA model of rate heterogeneity, ML estimate of alpha-parameter

GAMMA Model parameters will be estimated up to an accuracy of 0.1000000000 Log Likelihood units

Partition: 0

Alignment Patterns: 2988835

Name: No Name Provided

DataType: DNA

Substitution Matrix: GTR

RAxML was called as follows:

raxmlHPC -PTHREADS-AVX -f a -x 23084 -p 1840 -# autoMRE -m GTRGAMMA -n mafft-no-trim_all-coniidae_taxa_master_jan3_incomplete_fasta4_raxmlhpc_pthreadsavx3 -s mafft-no-trim_all-coniidae_taxa_master_jan3_incomplete_fasta4.phylip -o drilliidae -k -N 100 --asc-corr lewis 

Time for BS model parameter optimization 13719.333610

[Alexey Kozlov]

thanks but could you please also post raxml-ng log/output, since we do not actually support old
RAxML anymore...

> <https://groups.google.com/d/msgid/raxml/fff21bd7-ca19-4ee4-b2f9-f294f222ca9an%40googlegroups.com?utm_medium=email&utm_source=footer
> <https://groups.google.com/d/msgid/raxml/fff21bd7-ca19-4ee4-b2f9-f294f222ca9an%40googlegroups.com?utm_medium=email&utm_source=footer>>.

>
>
> --
> You received this message because you are subscribed to the Google Groups "raxml" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to
> raxml+un...@googlegroups.com <mailto:raxml+un...@googlegroups.com>.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/raxml/06ddc50c-de23-43cf-99c9-4b1e97a575d5n%40googlegroups.com
> <https://groups.google.com/d/msgid/raxml/06ddc50c-de23-43cf-99c9-4b1e97a575d5n%40googlegroups.com?utm_medium=email&utm_source=footer>.

[Katya Kringle]

Ok, that must've been the first issue, using the wrong raxml. Now I tried running with raxml-ng to get the logfile output. It still looks like it just calls one core, though with 16 CPU per core.

First I'll list the lscpu since maybe I am misunderstanding how that gets called!

lscpu

Architecture:          x86_64

CPU op-mode(s):        32-bit, 64-bit

Byte Order:            Little Endian

CPU(s):                32

On-line CPU(s) list:   0-31

Thread(s) per core:    2

Core(s) per socket:    8

Socket(s):             2

NUMA node(s):          2

Vendor ID:             GenuineIntel

CPU family:            6

Model:                 45

Model name:            Intel(R) Xeon(R) CPU E5-2650 0 @ 2.00GHz

Stepping:              7

CPU MHz:               1999.984

BogoMIPS:              3999.48

Virtualization:        VT-x

L1d cache:             32K

L1i cache:             32K

L2 cache:              256K

L3 cache:              20480K

NUMA node0 CPU(s):     0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30

NUMA node1 CPU(s):     1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31 

#########################

more T8.raxml.log

RAxML-NG v. 1.0.1 released on 19.09.2020 by The Exelixis Lab.

Developed by: Alexey M. Kozlov and Alexandros Stamatakis.

Contributors: Diego Darriba, Tomas Flouri, Benoit Morel, Sarah Lutteropp, Ben Bettisworth.

Latest version: https://github.com/amkozlov/raxml-ng

Questions/problems/suggestions? Please visit: https://groups.google.com/forum/#!forum/raxml

System: Intel(R) Xeon(R) CPU E5-2650 0 @ 2.00GHz, 16 cores, 236 GB RAM

RAxML-NG was called at 17-Jan-2022 17:05:05 as follows:

raxml-ng --msa mafft-no-trim_all-coniidae_taxa_master_jan3_incomplete_fasta4.phylip --model GTR+G --all --prefix T8 --bs-trees 100 --outgroup driliidae --threads auto --bs-metric tbe

Analysis options:

  run mode: ML tree search + bootstrapping (Transfer Bootstrap)

  start tree(s): random (10) + parsimony (10)

  bootstrap replicates: 100

  outgroup taxa: driliidae

  random seed: 1642464305

  tip-inner: OFF

  pattern compression: ON

  per-rate scalers: OFF

  site repeats: ON

  branch lengths: proportional (ML estimate, algorithm: NR-FAST)

  SIMD kernels: AVX

  parallelization: coarse-grained (auto), PTHREADS (auto)

[00:00:00] Reading alignment from file: mafft-no-trim_all-coniidae_taxa_master_jan3_incomplete_fasta4.phylip

[00:00:02] Loaded alignment with 106 taxa and 4830065 sites

Alignment comprises 1 partitions and 2988835 patterns

Partition 0: noname

Model: GTR+FO+G4m

Alignment sites / patterns: 4830065 / 2988835

Gaps: 56.89 %

Invariant sites: 36.74 %

NOTE: Binary MSA file created: T8.raxml.rba

Parallelization scheme autoconfig: 1 worker(s) x 16 thread(s)

Parallel reduction/worker buffer size: 1 KB  / 0 KB

[00:00:21] Generating 10 random starting tree(s) with 106 taxa

[00:00:21] Generating 10 parsimony starting tree(s) with 106 taxa

[00:04:15] Data distribution: max. partitions/sites/weight per thread: 1 / 186803 / 2988848

[00:04:33] Data distribution: max. searches per worker: 120

Starting ML tree search with 20 distinct starting trees

[Alexey Kozlov]

hm so according to the output, raxml-ng is running with 16 threads. what makes you think that it
only used a single core?

> <https://groups.google.com/d/msgid/raxml/06ddc50c-de23-43cf-99c9-4b1e97a575d5n%40googlegroups.com?utm_medium=email&utm_source=footer
> <https://groups.google.com/d/msgid/raxml/06ddc50c-de23-43cf-99c9-4b1e97a575d5n%40googlegroups.com?utm_medium=email&utm_source=footer>>.

>
>
> --
> You received this message because you are subscribed to the Google Groups "raxml" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to
> raxml+un...@googlegroups.com <mailto:raxml+un...@googlegroups.com>.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/raxml/c26f6fa4-fe64-47c3-91df-fcb7687743f3n%40googlegroups.com
> <https://groups.google.com/d/msgid/raxml/c26f6fa4-fe64-47c3-91df-fcb7687743f3n%40googlegroups.com?utm_medium=email&utm_source=footer>.

[Katya Kringle]

To be fair, I think the cores issue was really with the old raxml command, but here is a screenshot of top. I see that the % CPU is 16, but I admit was confused that this is how it runs. If this is the correct way, then I apologize for not understanding!

[Alexey Kozlov]

yeah this looks good.

so at least in case of raxml-ng, it uses all available cores as it is supposed to.

just keep in mind that your alignment is pretty large, and so tree inference with bootstrapping will
take quite some time on a single server, even with multiple cores.

On 19.01.22 18:29, Katya Kringle wrote:
> To be fair, I think the cores issue was really with the old raxml command, but here is a screenshot
> of top. I see that the % CPU is 16, but I admit was confused that this is how it runs. If this is
> the correct way, then I apologize for not understanding!

> > Latest version: https://github.com/amkozlov/raxml-ng <https://github.com/amkozlov/raxml-ng>

> <https://groups.google.com/d/msgid/raxml/c26f6fa4-fe64-47c3-91df-fcb7687743f3n%40googlegroups.com?utm_medium=email&utm_source=footer
> <https://groups.google.com/d/msgid/raxml/c26f6fa4-fe64-47c3-91df-fcb7687743f3n%40googlegroups.com?utm_medium=email&utm_source=footer>>.

>
>
> --
> You received this message because you are subscribed to the Google Groups "raxml" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to
> raxml+un...@googlegroups.com <mailto:raxml+un...@googlegroups.com>.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/raxml/e10c1cdc-7f13-42cb-8ffa-3ec8cec445dan%40googlegroups.com
> <https://groups.google.com/d/msgid/raxml/e10c1cdc-7f13-42cb-8ffa-3ec8cec445dan%40googlegroups.com?utm_medium=email&utm_source=footer>.