advice on *BEAST performance using BEAGLE and GPU's+CPU's

[sara]

Hi all,

after reading a few threads / documentation about BEAGLE I am still having a few doubts on how to make best use of it and of available CPU+GPU nodes.

I bet people are facing similar doubts nowadays, so... hope someone can advise me here:

I found these page/threads especially useful:

http://beast2.org/performance-suggestions/

https://groups.google.com/forum/#!searchin/beast-users/beagle$20usage/beast-users/P7OmgU_1qCo/5N2ki3n7a-AJ

https://groups.google.com/forum/#!searchin/beast-users/beagle_order/beast-users/RNbM-yRn5QQ/5vL4Rj0vB9IJ

but still have a few doubts:

Say our dataset is 30-50 genes (=partitions), each with around 100-150 site patterns (so, around 4500 in total)

and we have a cluster with several nodes (8-12 cores each) + a few nodes with GPU's (1 or 3)

GPU's:

1 : Tesla M2090

    Global memory (MB): 5375

    Clock speed (Ghz): 1.30

    Number of cores: 512

I am trying to figure out how best optimize runtimes (or where to start testing combinations) - need to run several long (500M gens) replicates, from what I see in my initial runs.

As I see it options are:

1) use CPU's (as more threads/cores the better) with _SSE option

2) make use of additional GPU's by assigning one (or a few?) partitions to it using -beagle_order

>>> question here is how best to do it. In the second thread I posted I see

If you had 3 partitions and you wanted to run the first two on the CPU (they would run on different cores) and the third on the GPU use:

-beagle_order 0,0,2

This command works by going through each partition in order and allocating it to the next device in the beagle_order list (wrapping around to the beginning if there are more partitions than numbers in the list). Generally you shouldn't put more than one partition per GPU device (it will work but will have bad performance - this is a limitation of BEAGLE at present). 

but the first one suggest we can pop a few more partitions in the GPU's. That should be based on what? Saturating the GPU cores with "site patterns" (say, assigning them partitions to to fill more than 512 site patterns. Is that correct? And how much more? Is there a thumb-up rule??

In the case of having a machine with more than 1 GPU, is there advantage of spreading "partitions" over several GPU's or not really? (and better use different GPU's and X threads of the CPU for different replicates?)

Finally, how does all this compares/combines with the "useThreads flag of CompoundDistribution" mentioned in the performance suggestions page?

http://beast2.org/performance-suggestions/

many many thanks in advance,

sara

[Miller, Mark]

Hi sara,

I  a little confused. Are you using beast or beast2 or both ?

Mark

--
You received this message because you are subscribed to the Google Groups "beast-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to beast-users...@googlegroups.com.
To post to this group, send email to beast...@googlegroups.com.
Visit this group at https://groups.google.com/group/beast-users.
For more options, visit https://groups.google.com/d/optout.

[Sara Rocha]

Beast2 (and would it make a difference?)

You received this message because you are subscribed to a topic in the Google Groups "beast-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/beast-users/KdFm5i-iW7Y/unsubscribe.
To unsubscribe from this group and all its topics, send an email to beast-users...@googlegroups.com.

[Miller, Mark]

Yes. The performance characteristics of the two codes in our hands are very different. We have had trouble using more than three cores effectively with beast2/beagle. I will leave it for others to provide their thoughts...

Mark

[Sara Rocha]

right... thanks... I'll wait for more advice also then.

[Tomas Hrbek]

Folks,

I observed the same thing. I did a simple test a while back using 1,
2, 4, 8, 16 CPU cores on two different datasets, and using more than 4
cores there was no improvement in performance. So I just use 4 cores
max for all BEAST2 analyses.

Tomas

On 02/15/2016 10:01 AM, Miller, Mark wrote:
> Yes. The performance characteristics of the two codes in our hands are
> very different. We have had trouble using more than three cores
> effectively with beast2/beagle. I will leave it for others to provide
> their thoughts...
>
> Mark
>
> On Feb 15, 2016, at 3:44 AM, "Sara Rocha" <spr...@gmail.com

> <mailto:spr...@gmail.com>> wrote:
>
>> Beast2 (and would it make a difference?)
>>
>> El 15/02/2016 04:14, "Miller, Mark" <mmi...@sdsc.edu

>> <mailto:mmi...@sdsc.edu>> escribió:

>>
>> Hi sara,
>> I a little confused. Are you using beast or beast2 or both ?
>>
>> Mark
>>
>> On Feb 14, 2016, at 2:52 PM, "sara" <spr...@gmail.com

>>> /
>>> /
>>> /If you had 3 partitions and you wanted to run the first two on

>>> the CPU (they would run on different cores) and the third on the

>>> GPU use:/
>>> /-beagle_order 0,0,2/
>>> /This command works by going through each partition in order and

>>> allocating it to the next device in the beagle_order list
>>> (wrapping around to the beginning if there are more partitions
>>> than numbers in the list). Generally you shouldn't put more than
>>> one partition per GPU device (it will work but will have bad

>>> performance - this is a limitation of BEAGLE at present). /
>>> /
>>> /

>>> but the first one suggest we can pop a few more partitions in the
>>> GPU's. That should be based on what? Saturating the GPU cores
>>> with "site patterns" (say, assigning them partitions to to fill
>>> more than 512 site patterns. Is that correct? And how much more?
>>> Is there a thumb-up rule??
>>> In the case of having a machine with more than 1 GPU, is there
>>> advantage of spreading "partitions" over several GPU's or not
>>> really? (and better use different GPU's and X threads of the CPU
>>> for different replicates?)
>>>
>>> Finally, how does all this compares/combines with the "useThreads
>>> flag of CompoundDistribution" mentioned in the performance
>>> suggestions page?
>>> http://beast2.org/performance-suggestions/
>>>
>>> many many thanks in advance,
>>> sara
>>>
>>> --
>>> You received this message because you are subscribed to the
>>> Google Groups "beast-users" group.
>>> To unsubscribe from this group and stop receiving emails from it,
>>> send an email to beast-users...@googlegroups.com

>>> <mailto:beast-users...@googlegroups.com>.

>>> To post to this group, send email to beast...@googlegroups.com

>>> <mailto:beast...@googlegroups.com>.

>>> Visit this group at https://groups.google.com/group/beast-users.
>>> For more options, visit https://groups.google.com/d/optout.
>>
>> --
>> You received this message because you are subscribed to a topic in
>> the Google Groups "beast-users" group.
>> To unsubscribe from this topic, visit
>> https://groups.google.com/d/topic/beast-users/KdFm5i-iW7Y/unsubscribe.
>> To unsubscribe from this group and all its topics, send an email
>> to beast-users...@googlegroups.com

>> <mailto:beast-users...@googlegroups.com>.

>> To post to this group, send email to beast...@googlegroups.com

>> <mailto:beast...@googlegroups.com>.

>> Visit this group at https://groups.google.com/group/beast-users.
>> For more options, visit https://groups.google.com/d/optout.
>>
>> --
>> You received this message because you are subscribed to the Google
>> Groups "beast-users" group.
>> To unsubscribe from this group and stop receiving emails from it, send
>> an email to beast-users...@googlegroups.com

>> <mailto:beast-users...@googlegroups.com>.

>> To post to this group, send email to beast...@googlegroups.com

>> <mailto:beast...@googlegroups.com>.

>> Visit this group at https://groups.google.com/group/beast-users.
>> For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to the Google
> Groups "beast-users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to beast-users...@googlegroups.com

> <mailto:beast-users...@googlegroups.com>.

> To post to this group, send email to beast...@googlegroups.com

> <mailto:beast...@googlegroups.com>.