Use of gpu nodes for BEAST/BEAST2 on CIPRES

[Mark Miller]

Dear Users,

 

CIPRES recently got access to some very fast gpu nodes on the Comet machine.

The upside is that is possible now to run BEAST/BEAST2 analyses that were impossible before.

The gpu speedup for some datasets is 35-fold relative to cpu cores.

However, the gpus nodes are in short supply, and quite costly. A run requiring 4 gpus will charge 84 cpu hours.

With the speedup these runs will cost less than an equivalent run on cpu. But it also means that a 120 hour run on

4 gpus will be charged 10800 core hours, or 1/3 to 1/5 of your annual allocation depending on whether you are in the US or outside the US.

 

This message is meant as a caution, one can use a lot of compute time in hurry with these new gpu nodes.

Their use is determined by how many partitions, patterns, and whether or not the run is amino acids.

When you submit, a warning message appears to advise you how many gpus your run will take, and of the potential cost of the job.

Their use will always give a faster result (save your allocation) relative to a run on cpus. So I encourage their use.

You can make runs that were previously impossible in the allowed time period. And runs that were already  possible will complete much faster.

Please just be aware that it is possible to burn through your entire year's allocation with 3-5 long runs.

 

We are adjusting our submission tools to prevent people from over consuming, these will be in place shortly.

Please contact me if you have questions.

 

Best,

Mark

[Maria Gavrutenko]

Hi Mark,

I am trying to run a BEAST job hoping to take advantage of the faster run time. I set maximum running time to 120 hours (max for gpu jobs). However, when I save my parameters, i get a warning message: "The job will run on 4 processors as configured. If it runs for the entire configured time, it will consume 2 X 120 cpu hours". Am I doing something wrong? Is there a different process to submit tasks to run on gpu nodes?

[Mark Miller]

Hi Maria,

It sounds like there is a bug in the code that creates that warning, since it is clearly incorrect. I will have a look. In the mean time you can just click through the warning and submit the job.

Let me know if you have further problems.

[Mark Miller]

I want to add that not all runs use GPUs. Only those where there is a real benefit (speedup). I found the bug, and will fix it. But your job will not use gpus, and could be run for 168 hours.

Mark

[Maria Gavrutenko]

Thank you for the fast reply. What determines if there is a real benefit? I ran this analysis before (a tree with 1170 terminals, 150 mil generations) and it was nowhere near finished after 168 hours. Right now I am forced to use a ML tree for my downstream analyses because it seems that there is no way to get the BEAST analysis completed in a reasonable time frame, but I was really hoping that this was feasible with gpus. 

[Mark Miller]

We do benchmarking on the different data set types. Amino acid v dna, and how many partitions and characters. The GPUs are costly, and dont help so much after a couple of partitions for DNA data.

There just isnt much speedup for your data after two threads.

[Magnus Nygård Osnes]

Hi, Mark. 

Are these GPU nodes still available?  I did not find any way to specify for example "beagle_GPU" in the parameter settings on the CIPRES server. 
I have been running some BEAST2 runs that do not finish within the time limit of 168 hours (DTA with more than > 3000 sites,  requiring at least 5*10^5 iterations to converge). I did some speed tests on my own computer and found that the runs are much faster when running on a GPU instead of the CPU. 

Sincerely,
Magnus Nygård Osnes

[Mark Miller]

I forgot to reply publicly on this. If you find your data runs faster on a GPU than 3 CPUs, we can accomodate that. In fact we want to know that. Magnus your case seems to be anomalous. Most data sets of this size run the same or slower than on GPUs (larger data sets get a huge benefit however). We tune CIPRES to configure for the average data set. A GPU costs 7x more than 3 CPUs for our project, So it is a big lose to run a GPU, pay more and get the same or lesser speed.

So we are still trying to understand which jobs of this size run faster on GPU.

Mark