'cores' argument

[Soren]

Dear Carl, 

I've been bench marking the time it takes for me to run a genome scan (scan1) by increase the number of cores that I want to use. I was quite surprised to find that the quickest was running on 1 core. In short, I ran scan1 with one phenotype and then increasing the number of cores. I got the following times:

1 core 1 phenotype: 29 sec

2 cores 1 phenotype: 46 sec

10 cores 1 phenotype: 2 min 51 sec.

Is this something you have seen before and are aware of? 

I'm doing the analysis on a 32-core processor with 128 GB RAM. 

Thanks in advance!
Soren

[Karl Broman]

This has not been my experience, but how the multiple cores get used depend on a number of things...mostly batched by chromosome.

What operating system are you using?

karl

On Feb 24, 2020, at 5:51 AM, Soren <smad...@gmail.com> wrote:



--
You received this message because you are subscribed to the Google Groups "R/qtl2 discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rqtl2-disc+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/rqtl2-disc/e4f8a9b9-5179-47c9-8224-3804f6c6e3f7%40googlegroups.com.

[Soren]

Hi again Carl, thanks for your quick reply. It's running on windows 10.

Soren

[Selcan Aydın]

Hi Karl,

I am seeing a similar issue when running on HPC cluster with newest qtl2 release using R version 4.0. It took me a while to figure it out but looks like once I increase the # of cores, the time it takes to run the scans get very large and the R session gets stuck. I don't understand the source of this issue but I can't replicate it locally on my Mac or on older R versions on the cluster. Seems to be R 4.0 specific issue. I use the function calc_kinship() in the same script and the # of cores doesn't seem to impact that function. 

Thanks!

Selcan

[Senthil Thillainadesan]

Hi Karl,

I'm not sure if what I am experiencing is exactly the same issue mentioned by Selcan but I have noted that scan1 and scan1perm code see to run ~5-10 time slower on HPC compared.

This is even when the cores are set to core=1. (With higher cores I run into problems with  high memory utilisation. So it is more efficient for me to set cores=1 and write my own code for clustering)

A scan which takes 1 minute on my personal computer takes about 10 minutes on on the cluster.

I've asked the people who run the cluster service to have a look at the idea and so far they haven't been able to figure how to address this.

Not sure that it is something that you will be able to help with as it may be something specific about our cluster set up. But I am curious to know if others have had any issues running there code on a cluster and whether they have any tips!

Cheers

Senthil

[Karl Broman]

Could you say more about the operating system on your cluster?  

There are two methods for parallel computing in R. R/qtl2 mostly uses mclapply(), but it can also use parLapply() if you first run makeCluster() to create a cluster object and then use that with the cores argument.

       https://github.com/rqtl/qtl2/blob/master/R/cluster_util.R#L39-L51  

It seems like the mclapply() method is not sharing memory they way it should, but rather duplicating objects.  

karl

[Selcan Aydın]

Hi Karl,

I am running things on the new JAX cluster, if that helps. Initially I thought this was specific to R 4.0 but I am seeing similar issues with R 3.6.2. This issue seems to lead to a larger memory utilization problem and causes errors on the cluster as well. IT is looking into it. Will keep you updated if I get an answer. I should note that I have not encountered this until recently, after switching to the new HPC cluster at JAX. 

Best,

Selcan

--
You received this message because you are subscribed to a topic in the Google Groups "R/qtl2 discussion" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rqtl2-disc/wOitJ2W2r00/unsubscribe.
To unsubscribe from this group and all its topics, send an email to rqtl2-disc+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/rqtl2-disc/53594170-e17d-4d00-b753-b1c578ab7a3eo%40googlegroups.com.

--

-selcan

[Senthil Thillainadesan]

Hi Karl,

Thanks for the quick response.

The cluster runs CentOS’ v6.9.

I should clarify that there are two issues I have noted.

1)The memory issues when using multiple cores with the "core= ..." argument. This is not specific to the the HPC cluster. Seems to occur on windows PC when running the analysis with core= 2+. I have a work around for this which basically involves using clusterapply to run scan1 or scan1perm with 3-4 chromosomes at a time across multiple phenotypes. This way I can use all cores on our PC without exceeding the available memory. 

The second issue which is really puzzling me is the fact that the both the scan1 and scan1perm codes seem to be run 5-10x slower on our HPC cluster compared to a windows PC with core=1. This does not seem to be a problem with clustering but some other clash perhaps with the code and maybe the cluster set up. 

Thanks

Senthil