Using foreach with RRO

[Robert Harlow]

Hi,

   I was playing around with RRO and I am very impressed!  I was, however, wondering how to resolve the following issue: It seems that the implicit multithreading coming from the MKL library can mess up explicit parallelism in foreach.  Consider the following code example:

library(foreach)

library(doMC)

registerDoMC()

n <- 1000

x <- matrix(rnorm(n^2), n, n)

##runs fine

tst <- foreach(i = 1:10) %do% {

    system.time(for(i in 1:10) x%*%x)

}

##hangs

tst <- foreach(i = 1:10) %dopar% {

    system.time(for(i in 1:10) x%*%x)

}

Should these two forms of parallelism not be used together?  I might just be naive...

Thanks,

Bob

sessionInfo()

R version 3.1.1 (2014-07-10)

Platform: x86_64-unknown-linux-gnu (64-bit)

locale:

 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C               LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8     LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   

 [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                  LC_ADDRESS=C               LC_TELEPHONE=C             LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

attached base packages:

[1] parallel  stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:

[1] doMC_1.3.0      iterators_1.0.6 foreach_1.4.1  

loaded via a namespace (and not attached):

[1] codetools_0.2-8 compiler_3.0.1 

[David Smith]

Hi Bob,

MKL and doMC shouldn't interfere with each other, but it might not be a good *idea* to use them both together. In your example, both MKL and foreach will be contending for CPU resources on all available cores, which might not be optimally efficient. It depends on how CPU-intensive the MKL operation and the foreach operation are. It's worth trying setMKLthreads(1) to restrict MKL to a single thread when using foreach to see if that improves performance.

Using foreach and MKL together should never cause the R engine to hang, though. doMC can have some unexpected interactions when you use R with a GUI, though. Are you using R from the command line, or from RStudio or Eclipse or anything like that? Can you try running your code directly from the command line (without a GUI or IDE) and see if the hang still occurs?

Thanks,
# David

-- 

David M Smith <da...@revolutionanalytics.com>
Chief Community Officer, Revolution Analytics  http://blog.revolutionanalytics.com
Tel: +1 (650) 646-9523 (Chicago IL, USA)

Twitter: @revodavid

--
You received this message because you are subscribed to the Google Groups "Revolution R Open" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rropen+un...@googlegroups.com.
To post to this group, send email to rro...@googlegroups.com.
Visit this group at http://groups.google.com/group/rropen.
To view this discussion on the web visit https://groups.google.com/d/msgid/rropen/783143db-ee35-4bf9-bf7b-0dcf0fecc5c8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Revolution R Plus

Subscribe to Technical Support & Indemnification for R

[Robert Harlow]

David,

   Thanks for the quick reply.  I am running RRO from the linux command line.  The code defintely hangs for large n, but if I make n smaller, say 10, the parallel version works fine.  Another way to make sure it doesn't hang is to use the setMKLthreads() function or play around with options()$core.  I am still not sure what the happy medium is here.

Thanks,

Bob