multi-core processing with Stan in a package

[jonatho...@gmail.com]

Hello,

I've been working on making an R package for my lab that fits a few precompiled Stan models.  https://github.com/ColtoCaro/compMS

I'm unclear on the best way to make parallel processing available to users of the package.  Right now I have a function which manipulates data, creates variables for Stan and calls sampling().  When working directly with Stan code, prior to calling the sampling function, I would usually use 

rstan_options(auto_write = TRUE)

options(mc.cores = parallel::detectCores())

This doesn't seem to work inside of a package.  

Are there any guidelines for using parallel processing in conjunction with pre-compiled Stan models?  

Thanks,

Jonathon

 

[Ben Goodrich]

On Tuesday, April 18, 2017 at 3:16:25 PM UTC-4, jonatho...@gmail.com wrote:

When working directly with Stan code, prior to calling the sampling function, I would usually use 

rstan_options(auto_write = TRUE)

That line is just unnecessary if your R package already has compiled models
 

options(mc.cores = parallel::detectCores())

That line should work, but maybe the user has to call it outside.

If you know the hardware on which the models are going to be called, you can just do

sampling(*, cores = 4)

or something.

Ben

[jonatho...@gmail.com]

Thanks Ben,

Something must be wrong with my package structure.  I tried both of the ways that you suggested using multiple cores and both give me the same error.

 Error in checkForRemoteErrors(val) : 

  4 nodes produced errors; first error: object 'model_allModels' not found 

Sampling works fine if I use a single core.  Any suggestions?

Jonathon

[jonatho...@gmail.com]

Some more information.

I'm calling the Stan model with this code

sMod <- compMS:::stanmodels$allModels

if(multiCore){
	model <- rstan::sampling(sMod, cores = parallel::detectCores())
	}else{
	model <- rstan::sampling(sMod)
	}

When multiCore == FALSE, the model begins sampling.  When TRUE I'm told that "model_allModels " can't be found.  Is there some other way to specify the model location?  Should this be done differently if multiple cores are being used?

Thanks,

Jonathon

[Henrik Singmann]

My guess is that you are using devtools::load_all() when you encounter the error, because exactly that happened to me as well.

Try to actually install your package (e.g., either via RStudio's 'Build and Reload' or create tar.gz and then install.packages) and then try again. The problem is that model_allModels is only available at the nodes if the package is actually installed (in most cases).

Hope that helps,
Henrik

[jonatho...@gmail.com]

Yes!  Thank you Henrik, that was driving me crazy.  

Build and Reload solves the problem.

Jonathon