Parallel chains in RStan

[Jökull Snæbjarnarson]

Hey guys, 

I'm running different sets of data on the same model in different chains in parallel, 

this can for example be very useful if you are doing some sort of cross validation on your model.

One way of doing this is for example : 

s1 <- stan_model(model_code = hcode) # compile the model

library(parallel)

sflist <- 

  mclapply(1:36, mc.cores = 36, 

           function(i) sampling(object = s1, data = foo_data[[i]], chains = 1,

                                chain_id = i, iter = 1000, warmup = 350,seed = seed)) 

But if you do it with the mclapply from the parallel package, you can't see the progress bars in Rstudio.

So in the newest version of Stan it is actually possible to run chains in parallel(without the parallel package), but I can't figure out if I can input a list of different data sets to run on in parallel(foo_data[[i]]), without the parallel package.

So can I do this with only the sampling function? And if not, can you show me how you did get the output to be printed in the console of Rstudio so I can implement this? 

Best regards,

Jökull Snæbjarnarson

[Ben Goodrich]

On Thursday, September 22, 2016 at 10:37:08 AM UTC-4, Jökull Snæbjarnarson wrote:

So can I do this with only the sampling function? And if not, can you show me how you did get the output to be printed in the console of Rstudio so I can implement this? 

I believe the only way that you can easily see the progress in a situation like this is to run R from a shell rather than in RStudio. Even still, the progress may look garbled because 36 processes are writing to the same stream.

Ben

[Robert Grant]

I suspect you want to stay in R, but you could do this with CmdStan by diverting the output of each chain to a different text file. If you look up the CmdStan manual in the section on running a Stan executable you'll see the relevant parallel commands for your operating system. Of course, writing to disk will slow things down. You could do this all with system() inside R.

[Jökull Snæbjarnarson]

Hmm ok but somehow we can run in parallel with the sampling function, and print out to the screen, and even the inits are specified as a list of lists per chain. It would be nice to be able to do the same with the data. I'll take a look at the source code to see how you guys solved this.

Jökull

[Bob Carpenter]

We don't want to use different data per chain because
it'll change the posterior and all of our diagnostics
are aimed at trying to reject cases where multiple
chains don't explore the same posterior volume.

- Bob

> --
> You received this message because you are subscribed to the Google Groups "Stan users mailing list" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to stan-users+...@googlegroups.com.
> To post to this group, send email to stan-...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.