Can update.brmsfit collect more iterations based on the old model?

[Fanyi Zhang]

Hi, 

I am working with a big dataset that it would take a week to run out 2000 iters per chain. And after I went through some convergence diagnosis it indicated it would need more iterations to get to my specified error bounds. Thus, I want to collect more samples based on my initial model, but I don't want to start from scratch. Can "update.brmsfit" help to achieve this purpose?

Thanks so much in advance!

[Paul Buerkner]

I would love to add such a feature in brms, but this requires (R)Stan to implement this first, which -- as far as I know -- they haven't done yet.

So, I am afraid to say that I currently see no reasonable way than starting from scratch (happy to be correct on this point).

[Michael Adkins]

Unfortunately, Paul is right on this one. I was wondering about the same thing a couple of years ago. Bob Carpenter commented on this a while back on the Stan user group list. He argued that the practical reason they haven't implemented it is the issue surrounding saving all the adaptation and RNG state, while keeping everything reproducible etc. Personally, I have my doubts it will be implemented at all given the focus of software development and the challenge of engineering a solution, but would definitely be happy to be proven wrong.

Think you might have to wait the full week. Hopefully you have a cluster available to do the work. Best of luck.

[Fanyi Zhang]

Thanks, and the explanation is very helpful. However, I still have a few follow-up questions here:

1. When you questioned if it would be implemented at all, what do you mean by "the focus of software development and the challenge of engineering a solution"? Do you indicate that if we can do massively parallel MCMC over thousands even millions of machines, then we don't care very much about updating any more?

2. You mentioned "a cluster available to do the work", do you indicate to implement MCMC methods in distributed and parallel settings? I know for Stan you can only parallel multiple chains but not on a single chain, and it's more of a statistical problem to be solved rather than an engineering one. Do you know if there's any new progress on this field that can be used off-the-shelf?

Thanks