Massive slowdown from setting threads=2 (not 1)?

[David Manheim]

I'm running Stan via R, with a single core. (This is just for testing for now.)

I have a model that ran in about an hour with the R command;

stan(file='fit_SIS_Generic.stan', data=data_trial,iter=100, chains=1, seed=194838, refresh=20)

(I am aware that this is too few iterations, etc. Again, this was for testing.)

I changed that slightly, to 

stan(file='fit_SIS_Generic.stan', data=data_trial, iter=100, chains=2, seed=194838, refresh=20),

and the first chain took about 5 times as long, per the updates delivered (the second chain was still running and I killed it after a dozen hours.)

There are other problems with the model, but I was surprised to see that running chains=2, where they run in sequence, not in parallel, should affect the timing - especially given the indetical random seed, which I would think gives the same process and results, though this may not be true when running multiple chains. 

I looked at the manual, and the only thing I could find was that there were differences that could occur in how the model runs due to thread safety for multiple chains.

Is there a different / better explanation?

[Ben Goodrich]

On Thursday, September 3, 2015 at 5:22:40 PM UTC-4, David Manheim wrote:

There are other problems with the model, but I was surprised to see that running chains=2, where they run in sequence, not in parallel, should affect the timing - especially given the indetical random seed, which I would think gives the same process and results, though this may not be true when running multiple chains. 

It should be identical even with multiple chains in parallel. The only guess I have is that doing two chains in parallel caused the computer to swap.

Ben

[Jonah]

I don't think they were run in parallel, just one after another (assuming mc.cores wasn't previously set). The intended meaning of the initial post I think is to ask why this happened when not running in parallel, unless I'm misunderstanding.

Unfortunately if I'm right then I have no clue why chain 1 would be any different. Were you able to compare the draws?

[Jonah]

And regarding the use of only 100 iterations, the dirty little secret is that sometimes - if the markov chain converges quickly and the posterior is easy to draw from without much autocorrelation and you don't need a larger effective sample size - you can be fine with 100 iterations for many purposes.

[Andrew Gelman]

Yes, exactly. 100 is the default, then if convergence is not rapid, people can run more. Much better than running 2000 as a start.

> On Sep 3, 2015, at 6:54 PM, Jonah <jga...@gmail.com> wrote:
>
> And regarding the use of only 100 iterations, the dirty little secret is that sometimes - if the markov chain converges quickly and the posterior is easy to draw from without much autocorrelation and you don't need a larger effective sample size - you can be fine with 100 iterations for many purposes.
>

> --
> You received this message because you are subscribed to the Google Groups "Stan users mailing list" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to stan-users+...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[David Manheim]

The default for the function "stan" in RStan is 2000, which seems strange, given this advice...

Also, it was running in sequence; as posited above, mc.cores was not set before running this.

[Andrew Gelman]

Yes, I agree.  I’m proposing to change the default to 100 with 4 chains.

[Daniel Lee]

Back to the original question: David, is your computer swapping? Do you get the same slowdown using CmdStan?

I suspect that the posterior for the Stan program you've written is ill-behaved and that might be causing your problems:

- how reasonable are your results? do they differ run to run? are there divergent iterations? are you hitting the tree depth limit?

- can you try adding stronger priors to see if that will help?

- I would suggest trying to run more warmup iterations. Perhaps the adaptation parameters are not being tuned anywhere reasonably.

- is the model non-identifiable?

Daniel

[David Manheim]

There are tons of problems with the model, which I'm trying to address. The version that is displaying this behavior is probably having multiple issues, including identifiability, and divergence - but within any one thread, given the same seed, I would think it shouldn't take different amounts of time depending on whether there was another thread scheduled to run afterwards. That was the reason I thought this was worth bringing up as a potential issue.

[Michael Betancourt]

If you’re running with multiple chains then the seed gets incremented for

each chain to ensure that they don’t all give the same results (i.e. the

RNG will be independent for each chain).  Consequently when running

multiple chains you should not expect the same exact results, and

one chain can easily get stuck in a pathological region if your model

is ill-posed.

[Ben Goodrich]

It is perhaps possible that the threads are executed out of order, i.e. the second one first. If there are multiple chains and a specified seeds, Stan winds the PRNG forward by some fraction of its period depending on the chain_id. This ensures that you are not using the same random numbers across multiple chains but it would also mean you can get very different behavior across chains.

Ben

[Michael Betancourt]

Just like in real life, if your model is decent then when you multithread you

similar but not exactly equal results.  :-p