Stan 2.10 through 2.13 have broken samplers

149 views
Skip to first unread message

Bob Carpenter

unread,
Dec 20, 2016, 11:05:05 AM12/20/16
to mc_stan+...@discoursemail.com, Stan users mailing list, stan development mailing list
As far as we can tell, Stan 2.09 is the latest version of Stan
with a properly functioning sampler.

Versions from 2.10 on are producing biased samples
that slightly underestimate posterior variance. Thanks to
Matthew R. Becker for filing the issue:

https://github.com/stan-dev/stan/issues/2178

Stan 2.10 changed the NUTS algorithm from using slice sampling
along a Hamiltonian trajectory to a new algorithm that uses
multinomial sampling:

https://arxiv.org/abs/1601.00225

We are mortified that after all of our nagging to get people
to use samplers that worked and weren't biased, we released
a biased sampler. The 2.10 version had a major bug which was
easy to see and fix, but that apparently didn't solve the
bigger problem.

Michael and I are poring over the proofs and the code, but
it's unfortunate timing with the holidays here as everyone's
traveling. We'll announce a fix and make a new release as soon
as we can. Let's just say this is our only priority at the moment.

Until then, the only thing I can recommend is using straight
up static HMC (which is not broken in the Stan releases)
or using something other than Stan or rolling back to Stan 2.09.

I'm not even sure how to do the latter for versions other than CmdStan,
which is just a source download and doesn't require any
installation.

If all else fails, we'll roll back the sampler to the 2.09 version
in a couple days.

- Bob

Michael Betancourt

unread,
Dec 20, 2016, 11:16:05 AM12/20/16
to stan...@googlegroups.com
Let me temper the panic by saying that the bias is relatively small and affects only variances but not means, which is why is snuck through all our testing and application analyses. Ultimately posterior intervals are smaller than they should be, but not so much that the inferences are misleading and the shrinkage will be noticeable only if you have more than thousands of effective samples, which is much more that we typically recommend.

Static HMC seems to be giving valid results on the simple test problems that we are considering, but it still performs horribly on hard problems and so I would advise again using it seriously.
> --
> We've moved the developers list to: http://discourse.mc-stan.org/. Please post new messages there.
> ---
> You received this message because you are subscribed to the Google Groups "stan development mailing list" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to stan-dev+u...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

Jonah Gabry

unread,
Dec 22, 2016, 12:45:30 PM12/22/16
to Stan users mailing list, mc_stan+...@discoursemail.com, stan...@googlegroups.com
From rstan you can set algorithm="HMC" when calling stan(), but I would trust Michael on this. That is, even with the bug NUTS should be better than static HMC (except for some trivial cases).
Reply all
Reply to author
Forward
0 new messages