diag vs. nondiag models in bug examples.

[Yuanjun Gao]

I compared the performance of three sampling schemes in fitting bug models,
1. default stan(diagonal matrix)
2. nondiagonal mass matrix shrinking to the diagonal part
3. nondiagonal mass matrix shrinking to both the diagonal and the identity matrix.

I choose to compare in terms of median time per n_eff since it tends to be stable comparatively. The result shows that nondiagonal sampling scheme generally performs better when dimension of the parameters are < 100. method 3(shrinking to identity) is a little better than method 2. However, for higher-dimensional paramters, the nondiagonal methods lose its power.

However, the result is noisy in that one stuck chain may lead to very poor mixing and n_eff. It is possible that randomness plays an important part in assessing the performance.

The result is attached.

Yuanjun

[Andrew Gelman]

Y
Interesting. We should discuss the "stuck chains" issue, as that seems worth resolving.
A

> --
> You received this message because you are subscribed to the Google Groups "stan development mailing list" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to stan-dev+u...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>
> <rlt_all.csv>

[Daniel Lee]

The "stuck chains" issue is a deep one and it is worth resolving. Michael and I are working on some diagnostics, which will (hopefully) tell us if the geometry is causing these issues. If it is, hopefully the RMHMC stuff will sort out the issue.

If we were looking to do something ad-hoc, maybe we can use geometry information to pick a starting point that won't get stuck.

[Andrew Gelman]

For the Jags examples we can use the Jags starting points.

[Daniel Lee]

That's one of the things I'm working on now -- making sure all of the examples use the provided initial values. Not all the models are taking advantage of that now.

[Nigel/essence]

I have had some experiences of chains getting stuck - occasionaly the epsilon value became very small. The way I fixed this was to restart the epsilon calculation, using a new estimate of the initial epsilon, under certain conditions during burnin. For example, if alpha became very small and the number of depths hit the maximum.

The initial epsilon estimate can be far from good if the curvature at the starting point is very different to that at the minimum.

Of course, you can get stuck in a mode, but that is another story.

Saddle points can cause problems with sticking also.

At the end of the day, my approach is to use a variety of samplers (NUTS, DREAM, RWM) with different characteristics, so they can unstick each other.

ps I did a lot of work 9 months ago on non diagonal mass matrices with multivariate distributions, they did indeed give great performance in high dimensions with high correlations, I also used BFGS during burinin which also gave great results, but of course this is only applicable with fixed hessian, and you need other methods to handle the general case, as y'all know.