Automated Diagnostics

[Michael Betancourt]

I whipped up an example of automated diagnostics that you can try out
at https://github.com/stan-dev/cmdstan/tree/feature/automated_diagnostics.
Obviously, having some misbehaving model output files will be handy!
Or just take a look at the logic in the source itself,
https://github.com/stan-dev/cmdstan/blob/feature/automated_diagnostics/src/cmdstan/print_diagnostics.cpp.

Any opinions on the text? It’s meant to be informative but make no hard
decision about the validity of the output (as theoretically we cannot do
that). The user will have to have a basic understanding of a Markov
chain — eventually we can reference the appropriate review section
in the manual.

Any diagnostics I’m missing?

For those that use CmdStan, is this best kept as a separate executable
(keeping bin/print agnostic to validity) or should we roll it together into
one single analysis executable?

[Krzysztof Sakrejda]

On Wednesday, April 1, 2015 at 10:46:51 AM UTC-4, Michael Betancourt wrote:
[snip]

> For those that use CmdStan, is this best kept as a separate executable
> (keeping bin/print agnostic to validity) or should we roll it together into
> one single analysis executable?

Pure user perspective: I've had bin/print crash with too many parameters in the model so I prefer modular tools if the maintenance overhead isn't too big. I'd rather not get held up just because one tool is having an issue.

Krzysztof

[Michael Betancourt]

Those crashes are due to how the expectations are calculated
and will affect these new diagnostics the same as in bin/print.

[Richard McElreath]

On Wednesday, April 1, 2015 at 7:46:51 AM UTC-7, Michael Betancourt wrote:
>
> Any opinions on the text? It’s meant to be informative but make no hard
> decision about the validity of the output (as theoretically we cannot do
> that). The user will have to have a basic understanding of a Markov
> chain — eventually we can reference the appropriate review section
> in the manual.

Re divergent iterations message: Maybe also note that this is often better fixed by un-centering (aka Matt tricking) the model. I haven't explored tweaking the sampler much, but I doubt the efficiency gain could approach what comes from a non-centered parameterization. Even without any Cholesky stuff, just moving the scale parameters to the linear model kills most of the divergent iters I've experienced, with the natural huge increase in n_eff/(iter-warmup).

[Michael Betancourt]

The problem is that (non-)centered parameterizations apply to only
hierarchical models and hence the advice doesn’t quite make sense
as a general solution.

> --
> You received this message because you are subscribed to the Google Groups "stan development mailing list" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to stan-dev+u...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Bob Carpenter]

If you have an example where bin/print crashes, please file a bug
report with a reproducible example. Of course, there's only so
much size a given program is going to be able to handle. bin/print
has to do an FFT on each column of the output to compute n_eff.

- Bob

[Krzysztof Sakrejda]

On Wednesday, April 1, 2015 at 8:18:07 PM UTC-4, Bob Carpenter wrote:
> If you have an example where bin/print crashes, please file a bug
> report with a reproducible example. Of course, there's only so
> much size a given program is going to be able to handle. bin/print
> has to do an FFT on each column of the output to compute n_eff.

I'll be back to working on a model that triggers this problem in the next week or two so I should be able to figure out what's up then.

Krzysztof

>
> - Bob