comparing fits vs. each other and simulations

[Bob Carpenter]

It'd be nice in R to be able to do something like this:

compare(c(fit1,fit2), sim, c("alpha","beta"))

to compare two (or more) fits, with a single (optional)
named list of "true" values, and an optional list of
a subset of parameters (like in print or traceplot in RStan).

I don't yet know enough about what I want to put into either
output or a visualization to make a feature request out of
this for RStan or PyStan (or shinyStan).

-Bob

[Andrew Gelman]

+1

> --
> You received this message because you are subscribed to the Google Groups "stan development mailing list" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to stan-dev+u...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Ben Goodrich]

Definitely compare() for posterior predictive distributions of multiple models against the observed outcome. But comparing the posterior distributions of parameters across models only makes goods sense when the models are nested, which we can't enforce but we could doc.

Ben
 

-Bob

[Bob Carpenter]

I'm not sure what you mean by "nested." I'm imagining
something where, for instance, I use two different priors or
different hyperprior distributions or maybe look at means
in a Poisson vs. negative binomial, or even predictions in
two wildly different models.

- Bob

[Ben Goodrich]

On Sunday, March 1, 2015 at 7:48:59 PM UTC-5, Bob Carpenter wrote:

I'm not sure what you mean by "nested."

One model is a special case or limiting case of the other
 

 I'm imagining
something where, for instance, I use two different priors or
different hyperprior distributions

Might make sense if the parameters that you are comparing are a priori independent of the priors that are different.
 

or maybe look at means
in a Poisson vs. negative binomial

That works since Poisson is a limiting case of the negative binomial
 

or even predictions in
two wildly different models.

For posterior predictive distributions, definitely. But the parameters wouldn't be comparable in wildly different models, even if they have the same "role" (e.g. alpha is an intercept, beta is a slope, etc.) because their interpretations would be different in wildly different models.

Ben