How to determine goodness of fit?

[He-chien Tsai]

As far as I know, AIC and BIC is for comparing several models, not for whether accepting a model. I can not make sure whether it's a good fit or not.
Did hypothesis testing like p-value or chi-squared test exist when using bayesian approach?

[Bob Carpenter]

There are lots of ways to evaluate the predictions of
models and to compare them, but we can't provide
a tutorial via e-mail. There are lots of resources
out there on Bayes. See, e.g., Gelman et al.'s
Bayesian Data Analysis book or the BUGS book. We
particularly like cross-validation-based measures, of
which WAIC is an approximation. And informal measures
like posterior predictive checking.

Yes, you can compute things like chi-squared tests and
also look at posterior intervals if you want.

- Bob

> On Jan 6, 2015, at 10:46 AM, He-chien Tsai <depo...@gmail.com> wrote:
>
> As far as I know, AIC and BIC is for comparing several models, not for whether accepting a model. I can not make sure whether it's a good fit or not.
> Did hypothesis testing like p-value or chi-squared test exist when using bayesian approach?
>

> --
> You received this message because you are subscribed to the Google Groups "Stan users mailing list" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to stan-users+...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[He-chien Tsai]

Thanks for reply. I think WAIC is similiar to AIC and BIC. They just tell me which one is the best in a collection of models and don't show anything about whether all models in that collection are bad.
I'm looking for a method for whether I should accept or reject a model like p-value or significance level. I guess something like that must exists in bayesian approach but I don't know the name.

[Bob Carpenter]

> On Jan 7, 2015, at 11:02 AM, He-chien Tsai <depo...@gmail.com> wrote:
>
> Thanks for reply. I think WAIC is similiar to AIC and BIC.

> They just tell me which one is the best in a collection of models and don't show anything about whether all models in that collection are bad.

Not quite. WAIC provides an approximation of cross-validation
performance. And it does it with the proper posterior instead of
point estimates.

You can use that to compare if you want.

> I'm looking for a method for whether I should accept or reject a model like p-value or significance level. I guess something like that must exists in bayesian approach but I don't know the name.

You can use things like chi-square tests if you insist on trying
to reject a model.

Nothing's going to tell you to accept a model, frequentist or Bayesian.

Models are typically approximations that can always be rejected with
enough data.

- Bob

[Avraham Adler]

On Wednesday, January 7, 2015 11:43:31 AM UTC-5, Bob Carpenter wrote:

Nothing's going to tell you to accept a model, frequentist or Bayesian.

Models are typically approximations that can always be rejected with
enough data.

George Box's maxim should be printed on the inside of every modeler's eyeballs.

"All models are wrong; some models are useful"

[He-chien Tsai]

> Nothing's going to tell you to accept a model, frequentist or Bayesian.

> Models are typically approximations that can always be rejected with
> enough data.

I don't get this. If that's true, what's p-value actually for? It also sounds like that every experiment's result by p-value are not confident because every model can always be rejected with enough data.
Despite that, I'd like to know what are the most common methods for bayesians to try rejecting a model. Although I found a paper titled "A BAYESIAN χ2 TEST FOR GOODNESS-OF-FIT", I seldom heard this test is used.

[Bob Carpenter]

> On Jan 8, 2015, at 12:33 PM, He-chien Tsai <depo...@gmail.com> wrote:
>
>
> > Nothing's going to tell you to accept a model, frequentist or Bayesian.
>
> > Models are typically approximations that can always be rejected with
> > enough data.
>
> I don't get this. If that's true, what's p-value actually for?

Publishing in journals.

> It also sounds like that every experiment's result by p-value are not confident because every model can always be rejected with enough data.

Correct --- this is just a consequence of models being approximate,
Gibbs inequality, and the central limit theorem.

- Bob

[Michael Betancourt]

I don't get this. If that's true, what's p-value actually for?

They’re for confusing poor statistics students.

It also sounds like that every experiment's result by p-value are not confident because every model can always be rejected with enough data.

Yup.  That’s why most practical Bayesians don’t take their models too seriously.

Despite that, I'd like to know what are the most common methods for bayesians to try rejecting a model. Although I found a paper titled "A BAYESIAN χ2 TEST FOR GOODNESS-OF-FIT", I seldom heard this test is used.

Proper Bayesians DO NOT reject models.  Like any hypothesis/significance test,

p-values require that you define both a null hypothesis and a competing hypothesis.

With the competing hypothesis you can compute the power of any testing procedure

which calibrates the resulting p-value.  In a Bayesian setting the null hypothesis is

the full model and there is no competing hypothesis (which would have to be something

like “every other possible model excluding the null” which may not even be expressible

mathematically).  This lack of calibration also effects cross-validation, WAIC, and the

like which is why they can only ever be used for comparisons.

Again, proper Bayesians do not reject models.  Instead we criticize models, looking

for tension with prior beliefs or predictive performance to expand and grow our model.

[Michael Betancourt]

I don't get this. If that's true, what's p-value actually for?

Publishing in journals. 

Hehe, between Bob and I, who was the snarkiest?

[Daniel Lee]

On Thu, Jan 8, 2015 at 2:07 PM, Michael Betancourt <betan...@gmail.com> wrote:

Hehe, between Bob and I, who was the snarkiest?

To whomever responds: please provide p-values.

[Jonah]

Haha. 

Both were snarky at a statistically significant level. The difference, however, was not statistically significant.  

[Jonah]

Crap, I forgot the p-values. But who can interpret those things anyway ...

On Thursday, January 8, 2015 at 2:24:14 PM UTC-5, Daniel Lee wrote:

[Andrew Gelman]

Hi, see chapter 6 and 7 of BDA3.
A

> On Jan 8, 2015, at 12:33 PM, He-chien Tsai <depo...@gmail.com> wrote:
>
>

[Andrew Gelman]

Bob.  Because we’re defining relative to each individual’s snarkiness baseline.

[Shravan Vasishth]

Snarkiness aside, here is what actually happened to me when I presented posterior credible intervals for the parameter of interest (using Stan) in my papers and alongside the interval, I presented p=Prob(b>0) (the posterior probability of the parameter being greater than zero given data), I got the comment, but this is a one-sided significance test; please present the two-tailed test: 1-2*min( P(b<0), P(b>0). 

Since that didn't make any sense at all, I just added the results of the likelihood ratio test based chi-squared using lmer. I think that's a reasonable compromise, because it gives the researcher used to p-values something to look at that they are familiar with at least for making their decision, and at the same time it just presents the amount of information available from the data as a credible interval. Maybe over time the shift will happen, but for now maybe that's good enough. The bottom line seems to be that if I don't want to just stick my results into the proverbial drawer (i.e., "publish papers"), I am going to have to report p-values *alongside* the posterior credible intervals.

If anyone has any better ideas on what to say/do to respond to the criticism that P(b>0) is a one-sided significance test---other than pointing out the obvious fact that it is not---I'm listening.