Bayesian goodness-of-fit and c-hat

[falaf...@gmail.com]

Hello,

I’ve noticed that in a couple of the hierarchical modeling books folks sometimes estimate the “lack of fit ratio” or c-hat, but I’ve only seen it done when the discrepancy statistic is chi-square. I’ve been estimating Bayesian p-values for mark-recovery data (within an integrated population model) using the Freeman-Tukey discrepancy statistic, but I wasn’t sure whether the ratio of the actual data discrepancy and the simulated data discrepancy could be interpreted in the same way as c-hats estimated from chi-square discrepancy statistic. I really do want to estimate c-hat somehow, and I can switch the discrepancy statistic to chi-square (or calculate both) if needed.

Thank you and happy new year, 

Jeff

Jeffrey A. Hostetler, PhD
US Fish and Wildlife Service
Division of Migratory Bird Management
Branch of Assessment and Decision Support
Patuxent Wildlife Research Center
11510 American Holly Drive
Laurel, MD 20708

[Daniel Hocking]

I don’t see a problem using that or any other discrepancy measure. I think the interpretation would be the same but not necessarily the same values. For example, 0.5 would be the ideal but the meaning of 0.7 might not be exactly comparable between metrics. That shouldn’t be an issue because we don’t compare models or papers that way and how far from 0.5 is problematic is more a rule of thumb than anything. While it’s a nice metric to calculate I hope ecologists don’t get too hung up on it. It’s really just one metric of fit for really complex models. Personally I do like to see observed vs predicted value plots as well. Ideally with held out validation data but that’s VERY rare with these models that require so much data to fit. Even then we are validating expected observations not abundance or detection individually. Simulations examine the edges of the model would be even better but again are usually beyond the scope of the study. 

All of that to say, I think you’re good with your discrepancy measure. Will be interested to hear if there’s reasons that’s not true. 

Happy New Year

Daniel Hocking

--
*** Three hierarchical modeling email lists ***
(1) unmarked: for questions specific to the R package unmarked
(2) SCR: for design and Bayesian or non-bayesian analysis of spatial capture-recapture
(3) HMecology (this list): for everything else, especially material covered in the books by Royle & Dorazio (2008), Kéry & Schaub (2012), Kéry & Royle (2016, 2020)
---
You received this message because you are subscribed to the Google Groups "hmecology: Hierarchical Modeling in Ecology" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hmecology+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/hmecology/4fa55b16-193c-4085-906e-cf92f1d95179n%40googlegroups.com.