Bayesian model checking in NIMBLE

[Cassidy Waldrep]

Hello all!

I've run a relatively simple logistic regression in NIMBLE that is trying to find the effects of proportions of habitat on whether or not an individual successfully incubated a nest (0 = failed, 1 = successful). My covariates are not correlated, VIF is under 3. My model converges nicely and traceplots look great. I have a few positive/negative effects which is cool to see! I'm not doing any sort of model selection. 

I'm wondering if anyone could guide me on how to check that the model has an okay "fit" and meets assumptions. I've been reading Conn et al., 2018, and it seems like there's a lot of different ways (cross validation, posterior predictive checks, etc). 

I see that NIMBLE has a runCrossValidate function. Is this what a statistician would recommend for checking model fit? 

Curious on opinions and if there's a standard in the ecology literature (asking as a current PhD student without much Bayesian model checking practice). Thank you in advance!

Cassidy Waldrep

[PierGianLuca]

Hi Cassidy,

Nice to hear that your calculations with Nimble went smoothly!

Your questions go beyond Nimble, and you'll probably receive many different, possibly even contrasting, answers. Model "fit" and model comparison is an area with still much debate. My personal take:

– Do you have particular reasons for using a logistic-regression model? for example, does it embody a mechanistic physico-ecological model? Otherwise, why not do a nonparametric study, with no model at all? It would also give you a more realistic picture of the uncertainty coming from your finite data sample. Running a nonparametric analysis is today not so computationally expensive, and can also be done with Nimble; it depends on the details of your study, of course.

– Model-fitting is at heart a *decision-making* problem: "do I want to use this model?". Which leads to further questions in need of answers:

• What are the alternative models? "fit better" than what?

• What are the gains & losses of the fitting errors? This question reveals that you need a utility function. Sure, you can use some general-purpose utility (accuracy, true-positive rate, and an infinity of others), but it could be very mismatched to the final purposes of your study.

The two questions above show that there's a lot of "subjectivity" in the "fitting" problem. Not in the sense of "whimsicality", but in the sense that the answer depends very much on the context of your specific problem.

Cheers!
Luca

On 260709 20:27, Cassidy Waldrep wrote:
> Hello all!
>
> I've run a relatively simple logistic regression in NIMBLE that is trying
> to find the effects of proportions of habitat on whether or not an
> individual successfully incubated a nest (0 = failed, 1 = successful). My
> covariates are not correlated, VIF is under 3. My model converges nicely
> and traceplots look great. I have a few positive/negative effects which is
> cool to see! I'm not doing any sort of model selection.
>
> I'm wondering if anyone could guide me on how to check that the model has

> an okay "fit" and meets assumptions. I've been reading *Conn et al., 2018,
> <https://esajournals.onlinelibrary.wiley.com/doi/10.1002/ecm.1314>* and it

[Chris Paciorek]

 Hi Cassidy,

I'll say at a high level that the NIMBLE development team has not done a lot to provide tools for model checking (and certainly we have not tried to provide a thorough set of such tools), though a couple of folks on the development team did make a substantial effort on calibrated posterior predictive checks that is in a package on GitHub. In general, we assume that users will either use existing packages to postprocess output from nimble for model checking or potentially write their own code (perhaps using nimble) for that. 

So as you suggested in your initial email, this is more of a stats question than a nimble question (I'm just confirming that from the perspective of the development team) and others may weigh in, in addition to Luca's comments.

-chris

--
You received this message because you are subscribed to the Google Groups "nimble-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nimble-users...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/nimble-users/de5e240b-91eb-4f1b-8e54-f2f15530136d%40magnaspesmeretrix.org.