Model Evaluation For Prediction

[Jordan Green]

Hi All, 

I am working on building occupancy models to predict habitat out of sample for a number of species to use as indicators in a forest management plan. When evaluating these models, many don't meet the AUC threshold (0.8) but even those that we see in the 0.7-0.8 range that are typically considered "acceptable" in the literature do a very poor job of accurately predicting presence. I've attached some figures made by colleagues (Buchkowski et al) that highlight the issues around predicting presence or absence.

Some suggestions we have received were to test different sampling designs and co-variates which we are in the process of implementing but I wanted to initiate some conversation around the issue of accurate prediction and whether AUC is the appropriate performance measure.

I appreciate any feedback! 

Cheers,

Jordan

[Marc Kery]

Dear Jordan,

one challenge when assessing the predictive success of an occupancy model is that you don't observe absence, only presence (at least if you can rule out false positives). Hence, I wonder whether it makes sense to use the standard calculation of the AUC. This gauges how well you can predict the actual detection/nondetection observations, and not presence and absence, which is what you would really like your model to predict well.

Ideally, you would have testing data for true presence and absence. That may perhaps be obtained for a subset of sites were you have such a large number of repeated visits, that the cumulative detection probability can be assumed to be equal to 1. Then you can compare the true presence and absence with the predicted occupancy under the model.

With respect to covariates in the model, I assume you are talking about detection covariates ? Because judging from your maps there must be some very informative occupancy covariates, because you have very strongly patterned maps of occupancy there, right ? (Although I can't really see much in the figures, which come out too small.)

Best regards  --- Marc

---

From: unma...@googlegroups.com <unma...@googlegroups.com> on behalf of Jordan Green <jsgre...@gmail.com>
Sent: Tuesday, September 9, 2025 14:53
To: unmarked <unma...@googlegroups.com>
Subject: [unmarked] Model Evaluation For Prediction

 

--
*** Three hierarchical modeling email lists ***
(1) unmarked (this list): for questions specific to the R package unmarked
(2) SCR: for design and Bayesian or non-bayesian analysis of spatial capture-recapture
(3) HMecology: for everything else, especially material covered in the books by Royle & Dorazio (2008), Kéry & Schaub (2012), Kéry & Royle (2016, 2021) and Schaub & Kéry (2022)
---
You received this message because you are subscribed to the Google Groups "unmarked" group.
To unsubscribe from this group and stop receiving emails from it, send an email to unmarked+u...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/unmarked/320b3c5b-ed1f-47dc-88e4-7f233be29682n%40googlegroups.com.

[Jordan Green]

Hi Marc,

Thanks for your reply. At each of the sites, a minimum of 5 repeated visits were conducted and for many species at many sites P* is > 0.95 so I should be able to do this comparison. The way I have rationalized it in my head is that if we calculate the AUC using imperfect observations, we are actually predicting detectability and sensitivity (or presence) is penalized due to low p, correct? Or would the low sensitivity indicate that detection covariates are missing?

Cheers,

Jordan