Question about null model tests of significance in relation to AUC

[CSB]

Hi - It's my understanding that the rule-of-thumb for AUC in habitat modeling exercises is, generally, that anything greater than or equal to 0.7 constitutes a model with some predictive value, and that anything less than 0.7, especially anything approaching 0.5, constitutes a model that has too high a probability of being no better than random that it's considered unsafe to draw conclusions from. But, I've also read that when modeling presence only (P/O) data an AUC of 1.0 is theoretically impossible. If that's true, does that mean that the 0.7 AUC threshold can be relaxed - especially if predictions and response curves make ecological sense?

This leads to my next question: If you have a P/O model with an AUC of less than, but close to, 0.7, lets say 0.685  (and so would normally be rejected), but you've found it to be statistically significant when compared to a null-model, is that model then safe to use/make inferences from? If the null-model shows that your model performance is statistically significantly better than random isn't it then useful? 

I know that the two statistics don't measure the same thing, but it seems like I often see one or the other used to justify a model. It just seems to me that if you had a model with an AUC of 0.95, but it failed the null-model test, it would basically not be able to tell you anything with any measure of certainty. But on the other hand, that if you had a model with an AUC of 0.6 that was statistically significant, you would have a model that you couldn't learn a lot from, but what little you did learn would be more defensible.  

I guess in the end the question is: Which outweighs the other in saying a model passes muster, a significance test or AUC?

Any paper or thread suggestions to help clarify would be good too.

Thanks!

[Jamie M. Kass]

You are right that a model that is useful should not be dismissed simply because a test of significance shows a probability too close to some seemingly arbitrary threshold. You hope for a model with a higher AUC, but if the best you can do with your data is something below 0.7, you may consider using it for predictions, albeit carefully and with caveats.
However, before resigning to this, you should certainly do a number of things (not exhaustive):
1) Make sure you are using independent test points for your evaluations and rating models via with the test AUC value, not the training AUC;
2) make sure your background extent is appropriate (see Anderson & Raza, 2010);
3) accounting for spatial autocorrelation between your occurrence points with spatial filtering, e.g. (you can use something like R package spThin -- see Aiello-Lammens et al. 2015);
4) make sure you are testing out many parameter combinations to make sure you have optimal settings for your dataset and extent (see Radosavljevic & Anderson, 2013; use something like R package ENMeval -- see Muscarella et al. 2014).
This way, you can choose between multiple models and find which model is the best fit for your study (via AIC or some sequential method like in Radosavljevic & Anderson). I cited papers I am familiar with (that my lab was involved with), but there are many more you might consider. Good overall overviews of this stuff are found in Elith et al. 2011 (A statistical explanation of Maxent for ecologists) and Merow et al. 2013 (A practical guide to MaxEnt for modeling species’ distributions).

-----------------------------------------------------------
Jamie Kass
PhD Student, Department of Biology
City College of New York, CUNY Graduate Center