Model evaluation independent of range extent?

[Alex Rebelo]

I would like have a quality control on a number of models I'm running for different species. Originally I was thinking of using models with an AUC of >.7, but recently came across this statement in the tutorial: 

"It is important to note that AUC values tend to be higher for species with narrow ranges, relative to the study area described by the environmental data." 

So I would like to know if there is another method for model evaluation that is not affected by the range extent, preferably without having to rerun the models?

Thanks in advance,

Alex

[Eliécer E. Gutiérrez]

Dear Alex,

If you are using a presence-only modeling technique, you should not use AUC at all. Why? The following articles and book will guide you:

Lobo, J. M„ A. Jiménez-Valverde, and R. Real. 2007. AUC: A misleading measure of the performance of predictive distribution models. Global Ecology and Biogeography 17:145-151.

Peterson, A. T., M. Pape, and J. Soberón. 2008a. Rethinking receiver operating characteristic analysis applications in ecological niche modeling. Ecological Modelling 213: 63-72.

Peterson, A.T., J. Soberón, R. G. Pearson, R. P. Anderson, E. Martínez-Meyer, M. Nakamura, and M. B. Araújo. 2011. Ecological niches and geographic distributions. Monographs in Population Biology, 49. Princeton University Press.

It is really concerning that the issues associated with AUC are serious and well documented, and yet authors continue to use AUC as their main metric of model evaluation, even to produce models that aim to guide applications of public health and conservation. Scary!

Consider using AIC and omission rates. An easy to use R package that allow using these metrics, comparing lots of models parameters (not only default ones), and geographic partition is called ENMeval. Take a look.

Best,

Eliécer

[Alex Rebelo]

Dear Eliécer,

Thanks for your reply, I will read up a bit more and checkout the R package.

Cheers,

Alex

[Sam Veloz]

Although there are issues with AUC, there are issues with any metric of model accuracy. Omission rate is obviously very sensitive to the threshold that you use to make binary maps and there is no consensus on the best way to select the threshold, particularly with presence only data. AIC is used for mode selection, not model evaluation so cannot serve as a substitute for AUC. 

Sam

Sent from my iPhone

--
You received this message because you are subscribed to the Google Groups "Maxent" group.
To unsubscribe from this group and stop receiving emails from it, send an email to maxent+un...@googlegroups.com.
To post to this group, send email to max...@googlegroups.com.
Visit this group at https://groups.google.com/group/maxent.
For more options, visit https://groups.google.com/d/optout.

[Eliecer E. Gutierrez]

Sure. My advice is to pick one or multiple thresholds that are reasonable for the question you are address.  Drop AUC. Read the articles I recommended.  Use AIC to select the best possible model.

To unsubscribe from this group and stop receiving emails from it, send an email to maxent+unsubscribe@googlegroups.com.

To post to this group, send email to max...@googlegroups.com.
Visit this group at https://groups.google.com/group/maxent.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "Maxent" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/maxent/_LCSGn5nEqk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to maxent+unsubscribe@googlegroups.com.

To post to this group, send email to max...@googlegroups.com.
Visit this group at https://groups.google.com/group/maxent.
For more options, visit https://groups.google.com/d/optout.

--

^v^______________________^v^

    Eliécer E. Gutiérrez, Ph.D.
    

    Research Associate

    PPG Ecologia - PNPD

    Departamento de Zoologia

    Instituto de Ciências Biológicas, Campus UnB

    Universidade de Brasília

    Asa Norte 70910-900

    Brasilia, DF, Brazil

    Telephone: (+55) (61) 9964-7744

    Research Associate

    National Museum of Natural History

    Smithsonian Institution

    Website: http://www.eliecergutierrez.com

    Smithsonian's page:  http://vertebrates.si.edu/mammals/mammals_staff_pages/gutierrez_eliecer.html

    ResearchGate: https://www.researchgate.net/profile/Eliecer_Gutierrez

    Lattes: http://lattes.cnpq.br/1828278925405193

    Twitter: https://twitter.com/ee__gutierrez

^v^______________________^v^

[Sam Veloz]

AUC is a valuable metric that balances omission and commission across all possible thresholds. None of the articles listed discounts this but they do provide information about how to use the metric properly. I don't recommend dropping it but I agree that multiple metrics can be useful. 

AIC and AUC are used for different purposes. AIC can help you select from a candidate set of models which by balancing fit and complexity. It tells you nothing about whether your model has high accuracy for predicting. So for choosing variables to use within a model and the form of responses (linear, hinge, etc) AIC is useful. But you will still need another metric to tell you if the best AIC model is any good. 

Omission is helpful but you can maximize omission by simply predicting all cells as present. I have seen cases where small differences in Maxent logistic thresholds can result in small or large areas of predicted presence. This is why you have to balance omission and commission and threshold which leads back to AUC as a valuable metric.

Sam

Sent from my iPhone

To unsubscribe from this group and stop receiving emails from it, send an email to maxent+un...@googlegroups.com.

[Alex Rebelo]

Thanks for the input Sam,

I am modelling for multiple species and I would like to exclude species that have models that cannot accurately predict suitable habitat, rather than select the best model for a particular species. 

So I guess my question is: is there an alternative metric that I can use to determine a consistent cutoff for model accuracy for species with different range extents? Or is this just not possible?

[Eliecer E. Gutierrez]

(A) "AUC is a valuable metric"?. Wrong. It is a really horrible metric, and using it is rather shameful . (1) AUC is extremely sensitive to the size of study region, which by the way continue to be defined in most studies in erroneous ways, not following the proper criterion for such an end (see Anderson Raza 2010; Barve et al 2011).  Besides, (2) AUC ignores the prediction probability values (prediction strengths) and the goodness-of-fit of the model; (3) AUC takes into account model performance over ROC space regions that are rarely (or never) used; (3) AUC gives equal importance to omission and commission errors, and for some [perhaps most] applications of ENM omission is a much more dangerous error than commission. If at least people would use partial ROC (introduced in an article I indicated in my previous email) the issue number "3" (see above) would not affect the AUC value, but the other issues persists. 

(B) "AIC and AUC are used for different purposes"? Obviously!  Advice to Alex: pick the best model with regard to goodness-of-fit based on AIC, and then examine omission rates to make sure model performance is fine. AIC allows you to fine tune not only feature classes but also beta-multiplier (this is important). Optimal balance between model performance and model complexity is critically important, particular if you plan to transfer the models into different climate scenarios than that used to train the model [see Warren and Seifert (2011) Ecological niche modeling in Maxent: the importance of model complexity and the performance of model selection criteria]. Of course, make sure you examine metrics of model performance, not AUC (= a mediocre choice). 

(C) "Omission is helpful but you can maximize omission by simply predicting all cells as present.' ? That is a problem only if you do not examine the proportion of the study area that is predicted suitable and the p-values produced by Maxent that show whether or not the omission is better than a random model considering the proportion of the study region predicted suitable. Well-trainned users would not ignore this step.

Alex, I encourage you not to do as most colleagues do, which is ignore advances in the field. Niche modeling is indeed way more than just "click, click, click".

My two cents. Send me an email, Alex, if you cannot get those articles I mentioned above and in my previous email, but I bet they are all on ResearchGate.

Eliécer

[Samuel Veloz]

Eliecer, interesting discussion but you are making black and white statements that are not supported. AUC has value, as with any metric, you need to be careful how you apply it. 

1) "AUC is extremely sensitive to the size of study region." Absolutely true, this is why you need to be extremely careful with selecting background points when creating your model. However this goes to best practices for modelling and less to do with AUC as a metric. All Maxent models will be sensitive to the size of the study region. This isn't an arbitrary decision and needs to be made carefully.  

"Besides, (2) AUC ignores the prediction probability values (prediction strengths) and the goodness-of-fit of the model. I don't know at what you are getting at here or which metric you propose to get at this. 

 (3) AUC takes into account model performance over ROC space regions that are rarely (or never) used.  This is from the Peterson paper and honestly I thought it made a pretty weak case for this when it came out. I believe the primary motivation was to justify the use of GARP, the authors felt GARP models were penalized more by AUC than Maxent models were. I think the literature since has demonstrated that GARP models are often pretty bad. The authors then make the argument that omission is more important because overprediction is often not really an error and may just represent suitable but unoccupied habitat. This is pretty weak argument since how do you know if it is really an overprediction or not? We need a metric to help. Go back and look at many of the GARP papers that relied on omission and p values and you will find that the maps tend to be huge areas of predicted presence that are evaluated with a clustered set of presence points resulting in incorrect application of testing for significance .   I am usually concerned with all area of ROC space in the models I work with. Also many applications of Maxent models use the continuous output, not a binary map, so omission is not applicable. What metric would you them recommend.

3):" AUC gives equal importance to omission and commission errors, and for some [perhaps most] applications of ENM omission is a much more dangerous error than commission." I really disagree with you here about weighting omission over commission. You have to account for both otherwise you just predict huge areas of your study area as present and assume you are good (see my comment on p-values below). You have to consider both and unfortunately we have a poor estimate of commission with presence only data.

The p-values produced by Maxent. These p-values are worthless. The statistic has assumptions that are rarely met with spatial presence only data. For one, errors will rarely be random and independent but rather spatially autocorrelated. If you want to get at significance your only reliable option is to use a null model randomization approach. These are useful but time consuming and may suffer some of the same issues above with selecting appropriate background and study size.

 

--

^v^______________________^v^

    Eliécer E. Gutiérrez, Ph.D.
    

    Research Associate

    PPG Ecologia - PNPD

    Departamento de Zoologia

    Instituto de Ciências Biológicas, Campus UnB

    Universidade de Brasília

    Asa Norte 70910-900

    Brasilia, DF, Brazil

    Telephone: (+55) (61) 9964-7744

    Research Associate

    National Museum of Natural History

    Smithsonian Institution

    Website: http://www. eliecergutierrez.com

    Smithsonian's page:  http://vertebrates.si.edu/ mammals/mammals_staff_pages/ gutierrez_eliecer.html

    ResearchGate: https://www.researchgate.net/ profile/Eliecer_Gutierrez

    Lattes: http://lattes.cnpq.br/ 1828278925405193

    Twitter: https://twitter.com/ee__ gutierrez

^v^______________________^v^

--
You received this message because you are subscribed to the Google Groups "Maxent" group.
To unsubscribe from this group and stop receiving emails from it, send an email to maxent+unsubscribe@ googlegroups.com.

To post to this group, send email to max...@googlegroups.com.
Visit this group at https://groups.google.com/ group/maxent.
For more options, visit https://groups.google.com/d/ optout.

--
You received this message because you are subscribed to a topic in the Google Groups "Maxent" group.
To unsubscribe from this topic, visit https://groups.google.com/d/ topic/maxent/_LCSGn5nEqk/ unsubscribe.
To unsubscribe from this group and all its topics, send an email to maxent+unsubscribe@ googlegroups.com.
To post to this group, send email to max...@googlegroups.com.
Visit this group at https://groups.google.com/ group/maxent.
For more options, visit https://groups.google.com/d/ optout.

--

^v^______________________^v^

    Eliécer E. Gutiérrez, Ph.D.
    

    Research Associate

    PPG Ecologia - PNPD

    Departamento de Zoologia

    Instituto de Ciências Biológicas, Campus UnB

    Universidade de Brasília

    Asa Norte 70910-900

    Brasilia, DF, Brazil

    Telephone: (+55) (61) 9964-7744

    Research Associate

    National Museum of Natural History

    Smithsonian Institution

    Website: http://www.eliecergutierrez.com

    Smithsonian's page:  http://vertebrates.si.edu/mammals/mammals_staff_pages/gutierrez_eliecer.html

    ResearchGate: https://www.researchgate.net/profile/Eliecer_Gutierrez

    Lattes: http://lattes.cnpq.br/1828278925405193

    Twitter: https://twitter.com/ee__gutierrez

^v^______________________^v^

--
You received this message because you are subscribed to the Google Groups "Maxent" group.

To unsubscribe from this group and stop receiving emails from it, send an email to maxent+un...@googlegroups.com.

[Eliecer E. Gutierrez]

I do recommend reading the articles I have pointed out in my previous email. After doing so, the recommendation I gave in previous email will be better understood and several flawed statements made by Samuel Veloz can be flagged as such by a wise reader. I lack the time to continue with this conversation, but appreciate the exchange of opinions. Best.

To unsubscribe from this group and stop receiving emails from it, send an email to maxent+unsubscribe@googlegroups.com.

To post to this group, send email to max...@googlegroups.com.
Visit this group at https://groups.google.com/group/maxent.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "Maxent" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/maxent/_LCSGn5nEqk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to maxent+unsubscribe@googlegroups.com.

To post to this group, send email to max...@googlegroups.com.
Visit this group at https://groups.google.com/group/maxent.
For more options, visit https://groups.google.com/d/optout.

[Samuel Veloz]

Agreed, read the papers and decide for yourself. The cited papers have been around many years and as noted people are still using AUC. So perhaps my reasoning is flawed or perhaps many modelers and reviewers agree with me that the metric is not as bad as these papers suggest. Ultimately you do have to choose and make the justification for your choice. My opinion is that there is no perfect metric that should be used, unfortunately, and we do have to take all of these results with a grain of salt. I use AUC but don't dwell on it too much. 

Cheers,

Sam

To unsubscribe from this group and stop receiving emails from it, send an email to maxent+un...@googlegroups.com.

[Alex Rebelo]

Thanks for both your inputs,

regards,

Alex

To unsubscribe from this group and stop receiving emails from it, send an email to maxent+unsubscribe@googlegroups.com.

To post to this group, send email to max...@googlegroups.com.
Visit this group at https://groups.google.com/group/maxent.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "Maxent" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/maxent/_LCSGn5nEqk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to maxent+unsubscribe@googlegroups.com.

[Alaaeldin Soultan]

It is really interesting topic, using the evaluation metric with presence-background model is a debatable points and one source of uncertainty in SDM.

So, I would recommend using different evaluation metrics, threshold dependent and independent.

In your case Alex, to avoid rerun your models again, you can report your normal AUC and also model sensitivity and specificity as they can gives inference about the size of the study area. Also it would be more informative if you calculate Boyce index as it adjusted for presence only model.

Cheers

Alaaeldin Soultan
PhD researcher
MaxPlanck Institute for Ornithology
Am Obstberg 1
78315 Radolfzell, Germany
Tel: +49 (0)7732 1501 911
Mob.+4915757474779