How select best models

[carlosed...@gmail.com]

Hello, I'm using maxent on wallace. I would like to know how to select the best models based on AUC and AICc. From what I saw, AUC must be greater than 0.5, while AICc must be less, but I'm still confused about that.

[Luca Gregnanin]



Hi,

I try to answer but maybe someone can help you better.

AUC represents the fitting of your model. Basically, the higher the AUC, the better your model (beware of overfitting, though).

AICc combines the information about the fitting of the model with the number of parameters included. The best model is the one with the lowest AICc. In your plot, you have the deltaAICc. In this case, the model with deltaAICc = 0 is the model with the lowest AICc, and thus the “best” model (according to this parameter).

If you need to choose a model based on one of these parameters, I suggest you to use the AICc, and to use the AUC only as a measure of how your model is able to discriminate between presences and background points. However, you should also compare the different models in order to see if they actually are different in their predictions (e.g. by the D statistics).

Hope this could help.

Luca Gregnanin

Il giorno 25 feb 2021, alle ore 14:32, carlosed...@gmail.com <carlosed...@gmail.com> ha scritto:

Hello, I'm using maxent on wallace. I would like to know how to select the best models based on AUC and AICc. From what I saw, AUC must be greater than 0.5, while AICc must be less, but I'm still confused about that.

<download (1).png>

--
You received this message because you are subscribed to the Google Groups "Maxent" group.
To unsubscribe from this group and stop receiving emails from it, send an email to maxent+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/maxent/8a4c1be4-5722-4e17-8fc0-b8addc4d368an%40googlegroups.com.

<download (1).png>

<2.png>

[carlosed...@gmail.com]

Great luca gregnanin

So, as I understand it, the most important would be AICc and AUC would be secondary. I am now looking for a tool to compare models. It is possible to compare the models in ENMeval or you have any suggestions.

Thank you for your help

[Jamie M. Kass]

I agree with most of what Luca said, but I would caution users to rely solely on AICc, as it does not tell you anything about how well your model can predict new data.

The metrics that use cross-validation (AUC, omission rates, etc.) do report on that, and AUC in particular has issues for presence-background models like Maxent -- namely, it is problematic to use as an absolute measure of performance. It is okay to use as a relative one that you use to compare models with different settings for the same dataset.

However, as Luca said, AUC can have very high scores for overfit models. This is why is it very important to use spatial partitioning approaches like "block" and compare the results with "random k-fold" trials. The spatial methods result in more extrapolation when evaluating folds, which means the model has to predict to data that looks more different from the training data than if you partitioned randomly. Overfit models have a harder time predicting to new data, so they do more poorly with spatial cross-validation than with random.

Hope I was able to help.

Jamie

[carlosed...@gmail.com]

Thanks, Jamie, I will now use AICc to decide on my models