AIC versus AUC as a basis model selection

[alqamy]

I am using the perl script developed by Dan Warren to calculate the
AIC for Maxent models. I found that in some of my runs (changing some
combinations of environmental variables) that there are some
contradiction between AUC and AIC. In most of the cases The model
selected on basis of lowest AIC is also the model that yeilds the
highest AUC, but just recently I came across a case where the model
providing exceptionally good AUC (0,960) is having a high AIC that is
about 40 points higher than the lowest AIC among other models. In my
knowledge AIC is a formal model selection criteria while AUC is a
measure of how different is the model prediction from chance. My
question is what to do in such a case? Favor AIC or AUC?
Hope any one can help in this regard

Husam El Alqamy, B.Sc., M.Phil.
Protected Area Coordinator,
Terrestrial Environmental Research Center, TERC
Environmental Agency Abu Dhabi
Antelope Specialist Group – ASG, IUCN.

[ZuZu]

It is my understanding that AIC cannot be applied to models that are
not nested (AIC will not be comparable). Perhaps that is the
difficulty?

ZuZu
CANPOLIN

[alqamy]

Hi ZuZu and List
Sorry I couldnt get what do you mean by your message. If "nested"
means that one model has a sub-group of the predictor variables while
the other have the same variables inaddition to others, then I have
nested models. Unfortunately your reply was very vague. Please kindly
elaborate.
Regards

Husam El Alqamy, B.Sc., M.Phil.
Protected Area Coordinator,
Terrestrial Environmental Research Center, TERC
Environmental Agency Abu Dhabi
Antelope Specialist Group – ASG, IUCN.

> > Antelope Specialist Group – ASG, IUCN.- Hide quoted text -
>
> - Show quoted text -

[gr...@zoology.ubc.ca]

Hi Husam - AIC is designed to penalise for overfitting. So its not
surprising that a model with a low AIC also has a lower (than the best)
AUC. In fact, even tho your experience seems to contradict this, I would
hypothesise that models with the highest AUC would never have the lowest
AIC because in my experience, MaxEnt has a strong tendency to overfit (tho
this clearly depends on many things). Upshot is that I would certainly use
AIC as a model selection criteria over AIC.

cheers,
ed.

>>
>> On Feb 23, 10:50�am, alqamy <alq...@gmail.com> wrote:
>>
>>
>>
>> > I am using the perl script developed by Dan Warren to calculate the
>> > AIC for Maxent models. I found that in some of my runs (changing some
>> > combinations of environmental variables) that there are some
>> > contradiction between AUC and AIC. In most of the cases The model
>> > selected on basis of lowest AIC is also the model that yeilds the
>> > highest AUC, but just recently I came across a case where the model
>> > providing exceptionally good AUC (0,960) is having a high AIC that is
>> > about 40 points higher than the lowest AIC among other models. In my
>> > knowledge AIC is a formal model selection criteria while AUC is a
>> > measure of how different is the model prediction from chance. My
>> > question is what to do in such a case? Favor AIC or AUC?
>> > Hope any one can help in this regard
>>
>> > Husam El Alqamy, B.Sc., M.Phil.
>> > Protected Area Coordinator,
>> > Terrestrial Environmental Research Center, TERC
>> > Environmental Agency Abu Dhabi
>> > Antelope Specialist Group � ASG, IUCN.- Hide quoted text -
>>
>> - Show quoted text -
>

> --
> You received this message because you are subscribed to the Google Groups
> "Maxent" group.
> To post to this group, send email to max...@googlegroups.com.
> To unsubscribe from this group, send email to
> maxent+un...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/maxent?hl=en.
>
>

[alqamy]

Many Thanks Ed but I think there is a typo in the last pharse in your
message.

"Upshot is that I would certainly use AIC as a model selection

criteria over AIC" can you please kindly verify whic is AIC and which
is AUC. I know it is confusing to keep the separate when typing fast.
Regards

Husam

> >http://groups.google.com/group/maxent?hl=en.- Hide quoted text -

[gr...@zoology.ubc.ca]

oops. and I tried to be careful. AIC over AUC.

[Tereza]

Husam,

Can you, please, advice me where to obtain the script to calculate
AIC?

Thanks, Tereza

[Chris E]

Could you guys give us directions how to calculate AIC from maxent
outputs. I'm comparing models varying independently few parameters to
the same data set. I found, better graphically (based in my own
knowledge of specie and study region) models has lower AUC (with very
close range between each other) . I understand AUC is a measure of
how different is the model prediction from chance, so in my case i
think is not a good model selection index.

Thanx, Chris

[alqamy]

Dear Chris and list
Here I am quoting Dan Warren (Author of the AIC script) on how to use
the script for calculating AIC for maxent models. You have to contact
him for the script.

"You should drop these into a directory with your occurrence points,
lambdas files, and ascii files. You need to go to a command line and
type "AIC.pl inputfile", where "inputfile" is the name of your input
file. I'm assuming your computer already knows how to run Perl
scripts.

The input file is a comma separated file that tells the script where
to find the occurrence points, ascii file, and lambda file associated
with a specific model. At present, you need to have each species in
its own separate .csv file. Once the AIC script is done, it will
output a file starting with AIC_.
That gives you likelihoods, BIC, AIC, and AICc values. The script is
currently set up so that it automatically rejects any model with more
parameters than it has data points, as Maxent will happily produce
models like this and they make AICc exhibit pathological behavior. In
response to your question: I am treating each nonzero lambda value as
a parameter in calculating information criteria, as for most purposes
I strongly prefer simpler marginal suitability functions than those
that Maxent produces by default (one of the reasons I've been playing
with AIC/BIC).

One thing I do ask is that you contact me before publishing anything
with this script to see if I've gotten a paper out on it yet. This
script was developed for part of my dissertation work, which is not
yet complete. I hope to have a paper submitted on it within a couple
of months, but obviously I would like to have my paper be the first to
discuss the method if possible. Finally, as I said before, this
script is very new and may still be buggy, but it's worked fine for me
so far. Use at your own risk!"

Dan Warren
Population Biology Graduate Group
University of California, Davis
Email: danw...@ucdavis.edu
Phone: 530-848-3809

[dan.l....@gmail.com]

I am now very close to being finished with a rough first draft of the
model selection paper that this AIC/BIC script was developed for.
While I obviously don't want to announce all of the results here, I
will say that AIC (actually AICc) seems to perform very well at
selecting Maxent models, and for the simulated data my study was based
on it is a better indicator of model quality than either training or
test AUC.

Regarding nested and non-nested models, my understanding is that AIC
and BIC are valid for non-nested models but LRTs are not (e.g.,
Burnham and Anderson p. 88).

[Steven Dempsey]

Has that paper been published?

[Samuel Bosch]

For those wondering: the paper referred to is this one: http://www.esajournals.org/doi/abs/10.1890/10-1171.1 (Ecological niche modeling in Maxent: the importance of model complexity and the performance of model selection criteria) and the AIC calculation functions have been added to ENMTools (http://enmtools.blogspot.be/)

[Jamie M. Kass]

I also wanted to inform the R users in the audience that there are two packages you may want to take a look at. The first is "dismo", which has a nicheOverlap() function that allows one to run both Schoener's D and Moran's I on prediction rasters. Another is "ENMeval", which my lab helped put out. It will iterate over combinations of feature classes and regularization multipliers, while cross-validating in a user-defined way, and return a table of evaluation statistics that includes both test AUC and AIC, among others. The idea is to let you make the decision as to what your optimal model settings are. The paper is here: 

http://onlinelibrary.wiley.com/doi/10.1111/2041-210X.12261/abstract

and cited as so:

Muscarella, R., Galante, P. J., Soley-Guardia, M., Boria, R. A., Kass, J. M., Uriarte, M., Anderson, R. P. (2014), ENMeval: An R package for conducting spatially independent evaluations and estimating optimal model complexity for Maxent ecological niche models. Methods in Ecology and Evolution, 5: 1198–1205.