ENMevaluate: Hinge features + low RM values results in NA AIC values for Maxent models

[Sharla Foster]

Hi all, 

I was using ENMevaluate to run maxent.jar with the feature class combinations set to "L", "LQ", "H", "LQH", "LQHP", "LQHPT"  (the default settings) and the RM values set to run from 0.5 to 8 in steps of 0.5. I have 23 occurrence points and 10 environmental variables. This resulted in 96 models and for models with the first 3 RM values (0.5, 1, 1.5) that included a hinge feature class ("H", "LQH", "LQHP", and "LQHPT") the AICc, delta.AICc and w.AICc results were not able to be calculated resulting in NA values. These models still had results for all other columns. 

Interestingly the models for which the AIC was not able to be calculated had much higher parameter values (around 80-99) than the rest of the models (typically 8-40). I was wondering if there is something particular about the hinge feature class that might explain these results. 

I am using the lowest AIC value to determine the optimal model, so NA values make me concerned that I could be missing out on potentially good models. Especially since the optimal model was an LQ model with a low RM value. However the extremely high number of parameters makes me question that. Maybe someone could tell if the fact that the AIC is incalculable is an indication that the model is not good?

 

Thanks so much for any help you can provide, 

Sharla

[Adam B. Smith]

Hi Sharla,

If the model has more coefficients than presence points, then these models should be discarded (as per Warren & Siefert who explained how to use AIC to tune Maxent... ENMEval is not really discarding the models but setting their AIC to NA to have the same effect).  In your case, this is occurring when the regularization parameter is very low, which is why you are getting so many parameters (lower RM ==> more complex models).  Hinge features are especially "parameter-hungry" because in theory there can be a new hinge at every value of the environment in the training set.  In contrast, for example, you only get one linear or quadratic feature per predictor in the model.  Hence, models with hinges (but also threshold features) with low RMs tend to be overly complex, so much so that there are more coefficients than occurrence points.

Technically you could still calculate AIC for these models, but Warren and Siefert recommended discarding them because it's weird to say, have just 23 data points but be able to estimate 90 things from them.  Maxent is the only SDM I know of that can do this (besides BIOCLIM)--it's possible to train a model on a single presence site... but it's not credible!  So ENMEval is protecting you from this statistical pitfall.

Best,

Adam

Warren, D.L. and S.N. Siefert.  2011.  Ecological niche modeling in Maxent: The importance of model complexity and the performance of model selection criteria.  Ecological Applications 21:335-342.

~~~~~~~~~~~~~~~~~

Adam B. Smith, Associate Scientist in Global Change

Missouri Botanical Garden Global Change Lab:

* Best practices in species distribution modeling: A stand-alone workshop in R

* Using museum and herbarium data in ecology and conservation: A webinar

* A database of biodiversity databases

[Sharla Foster]

Hi Adam, 

That makes a lot of sense! Thanks for explaining that so clearly, I really appreciate it. 

Best, 

Sharla