What is that k= number of parameters?

[Ugyen Thinley]

Dear Experts,

The formula for AIC calculation:

I failed to understand k= number of parameters. Is it being referred to the number of independent variables or what the features being used during model development, such as training set, test set, thresholding of omission, commission errors etc? Please help!

Regards,

Ugyen

[Jamie M. Kass]

The k in AIC refers to the number of parameters used by the model. For regression, this is simply the number of coefficients you are estimating based on those you’ve defined for the model. So if you specified a variable “temperature” and a quadratic term for it, you’d have 2 parameters. With regularized models like Maxent, internal variable selection is taking place, so not all variables and their available features make it into the model. If you look at the lambdas file created with fhe output, you can see that the second column has 0’s for some variables — these have dropped out of the model because they were deemed weaker than the others at explaining the signal during the model learning phase. So we usually use all the variables with non-zero coefficients in the lambdas file as the number for k in this calculation.

However, it’s not clear what to do with hinge features — should they be counted with the same weight as a quadratic, for example? The literature on GAMs and other spline techniques also has trouble with AIC for this reason, and I’m not sure if anythingms been fully resolved. A recent paper focused on comparing AIC and other evaluation methods in Maxent. The citation is below.

Galante, P. J., Alade, B., Muscarella, R., Jansa, S. A., Goodman, S. M., & Anderson, R. P. (2017). The challenge of modeling niches and distributions for data‐poor species: a comprehensive approach to model complexity. Ecography.

http://onlinelibrary.wiley.com.ezproxy.gc.cuny.edu/doi/10.1111/ecog.02909/full

And as for your questions about cross validation, AIC is only calculated on the full model, not on testing groups, so cross validation is irrelevant for this conversation.

Jamie Kass
PhD Candidate
City College of NY