AIC and BIC

[Hartman, Rosemary@DWR]

Hi Folks,

 

At last week’s meeting Jereme brought up the fact that AIC was not appropriate for mixed models, and BIC was more appropriate. I hadn’t heard this before so I did a little digging around and found some articles that compare the two methods (attached), so I thought I’d share them. There are definitely philosophical differences between them, though BIC can be used on frequentist models and AIC can be used on Bayesian models.

 

I don’t have a strong foundation in statistical theory, so I definitely don’t understand all the nuances, but in my experience I’ve never seen a big difference between the results of AIC and BIC for any of the models I have run. That is, the two numbers are usually only slightly different, and I’ve never had BIC support a different model from the AIC value. Does anyone have a good example of a type of model with very different AIC versus BIC results?

 

I also found a few articles about using ‘conditional AIC’ for mixed models:

https://www.researchgate.net/profile/Miguel-Jimenez-Bravo/publication/343343412_Multimodal_Perception_of_Prominence_in_Spontaneous_Speech_A_Methodological_Proposal_using_Mixed_Models_and_AIC/links/5f3a6828299bf13404cb3162/Multimodal-Perception-of-Prominence-in-Spontaneous-Speech-A-Methodological-Proposal-using-Mixed-Models-and-AIC.pdf

https://www.sciencedirect.com/science/article/pii/S0047259X14000736

 

Thanks!

Rosie

 

PS: My slides and code from the mixed model presentation are available here: https://interagencyecologicalprogram.github.io/DataScience/agendas

[Bashevkin, Sam@Waterboards]

I hadn’t heard of that distinction before, although I know there are various corrections to AIC that can be used. I know that the mgcv R package that is used to fit generalized additive models (which can be thought of as a type of mixed effect model) uses AIC to compare models, although they apply a correction to the AIC formula (See ?AIC.gam if you have mgcv loaded). When I first started using mgcv, I was using BIC because I had learned it was a better metric for model comparison, until I read more and understood that the mgcv package had created their variation on AIC to be used with GAMS, which (as I recall) was better than applying a standard BIC formula. It’s likely that other packages for mixed models also have variations on AIC that may be improved over vanilla AIC.

 

From: bay-delta-...@googlegroups.com <bay-delta-...@googlegroups.com> On Behalf Of Hartman, Rosemary@DWR
Sent: Monday, August 14, 2023 12:02 PM
To: Bay-Delta Data Science Fun <bay-delta-...@googlegroups.com>
Subject: [bay-delta-datascience] AIC and BIC

 

EXTERNAL:

 

--
https://interagencyecologicalprogram.github.io/DataScience/
---
You received this message because you are subscribed to the Google Groups "Bay-Delta Data Science Project Work Team" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bay-delta-datasc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bay-delta-datascience/SJ0PR09MB6864EE57224F25E215216F31BC17A%40SJ0PR09MB6864.namprd09.prod.outlook.com.

[Gaeta, Jereme@Wildlife]

Hi all, 

To follow up, I have a couple of slides from my old mixed effects regression course:

The main reason to choose BIC (or DIC) over AIC is that AIC does not take sample size into account:

Why does this matter? Well:

So, if you are taking an information theoretics approach in which you are performing a model selection procedure allowing fixed and random effects structure to vary, you need to change how you are fitting the model depending on the selection step (i.e., are you doing the random effects selection or are you doing the fixed effects selection?)

So, from my experience, the patterns of AIC and BIC are very similar, but the big difference is if you are performing a model selection procedure. You want to do it correct, not good enough. 

As Rosie mentioned, there are multiple schools of thought, but I prescribe to the Gelman and Hill as well as Zuur philosophies. If you are not taking this approach, just be prepared to justify your methods.

I hope this clears up my comments.

Cheers,

-J

Jereme W. Gaeta, PhD    Environmental Program Manager – Managerial  CA Department of Fish and Wildlife – Water Branch 1010 Riverside Pkwy West Sacramento, CA 95605  Mobile: 209-403-6935

---

From: bay-delta-...@googlegroups.com <bay-delta-...@googlegroups.com> on behalf of Hartman, Rosemary@DWR <Rosemary...@water.ca.gov>

Sent: Monday, August 14, 2023 12:02 PM
To: Bay-Delta Data Science Fun <bay-delta-...@googlegroups.com>
Subject: [bay-delta-datascience] AIC and BIC

 

WARNING: This message is from an external source. Verify the sender and exercise caution when clicking links or opening attachments.

--

[Hartman, Rosemary@DWR]

From now on, all papers I write will include the term ‘vanilla AIC’