BIC vs. DIC

[Carlo Sestieri]

Hi,

as you wrote in the tutorial, I found the DIC somewhat biased to more complex models.

I would like to try model comparison using BIC instead (using a MATLAB script). So I have a couple of questions:

> Is there any reason why BIC may not be the right measure?

> How can I extract from the model the info to calculate it?

Thanks a lot,

Carlo

[Michael J Frank]

Carlo, 

The BIC is not defined for hierarchical models. The DIC can be seen as a generalization for this setting (but more like a generalization of AIC, which is meant as a metric to be able to predict out of sample data rather than a metric for defining the 'true' model). It generally can be a reasonable approximation when pD (the number of effective parameters, which is part of the DIC calculation) is << n, the number of observations.

The bias toward complexity can be resolved a little bit with subsequent versions of DIC, e.g. by Plummer (2008). This is not calculated automatically in pyMC (it is computationally costly) but is available in JAGS  -- and from my experience there, and also some notes from Plummer himself, it almost always penalizes by very close to twice as much as standard DIC. So if you wish you can simply multiply the pD that HDDM gives you (the penalty term) by 2 and then adjust DIC accordingly.  I haven't verified that this selects the appropriate model though given known generative data with DDM.  

As mentioned in previous posts, I would suggest also doing a posterior predictive check to complement any model selection metric, and also note that with more complex models the posteriors will generally be wider and hence that also provides a form of penalty in terms of how easy it will be to detect a difference in parameters (between conditions groups etc). Thus if you see a significant difference (little to no overlap in posteriors), this is in spite of the increased complexity. (This is roughly Kruschke's argument).

Michael

Michael J Frank, PhD, Associate Professor
Laboratory for Neural Computation and Cognition
Brown University
http://ski.clps.brown.edu
(401)-863-6872

--
You received this message because you are subscribed to the Google Groups "hddm-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hddm-users+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Carlo Sestieri]

Michael,

thanks for the explanation.

I do find a significant drift rate difference between conditions only when I use a simple model but not when using the more complex model favored by DIC.

However, when I used DMAT on the same data, both the AIC and the BIC favored the simpler model. 

I had a look at other papers that used HDDM and noticed that the DIC favored the most complex model among those that were tested.

That's why I wanted to see if other methods give me the same answer.

Could you tell me how to obtain the pD and the penalty term in order to calculate the modified DIC according to Plummer?

Thanks a lot,

Carlo

[Michael J Frank]

 pD should be output by HDDM when it reports DIC.  The normal formula for DIC is 

DIC = D + pD

so to get the version where pD is doubled (which should be close to what you would get from Plummer's method, but I can't guarantee this would always be the case) you would then simply add another pD to the DIC that HDDM reports: DIC_new = DIC + pD = D+ 2*pD

Of  course you do this for each model, the more complex one and the more simple ones, with their own pD and DIC values, and it would penalize more complex models more.  (Note also that more parameters doesn't always mean proportionally more complex in terms of degrees of freedom, this is what the AIC and BIC approximations get wrong). 

Michael