Evidence Lower Bound as model selection criterion

[Marcel Falkiewicz]

Hello, 

 in the field of neuroscience there is a 'school', which claims that evidence lower bound can be considered as the ultimate criterion for model comparison and selection. For example, see http://www.sciencedirect.com/science/article/pii/S1053811911008160# . However, I've read many papers from different fields and haven't encountered such large enthusiasm towards it. Since my statistical knowledge is pretty basic, I would like to ask your opinion - can ELBO be considered as a model selection criterion? If so, what are the neccessary conditions?

Regards,

 Marcel

[Michael Betancourt]

The ELBO is an approximation to the evidence (sometimes called

the marginal likelihood).  It tends to be a better approximation then

BIC, but even if it is very accurate you have to deal with the subtleties

of the evidence itself.  For that see the numerous relevant posts on

this list as well as the discussion in Bayesian Data Analysis.

--
You received this message because you are subscribed to the Google Groups "Stan users mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to stan-users+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Dustin Tran]

Aside from subtleties with using the marginal likelihood itself, the ELBO can indeed be used. The bigger issue to keep in mind though is the sensitivity of the variational approximation to the posterior. That is, using the ELBO as a replacement for the marginal likelihood makes the assumption that the KL divergence between the posterior and the variational distribution are the same for every model. This is rarely true, and it is easy to construct scenarios where the marginal likelihood is actually lower for model 1 but the ELBO is higher for model 2, because the variational family cannot fit model 1 very well. This is not nearly as bad as AIC or BIC in "most cases”, however, so I’m not surprised by that paper’s stance.

Dustin

[Dustin Tran]

> it is easy to construct scenarios where the marginal likelihood is actually _higher_ for model 1 but the ELBO is higher for model 2,

fixed typo

Dustin

[Marcel Falkiewicz]

Thank you guys for clarification, despite the sad conclusion:) Anyway,
it's good to know that ELBO could be useful in model selection,
considering all the limitations. Maybe this should be mentioned in
documentation?

[Andrew Gelman]

Yes, to start with I think the term “evidence” is horrible; second, the whole “lower bound” thing is pretty meaningless in that if you are comparing 2 models, you’ll use the difference so a “lower bound” is no better than an “estimate”; third, it typically depends on aspects of the prior that are assigned arbitrarily, for example if you change your prior from normal(0,10) to normal(0,100), your marginal likelihood changes by a factor of 10.  So, yeah, basically don’t do it.  If you want to do model comparison I recommend predictive criteria such as disucssed here:

http://www.stat.columbia.edu/~gelman/research/unpublished/loo_stan.pdf

A

On Aug 5, 2015, at 10:20 AM, Michael Betancourt <betan...@gmail.com> wrote:

[Andrew Gelman]

Ummm, no!