Calculating Bayes Factor with Stan

[Xulong Wang]

Hi all, 

How would you calculate the Bayes Factor between two models with Stan? I am thinking of using the log posterior probability estimated from Stan. However, not sure whether this is legitimate. 

Thanks,

[Bob Carpenter]

The problem is that our probably functions drop all the constant
terms in sampling statements

x ~ distribution(...);

so you're not getting the entire log probability function.
If you replace the sampling statements with

increment_log_prob(distribution_log(x, ...));

that will not drop the constant terms.

- Bob

> --
> You received this message because you are subscribed to the Google Groups "Stan users mailing list" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to stan-users+...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Michael Betancourt]

Even with the proper likelihood normalization there is no proper way of computing Bayes factors in Stan at the moment, nor will there be in the immediate future.

--

[Bob Carpenter]

Why not? I don't know much about Bayes factors, but I looked
at the Wikipedia page

http://en.wikipedia.org/wiki/Bayes_factor

and I don't see any reason we can't compute the evidence,
p(data|model), if we isolate the likelihood term ll = p(data|theta)
and then look at its posterior mean.

I assume I'm missing something here, but don't know what.

- Bob

[Michael Betancourt]

No. We have had this discussion multiple times on the list.

[Xulong Wang]

Yes. This topic has been discussed in the thread below but no definite practical conclusion was made.

https://groups.google.com/forum/#!searchin/stan-users/bayes$20factor/stan-users/nmh5oy-Ox9U/-U-ZACF8qusJ

In this post, Michael proposed to write a manual of BF calculation with Stan. Has this been done yet? I do not know how to proceed to calculate the BF at this moment.

[Bob Carpenter]

Thanks for finding and linking.

To summarize the thread and answer my own question:

1. Michael's point was that you can't compute the relevant
integral for the marginal probability of the data
using MCMC because it has infinite variance.

2. Andrew's point was that these factors don't make
much sense with very broad priors, and that if you want
to compare models, cross-validation or WAIC are better
(though still imperfect) tools.

- Bob

[Xulong Wang]

Thanks Bob for the consistent patience in replying posts. Appreciated. I noticed Stan has WAIC implementation, and I will try to implement it in my work. 

[Sachintha Karunaratne]

Hi guys, 

I don't know whether this has been discussed earlier, but there are methods

to calculate the marginal likelihood from the output of MCMC.

See the work by Chib https://stat.duke.edu/~scs/Courses/Stat376/Papers/NormConstants/ChibJASA1995.pdf

He uses the output from Gibbs to calculate the marginal likelihood. However, you need to do some additional 

sampling that you wouldn't normally do,  to be able to calculate the marginal likelihood.

I've personally used his method to calculate the normalising constant of a particular generalisation

of a multivariate von mises distribution.

Regards,

Sachintha.

[Michael Betancourt]

We are aware of the efficient methods for computing the evidence/marginal likelihood

but they will not be available in Stan until some internal changes are made to the code.

[Soga Shota]

Hi,

   How about WBIC instead of bayes factor? Is there any reason not to prefer WBIC?

Importantly, WBIC can be computed with stan. http://jmlr.org/papers/v14/watanabe13a.html

Shota

2014年8月6日水曜日 16時57分01秒 UTC+9 Michael Betancourt: