Bayes Factors

[sara]

Hi!
I am interested in using bayes factors to compare runs performed with
different rate priors (a range of values - 2% to 4% - normal
distribution), different clock models (strict/relaxed) and different
treepriors (cte size vs exp growth). I have several combinations of
runs with these parameters.

Can I compare a single run with any other or only runs which only
differ in one of the parameters are comparable? And if it is the case
that only comparisons differing in one of the parameters are
appropriate, should the order of the comparisons matter?

hope I've been clear enough,
thanks,
sara

[alexei]

Dear Sara,

I am not sure how much discussion has been had on this forum regarding
the estimation of marginal likelihoods for the purpose of computing
Bayes factors. However my understanding is that the method of
estimating Bayes factors implemented in Tracer (using the harmonic
mean of posterior likelihoods to estimate the marginal likelihood) is
so problematic as to be essentially useless. Of course I have used it
myself in previous publications, but as far as I can tell most serious
Bayesian statisticians now laugh at its use. My understanding is that
the estimator has infinite variance and so in practice the uncertainty
in the estimate of the marginal likelihood is so great that any result
you get is practically useless. The error estimates of the marginal
likelihoods produced by bootstrapping are almost certainly
unreliable.

I have been very tempted to suggest that this estimator is removed
from Tracer for these reasons. Luckily, some of the BEAST development
team are working on alternative methods of estimating Bayes factors
such as thermodynamic integration, which have much better statistical
properties. In the mean time I would personally avoid the use of the
harmonic mean estimator.

With regards to your particular questions here are a few thoughts:

(1) Comparing strict versus relaxed clocks:

The comparison between lognormal relaxed and strict is *relatively*
easy. Use a lognormal relaxed clock first. If there is no appreciable
probably mass near zero in the marginal posterior distribution of
ucld.stdev then you can't use a strict clock. However if the marginal
distribution of ucld.stdev extends down to (abuts) zero, then the data
can't reject a strict clock.

(2) Constant size versus Exponential growth:

Use exponential growth first. If the marginal posterior distribution
of the growthRate includes 0, then your data is compatible with
constant size. You must ensure that the operator on the
exponential.growthRate is a randomWalkOperator and not a scaleOperator
for this "test" to be valid.

FINALLY: the important thing about model choice is the sensitivity of
the estimated *parameter of interest* to changes in the model and
prior. So in many respects its more important to identify which
aspects of the modeling have an impact on the *answer you care about*
than to find the "right" model.

Cheers
Alexei

[sara]

Hi Alexei,

thanks for your answer. really usefull. I was not aware of that
problem with bf estimation. I am wondering if it also applies to bf
calculation used in phylogenetics, now that partitioned analysis are
so used (and bf to choose partitions). or are the marginal lk of runs
in mrbayes estimated in a different way?

thanks!

[Marc Suchard]

The ideas that Alexei lays out for comparing these models are very
good.

In addition to looking at the posterior distribution of your parameter
of interest near it's restriction (i.e, ucld.stdev near zero), you
should compare this distribution to the prior of your parameter of
interest. The change in the posterior distribution from the prior
distribution near the restriction is actually the Bayes factor.
There are many ways to compute Bayes factors -- some much better than
others. The harmonic mean estimator of the marginal likelihood is not
a perfect solution. Here, comparing the posterior to prior
distribution near (at) the restriction is the Savage-Dickey ratio
estimator of the Bayes factor (a generally very good method). Most
of my own research involves using the Savage-Dickey ratio estimator.

Key take home message: don't just look at your posterior
distributions. Compare these distributions to your priors. The
comparison tells you how much the data inform your inference.

best, Marc

[alexei]

Hi Sara,

The same problem exists for marginal likelihoods in MrBayes as the
same basic method for estimating them is employed as far as I know.
How bad the problem is depends on who you ask :-)

Cheers Alexei

[alexei]

Marc,

Yes -- very important point. Thanks - I forgot to say that. Of course,
the comparison between posterior and prior is made easier if you have
been careful about choosing your priors (i.e. proper for a start, and
preferably not so diffuse that its hard to estimate the prior density
at parameter values of interest). I know you have previously made a
lot of noise about the need to use more informative priors in BEAST
analyses. I tend to agree. We should change some of the defaults to be
more informative, or not let BEAUti generate a BEAST script until the
user has specifically chosen a proper prior for every parameter :-)

Cheers
Alexei

[Nelson Fagundes]

Dear all,

 

Still on this subject, is there any way of comparing topologies using Beast outputs without relying on BF estimation? For instance, testing the monophyly of a given group of organisms? Or should one use a more traditional test based on likelihood, such as Shimodaira-Hasegawa?

 

Cheers

Nelson

 

2009/9/22 alexei <alexei....@gmail.com>

[alexei]

Nelson,

Since the prior probability of a particular clade (given a uniform
distribution on labeled histories) is (relatively) easy to compute,
the bayes factor for a particular clade is also easy to compute from
the prior and the posterior clade probabilities.

For example in BEAST, the prior probability of the (A,B) grouping in a
four taxa tree of (A,B,C,D) is 4/18 = 2/9, because 4 of the 18
possible labeled histories have the AB grouping:

Labeled histories with AB grouping:

((A, B),C),D)
((A, B),D),C)
((A, B):1,(C,D):2)
((A, B):2 (C,D):1)

Labeled histories *without* AB grouping:

((A, C),B),D)
((A, C),D),B)
((A, D),B),C)
((A, D),C),B)
((B, C),A),D)
((B, C),D),A)
((B, D),A),C)
((B, D),C),A)
((C, D),A),B)
((C, D),B),A)
((A, C):1,(B,D):2)
((A, C):2 (B,D):1)
((A, D):1,(B,C):2)
((A, D):2 (B,C):1)

(balanced labeled rooted trees of four taxa are represented by two
labeled histories, depending on which cherry is older)

Cheers
Alexei

On Sep 23, 1:05 pm, Nelson Fagundes <nrosa1...@gmail.com> wrote:
> Dear all,
>
> Still on this subject, is there any way of comparing topologies using Beast
> outputs without relying on BF estimation? For instance, testing the
> monophyly of a given group of organisms? Or should one use a more
> traditional test based on likelihood, such as Shimodaira-Hasegawa?
>
> Cheers
> Nelson
>

> 2009/9/22 alexei <alexei.drumm...@gmail.com>

[Andrew Rambaut]

I will add a couple of other points to this discussion:

If there are any constraints on the ages of nodes then the prior will
definitely not be uniform across labelled histories. In these cases
you will need to run the BEAST without data to obtain the prior
probability. The second point is that running the prior in BEAST may
require very long runs as the prior tree space is very large. The good
news is that it is much quicker to evaluate each state.

Andrew