Any experiences with ascertainment bias on bootstrap support patterns?

[Grimm]

Hi,

for one of our next posts, I run a binary matrix coding for Dravidian languages with RAxML. These matrices (being the product of a pretty unique sequencing process) have nearly no invariant sites (like morphological matrices).
So I run for the fun of it the analysis with (-m ASC_BINGAMMA) and without (-m BINGAMMA) ascertainment bias correction.

The original paper on the ascertainment bias deals with branch-lengths and topology effects, is there anything known about how the correction affects bootstrap supports?

My guess was, it cannot matter too much, and indeed I found a quite high pearson correlation (0.94) between the two bootstrap samples when comparing all bipartition frequencies. But for individual alternatives the difference could be 20 or more, which strikes me quite a bit.

Overall, the corrected analysis appears to produce higher branch support than the standard analysis, and I started to wonder whether higher decisiveness is an expected, even wanted (?) by-effect of the ascertainment bias correction. Or, is there a risk that it masks internal data conflict? (Languages and morphologies are usually not the product of a strictly dichotomous process)?

Any thoughts welcomed.

Cheers, Guido

Image (at top, I'd loved it below, but doesn't work) shows the BS uncorrected vs. BS corrected for asc. bias plot

[Alexandros Stamatakis]

Hi Guido,

I think the first question to ask is how stable the ML tree is.

So, if you did, say 20 ML tree searches on random and parsimony starting
trees, how many distinct trees do you get and how different are they
from each other (i) topologically and (ii) statistically.

Also, what happnes if you run the BS analyses with the bootstopping
criterion?

It might well be that the signal itself is rather weak and as a
consequence what you are observing might be random effects.

That's what I'd try to analyse first.

Hope that helps,

Alexis

On 03.05.2018 17:39, Grimm wrote:
> <https://lh3.googleusercontent.com/-1It3cwY1kbk/WussaS78uxI/AAAAAAAAGP0/upAnZnrZ_XMYFw6BSmO9N_ABrnGoNLwPwCLcBGAs/s1600/PlotNormalBSvsABcorrected.png>

> --
> You received this message because you are subscribed to the Google
> Groups "raxml" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to raxml+un...@googlegroups.com
> <mailto:raxml+un...@googlegroups.com>.
> For more options, visit https://groups.google.com/d/optout.

--
Alexandros (Alexis) Stamatakis

Research Group Leader, Heidelberg Institute for Theoretical Studies
Full Professor, Dept. of Informatics, Karlsruhe Institute of Technology

www.exelixis-lab.org

[Grimm]

Hi Alexi,

naturally used the bootstop criterion (always do).
The ASC run fulfilled it after 900 replicates (WRF average of 1.43) being more decisive (?)
The uncorrected one used up the max.1000 without converging; WRF average of 2.33; a value in the usual range for (non-molecular) not overly tree-like matrices.

I think the first question to ask is how stable the ML tree is.

It's languages :) The signal from the matrix is not overly tree-like, trees are usually as instable as for morphological data.

In this particular case some terminal relationships are trivial, signal-wise (relatively young splits), the deeper relationships are chronically difficult to resolve (usually two or three equally valid alternatives). Even Bayesian analysis doesn't manage to converge to PP ~ 1 for most branches. Hence, my focus on the bootstrap results.

 

So, if you did, say 20 ML tree searches on random and parsimony starting
trees, how many distinct trees do you get and how different are they
from each other (i) topologically and (ii) statistically.

Will have to verify for this one, but usually you get (i) 20 somewhat different trees, aspect-wise, that (ii) are not significantly different using RAxML's in-built SH-test
 

It might well be that the signal itself is rather weak and as a
consequence what you are observing might be random effects.

I compared the result with the original analysis which used more or less complex linguistic substitution models in a Bayesian framework. The splits that received higher support from the ASC run, are the ones preferred by Bayes.

If it's randomness, would it make sense to re-run full analyses (tree inference + bootstrapping) from different seeds (for -x and -p), and compare the frequencies across the bootstrap runs?

Hypothesis being that variation let's say in 10 standard analyses and 10 corrected for the ascertainment bias should differ among each other as much as between each other, when it's random. Would 10 each be enough to discern between stochasticity and genuine? And should I change to standard bootstrapping rather than using the fast-implementation? I remember having read somewhere that even for molecular data, we'd expect a fuzzyness of about +-5 in the BS supports.

/G.

[Alexandros Stamatakis]

Hi Guido,

> naturally used the bootstop criterion (always do).
> The ASC run fulfilled it after 900 replicates (WRF average of 1.43)
> being more decisive (?)
> The uncorrected one used up the max.1000 without converging; WRF average
> of 2.33; a value in the usual range for (non-molecular) not overly
> tree-like matrices.

I see.

> I think the first question to ask is how stable the ML tree is.
>
>
> It's languages :) The signal from the matrix is not overly tree-like,
> trees are usually as instable as for morphological data.

That's what I guessed, and I did notice it's about languages :-)

> In this particular case some terminal relationships are trivial,
> signal-wise (relatively young splits), the deeper relationships are
> chronically difficult to resolve (usually two or three equally valid
> alternatives). Even Bayesian analysis doesn't manage to converge to PP ~
> 1 for most branches. Hence, my focus on the bootstrap results.
>
>
> So, if you did, say 20 ML tree searches on random and parsimony
> starting
> trees, how many distinct trees do you get and how different are they
> from each other (i) topologically and (ii) statistically.
>
>
> Will have to verify for this one, but usually you get (i) 20 somewhat
> different trees, aspect-wise, that (ii) are not significantly different
> using RAxML's in-built SH-test

Okay, that's also more or lesse what I'd expect.

> It might well be that the signal itself is rather weak and as a
> consequence what you are observing might be random effects.
>
>
> I compared the result with the original analysis which used more or less
> complex linguistic substitution models in a Bayesian framework. The
> splits that received higher support from the ASC run, are the ones
> preferred by Bayes.

Okay.

> If it's randomness, would it make sense to re-run full analyses (tree
> inference + bootstrapping) from different seeds (for -x and -p), and
> compare the frequencies across the bootstrap runs?

Yes, that's a good idea.

> Hypothesis being that variation let's say in 10 standard analyses and 10
> corrected for the ascertainment bias should differ among each other as
> much as between each other, when it's random. Would 10 each be enough to
> discern between stochasticity and genuine?

Id' do at least 100.

> And should I change to
> standard bootstrapping rather than using the fast-implementation?

Yes that would have been my next suggestion.

> I
> remember having read somewhere that even for molecular data, we'd expect
> a fuzzyness of about +-5 in the BS supports.

Never heard of that but sounds plausible.

We did some experiments with bootstraps for that ascertainment bias
paper, but as far as I remember they were somewhat unconclusive.

Alexis

>
> /G.

[Grimm]

Morning Alexi,

> If it's randomness, would it make sense to re-run full analyses (tree
> inference + bootstrapping) from different seeds (for -x and -p), and
> compare the frequencies across the bootstrap runs?

Yes, that's a good idea.

> Hypothesis being that variation let's say in 10 standard analyses and 10
> corrected for the ascertainment bias should differ among each other as
> much as between each other, when it's random. Would 10 each be enough to
> discern between stochasticity and genuine?

Id' do at least 100.

> And should I change to
> standard bootstrapping rather than using the fast-implementation?

Yes that would have been my next suggestion.

OK. Then I'll set it up accordingly at some point, and post the results.

For some reason morphologists and linguists seem to ignore ML (it's usually either MP or Bayes, and this is what one is asked for during review; just read a fresh simulation study testing for the effect of character dependency, again: MP vs. Bayes), but I find it the most appropriate tree-based approach regarding the primary signal in the data one has to work with.

Cheers, and thanks, Guido

[Alexandros Stamatakis]

Hi Guido,

> OK. Then I'll set it up accordingly at some point, and post the results.

That would be very nice.

> For some reason morphologists and linguists seem to ignore ML (it's
> usually either MP or Bayes, and this is what one is asked for during
> review; just read a fresh simulation study testing for the effect of
> character dependency, again: MP vs. Bayes), but I find it the most
> appropriate tree-based approach regarding the primary signal in the data
> one has to work with.

I agree, but there are some linguists who do, we are collaborating with
one :-)

Alexis

>
> Cheers, and thanks, Guido

[Grimm]

A little update (tiny observation) I made in this context

Am Montag, 7. Mai 2018 14:17:39 UTC+2 schrieb Alexis:

Hi Guido,

> OK. Then I'll set it up accordingly at some point, and post the results.

That would be very nice.

Have not found the time to do it (yet; even unemployed, I don't get bored). But I run both methods on the trivial (signal-wise) matrices for one of our network posts: https://phylonetworks.blogspot.com/2018/08/trivial-data-but-not-so-trivial-graphs.html

For the straightforward trivial case, a linear evolutionary sequence (A->B->C etc.), it makes little difference.

But it does for the tricky trivial case, the classic dichotomous sequence (A splits into B and C etc.) Some for the supports (different "para-clades" are preferred with expected split support pattern), and a lot for branch-lengths (see phylograms in the post).

No idea how this phenomenon scales up, was to be expected, or has any lesson to learn when inferring trees from non-molecular data sets with no invariable sites.