Hi Nick,
first, you cannot have a poorly supported "node" in your backbone tree, only a poorly supported branch. When we establish e.g. bootstrap support, we count taxon bipartitions, which are represented by internodes in phylogenetic trees. When using a backbone constraint in RAxML, we do the same: we define taxon bipartitions.
Phylogenetically it makes hence no sense to constrain a hard polytomy, because a phylogenetic tree cannot have one: it's a dichotomous graph – a node is the connective (vertex) of three internodes per definition, not four.
Let's say you have four groups of taxa: A, B, C, D – the latter three making your polytomy: the only possible background constraint would be A | B + C + D, RAxML would then infer only trees with A | BCD split, and optimise either AB | CD, AC | BD or AD|BC as subsequent splits. Rooting with A following the prior constraint, either B, C, or D would be sister to the other two.
In other words, if you want to resolve the BCD polytomy, you define all three splits as backbone constraints, and then, e.g. to a topology test, which one outperforms the other two or if they are equally probable.
Note that there are two kinds of poorly supported branches:
- indicisiveness: there's little signal in your data to resolve this part of the tree. Constraining one alternative wouldn't hurt, but doesn't help either. There's simply no signal in the data to resolve this soft polytomy.
- internal signal conflict: eg. some part of the data prefers a AB | CD split, another part a AC | BD split: you get low support, because it's split between the two alternatives.
Pending on the signal in your matrix, your inferred tree may actually show a AD | BC split, although it's the worst supported alternative of all three. Because internal signal conflict may inflict branching artefacts since I force incongruent genealogies in a single tree.
To see which is the case for your 3-way polytomy, just read in the
RAxML bootstrap sample in SplitsTree and check out the consensus
network.
An example from bears: (a), (b) and (c) are genuine genealogies based on Y-Chromosome (paternal lineages), mitochondrial coding gene regions (maternal lineages) and nuclear-encoded autosomal introns (inherited from both parents). Bears are solitary, the females are prone to not migrate, the males like too. The species are only semi-biological species and their ancestors may not have been at all (e.g. polar bears and brown bears produce interfertile offspring). All these tree are true trees but incompatible in aspects.
If I combine the data, the support collapses for some branches (clades in Giand Panda rooted trees), and I get the poorly supported false red clade: a sister relationship between Sloth and Sun Bear not supported by in any of the individual three data sets that were combined.
And here's how the RAxML bootstrap replicate sample consensus network looks like for the combined data, and with the (a), (b), (c)-individual supports mapped.