Hi Rutger,
while RAxML allows for multifurcating constraints I don't think it can do nested, gold tree-like constraints.
Defining an outgroup only roots the tree for post-inference interpretation, it has no effect on the topology optimisation. To be able to root a tree with a multi-tip outgroup, we assume that in- and outgroup are reciprocally monophyletic, i.e. the tree has a branch (internode)/ supports a taxon bipartition ingroup | outgroup; to enforce this we would in your example use the following constraint
(A,B,C,D,E) [or (O1,O2,O3)
If the outgroup taxa don't have that quality, and, in an unconstrained analysis some of them mingle with part of the ingroup, it cannot be used to root the ingroup tree. Similarily, any outgroup sample that changes ingroup relationships may be problematic.
But the question I have is why you want to enforce a gold tree [
(((A,C),E),F) ] during analysis at all?
If your primary interest is to test where B and D connect to the already known backbone topology, you can just use the "evolutionary placement algorithm" (EPA) reading in the known topology as guide tree and testing any to place tip as query.
For the example, we read in
(((A,C),E),F)Ox as reference tree, an alignment including all tips (i.e. incl. B and D as well) and test B and D as queries. EPA will then give you a ML score for where B and D best-place (and 2nd-, 3rd-best...) within this reference tree.
Since you want to have your tree rooted, including the outgroup closest to the ingroup will be enough to see if B or D would be early branching sisters or nested within the ingroup. Note that ML has a 50:50 chance to escape long-branch attraction, i.e. a long branching ingroup query may be attracted by a too distant outgroup, or outgroups.
EPA is implemented in classic RAxML but also available as stand-alone tool for very large query numbers: EPA-ng
If your data doesn't support well enough the gold tree (we used EPA for instance to place fossils within a molecular-based tree; morphological data usually reflect phylogenetic relationships imperfectly), but you want to keep it for e.g. trait mapping purposes but with B and D, EPA would also be the tool to place B and D within your gold tree, and then you just take this topology, feed it into RAxML and optimise only the branch-length and model(s).
Cheers, Guido.