Dear Luiz,
Thank you much for the Q and ur interest in paml!
Is it related to the Fig. S1D in
the SI pointed by sandra? If i didn't misunderstand anything, my gut feeling is that your Point 2 is correct that you might want to use an unrooted tree for testing a homogeneous model say M0. That said, a rooted tree for a foreground-background case, and an unrooted tree for M0 (and they're nested models so can use LRT). Perhaps u already know the following but pls forgive my verbosity, for the left tree in Fig. S1D, there r 2 omega values, plus 4 branch lengths to estimate, while the one on the right has 3 branch lengths plus one omega to estimate. That's why the d.f. diff is 4+2-(3+1)=2. Actually, you can find similar things in Table S5 in
Sandra et al. 2023 where except for test 4 all have a diff of d.f. equal to 1.
I actually don't know what will happen if you use a rooted tree for M0 but i guess the loglikelihood will be (almost) the same to using an unrooted tree, but the program might report one more parameters (and there could be some numerical problems such that the lnL might not be accurate). This tentative view, if true, would suggest that in Fig. S1D right, the program will report 5 free parameters but you should keep in mind that actually it's 4 (b/c in that case the model is not identifiable so the model that is specified has 4 free parameters despite the use of a rooted tree). As to an example of a non-identifiable model, again pls forgive my verbosity, imagine a case where one wants to minimize f(x,y)=(xy)^2 + 2xy , apparently x and y cannot be separatly estimated so one might want to reparameterize the model as z=x*y and so when z=x*y=-1 the min of f(x,y) is obtained, however in practice if you ask the code or program to estimate the MLE of x and y separately the computer will report sth, including probably the same likelihood, and the MLEs of x and y thus d.f. = 2 but actually there is only one free param to estimate, z).
Sorry for the above lengthy and probably a bit messy answer. Does it help? Feel free to let us know should u have further Qs. btw, I guess a clearer way is to upload your tree or a schematic graph of the situation so that other might know about it better. Thanks!
Dear Sandra,
Thanks much for your help! Btw, i might be mistaken but is the description of the rooted and unrooted tree in the 1st para under the section "Rooted versus unrooted trees" reverse?
best,
sishuo