Gim statistic, model comparisions

[Hana Majerova]

Hi Ryan, 

 

I am using dadi-cli to generate my model data, successfully I hope! However, as statistics is quite new to me, I have some questions regarding the results. It would be very helpful to get some rules of thumb on how to handle the following:

Briefly, I am working with pseudo-diploidized, subsampled, and composite RadSeq data that has been cleared of paralogues. I have tested many models, and most yield similar results to the one appended here.

1. Uncertainty in theta: For all my parameters except theta, I get reasonable confidence intervals (CIs). Theta, however, is always 'unconfident' (wide CIs). Is this acceptable? I understand theta is used primarily to calculate real-world values (times, population sizes, etc.), but can I still trust the rest of my demographic parameters? Is there a detectable reason why theta is consistently so uncertain? I have seen this behavior even with datasets containing paralogues, as well as on projected and unlinked datasets.
2. Step Sizes: The confidence intervals for the parameters differ by an order of magnitude depending on the step size used. Is this normal? Should they be approximately the same, or is it okay to select one step size and present that as the result?
3. Small CIs: Sometimes the confidence intervals are so small they seem biologically unrealistic. Is that okay? I’ve heard this can be due to a 'mathematical collapse' when a likelihood peak is too sharp, but I am unsure how to interpret this.
4. Model Comparison: Is there a way to compare models generated by dadi-cli on composite data using their log-likelihoods? I know it should be possible to use CLAIC for non-nested models and LRT for nested models, but I cannot find a way to perform these in dadi-cli. Can these be inferred from the GIM (Godambe Information Matrix) results? If dadi-cli doesn’t support this directly, what is the best alternative?                 

Any suggestions or guidance would be greatly appreciated.

Thank you in advance and all the best,

Hana

[Ryan Gutenkunst]

Hello Hanna,

1. It is surprising to see that estimates with that large an uncertainty. It’s typically an easy parameter to estimate, since it just relates to the overall scale of the data.

2. You want to see consistency among the step sizes, to have confidence in the results.

3. That’s a sign of problem as well.

4. Dadi-cli does not have the composite nested LRT implemented right now, but is in the dadi Python version: https://dadi.readthedocs.io/en/latest/user-guide/likelihood-ratio-test/ 

If you have the computational resources, you can also just do a brute force bootstrap fitting of your model to many of the bootstrap data sets.

Best,

Ryan

--
You received this message because you are subscribed to the Google Groups "dadi-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dadi-user+...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/dadi-user/01c3d5c0-8484-4ba6-90e2-2837644512b5n%40googlegroups.com.
<founder_nomig_with paralogues.InferDM.bestfits.pdf><founder_nomig.Uncertainty_T1.txt><founder_nomig.bestfits.pdf><founder_nomig.InferDM.bestfits>

[Hana Majerova]

Hello Ryan, 

thank you for your answer. 

I understand that you wouldn't move forward with this statistic. 

Do you have any ideas about what might be wrong with my data or approch? It always outputs this same result, regardless of the filters or models I use. 

Best, 

Hana.

To view this discussion visit https://groups.google.com/d/msgid/dadi-user/F95E1ECD-F907-4439-98CA-2B1B93DB6884%40gmail.com.

[Hana Majerova]

HI Ryan, 

I append .bestfits results, as I think it might be helpful.

Best, 

Hana

[Ryan Gutenkunst]

Hello Hana,

Take a look at the bootstrap datasets generated as part of the analysis. They should be similar to the initial input data, both visually and in terms of total segregating sites. A large uncertainty in theta could be caused by some issue with the bootstrapping that yields some data sets with very few SNPs.

You might also try rerunning the uncertainty analysis using the —logscale option. That will enforce positivity of the parameters.

Best,

Ryan

To view this discussion visit https://groups.google.com/d/msgid/dadi-user/CAMkjdSHh7vzdZ6KrVA341ab6HjmTK3zxFCQeVDU6t%3DU%2BoaZCoA%40mail.gmail.com.
<founder_nomig.InferDM.bestfits>

[Hana Majerova]

Hi Ryan,
 
thank you very much for you hint, it helped a lot. I checked my bootstraps and I realized that my problem was actually a bug in dadi-cli and that it was already solved (https://groups.google.com/g/dadi-user/c/JtWWy7w5Mvo/m/4XDjjbxABQAJ) . So I just updated my dadi-cli version.
 
I wonder if I could bother you with one more question regarding theta and step sizes: would you consider the confidence intervals in the appended files reliable? I am still not 100% sure how to interpret them, but I would say that those generated for founder_nomig are not reliable, as they do not follow the decrease in step sizes in the same direction. Those for vic_no_mig seem more reliable, as they decrease consistently with decreasing step sizes (though the decrease is small).
I believe this should be acceptable and interpretable at a step size of 0.001, as the RADSeq data are robust, or is it not?  The theta CI should also be correct as it represents approximately 7% of the value estimated by the model.
Please, is my logic correct?
Thank you very much for your answers.
 
Best,
Hana

To view this discussion visit https://groups.google.com/d/msgid/dadi-user/85E9B372-E9FD-4634-9D89-07E486327C88%40gmail.com.

[Ryan Gutenkunst]

Hi Hana,

Regarding the uncertainties and step size, what you’re looking for is (rough) consistency among the step sizes. I don’t think these fit that bill for the first several parameters. Theta is well constrained in all the cases.

Best,

Ryan

To view this discussion visit https://groups.google.com/d/msgid/dadi-user/CAMkjdSGJkCqy58gv4hYYOBcMY-Y_-4vr8C62YPs_qCDtUjadPg%40mail.gmail.com.
<vic_no_mig_Uncertainty_T1_updated.txt><founder_nomig_Uncertainty_T1_updated.txt>