Permutation test LOD thresholds and Bonferroni correction

[Mark Sfeir]

Hi,

I'm looking for some insight or references to reading materials to help me think about how to manage the multiple comparisons problem in a study where each of many phenotypes is being permutation tested to identify a significant LOD score threshold. Is it reasonable to do a normal Bonferroni correction by dividing the target alpha by the number of phenotypes tested? 

Another more general sense in which the multiple comparisons problem seems to show up here is in the sheer number of positions in the genome being tested for a QTL. I could use some guidance on how to interpret LOD scores and statistical significance in relation to the millions of candidate marker/pseudomarker positions in question. Should each of these candidate positions be considered a separate hypothesis for establishing a significance threshold? How do the qtl2 and other statistical tools manage these problems together? 

I'd appreciate input from anyone interested,

-Mark Sfeir

Research tech at UNC

[Dan Gatti]

The permutations that we do to assess significance thresholds are correcting for the multiple testing issue across the genome. We get the empirical distribution of maximum LOD scores under permutation and then get a “genome-wide p-value” from quantiles of that distribution. So I you map one phenotype and use permutations to assess your significance threshold, this is enough. Churchill & Doerge, 1994 is the reference that I use (https://pubmed-ncbi-nlm-nih-gov.ezproxy.jax.org/7851788/).

 

For multiple phenotypes, including eQTL mapping, you could use a Benjamini & Hochberg FDR.  A Bonferroni correction is safe, but may be too conservative. If you have 20 phenotypes and only one has a peak that crosses the permutation-derived threshold at an alpha of 0.05 (or 1 in 20), then you might treat that peak with suspicion.  But the FDR approaches seem to strike a good balance between power and Type I errors.

 

Dan

--
You received this message because you are subscribed to the Google Groups "R/qtl2 discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rqtl2-disc+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/rqtl2-disc/3475b50b-c48a-44ed-925f-5a92597c9726n%40googlegroups.com.

---

The information in this email, including attachments, may be confidential and is intended solely for the addressee(s). If you believe you received this email by mistake, please notify the sender by return email as soon as possible.

[Dan Gatti]

PS: q-value is another good FDR approach. It’s available through Bioconductor: https://www.bioconductor.org/packages/release/bioc/html/qvalue.html

 

From: rqtl2...@googlegroups.com <rqtl2...@googlegroups.com> On Behalf Of Mark Sfeir
Sent: Thursday, November 10, 2022 3:03 PM
To: R/qtl2 discussion <rqtl2...@googlegroups.com>
Subject: [SOCIAL NETWORK] [Rqtl2-disc] Permutation test LOD thresholds and Bonferroni correction

 

Hi,

--

You received this message because you are subscribed to the Google Groups "R/qtl2 discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rqtl2-disc+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/rqtl2-disc/3475b50b-c48a-44ed-925f-5a92597c9726n%40googlegroups.com.

[Mark Sfeir]

Thank you! I'll look into these. 

[Mark Sfeir]

I'm still planning to look at the Benjamini-Hochberg and q-value approaches to assessing significance, but just to make sure I know how to do the more conservative Bonferroni method right, a couple more questions here:

1) If only a subset of the total phenotypes measured in a study are considered interesting enough for permutation testing and assessment of significance, does that mean that the Bonferroni factor by which to divide the target alpha level would be only the number of phenotypes in that selection, or should the total number of phenotypes in the study be used regardless?

2) Maybe it is ok to use the smaller number of phenotypes in the subset as a Bonferroni factor only if we truly disregard the other phenotypes beforehand, as opposed to running the permutations on all phenotypes and then cherry-picking only those that will yield a significant result when using that smaller Bonferroni factor? 

-Mark