about High FDR

[Giulia Gentile]

Hi all,

in GSEA user guide there is a sentence regarding the case of a gene set with a small nominal p value and a high FDR value: "On the other hand, the FDR is based on two distributions of all gene sets; if only one of many gene sets is significantly enriched, that gene set is likely to have a high FDR". What is the meaning of this sentence in terms of biological significance? Is there an higher probability of being a true enrichment compared to the other gene sets that are not enriched or it is still likely to be a false positive? 

Thank you in advance for your help.

Giulia Gentile

[Anthony Castanza]

Hi Giulia,

The FDR is a competitive test that effectively estimates how likely that set is to be the driving factor of your perturbation relative to the other sets tested, thus changing the gene set universe affects the FDR, it is not a simple multiple testing corrected pValue like many FDRs are. Could you provide the source for the quotation you're referencing here, I think it's not quite correct, but the context would help.

Generally speaking, a small NOM pValue means that, the genes in the set are significantly enriched relative to their self-null distribution (e.g. unlikely to be by chance), and a small FDR means that, among all the sets tested the small FDR sets were the strongest signal.

Does that make sense?

-Anthony

Anthony S. Castanza, PhD

Curator, Molecular Signatures Database

Mesirov Lab, Department of Medicine

University of California, San Diego

--
You received this message because you are subscribed to the Google Groups "gsea-help" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gsea-help+...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/gsea-help/5dada5cf-7a8b-4704-96d1-60688ba98da8n%40googlegroups.com.

[Giulia Gentile]

Hi Anthony,

Thank you for your answer.

You can find that sentence in the GSEA user guide (https://www.gsea-msigdb.org/gsea/doc/GSEAUserGuideFrame.html) at nominal p-value paragraph, as follows:

Nominal P Value

The nominal p value estimates the statistical significance of the enrichment score for a single gene set. However, when you are evaluating multiple gene sets, you must correct for gene set size and multiple hypothesis testing. Because the p value is not adjusted for either, it is of limited value when comparing gene sets. The Gene Set Enrichment Analysis PNAS paper describes the p value statistic in the section titled Appendix: Mathematical Description of Methods.

The FDR is adjusted for gene set size and multiple hypotheses testing while the p value is not. When a top gene set has a small nominal p value and a high FDR value, it generally indicates that it is not as significant when compared with other gene sets in the empirical null distribution. This could be because you do not have enough samples, the biological signal is subtle, or the gene sets do not represent the biology in question very well. On the other hand, the FDR is based on two distributions of all gene sets; if only one of many gene sets is enriched, that gene set is likely to have a high FDR. Finally, a top gene set with a high nominal p value and a low FDR value, generally indicates a negative result: the gene set itself is not significant and other sets are weaker.

In the GSEA report, a p value of zero (0.0) indicates an actual p value of less than 1/number-of-permutations. For example, if the analysis performed 100 permutations, a reported p value of 0.0 indicates an actual p value of less than 0.01. For a more accurate p value, increase the number of permutations performed by the analysis. Typically, you will want to perform 1000 permutations (phenotype or gene_set). (If you attempt to perform significantly more than 1000 permutations, GSEA may run out of memory.)

I am in doubt about the interpretation of that sentence.

Regards,

Giulia Gentile

[Anthony Castanza]

Hi Giulia,

Thanks for bringing this to my attention, the documentation there is misstating things slightly and needs to be updated, when it says a "low FDR" it means a "low ranked FDR", (e.g. a large FDR low on the ranked list of enrichment results). We should definitely clarify this to refer exclusively to "large" and "small" FDR values specifically.

-Anthony

Anthony S. Castanza, PhD

Curator, Molecular Signatures Database

Mesirov Lab, Department of Medicine

University of California, San Diego

To view this discussion visit https://groups.google.com/d/msgid/gsea-help/1a1cfe9a-e154-4e8f-9aa6-ef3ea53a9ba8n%40googlegroups.com.

[Giulia Gentile]

Hi Antony,

Thanks for clarifying the meaning of high and low in ranking, which should be replaced by large and small in value. So, if I understand well, the sentence  "if only one of many gene sets is significantly enriched, that gene set is likely to have a high FDR", should be changed to " if only one of many gene sets is significantly enriched, that gene set is likely to have a large FDR value".
In this peculiar case, for example, only one of many gene sets is significantly enriched (the only one has a nominal p-value <0.01 and all the others are greater than 0.1) and with a large FDR (FDR q-value > 0.26), could we take into account this result as a significant enrichment (FDR q-value with the smallest value among the gene set tested even if >0.26)?

Regards, 

Giulia Gentile

[Anthony Castanza]

 Hi Giulia,

Sorry for the delay getting back to you here.

I would probably consider the set a marginal hit; the FDR isn't a strict multiple testing correct of the pValue (note that we don't refer to it as a qValue as is done for "corrected" pValues). However, like I mentioned, the FDR can be affected by many things, among them the gene set universe tested. Without knowing a bunch of things about your data; how many samples you had in each phenotype, what permutation mode was used, how many genes in your dataset, how many sets were tested, I couldn't really say for sure.

-Anthony

Anthony S. Castanza, PhD
Curator, Molecular Signatures Database
Mesirov Lab, Department of Medicine
University of California, San Diego

To view this discussion visit https://groups.google.com/d/msgid/gsea-help/1f8dfe5a-ea70-4443-b7c2-89249d06699an%40googlegroups.com.

[Giulia Gentile]

Hi Anthony,

thank you for your help.

I had three biological replicates for the treated and three for the control samples, gene set as permutation type used, 16750 genes, and 50 gene set hallmarks.

Giulia

[Anthony Castanza]

Hi Giulia,

Without seeing some of the diagnostic plots myself it's hard to say for sure but generally your configuration seems okay. It's possible that the Hallmarks are just not a particularly good fit for your experiment. Have you tried any of the other more generally biologically applicable sets? Many of them are less perturbationally focused than the Hallmarks, but their breadth of coverage might make up for it. Something like Reactome/WikiPathways/KEGG_Medicus with their much broader coverage might end up having sets that have more signal in your data.

-Anthony

Anthony S. Castanza, PhD
Curator, Molecular Signatures Database
Mesirov Lab, Department of Medicine
University of California, San Diego

To view this discussion visit https://groups.google.com/d/msgid/gsea-help/850664e4-a818-4b87-8c17-3b25a12edcb8n%40googlegroups.com.

[Giulia Gentile]

Thank you, Anthony.

No, I haven't tried other gene sets.

This pair comparison (treated vs control) is the last time point of a time series, and I have obtained good results using the 50 hallmarks in the other time points. I have the NES vs. significance plot of the last time point for you here. The enriched gene set I was talking about was significantly enriched as upregulated in the first time point and then decreased till reaching these values at the last time point. For these reasons, I was in doubt whether to consider it or not. Giulia

[Anthony Castanza]

Hi Giulia,

I think it's going to be difficult to make a firm call for this time point. It certainly appears to be a strong enrichment and FDR is puzzling considering the lack of obvious competitor sets in the sets immediate neighborhood.

I might suggest plotting the enrichment scores of this set across your time series and seeing if there is an apparent trend.

You might also look at the overlaps in the core enrichment genes for this set at each of the time points. This tells you the specific genes that are driving the score, perhaps there are some cryptic sub programs in the set that are changing in the later time points and driving the reemergence of the enrichment.

Additionally, GSEA does have some support for direct analysis of time series data. You can construct a "continuous metric" CLS file (the formatting is a little different) then compute log ratios vs control and run GSEA with the cosine metric across your time points 

"Cosine is a variant of Pearson’s that is defined by Eisen et al. (1998). It is appropriate only when the expression profiles are based on log-expression ratios relative to a control." (GSEA User Guide)

Perhaps this would give some more insight into what's happening?

-Anthony

Anthony S. Castanza, PhD
Curator, Molecular Signatures Database
Mesirov Lab, Department of Medicine
University of California, San Diego

To view this discussion visit https://groups.google.com/d/msgid/gsea-help/52353932-46f7-45b7-ba92-892739d963d9n%40googlegroups.com.

[Giulia Gentile]

Hi Anthony,

 Thanks a lot for your feedback.

 Yes, there is an interesting trend for that gene set in the time series, and I have also found it among the gene sets positively correlated with the phenotype in the GSEA continuous analysis. Now I have a picture of those gene sets that are exclusive to a certain time point and those that show an involvement across the time series.

Again, thank you for your support.

Giulia