Different results running GSEA on the same data

[Mferreira]

Hello,

I ran a few GSEAs a couple of months back using the R package. Just today I decided to rerun the same analyses, on the same data, using the exact same script I wrote to automate the process and I keep getting different results. Seems like every time I run the analysis the outcome is different: in one run I get 16 categories, while the next I get half of that, with relatively little overlap between both.

Am I doing something wrong? How am I supposed to replicate my own results here?

[Lukas Behm]

Hello, I'm using the web version of WebGestalt (for a lack of coding skills ;-) and experienced kind of the same issue.

I ran a GSEA with following settings

- Organism: Mus musculus

- Databases: geneontology (BP, CC, MF) and pathway (KEGG, Reactome, WikiPathways)

- AnalyteType: Gene/Protein

- Upload: Gene Lists of 8 different conditions (ranked files), but tried them separately as well

The problem I encountered was I got different results (diff number of sign categories and a bit diff values in the tables), when I ran the same file with the same settings. (-> see screenshots as example)

What's the reason for that?

Is there a problem with my settings or is that just normal for the algorithm/calculation method?

Advanced Parameters

- Redundancy Removal: Weighted set cover (fast)

- Minimum number of analytes for a category: 3 (as its the lowest possible)

- Maximum number of analytes for a category: 2000 before, but now changed to 500 as I read the forum entries

- Significance Level: FDR 0,05 (when I ran all 8 conditions/list at once: FDR doesn't work and i have to select Top100)

- Number of permutations: 1000

- p: 1 (What is it? I do not find a explanation in the manual)

- Collapse Method: Mean

- Number of categories expected from set cover: 10

- Number of clusters (k) for k-Medoid: 10

- Number of categories visualized in the report: 40

- Color in DAG: Continous

I hope you can help me to find a reliable solution

Best

Lukas

[John Elizarraras]

This change in results is due to the nature of GSEA. GSEA uses random permutations to get a distribution which is used for calculating the statistical significance. This results in the effect you see here, where the results from GSEA are not consistent every run. The amount of variability can depend on a lot of factors, such as the size of the GO term, but the largest factor seems to be the input list size. A way to combat this is to modify the number of permutations in the advanced parameters section of the website, or if using the R package, the parameter is perNum. 2000 may be a better number and will provide a more consistent output.

One feature we are working is to have the ability to set the random seed so the results are consistent, but I think the difference in results you are observing can still be informative.

As far as the p variable Lukas mentioned, that is a variable used when calculating the enrichment score. The description from the documentation (https://bzhanglab.github.io/WebGestaltR/reference/WebGestaltR.html#arg-gseap) is: "the exponential scaling factor of the phenotype score. The default is 1. When p=0, ES reduces to standard K-S statistics (See original paper for more details).". I would leave this as 1, unless you have reasons to not want the ranking metric to affect the enrichment score, and only be used for sorting the list.

Each advanced parameter should have a help symbol that you can hover over to learn more about it.

Please let me know if you have any questions.

Best,

John