Change in meta analysis?

121 views
Skip to first unread message

vanessa sochat

unread,
Nov 8, 2014, 2:52:07 PM11/8/14
to neurosy...@googlegroups.com, Matthew Sacchet
Hi Tal,

Were there changes when you updated the neurosynth methods?  I noticed that the default threshold for meta analysis is now 0.01 (that's not actually an issue) - the problem I'm running into is that when I do a meta analysis with the same sets of IDS I had before, same threshold of 0.05, the entire image map is white (0 and -0 values).  I'm not really sure what is going on - it's definitely put me at a stop in my analyses for this weekend. Do you have any ideas?

I am doing:
ma = meta.MetaAnalysis(ns.db,members,q=float(fdr))
ma.save_results(outdir, prefix=outprefix, image_list=image_list)

Best,

Vanessa

--
Vanessa Villamia Sochat
Stanford University

Tal Yarkoni

unread,
Nov 8, 2014, 3:02:21 PM11/8/14
to neurosy...@googlegroups.com, Matthew Sacchet

Hi Vanessa,

The default FDR threshold changed to 0.01 a long time ago. The main reason for that is that I suspect that the chi-square p values are (slightly) anti-conservative relative to a full-blown permutation analysis (which is not feasible to run given the scale of the dataset and number of terms). So I lowered it to a level where that shouldn't be an issue.

Re: your discrepant analyses, that's harder to say. One easy way to determine if the meta-analysis code is working properly for you is to run an analysis on some "big" features that are on the website and compare them (e.g., memory, language, emotion, etc.). If you're using the latest dataset with default parameters, the local images should be identical to what's on the website. If they are, and you still get an empty image now whereas you didn't before, then most likely the earlier result was noise. If you were running with the wonky ~3,800-feature dataset produced by the parsing error, those results would definitely not be trustworthy (sorry!). If that's not the issue, then I'm not sure; nothing else should have changed...

Tal



--
You received this message because you are subscribed to the Google Groups "neurosynth" group.
To unsubscribe from this group and stop receiving emails from it, send an email to neurosynthlis...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

vanessa sochat

unread,
Nov 8, 2014, 3:04:18 PM11/8/14
to neurosy...@googlegroups.com, Matthew Sacchet
Ok - I'm doing this debugging right now, will report back soon! :)

vanessa sochat

unread,
Nov 8, 2014, 3:16:36 PM11/8/14
to neurosy...@googlegroups.com, Matthew Sacchet
Ok, I just tested for the term "anxiety" at a 0.01 threshold.  The only difference from above is getting the anxiety ids first:

ids=ns.db.get_ids_by_features("anxiety")

the pFgA FDR corrected map (right) is completely white, and the pAgF FDR 0.01 map (left) looks like it is too robust (note that I thresholded at 0 because it went down to -8 and looked really weird!), at least compared to the web interface.  What is the thresholding on the web maps?
maps.png

vanessa sochat

unread,
Nov 8, 2014, 3:17:55 PM11/8/14
to neurosy...@googlegroups.com
Here is the map without any thresholding (pAgF FDR 0.01)
nothreshold.png

Tal Yarkoni

unread,
Nov 8, 2014, 3:34:56 PM11/8/14
to neurosy...@googlegroups.com

Oh, the problem there is you're not passing a threshold when getting IDs in get_ids_by_features(). If you don't pass one, it retrieves all studies, which is why your map is empty--you're contrasting 9,700 studies against nothing. That's always been that way, but it's probably not ideal. I'll give it a default value of 0.001 to be consistent with other get_ids_by_* methods. Try again with get_ids_by_features('anxiety', threshold=0.001) and it should work fine.

Tal

vanessa sochat

unread,
Nov 8, 2014, 3:46:45 PM11/8/14
to neurosy...@googlegroups.com
Hi Tal,

Ok, the anxiety map looks good.  But I'm not yet convinced that every single set of my ids that previously had significant results is just "noise."  The images I have were produced with the (old code), however I didn't use the feature_table to get any IDs - I did my own text parsing to get custom sets.  The meta analysis (with the old code but the newest database) and FDR 0.05 produced maps with significant activation (this was yesterday morning, because I did pip install neurosynth --upgrade).  After that, the same process leads to the white maps for all pAgF images.  Can we talk about potential changes that would lead to this, or things to try, to rule out some bug on my end?

Best,

Vanessa

Tal Yarkoni

unread,
Nov 8, 2014, 5:57:59 PM11/8/14
to neurosy...@googlegroups.com

Just so I make sure I understand, are you saying that before the upgrade (a) you were getting non-zero results without passing a threshold argument to get_ids_by_features(), or that (b) for a custom analysis where you manually selected studies, you were getting non-zero results, which then went away following the upgrade?

Tal


priyanka gorijala

unread,
May 5, 2020, 9:50:46 PM5/5/20
to neurosynth
Hello I am doing something similar custom analysis, can you help me set FDR threshold 0.05 for meta analysis. I am not sure where should I specify that threshold.
To unsubscribe from this group and stop receiving emails from it, send an email to neurosy...@googlegroups.com.

Sudhakar

unread,
May 6, 2020, 1:06:56 AM5/6/20
to neurosy...@googlegroups.com
Can you post your code snippet.  That will be helpful. 

Sudhakar
Research scholar
IIIT-ALLAHABAD
https://brainnetworkresear.wixsite.com/brainnetworks

To unsubscribe from this group and stop receiving emails from it, send an email to neurosynthlis...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/neurosynthlist/dca83108-ffea-4aeb-9f0a-a1cb9f1a257e%40googlegroups.com.

priyanka gorijala

unread,
May 10, 2020, 10:46:16 AM5/10/20
to neurosy...@googlegroups.com
Hello,
Thanks for your reply. 
My code snippet:
import neurosynth as ns
from neurosynth.base.dataset import Dataset
from neurosynth.analysis import meta 
dataset = Dataset('/data0/research/neurosynth/neurosynth-data/database.txt') 
dataset.add_features('/data0/research/neurosynth/neurosynth-data/features.txt')
idfile = open('/data0/research/neurosynth/neurosynth-data/Ids.txt',"r") ## I gave list of pubmed IDs in IDs file
ids=idfile.readlines()
ma = meta.MetaAnalysis(dataset, ids, q=(float(0.05))) 
## I want to set FDR threshold of 0.05 for my meta analysis I used above code for that but it gave me a result which have p values=0 for all the data points.
So I am not sure whether I am doing it right or not. Hope you could help me. Thanks in advance for your help.
To unsubscribe from this group and stop receiving emails from it, send an email to neurosy...@googlegroups.com.

priyanka gorijala

unread,
May 10, 2020, 10:46:22 AM5/10/20
to neurosy...@googlegroups.com
Hello,
Thanks for your reply. 
My code snippet:
import neurosynth as ns
from neurosynth.base.dataset import Dataset
from neurosynth.analysis import meta 
dataset = Dataset('/data0/research/neurosynth/neurosynth-data/database.txt') 
dataset.add_features('/data0/research/neurosynth/neurosynth-data/features.txt')
idfile = open('/data0/research/neurosynth/neurosynth-data/Ids.txt',"r") ## I gave list of pubmed IDs in IDs file
ids=idfile.readlines()
ma = meta.MetaAnalysis(dataset, ids, q=(float(0.05))) 
## I want to set FDR threshold of 0.05 for my meta analysis I used above code for that but it gave me a result which have p values=0 for all the data points.
So I am not sure whether I am doing it right or not. Hope you could help me. Thanks in advance for your help.




On Wed, May 6, 2020 at 1:06 AM Sudhakar <rs...@iiita.ac.in> wrote:
Can you post your code snippet.  That will be helpful. 

Sudhakar
Research scholar
IIIT-ALLAHABAD
https://brainnetworkresear.wixsite.com/brainnetworks

Hello I am doing something similar custom analysis, can you help me set FDR threshold 0.05 for meta analysis. I am not sure where should I specify that threshold.

On Saturday, 8 November 2014 15:02:21 UTC-5, Tal Yarkoni wrote:

Hi Vanessa,

The default FDR threshold changed to 0.01 a long time ago. The main reason for that is that I suspect that the chi-square p values are (slightly) anti-conservative relative to a full-blown permutation analysis (which is not feasible to run given the scale of the dataset and number of terms). So I lowered it to a level where that shouldn't be an issue.

Re: your discrepant analyses, that's harder to say. One easy way to determine if the meta-analysis code is working properly for you is to run an analysis on some "big" features that are on the website and compare them (e.g., memory, language, emotion, etc.). If you're using the latest dataset with default parameters, the local images should be identical to what's on the website. If they are, and you still get an empty image now whereas you didn't before, then most likely the earlier result was noise. If you were running with the wonky ~3,800-feature dataset produced by the parsing error, those results would definitely not be trustworthy (sorry!). If that's not the issue, then I'm not sure; nothing else should have changed...

Tal



On Sat, Nov 8, 2014 at 1:51 PM, vanessa sochat <vso...@stanford.edu> wrote:
Hi Tal,

Were there changes when you updated the neurosynth methods?  I noticed that the default threshold for meta analysis is now 0.01 (that's not actually an issue) - the problem I'm running into is that when I do a meta analysis with the same sets of IDS I had before, same threshold of 0.05, the entire image map is white (0 and -0 values).  I'm not really sure what is going on - it's definitely put me at a stop in my analyses for this weekend. Do you have any ideas?

I am doing:
ma = meta.MetaAnalysis(ns.db,members,q=float(fdr))
ma.save_results(outdir, prefix=outprefix, image_list=image_list)

Best,

Vanessa

--
Vanessa Villamia Sochat
Stanford University

--
You received this message because you are subscribed to the Google Groups "neurosynth" group.
To unsubscribe from this group and stop receiving emails from it, send an email to neurosy...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "neurosynth" group.
To unsubscribe from this group and stop receiving emails from it, send an email to neurosynthlist+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "neurosynth" group.
To unsubscribe from this group and stop receiving emails from it, send an email to neurosynthlist+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/neurosynthlist/CAPVJSddYemcN4fKB07rN9gaF2T8iDnfcZ_XkLVWAidZNqQtBvw%40mail.gmail.com.

Tal Yarkoni

unread,
May 11, 2020, 4:35:02 PM5/11/20
to neurosy...@googlegroups.com

Hi,

The Neurosynth Python core tools are basically deprecated at this point; they're still available, but I'm not actively maintaining them. I'd recommend looking into NiMARE (https://github.com/neurostuff/NiMARE), which does most of what Neurosynth did, but also much more.

Best,

Tal

On Sun, May 10, 2020 at 9:46 AM priyanka gorijala <priyanka...@gmail.com> wrote:
Hello,
Thanks for your reply. 
My code snippet:
import neurosynth as ns
from neurosynth.base.dataset import Dataset
from neurosynth.analysis import meta 
dataset = Dataset('/data0/research/neurosynth/neurosynth-data/database.txt') 
dataset.add_features('/data0/research/neurosynth/neurosynth-data/features.txt')
idfile = open('/data0/research/neurosynth/neurosynth-data/Ids.txt',"r") ## I gave list of pubmed IDs in IDs file
ids=idfile.readlines()
ma = meta.MetaAnalysis(dataset, ids, q(float(0.05))) 
## I want to set FDR threshold of 0.05 for my meta analysis I used above code for that but it gave me a result which have p values=0 for all the data points.
So I am not sure whether I am doing it right or not. Hope you could help me. Thanks in advance for your help.



Priyanka. G
Research Assistant
IUPUI- Indianapolis


On Wed, May 6, 2020 at 1:06 AM Sudhakar <rs...@iiita.ac.in> wrote:
Can you post your code snippet.  That will be helpful. 

Sudhakar
Research scholar
IIIT-ALLAHABAD
https://brainnetworkresear.wixsite.com/brainnetworks

On Wed, 6 May 2020, 07:20 priyanka gorijala, <priyanka...@gmail.com> wrote:
To unsubscribe from this group and stop receiving emails from it, send an email to neurosynthlis...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "neurosynth" group.
To unsubscribe from this group and stop receiving emails from it, send an email to neurosynthlis...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "neurosynth" group.
To unsubscribe from this group and stop receiving emails from it, send an email to neurosynthlis...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/neurosynthlist/CAFPSfNc-KoqZ67a%3D_uVbN1MwMyURw9zcGtiNKTkWzZ%2BZ9r8okw%40mail.gmail.com.

priyanka gorijala

unread,
May 12, 2020, 9:18:50 AM5/12/20
to neurosynth
Thank you so much I will give it a try.
To unsubscribe from this group and stop receiving emails from it, send an email to neurosy...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "neurosynth" group.
To unsubscribe from this group and stop receiving emails from it, send an email to neurosy...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "neurosynth" group.
To unsubscribe from this group and stop receiving emails from it, send an email to neurosy...@googlegroups.com.

priyanka gorijala

unread,
Jun 23, 2020, 3:58:21 PM6/23/20
to neurosy...@googlegroups.com
Hello,
Can someone please help me how to visualize the list of MNI coordinates ( in csv file format) which I extracted from meta analysis. I can't use neurosynth webtool to do this because my file format is not .nii.
I tried using MRIcron but it only allows me to visualize one row at a time but I want to visualize ALL the coordinates and see in which regions they are.
Priyanka. G
Research Assistant
IUPUI- Indianapolis

To unsubscribe from this group and stop receiving emails from it, send an email to neurosynthlis...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/neurosynthlist/c2f222e8-b590-41e9-8005-da54058258aa%40googlegroups.com.
Reply all
Reply to author
Forward
0 new messages