Re: Cluster corrections for ROI analysis

[Thomas Nichols]

Hi Maria (CC: SnPM group),

Ah, I've had a little read and I can see that randomise t-contrasts are one sided.  Apologies, I'm quite new to the technique.

From what I'm reading on one sample t-tests in randomise I think our case might be slightly different though.  i'll try to be a bit clearer about what we were testing.

We are using a task which has, in previously published papers shown a strong baseline directional effect in one of the conditions of interest (trial type A > trial type B for simplicity).  Therefore, to confirm this in our placebo group we ran a one sample t-test (I assumed this was two tailed) on the placebo group data only.  In randomise I did this by setting up the design matrix to have only 1s, as I am interested in the directional effect in the lower level contrast (trial A - trial B).  Now, I assumed that this lower level contrast (trial A - trial B) tests for a significant difference in either direction (a positive or negative difference if that makes sense), each with 2.5% in each tail at a 5% overall significance level. 

No, I'm afraid that isn't the case.  In SPM, SnPM (and FSL) there are only one sided t tests; if you explicitly want a 2-sided t-test you must either run two separate analyses with complementary contrasts (e.g. 1 and then -1) and use half the threshold (e.g. FWE 0.025 instead of 0.05).  Alternatively, you can use a F-test that will test both directions, though that isn't in most of the SnPM plugins.

Therefore, I thought I could just do p/2 for a one tailed, directional test?  This may be statistically naive of me so any advice would be very welcome.  FYI, this effect is not our main effect of interest but just a replication for baseline which has been previously observed in other studies with larger samples so we feel the one-tailed test is justified, given our small sample

So, to restate, the p/2 is indeed a suitable solution, but for the two different tests.

Does this help?

-Tom

 

Thanks for all your time on this.

Best,

Maria

On Mon, 31 Oct 2016 at 01:34 Thomas Nichols <t.e.n...@warwick.ac.uk> wrote:

Hi Maria,

One last (i hope) question.

We care testing some replication baseline data as well as our manipulation effect.  To this end we have a strong directional hypothesis and want to look at a one-tailed test.  How would I do this with the randomise data?  Thresholding the t-stat image at .9 rather than .95?

I'm afraid your question isn't clear enough.  How exactly do you want to use the replication data?  Note that all randomise t-contrast inferences are one sided.   So if you want the other direction the best thing is to simply run it again with the sign-flipped contrast.

-Tom

 

Thanks,

Maria

On Mon, 24 Oct 2016 at 13:32 Thomas Nichols <t.e.n...@warwick.ac.uk> wrote:

Hi Maria,

Oh! You're using randomise!  In that case, whatever you do is valid and you're safe.  With randomise you're free to use voxel, cluster or TFCE, as you see fit while confidence of good false positive control.

Does this clear it up?

-TOm

On Mon, Oct 24, 2016 at 5:19 PM, Maria Ironside <maria.i...@psych.ox.ac.uk> wrote:

Hi Tom,

Thanks so much for your helpful replies.

I had one more question for clarification because I think I wasn't very clear in describing the ROI randomise analysis I had done in my previous email.

I ran randomise (two-sample paired t-test) with TFCE and a pre-thresholding mask (that's my amygdala ROI).  Is that what you are recommending below by voxel wise inference?  Or should I use the voxel based thresholding option in randomise instead of TFCE?  Does randomise (using these parameters) have the same problems as cluster correction with smaller ROIs?

Best,

Maria

On Mon, 24 Oct 2016 at 06:34 Thomas Nichols <t.e.n...@warwick.ac.uk> wrote:

Hi Jacinta,

As it's hard to think of an intervention that
wouldn't yield interindiv diffs in strength of effect,
it seems this issue will always arise.

In your talk I understood there were 2 alternative
ways to try and deal with this: 1) randomise 2) FEAT
with an adjusted p level that is more conservative.

That's right; a cluster forming threshold of 3.1 with FEAT should be fine.

 

In our study, we have strong a priori predictions that
our stim effect will be in the amygdala. So we have
analysed our data using FEAT (Flame) defaults combined
with a pre-threshold mask of the amygdala.

As the search region gets smaller and smaller, cluster size inference often loses power, simply because it is difficult for clusters to get large (they bump into the edge of the mask and essentially are truncated).  While permutation should suffer, the RFT can not do so well with this.  Thus for really small ROI's I usually recommend switching to voxel-wise inference.

 

Since we are confining our search to a single ROI (rather than whole brain)
I am wondering if we still need to adjust our analysis from previous FEAT defaults on foot of
this concern? If so, are you able to advise on how to do this?
If not we can ask in-house.

Without running a new raft of simulations/evaluations, I can't predict how an amygdala masked analysis is performing.  So all I can say is that if you were doing cluster inference with 2.3 cluster forming threshold, I'd be worried and would revisit those analyses to be sure.

Does this help?

-TOm

 

Many thanks for your prompt and very helpful replies.

Jacinta

Dr Jacinta O'Shea
Centre for Functional Magnetic Resonance Imaging of the Brain (FMRIB)
Nuffield Department of Clinical Neurosciences
University of Oxford
Email: jacint...@ndcn.ox.ac.uk  

Tel: +44 (0) 1865 222736

http://www.ndcn.ox.ac.uk/team/jacinta-oshea 

---

From: ten.p...@gmail.com [ten.p...@gmail.com] on behalf of Thomas Nichols [t.e.n...@warwick.ac.uk]
Sent: 22 October 2016 19:41
To: Jacinta O'Shea
Cc: Maria Ironside

Subject: Re: Cluster corrections for ROI analysis

Hi Jacinta,

I have a quick qst of clarification.
From what I understood of your recent FMRIB talk
on this, and from your response below, the problem
you have identified refers (only?) to the case of comparing
2 groups of participants. And that is the only case
tested so far -?

Yes.  It gets into the weeds though... one-sample FEAT/FLAME is *safe* under the total null hypothesis... i.e. if there's nothing there at all, all is well, and if anything things are a little conservative.  However, if there *is* some signal, that means there is both a true non-zero mean effect but there might also be some subject-to-subject variability in that effect, and thus FEAT/FLAME starts to behave like all the other tools and your significances may be inflated.  So... in terms of controlling false alarms when nothing is there, you're fine; but in terms of accurately reporting the strength of the effect, there could be some inflation.

In our study we have a *single sample of participants,
who undergo 2 fmri sessions on 2 different days (combined with real vs sham tdcs).
Our analysis compares across these sessions to test for a predicted effect of stimulation. 

So we perform a t test, but it's on a paired sample,
rather than across 2 independent samples.

Do the same issues arise in this case?

This is a paired t-test, which boils down to a one-sample t-test on differences, so it's basically in the "one sample t-test" camp.

So, "safe" against false alarms, but possibly juiced P-values when there is an effect.

Does this make sense?

-Tom

 

Thanks for your help

Jacinta

Dr Jacinta O'Shea
Centre for Functional Magnetic Resonance Imaging of the Brain (FMRIB)
Nuffield Department of Clinical Neurosciences
University of Oxford
Email: jacint...@ndcn.ox.ac.uk  

Tel: +44 (0) 1865 222736

http://www.ndcn.ox.ac.uk/team/jacinta-oshea 

---

From: Maria Ironside [maria.i...@psych.ox.ac.uk]
Sent: 22 October 2016 00:04
To: Thomas Nichols
Cc: Jacinta O'Shea
Subject: Re: Cluster corrections for ROI analysis

Hi Tom,

Thanks for your advice those months ago, I have reanalysed my results using randomise as you suggested and I had a few questions if you don't mind.

I would like some advice on how to report randomise results.  Because this controls for family wise error rather than cluster correction is it still appropriate to call the ROI results small volume corrected?  Also, should I be reporting the max t stat or the mean t stat?  I may be just a bit confused about the terminology here so advice appreciated.  I was struggling to find some relevant examples in the literature.

I have some nice amygdala results using randomise but I also have some marginal ones, which I am unsure how to report.  For example, for a specific contrast when when I use .94 as the cut off I get just 2 sig voxels but when I use .93 as a cut off I get 9 voxels. This may be statistically naive of me but previously I would use the significant blobs from the analysis to generate the masks from which to extract the values to plot.  So I'm not sure what to use for this as two voxels is tiny.  Should I use a sig level which generates a certain number of voxels to report marginal stats?  Much easier when it is p<.05!  

Thanks very much for your time.

Best,

Maria

On Fri, 19 Aug 2016 at 09:37 Thomas Nichols <ten.p...@gmail.com> wrote:

Dear Maria,

While I would recommend comparing these results to a nonparametic analysis, this analysis *should* be safe. It's complicated thought: The cluster inference procedure, using a low (t=2.3 equivalent to P=0.01) cluster forming threshold with SPM or FSL's OLS is exactly the one that showed bad behavior (inflated false positives).  With FSL Flame1 we found conservative behavior with total null data, due to an artifact of the way it estimates the between subject variance. For two sample t-tests on homogeneous task data (i.e. Effect present overall but no group difference) we found the same bad inflated false positives with Flame1.

So under complete null data, one sample t-test, FSL Flame1 is safe. But if you are so very unlucky as to have *mean* *zero* effect, but a non-zero between subject variance, you could also get inflated false positives with one-sample flame1.

And, finally, our results were specific to a brain analysis, but there are yet more assumptions/approximations when the flame1 parametric random field theory is used to find p-values with a reduced volume like you have done.

So! The answer basically is: Yeah, probably, maybe, it's fine, but why not rule out the potential reviewed criticism and try the analysis in randomise! If you need some help with this just let me know!

-Tom

 

On Aug 17, 2016, at 7:01 PM, Maria Ironside <maria.i...@psych.ox.ac.uk> wrote:

Dear Prof. Nichols,

Sonia Bishop advised that I should write to you and ask for advice regarding some fMRI data we are writing up, particularly following your recent excellent PNAS paper, challenging cluster based corrections.  I'm hoping you can help.

In our study, 16 female participants carried out an attentional control task in the scanner in a two session (active vs placebo), within subjects design.  We had an a-priori hypothesis about the effect of our intervention on amygdala activation in a specific task contrast (event related task design) and so we did ROI analysis in FSL by running a mixed effects analysis (Flame 1+2) and applying a pre-thresholding amygdala mask using the Harvard-Oxford structural atlas.  We used cluster thresholding, with a Z threshold of 2.3 and a cluster P threshold of  0.05.

We have found the expected effect in the amygdala but want to check our methods before submitting.  Would you say that using cluster correction in ROI analysis is a bad idea?  Any advice most appreciated.

Best,

Maria Ironside

__________________________________________________________
Thomas Nichols, PhD
Principal Research Fellow, Head of Neuroimaging Statistics
Department of Statistics & Warwick Manufacturing Group
University of Warwick, Coventry  CV4 7AL, United Kingdom

Web: http://go.warwick.ac.uk/tenichols
Email: t.e.n...@warwick.ac.uk
Phone, Stats: +44 24761 51086, WMG: 

--

__________________________________________________________
Thomas Nichols, PhD
Professor, Head of Neuroimaging Statistics
Department of Statistics & Warwick Manufacturing Group
University of Warwick, Coventry  CV4 7AL, United Kingdom

Web: http://warwick.ac.uk/tenichols

Email: t.e.n...@warwick.ac.uk
Tel, Stats: +44 24761 51086, WMG: +44 24761 50752
Fx,  +44 24 7652 4532

[Maria Ironside]

Thanks for getting back to me Tom,

I may be statistically naive here but isn't this just what I described?  Our analysis is one of the two separate analyses? The one with all 1s?

Therefore we could use a FWE of 0.025 in this case?

Apologies if I'm being a bit slow to get this and thanks so much for all your help.

Incidentally, I am using FSL (not SPM or SnPM) to run randomise.

Best,

Maria

[Thomas Nichols]

Maria,

Sorry not following.  Yes, if you have indeed run the two one-sided contrasts, -1 & 1, examining them both with the more stringent threshold 0.05/2 is indeed exactly what you need.

-Tom

--
You received this message because you are subscribed to the Google Groups "Statistical Nonparametric Mapping" group.
To unsubscribe from this group and stop receiving emails from it, send an email to snpm-support+unsubscribe@googlegroups.com.
To post to this group, send email to snpm-s...@googlegroups.com.
Visit this group at https://groups.google.com/group/snpm-support.
To view this discussion on the web, visit https://groups.google.com/d/msgid/snpm-support/CADtSoRS%2B7kN5pwzbfuF-auLVr4aYhTdKNuq72VtQXVprvxNUQQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Carlos Murillo]

Dear Prof Thomas Nichols and spm users, 

I have a follow-up inquiry in this old post on cluster correction for ROI analysis.

You mention that when small ROI's you usually recommend switching to voxel-wise inference because cluster size inference loses power. 

I am doing a VBM whole-brain analysis and for a secondary analysis using a mask of around 10-15% of the brain. For correction for multiple comparisons and cluster inference I am using TFCE toolbox from Prof Gasser.

For what I understand from your post, for the ROI analysis I should change to voxel-inference. Or that rationally refers to smaller masks?

Do you know of a study that demonstrates the lack of power for cluster size inference  as the search regions gets smaller?

Thank you so much for your time and consideration, 

Kind regards

Carlos