Comparing one sample t-test output in SnPM and randomise

[Hill, Donal]

Dear experts,

I have analysed a set of 31 difference images (the difference calculated between two conditions) using a one sample t-test. I have implemented the t-test both within SnPM and using FSLs randomise option with “-c”.

The options I specify in SnPM are shown in the attached screenshot. 

 

I have fed in the 31 different images as input, and I subsequently analyse the output using the following options in the “Inference” batch.

Meanwhile, in FSL, I feed in precisely the same difference images as a single 4D nii file. I run randomise with the following options:

randomise -m mask -i data -o t_test_results -v 8 -c 3.1 -R -n 5000 -N -1 

where “mask” is a binary mask to exclude voxels outside the brain, and “data” is my 4D nii file containing the same 31 images that I feed into SnPM.

I subsequently run an instance of FSLs “cluster” on the to extract the clusters found by randomise, where I define the threshold to be “ -t 0.95" in line with the p < 0.05 SnPM requirement above. I pass the “clustere_corrp_tstat1” output file from randomise to the cluster function.

I find a single significant cluster in the data (the same one is found by both SnPM and randomise), but the number of voxels in the cluster and the cluster p-values reported by both method are somewhat different.

In SnPM :        Cluster size = 2579 voxels    ,   FWE corrected p = 0.0370

In FSL :           Cluster size = 2722 voxels    ,   FWE corrected p = 0.0300

I would like to try and understand the source of the difference, and whether it is due to something I have done wrong using either method. I think the number of permutations performed is sufficiently large that this is not due to an under-defined null distribution. Any help would be very much appreciated!

Kind regards,

Donal

[Hill, Donal]

My apologies, I quoted incorrect values for the FSL results below. The correct values are:

SnPM : Cluster size = 2579 voxels , FWE corrected p = 0.0370

FSL : Cluster size = 2709 voxels , FWE corrected p = 0.0330


So the two softwares seem to be close but not in complete agreement.

> On 7 Apr 2016, at 11:52, Hill, Donal <donal...@kcl.ac.uk> wrote:
>
> Dear experts,
>
> I have analysed a set of 31 difference images (the difference calculated between two conditions) using a one sample t-test. I have implemented the t-test both within SnPM and using FSLs randomise option with “-c”.
>
> The options I specify in SnPM are shown in the attached screenshot.
>

> <Screen Shot 2016-04-07 at 11.39.41.png>

>
>
> I have fed in the 31 different images as input, and I subsequently analyse the output using the following options in the “Inference” batch.
>

> <Screen Shot 2016-04-07 at 11.41.17.png>

[Thomas Nichols]

Dear Donal,

The slight difference is due to the different ways that SnPM/SPM and FSL define clusters.  SnPM and SPM use an order 18 neighborhood (see spm_clusters.m line 22), while FSL's randomise uses an order 26 neighborhood (CLUST_CON variable in randomise.cc).

Have you checked the voxel-wise inference results?  Way back when we benchmarked randomise against SnPM and got the same answers for voxel-wise, so you still should see the same thing for simple models.  For more complex models, i.e. with nuisance covariates, there are subtle differences between the way that SnPM and randomise handles the nuisance that may give rise to differences.

-Tom

--
You received this message because you are subscribed to the Google Groups "Statistical Nonparametric Mapping" group.
To unsubscribe from this group and stop receiving emails from it, send an email to snpm-support...@googlegroups.com.
To post to this group, send an email to snpm-s...@googlegroups.com.
Visit this group at https://groups.google.com/group/snpm-support.
To view this discussion on the web, visit https://groups.google.com/d/msgid/snpm-support/7C5EBEFE-326D-4868-98FB-2453029A2AB3%40kcl.ac.uk.

For more options, visit https://groups.google.com/d/optout.

--

__________________________________________________________
Thomas Nichols, PhD
Professor, Head of Neuroimaging Statistics
Department of Statistics & Warwick Manufacturing Group
University of Warwick, Coventry  CV4 7AL, United Kingdom

Web: http://warwick.ac.uk/tenichols

Email: t.e.n...@warwick.ac.uk
Tel, Stats: +44 24761 51086, WMG: +44 24761 50752
Fx,  +44 24 7652 4532

[Hill, Donal]

Hi Tom,

Thanks so much for your quick reply! Could I ask what the neighbourhood order is, and why the specific values are chosen? The goal of my comparison was to see what you get “out of the box” so to speak using both softwares, so I would be hesitant to alter the default settings for either method. Does a higher order neighbourhood imply better performance?

 

I’ve not done any voxel-wise inference comparisons so far, but I could certainly check to see that I see the same results there. 

I have also checked randomise using “-T” and compared it to the SPM TFCE implementation, and find larger discrepancies there. I know that you did not author the TFCE implementation for SPM, but I’m also unsure why those particular differences arise. Perhaps it is also due to this neighbourhood order difference. 

I’m ultimately just trying to understand if the differences I see are due to a) me messing something up in the analysis or b) an underlying difference in setup behind the scenes. I can say, however, that the cluster sizes found by the parametric GLM and FLAME1 in my work agree entirely, albeit with slightly different (and both very non-sensible looking!) p-values.

Cheers,

Donal

[Thomas Nichols]

Hi Donal,

Thanks so much for your quick reply! Could I ask what the neighbourhood order is, and why the specific values are chosen? The goal of my comparison was to see what you get “out of the box” so to speak using both softwares, so I would be hesitant to alter the default settings for either method. Does a higher order neighbourhood imply better performance?

No, just about how you define 'connectivity' (see http://nidm.nidash.org/specs/nidm-results_120.html#dfn-nidm_0000080 ).  Once you threshold a map, you still need some rule to define what are clusters, and this tells you how to do it.

I’ve not done any voxel-wise inference comparisons so far, but I could certainly check to see that I see the same results there. 

I have also checked randomise using “-T” and compared it to the SPM TFCE implementation, and find larger discrepancies there. I know that you did not author the TFCE implementation for SPM, but I’m also unsure why those particular differences arise. Perhaps it is also due to this neighbourhood order difference. 

Aside from neighborhood connectivity, there is the issue of the "dt".  Randomise uses dt = Max(T)/100, where Max(T) is in the test statistic image (unpermuted).

I’m ultimately just trying to understand if the differences I see are due to a) me messing something up in the analysis or b) an underlying difference in setup behind the scenes. I can say, however, that the cluster sizes found by the parametric GLM and FLAME1 in my work agree entirely, albeit with slightly different (and both very non-sensible looking!) p-values.

Hmmm... well... hopefully SnPM is more sensible!

-Tom

 

To post to this group, send email to snpm-s...@googlegroups.com.

Visit this group at https://groups.google.com/group/snpm-support.

To view this discussion on the web, visit https://groups.google.com/d/msgid/snpm-support/6B7497C7-770C-43BD-A5A9-AE7E9BC06095%40kcl.ac.uk.

For more options, visit https://groups.google.com/d/optout.