ROI analysis

[Georgette Argiris]

Hello! I have another question regarding just a simple ROI contrast analysis and was wondering if this is something that I can do in cosmo.

I was previously using Brainvoyager for the whole-brain contrast analysis but have selected five ROIs that I create as a mask on which I would

like to do a comparison between my conditions. So essentially, the mask is just a reduction of the voxels I am considering in order to reduce

the number of comparisons I'm making. 

I wanted to do this analysis in cosmo because I wanted to have control over the correction for multiple comparisons. But I'm wondering now

if this is something that makes sense to do in cosmo as I'm not doing any classification, just a comparison between my conditions at the 

individual GLM level that I would then like to combine at the group level and use TFCE.

Can you let me know if it makes sense to do this in cosmo?

Thank you!

Georgette

[Georgette Argiris]

Thinking more on this, I was wondering if this could be a simple possibility:

-per subject, I will have my GLM that has betas per condition. I could subtract my conditions (like a normal contrast analysis) and then at the group level,

I can combine these contrasts and do permutation testing against a chance level of 0 (a one-sample t-test like I would have done if the values that I was inputting

were correlation). 

[Nick Oosterhof]

On 11 August 2017 at 12:41, Georgette Argiris <georg...@gmail.com> wrote:

Thinking more on this, I was wondering if this could be a simple possibility:

-per subject, I will have my GLM that has betas per condition. I could subtract my conditions (like a normal contrast analysis) and then at the group level,

I can combine these contrasts and do permutation testing against a chance level of 0 (a one-sample t-test like I would have done if the values that I was inputting

were correlation). 

At the group level you could do a t-test probably. See cosmo_stat for doing stats directly at the group level (including a paired t-test, which is basically a contrast between betas from two conditions), without correction for multiple comparisons.

If you have multiple ROIs you may want to correct for multiple comparisons. Bonferroni is fine but conservative; less conservative would be using cosmo_montecarlo_cluster_stat with a 'singleton' neighbourhood definition where each feature is only neighbour of itself and all features have the same 'size'. If you want to pursue this avenue and need more details, please let us know. 

[Georgette Argiris]

Thank you! Yes I would like to try using cosmo_montecarlo_cluster_stat. I tried to look up singleton neighborhood but couldn't find anything. So it is a neighborhood that should have only itself as a neighbor right? Therefore it will be the same dimensions as sample?

Also, if I have multiple ROI masks per subject, how would I go about correcting them for multiple comparisons? Right now what I did was create a mask with all of my ROIs together, but what I would like to do is to have different ROI masks and compare between them.

How should these be loaded in order to compare?

Thank you again!

[Nick Oosterhof]

> On 11 Aug 2017, at 20:22, Georgette Argiris <georg...@gmail.com> wrote:
>
> Thank you! Yes I would like to try using cosmo_montecarlo_cluster_stat. I tried to look up singleton neighborhood but couldn't find anything. So it is a neighborhood that should have only itself as a neighbor right? Therefore it will be the same dimensions as sample?
> Also, if I have multiple ROI masks per subject, how would I go about correcting them for multiple comparisons? Right now what I did was create a mask with all of my ROIs together, but what I would like to do is to have different ROI masks and compare between them.

I've added some new code to define a neighborhood for ROI multiple comparison correction: https://github.com/CoSMoMVPA/CoSMoMVPA/pull/153
Unfortunately the testing system (travis) shows some issues for other parts of CoSMo at the moment, that's why it is not incorporated in the official master branch yet (hopefully that will happen soon though). But if you're impatient, you could try using the raw file from here:

https://raw.githubusercontent.com/nno/CoSMoMVPA/0f7bb90024012e142f2aa0c98f1613b114a7a9e7/mvpa/cosmo_singleton_neighborhood.m

and add it to the mvpa/ directory.

For ROI analysis, the idea is to compute an output value (correlation difference / classification accuracy) for each ROI and for each participant separately. So you would load the fMRI data using a mask for a single ROI for one participant, do MVPA, and store the output value. Repeat for each ROI and each participant. Assuming you are just interested in correlations above zero / classification above chance level a one-sample t-test is fine. To do so, just store the data in an N_participants x N_ROI .samples matrix and set .sa.chunks and .sa.targets appropriately The new cosmo_singleton_neighborhood.m gives an example in the help section, also with respect to multiple comparison correction.
Does that answer your question?

[Georgette Argiris]

Thank you for the code regarding the singleton neighborhood, I will try it out!

Regarding the ROI analysis, I don't plan on doing MVPA classification on this data. I just wanted to do a simple contrast

analysis in five different ROIs that would be corrected for multiple comparisons using permutation testing. 

What I ended up doing was creating one mask composed of all of my five ROIs, subtracting one condition from the

other (as one would do in a contrast analysis) and checking against chance (which would be 0- ultimately a one-way t-test 

where 0 would mean no difference) using cosmo_monte_carlo_stat. However, my concern was that 1.) theoretically, there may be

a difference in creating one big mask of my ROIs as opposed to single masks (one for each ROI) and controlling for multiple comparisons

in that way and 2.) that cosmo_monte_carlo_stat is supposed to be used for classification accuracies/correlations and not simple

contrast analysis where subtraction is used. Does this make sense?

Thank you!!

[Nick Oosterhof]

> On 16 Aug 2017, at 12:46, Georgette Argiris <georg...@gmail.com> wrote:
>
> Thank you for the code regarding the singleton neighborhood, I will try it out!
> Regarding the ROI analysis, I don't plan on doing MVPA classification on this data. I just wanted to do a simple contrast
> analysis in five different ROIs that would be corrected for multiple comparisons using permutation testing.
> What I ended up doing was creating one mask composed of all of my five ROIs, subtracting one condition from the
> other (as one would do in a contrast analysis) and checking against chance (which would be 0- ultimately a one-way t-test
> where 0 would mean no difference) using cosmo_monte_carlo_stat.

If you want to do univariate ROI analysis, in the usual case one computes the average response value over features (voxels) for each condition. If you have ROIs with, say, 100, 110, 120, 130, and 140 voxels for 3 conditions,
then first one would take the average in each ROI separately. Thus if the first ROI has ds.samples 100x3, use

ds_avg=cosmo_fx(ds,@(x)mean(x,2),{},2)

to compute the average, which is a 3x1 dataset.
You could then also compute differences between conditions, for example condition 1 minus condition 2:

ds_1vs2=cosmo_slice(ds_avg,1);
ds_1vs2.samples=ds_avg.samples(1,:)-ds_avg.samples(2,:)

where ds_1vs2.samples is a scalar.
Averages from multiple ROIs, if put in a cell ds_avg_cell can be combined into a single dataset using cosmo_stack(ds_cell,2)
Then averages from multiple participants can be stacked along the first dimension using cosmo_stack(...,1). After setting .sa.chunks and .sa.targets, cosmo_stat or cosmo_montecarlo_cluster_stat can be used.

> However, my concern was that 1.) theoretically, there may be
> a difference in creating one big mask of my ROIs as opposed to single masks (one for each ROI) and controlling for multiple comparisons
> in that way

See above - for univariate ROI analysis the typical approach is to compute the average response over voxels.

> and 2.) that cosmo_monte_carlo_stat is supposed to be used for classification accuracies/correlations and not simple
> contrast analysis where subtraction is used.

cosmo_montecarlo_cluster_stat is quite universal and can - with an appropriate neighborhood - also be used for ROIs and for simple univariate contrast group analysis.
Does this answer your questions?

[Georgette Argiris]

Thanks you so much! We will try by taking the average value over the entire ROI and use cosmo_monte_carlo_stat as you suggest and will also try 

with just one mask of all of the regions and do a voxel-by-voxel contrast with cosmo_monte_carlo_stat, using the singleton neighborhood that you provided.

[Julia Landsiedel]

Hi Nick,

I hope it’s okay that I’m reviving this thread. I used exactly what you described above to do multiple comparison correction across 4 ROIs after svm classification. Create the singleton neighbourhood with 4 ROIs and then used cosmo_monte_carlo_stat with TFCE and null data based on the actual sample data to do the multiple comparison correction.

I recently submitted a paper describing this method and I am really struggling with one of the reviewers who is criticising this method. As I don’t feel like an expert on TFCE, I was wondering whether you could kindly help me with the below:

“The applicability of TFCE to individual ROIs without any other neighbours still seems unclear. It is indeed possible to just apply the different "cluster thresholds" of TFCE to individual ROI values, but if there is no cluster extent to consider, it is not clear to me why TFCE would be preferable to just comparing the voxel/ROI values to the null distribution directly. That is, the TFCE integral will just be missing the cluster size term and only evaluate the "cluster height" at various h-levels rather than using the exact values. This additional set of arbitrary thresholds just has the effect of reducing the fidelity of the statistics compared to the raw values. While this reduced fidelity together with the thresholding at multiple arbitrary levels does make the statistics slightly more conservative if the delta h is relatively large, this does not seem to fit the definition of cluster correction, nor does the slight conservative bias have anything to do with multiple comparisons. Rather, it is just uncorrected statistics compared against a null distribution after reducing the precision. Please correct me if I am wrong, but this is how I would interpret the TFCE equations.
So, effectively, the TFCE statistic is probably fine for evaluating the uncorrected significance of the observed statistics, but I believe additional correction for multiple comparisons would still be required, e.g. with Bonferroni correction as with the other analyses.”

My rough understanding of this is, that the TFCE is looking for clustering of the four singleton neighbourhoods during the simulations of the null data, i.e. I have four ROIs, hence the cluster size is 4, and the z-values for each singleton neighbourhood from cosmo_monte_carlo_stat provide an indicator for which ROI clustered together with the others.

Any insight or additional explanation would be greatly appreciated.

Thanks,
Julia

[Nick Oosterhof]

Greetings,

> On 5 Apr 2023, at 15:38, Julia Landsiedel <landsied...@gmail.com> wrote:
>
> I hope it’s okay that I’m reviving this thread. I used exactly what you described above to do multiple comparison correction across 4 ROIs after svm classification. Create the singleton neighbourhood with 4 ROIs and then used cosmo_monte_carlo_stat with TFCE and null data based on the actual sample data to do the multiple comparison correction.
>
> I recently submitted a paper describing this method and I am really struggling with one of the reviewers

I assume the third reviewer, yes?

https://www.youtube.com/watch?v=-VRBWLpYCPY. ['Scientific Peer Review, ca. 1945’]

:-)

> who is criticising this method. As I don’t feel like an expert on TFCE, I was wondering whether you could kindly help me with the below:
>
> “The applicability of TFCE to individual ROIs without any other neighbours still seems unclear. It is indeed possible to just apply the different "cluster thresholds" of TFCE to individual ROI values, but if there is no cluster extent to consider, it is not clear to me why TFCE would be preferable to just comparing the voxel/ROI values to the null distribution directly. That is, the TFCE integral will just be missing the cluster size term and only evaluate the "cluster height" at various h-levels rather than using the exact values. This additional set of arbitrary thresholds just has the effect of reducing the fidelity of the statistics compared to the raw values. While this reduced fidelity together with the thresholding at multiple arbitrary levels does make the statistics slightly more conservative if the delta h is relatively large, this does not seem to fit the definition of cluster correction, nor does the slight conservative bias have anything to do with multiple comparisons. Rather, it is just uncorrected statistics compared against a null distribution after reducing the precision. Please correct me if I am wrong, but this is how I would interpret the TFCE equations.
> So, effectively, the TFCE statistic is probably fine for evaluating the uncorrected significance of the observed statistics, but I believe additional correction for multiple comparisons would still be required, e.g. with Bonferroni correction as with the other analyses.”

There is some merit in what the reviewer writes, but I would not agree that this analysis would only result in uncorrected significance. Indeed, if that were the case, then there is little benefit compared to just a standard univariate test.

The point about cluster-height with various h-levels should not matter as long as the dh (delta height) value is small enough. I may have to think a bit longer, but my first impression is that any value smaller than the smallest absolute differences across the ROIs should do. Or just try a tiny value such as 1e-4 or 1e-5, I would not expect results to differ much between these choices (since there are only 4 features, this seems computationally quite feasible; more so than in whole-brain analyses)

More in general, it may be confusing to consider this as a true TFCE analyses though. Instead, the cosmo_monte_carlo_stat with the TFCE analyses can be used to do a multiple comparison correction. The key difference with Bonferroni would be that in case of non-independence between the ROIs, Bonferroni is stricter than necessary (i.e. is more likely to result in false negatives). (The same argument would apply to whole-brain multiple comparisons, where Bonferroni correction is definitely too strict).

That said, if your results are strong enough to survive Bonferroni, then that may help in reassuring the reviewer that the results are not likely to be a false positive.

>
> My rough understanding of this is, that the TFCE is looking for clustering of the four singleton neighbourhoods during the simulations of the null data, i.e. I have four ROIs, hence the cluster size is 4, and the z-values for each singleton neighbourhood from cosmo_monte_carlo_stat provide an indicator for which ROI clustered together with the others.

This does not seem correct to me: the cluster size of each ROI is set to one, and since they are not connected, you can never find a cluster size of 4. The TFCE score for each ROI has a direct (monotonic) relationship with the strength of the effect in that ROI.