Group-level cluster-based permutation testing for correlations

[Petra Fischer]

Hello Mike,

First of all, thanks a lot for writing this incredibly helpful book!

I just wanted to make sure that I understood the procedure for cluster-based permutation tests across subjects on correlations correctly, as the example for figure 34.4 shows only the analysis within one subject.

If I have 15 subjects, I could run 1000 permutations, where I take the data from each subject, permute the RTs within each subject (i.e. randomly assign EEG power to RTs) and compute a linear least-squares fit for each pixel in the time-frequency map. It means I would need another loop to go through subjects within the permutation loop. Could I then compute t-values (following the logic of one-sample t-tests with 15 beta weights, one for each subject, as independent variable to assess whether they are significantly different from 0) and perform the clustering on this map of t-values?

I would then nevertheless plot Spearman's rho of my original data if I want to show the size of the correlations, and highlight only those pixels that were significant when comparing the original least-squares fit to the permuted one.

Would you say this is fine?

 

And finally, my last question: Would you also recommend me to report for how many of those 15 subjects the individual correlations were significant in a certain time-frequency window in addition to the evaluation on the group-level?

 

Many thanks for your help,

Petra

[Mike X Cohen]

Hi Petra. Do you mean you are performing within-subjects cross-trial correlations and then you want to see whether those brain-behavior correlations are consistent across the group of 15 subjects (thus, you have one correlation value per subject [per TF point, per electrode, etc.]), or do you mean you are performing individual differences analyses and you are correlating trial-averaged power with trial-averaged behavior at the group level (thus, you have one correlation value for the entire group)? 

If the former, you can follow the procedure in the book for each subject individually, and then at the group level you can test whether the correlation coefficients over all subjects are significantly different from zero. You can also perform within-subjects statistics and then bring the z-values (or t-values) to the group level to test against zero (this might be a little bit cleaner than the raw correlation coefficients). This will also allow you to examine the consistency of the effects within-subject. 

When you write "randomly assign EEG power to RTs" do you mean you randomly shuffle the mapping between RT and trial order? Or are you shuffling something else?

As for the visualization, you could plot either the raw correlation coefficients or the z/t-values. They should convergence fairly nicely. 

So basically: Yes, the procedure you describe sounds fine. 

Mike

--
You received this message because you are subscribed to the Google Groups "AnalyzingNeuralTimeSeriesData" group.
To unsubscribe from this group and stop receiving emails from it, send an email to analyzingneuraltimes...@googlegroups.com.
Visit this group at http://groups.google.com/group/analyzingneuraltimeseriesdata.
For more options, visit https://groups.google.com/d/optout.

--

Mike X Cohen, PhD
mikexcohen.com

[Petra Fischer]

Hi Mike,

Thanks a lot for your reply!

Yes, I meant shuffling the mapping between RT and trial order. And yes, I was looking to do the former, so your suggestion to compute the within-subjects statistics first should work well for me. Was that also exactly what you've done in the "Subthreshold muscle twitches dissociates... " paper with the RT and power?

I have only wondered whether I need to correct the multiple t-tests on the z-values (for each timepoint*frequency) at the group level even if I already correct the individual maps for multiple-comparisons. I think that was why I initially came up with the other idea, which avoids this question. I assume computing those t-tests should be okay because the maps already are thresholded?

I notice that you prefer using z-values (33.1 in the book) rather than counting the values more extreme than the observed value, where it would be possible to have a p-value of exactly zero. I have always preferred the method of adding 1 to both the nominator and denominator (e.g. 4+1/1000+1 in your example), which you didn't mention in chapter 33.4. Wouldn't that be a good option as well? I imagine that now and then distributions arising from permutations are not symmetric even when a large number of permutation samples are generated. Wouldn't that invalidate the z-value approach?

Thanks again,
Petra

To unsubscribe from this group and stop receiving emails from it, send an email to analyzingneuraltimeseriesdata+unsub...@googlegroups.com.

Visit this group at http://groups.google.com/group/analyzingneuraltimeseriesdata.
For more options, visit https://groups.google.com/d/optout.

[Mike X Cohen]

Hi Petra. Within-subjects and group-level analyses should be statistically thresholded separately. If you want to test whether the correlations are significant within-subject, you should threshold each subject's results map. But if you want to do group-level analyses, the non-thresholded statistics maps (z, t, or r) should be used, and each pixel is tested against zero. It would be awkward to have the single-subject thresholded maps at the group level, because you will be comparing non-zero to zero pixels. 

Keep in mind that the goals of within-subject vs. group-level statistics are different. Within-subjects analyses tell you about the amount of cross-trial variability relative to the effect size, whereas group-level analyses tell you about the consistency of the direction of the effect over subjects. If you have a consistent effect over subjects, but there is a lot of variability, you might get no within-subjects effects but strong group-level effects. Anyway, the main point is that you shouldn't use the thresholded single-subject maps in group-level analyses. If you want to do both, then you'll need to have access to both the thresholded and non-thresholded single-subject results.

As for computing the p-values: You are correct that conversion to z-statistic is interpretable only if the data are "sortof" normally distributed. Whenever there is a null hypothesis of 0 and the values can be negative or positive, then z values are very likely to be interpretable. This is the case, for example, with baseline-normalized power, correlations, and condition differences of nearly any metric. In these situations, I prefer z values over count-based p values, because z values are a commonly used and easily interpretable metric (standard deviation units away from the center of the null hypothesis distribution), whereas a p-value is harder to interpret on its own. I don't immediately see the advantage of adding a 1 to the p-value computation -- essentially you are adding an additional permutation that you fix to be suprathreshold -- but given enough permutations, it won't be detrimental. 

Mike

To unsubscribe from this group and stop receiving emails from it, send an email to analyzingneuraltimes...@googlegroups.com.

Visit this group at http://groups.google.com/group/analyzingneuraltimeseriesdata.
For more options, visit https://groups.google.com/d/optout.

--

Mike X Cohen, PhD
mikexcohen.com

--

You received this message because you are subscribed to the Google Groups "AnalyzingNeuralTimeSeriesData" group.

To unsubscribe from this group and stop receiving emails from it, send an email to analyzingneuraltimes...@googlegroups.com.

Visit this group at http://groups.google.com/group/analyzingneuraltimeseriesdata.
For more options, visit https://groups.google.com/d/optout.

[Petra Fischer]

Hi Mike,

Okay, thanks a lot, that clarifies everything. I have misunderstood what you have said before even though it should have been clear if I just would have considered your points.

Yes, I think, adding the 1 is just supposed to ensure validity by strictly avoiding p-values of exactly 0. This might happen in a pairwise comparison of reaction time means with 2*500 trials (from two conditions) that are permuted 1000 times, and let's say in all of those trials (or at least very many of them) condition A led to longer reaction times than condition B. It would be quite unlikely to obtain exactly the "original configuration" even when performing 1000 permutations. In this case any other permutation (shuffling one element in A<>B) would lead to a difference in means that is smaller than the originally obtained difference value, and ultimately lead to the nominator remaining 0. But practically, and especially with EEG data, this is unlikely to happen. 
I guess it is just personal preference but to me, p-values from permutation tests - as probabilities of observing such a data sample assuming that the null hypothesis is true - feel slightly more intuitive than z-scores, probably just because I don't need to picture the extent of one standard deviation first. In the case of p-values obtained from parametric tests that strongly depend on the sample size, I would also prefer z-scores as you do.

Thanks a lot for shedding light on the matter! Getting answers so quickly makes the book even more invaluable, I am already looking forward to read the chapters I haven't so far. It is such a good guide for performing goal-directed data exploration - I am often reminding myself of how you mentioned "analysis paralysis", which has definitely made me much more aware of periods when I have been just loosing time by trying around too much.

Best wishes,
Petra

To unsubscribe from this group and stop receiving emails from it, send an email to analyzingneuraltimeseriesdata+unsub...@googlegroups.com.

Visit this group at http://groups.google.com/group/analyzingneuraltimeseriesdata.
For more options, visit https://groups.google.com/d/optout.

--

Mike X Cohen, PhD
mikexcohen.com

--
You received this message because you are subscribed to the Google Groups "AnalyzingNeuralTimeSeriesData" group.

To unsubscribe from this group and stop receiving emails from it, send an email to analyzingneuraltimeseriesdata+unsub...@googlegroups.com.

Visit this group at http://groups.google.com/group/analyzingneuraltimeseriesdata.
For more options, visit https://groups.google.com/d/optout.

[Andrew Chang]

Hi Mike,

Thanks for your excellent book and this awesome discussion platform.

I want to ask a stats question, which is exactly the same as posted here: https://stats.stackexchange.com/questions/33933/group-level-correlation-effect

Also, I found that you have suggested "at the group level you can test whether the correlation coefficients over all subjects are significantly different from zero" in a post a few years ago (see below), which is essentially the method (b) in the link.

May I ask whether you have any references supporting the method (b)?

Best,

Andrew

To unsubscribe from this group and stop receiving emails from it, send an email to analyzingneuraltimeseriesdata+unsub...@googlegroups.com.

Visit this group at http://groups.google.com/group/analyzingneuraltimeseriesdata.
For more options, visit https://groups.google.com/d/optout.

[Mike X Cohen]

Hi Andrew. Options b and c in that stack link are good. I don't know if there is a specific reference for this approach. It's probably not fair to cite my own book ;)  But the reasoning is no different for a correlation than for a discrete condition effect. Essentially, you are are interested in whether some "effect" that you measure within each individual is consistent across individuals. That effect can be a condition difference or a correlation. Still, you obtain a measure of effect size and direction (e.g., a t-value or a z-value) per subject and then apply a statistic at the group level to make an inference (p<.05). 

The two advantages of using permutation testing on the within-subject correlation coefficient are (1) account for the empirical data distribution by comparing against the empirical null-hypothesis distribution, which is likely to be different for different TF-points, electrodes, and people; (2) the resulting z value will be normally distributed under the null-hypothesis whereas the correlation is not technically normally distributed. This makes parametric group-level statistics more valid.

For more general references about permutation testing, you might look up the Maris and Oostenveld 2007 paper (focused on M/EEG), or the Nichols and Holmes 2001 paper (focused on fMRI).

Hope that helps.

Mike

Visit this group at https://groups.google.com/group/analyzingneuraltimeseriesdata.

For more options, visit https://groups.google.com/d/optout.

--

Mike X Cohen, PhD
New online courses: mikexcohen.com

[Andrew Chang]

Thank you, Mike!