Permutation method not working properly for correlation analysis?

[Alica Rogojin]

Hi all,

I recently ran AFQ_MultiCompCorrection for two participant groups (let's say males and females) using 100 nodes.  I also ran it with 50 nodes to compare what I got for alphaFWE. 

100 nodes:

Females --> Chevrud gives 2.9658e-05 vs Permutation gives 7.1945e-05

Males --> Chevrud gives 2.9151e-05 vs Permutation gives 6.3568e-05

50 nodes:

Females --> Chevrud gives 5.9277e-05 vs Permutation gives 8.2081e-05

Males --> Chevrud gives 5.8251e-05 vs Permutation gives 6.9033e-05

These numbers don't make sense to me.  If I'm doing less comparisons (cutting them by half in the case of 50 nodes), I would expect my corrected alpha values to be much larger.  I get this with the Chevrud method, but not with the recommended permutation method.  Sometimes I get almost exactly the same value for alphaFWE using the permutation method.

Is there something wrong with the code when trying to do correlations?  I tried it with version 0.1 and 1.2, and got the same thing.  Or am I misunderstanding how the permutation method actually works and it makes sense to get alpha values that are very close even after cutting comparisons by half?

Best,

Alica

[Jason Yeatman]

I never actually tested this but what you report is exactly what would be expected. Neighboring nodes are highly correlated in fact, almost perfectly correlated. Therefor, a multiple comparison correction should not care how many nodes you put in but how many independent dimensions there are in the data. This is the exact goal and argument for permutation based approaches: you don’t have to make any assumptions but, rather, allow the distribution of the data to inform your correction. I’ve always said that sampling you fewer nodes should not improve your statistical power and that seems to be what you show

--
You received this message because you are subscribed to the Google Groups "Automated Fiber Quantification (AFQ)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to automated-fiber-quantif...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/automated-fiber-quantification-afq/0f3e3e7c-a45c-4e88-a7f5-3202db8f7e4e%40googlegroups.com.

--

Sent from a mobile device. Please excuse typos