more on whole brain pattern significance

[annib]

Hi Chris,
So Graham and I were talking to John about how one could get a
significance inference for the R squared fit of PCs and he said that
you could do a leave one out testing to get a null distribution and I
was sure I remembered that you've already got that as one of your
choices. Would I be right in saying that I would choose option 11 to
get a p value for the behav_fit_rsq.

Cheers
Anna

[Christian Habeck]

I think so, but I have to check with my RA.

This behav_fit_rsq histogram would not be a null-distribution (since the bootstrap does NOT create null conditions but leaves the group assignment of allele status intact), but it would try to estimate the actual underlying distribution variability. It would give you a confidence interval for R^2, i.e. R^2 is bigger than 0.05 in 95% of the cases. it doesn;t really classify a p-value.

The p-value on the other would be more easily computable by doing an actual permutation test where null-conditions were created many times.

We have the group assignment as the dependent variable, we have the pattern expression as the independent variable Because no design matrix is used prior to the PCA, a permutation of group status would leave the PC structure totally invariant. This means we can just use the dependent and independent variable as they are now, permute the dependent variable randomly, run a regression and keep the F-value or R^2 value as statistic. Do this 10,000 times and see how it compares to the F or R^2 of the correct model.

So again,

independent variable = group status
dependent variables = subject expression of the particular PCs + 1 offset


Do regression

dependent variable = independent variables * beta + e

for the  point estimate and compute F.


Now do 10,000 times

dependent variable with perturbed rows = independent * beta' + e'

Compute F'


Look whether F falls in the tail of the F' distribution. If so, the result is significant.

If you give me the bolded variables, I can do this. Or John could do this I am sure.

Chris

-- 
Christian Habeck
ch...@columbia.edu
http://www.cumc.columbia.edu/dept/sergievsky/cnd/habeck.html

[Christian Habeck]

Hi ANNA,

Ok, sorry I had to look at the package to refamiliarize myself. YES,
option 11 should do what you want and the results should be similar to
what I had outlined before. we coujld do both do make sure everything is
kosher, i.e. you have good results and the software package works.

Also, my ramblings about the bootstrap are irrelevant here because this
option 11 would do a permutation test.


Chris

[annib]

Thanks Chris,
I just tried running it and got a matlab error so I haven't got any
results yet. Will let you know when I do :)

Anna

[Christian Habeck]

Hi,

BTW Tony update this part yesterday because the program unnecessarily performed a PCA for each iteration and then did the behavioral fit. The PCA step can be cut out, since a permutation will not change the results if no design matrix is used.

Let me know if there are some problems, then I can take a look.

Chris