Advice on directionality and cosmo_stat outputs

[Carolyn McGettigan]

Dear CoSMoMVPA list

Thanks for adding me to the group. I've been running searchlight RSA on fMRI data and have been encountering issues in interpreting the directionality of effects in the outputs of cosmo_stat.

In brief: When running a group paired t test in cosmo_stat, how does cosmo_stat determine which condition (target) is the reference and which the comparison? I'm particularly interested in what should be done about assigning values to the 'targets', because these have an effect and not apparently in the intuitive direction (I think).

The background to this specific query is below, if that helps!

Best wishes

Carolyn

-------------

Details:

I ran a group ANOVA to compare observed neural dissimilarity scores across 3 conditions in my experiment. For each participant, I ran a searchlight analysis to extract the observed neural dissimilarity for all possible comparisons of items in my expt (3 conditions A, B, and C x 8 items each: 276 comparisons in the RDM). From these data, I separately extracted the mean dissimilarity value per searchlight volume for within-condition item comparisons per condition - i.e. for condition A, I calculated the mean of distances A1-A2, A1-A3... A7-A8; and similarly for condition B, and condition C). This resulted in 3 maps per participant, each containing mean dissimilarity values per searchlight for 1 of the conditions A, B, or C.

I stacked the data with ds_stack and applied cosmo_stat to run an ANOVA (1 chunk per ppt (N=26); target values 1 = condition A, 2 = condition B, 3 = condition C).

The analysis showed significant results, and plotting out the dissimilarity values for peak coordinates showed the trend as A > B > C. I decided to break this down using paired t tests in cosmo_stat (using stat_map=cosmo_stat(ds_stack, 't', 'z')), to check if A > B, and B > C. At this point I get apparently conflicting results e.g. when I weight A as +1 and B as +2, I get positive z values suggesting B > A in most searchlights. But when I plot out the peaks from the t test, I see that the observed dissimilarity data from ds_stack show in fact A > B (in line with the peaks from the ANOVA).

So maybe the trouble lies with the values being assigned to the targets in the paired t test - I would like to have some clarity on that.

[Nick Oosterhof]

Greetings,

At the moment I have limited computer access, yet I hope my reaction below is helpful:

On 16 Apr 2024, at 17:11, Carolyn McGettigan <mcgettiga...@gmail.com> wrote:

Thanks for adding me to the group. I've been running searchlight RSA on fMRI data and have been encountering issues in interpreting the directionality of effects in the outputs of cosmo_stat.

In brief: When running a group paired t test in cosmo_stat, how does cosmo_stat determine which condition (target) is the reference and which the comparison? I'm particularly interested in what should be done about assigning values to the 'targets', because these have an effect and not apparently in the intuitive direction (I think).

I cannot try this out myself at the moment. 

Could you try to construct a minimal example dataset (and some matlab code) that illustrates the behaviour of cosmo_stat that you find confusing? A dataset with one feature (column in .samples) would be good. The number of participants (unique targets) could be equal to what you have in the real data, or potentially less. You could use either real or synthetic data.

[Carolyn McGettigan]

Dear Nick

Thanks for your quick response! I hope my reply here will help clarify.

I used cosmo_synthetic_dataset to create a dataset containing 6 features (voxels) and resembling my own group dataset (N=26 participants; 2 within-subject conditions). I've named the dataset 'ds_stack' (struct attached), as if it were the stacked group data in my experiment. Pasted values for ds_stack.sa.targets, ds_stack.sa.chunks, and ds_stack.samples are below.

In ds_stack, the first 26 rows of .samples, .sa.targets, and .sa.chunks correspond to Condition A (target values = -1; chunks = 1:26), and rows 27:52 correspond to Condition B (target values = +1; chunks = 1:26). To demonstrate my query, I've set Condition A to range between 0.6 and 0.8, and Condition B to range between 0.2 and 0.65. So, we have a very clear trend A > B for all 6 voxels.

Running cosmo_stat(ds_stack,'t','z') gives:

  struct with fields:

          a: [1×1 struct]
         fa: [1×1 struct]
    samples: [6.4824 6.7025 6.2338 5.4922 6.6919 6.0593]
         sa: [1×1 struct]

The z values are positive, which would suggest (to me) that B (target values +1) > A (target values -1). But we know that mean A > mean B in every voxel. So, I played around with the assignment of target values to see what the impacts would be. This included making BOTH target values negative, BOTH positive, and of course flipping the magnitude with target (A) > target (B) and vice versa.

The general pattern:

Cond A target value < Cond B target value ==> z values POSITIVE

Cond A target value > Cond B target value ==> z values NEGATIVE

It feels like I must be missing something very obvious!

Best wishes

Carolyn

ds_stack.sa.targets = [-1*ones(26,1); 1*ones(26,1)]

ds_stack.sa.chunks = [1:26,1:26]'

ds_stack.samples = 

[0.6696    0.7678    0.7953    0.6119    0.7196    0.7745
    0.6768    0.6730    0.6179    0.7168    0.6211    0.7965
    0.7977    0.6823    0.7444    0.6335    0.7834    0.6466
    0.6238    0.7513    0.7081    0.7009    0.6176    0.6540
    0.6343    0.6413    0.6187    0.7802    0.7679    0.7186
    0.6359    0.6852    0.6025    0.6554    0.7672    0.6110
    0.7742    0.7269    0.6286    0.7997    0.6893    0.7825
    0.7816    0.7061    0.7754    0.7474    0.7581    0.7919
    0.7326    0.7139    0.7686    0.6786    0.6548    0.7603
    0.7878    0.7079    0.7449    0.6612    0.6990    0.6525
    0.7569    0.7428    0.7129    0.7830    0.6842    0.6044
    0.7899    0.7639    0.6916    0.6791    0.6192    0.6761
    0.6167    0.6943    0.6959    0.6830    0.6657    0.7749
    0.7242    0.7392    0.6943    0.6064    0.7676    0.7641
    0.7040    0.6189    0.7251    0.6438    0.6132    0.7419
    0.7864    0.6718    0.7067    0.7167    0.6804    0.7639
    0.6227    0.7743    0.6305    0.7686    0.7229    0.6923
    0.7456    0.6337    0.6686    0.7551    0.7820    0.7228
    0.7550    0.7080    0.6645    0.6891    0.7694    0.6870
    0.7889    0.7872    0.7825    0.6365    0.6546    0.7905
    0.6720    0.7351    0.6220    0.6434    0.6125    0.6731
    0.7741    0.6639    0.7134    0.7003    0.7355    0.7357
    0.6441    0.7680    0.7176    0.7045    0.6585    0.7836
    0.7666    0.7473    0.7670    0.6038    0.7061    0.6420
    0.6319    0.7175    0.6980    0.6271    0.7516    0.6445
    0.6490    0.6810    0.7571    0.7487    0.6404    0.6307
    0.3095    0.2613    0.3476    0.4691    0.2118    0.2461
    0.6362    0.5109    0.5052    0.5469    0.2877    0.2435
    0.5443    0.3114    0.3890    0.4293    0.4205    0.3425
    0.2350    0.3128    0.4073    0.6299    0.6458    0.3076
    0.5504    0.4123    0.4940    0.2856    0.2206    0.4541
    0.4296    0.4920    0.5561    0.4874    0.2352    0.3969
    0.4405    0.3295    0.5516    0.6450    0.4953    0.6485
    0.4939    0.2499    0.3208    0.6207    0.6203    0.2770
    0.2549    0.2373    0.3717    0.6262    0.2065    0.2578
    0.2246    0.2818    0.3954    0.6050    0.4561    0.5791
    0.5825    0.2958    0.6403    0.6031    0.5033    0.5733
    0.4013    0.3973    0.2782    0.2171    0.2943    0.2147
    0.5787    0.5283    0.2088    0.2774    0.4308    0.5887
    0.4926    0.5784    0.3333    0.4153    0.2821    0.5671
    0.5598    0.3679    0.4205    0.5002    0.2763    0.3755
    0.3533    0.5536    0.4766    0.2404    0.5929    0.2519
    0.3304    0.3700    0.3990    0.5457    0.5125    0.5933
    0.4891    0.3978    0.5542    0.4737    0.4360    0.6456
    0.2473    0.6369    0.5050    0.4515    0.4603    0.3003
    0.4678    0.2686    0.4028    0.5125    0.2522    0.5056
    0.3410    0.5604    0.2573    0.3553    0.2737    0.4768
    0.5531    0.2794    0.5404    0.3434    0.4718    0.5970
    0.2359    0.2462    0.3010    0.4043    0.4399    0.4967
    0.6111    0.5767    0.2037    0.5704    0.2077    0.3334
    0.2070    0.4963    0.5783    0.6478    0.3408    0.2862
    0.4326    0.4078    0.4039    0.3062    0.2536    0.4826]

[Nick Oosterhof]

Dear Carolyn,

On 17 Apr 2024, at 12:10, Carolyn McGettigan <mcgettiga...@gmail.com> wrote:

Thanks for your quick response! I hope my reply here will help clarify.

I used cosmo_synthetic_dataset to create a dataset containing 6 features (voxels) and resembling my own group dataset (N=26 participants; 2 within-subject conditions). I've named the dataset 'ds_stack' (struct attached), as if it were the stacked group data in my experiment. Pasted values for ds_stack.sa.targets, ds_stack.sa.chunks, and ds_stack.samples are below.

In ds_stack, the first 26 rows of .samples, .sa.targets, and .sa.chunks correspond to Condition A (target values = -1; chunks = 1:26), and rows 27:52 correspond to Condition B (target values = +1; chunks = 1:26). To demonstrate my query, I've set Condition A to range between 0.6 and 0.8, and Condition B to range between 0.2 and 0.65. So, we have a very clear trend A > B for all 6 voxels.

Running cosmo_stat(ds_stack,'t','z') gives:

  struct with fields:

          a: [1×1 struct]
         fa: [1×1 struct]
    samples: [6.4824 6.7025 6.2338 5.4922 6.6919 6.0593]
         sa: [1×1 struct]

The z values are positive, which would suggest (to me) that B (target values +1) > A (target values -1). But we know that mean A > mean B in every voxel. So, I played around with the assignment of target values to see what the impacts would be. This included making BOTH target values negative, BOTH positive, and of course flipping the magnitude with target (A) > target (B) and vice versa.

The general pattern:

Cond A target value < Cond B target value ==> z values POSITIVE

Cond A target value > Cond B target value ==> z values NEGATIVE

It feels like I must be missing something very obvious!

Thanks for the follow-up. This is indeed the pattern. 

Looking at it now, I would agree that this behaviour is somewhat counterintuitive (or at least surprising): the sign is opposite of what one might expect. On the positive side: this behaviour is consistent and predictable :-).

"What is causing this”, one might ask. Well, the unique targets are found using the “unique” function, which gives the values 1 to N for the N target values in their sorted order. Thus sorting means that if target value A < target value B, then condition A gets the internal label 1 and condition B the internal label 2. The t-test is performed on internal label 1 minus internal label 2. Indeed if true A > true B, then this difference is positive, and thus z values are positive. 

This behaviour is present from at least 2015. As such I think it would be a bad idea to change it now, as re-running existing analyses would yield different results. So for now, I’m putting this in the “won’t fix” the code bucket. However I did update the documentation and added a longer explanation of this behaviour, which is included in versions of CoSMoMVPA post-dating this message. 

best,

Nick