Cross-validated distances

[Michael Peer]

Hi,

first of all, thanks for this amazing tool - it is very useful, well documented and easy to use.

I have a question: for computation of a neural dissimilarity matrix for RSA, I want to compute a cross-validated distance between patterns. That is, if I have two runs, I want to measure the distance between each pattern (condition) in run 1 and each pattern in run 2, to obtain the dissimilarity matrix. Alternatively if there are more than two runs, this can be done in a leave-one-out cross-validation way. This method appears to have the advantage of getting a measure of pattern consistency across runs (in the matrix diagonal), and removing a positive bias in distance values (Walther and Diedrichsen 2016).

I cannot seem to find any way to implement this kind of neural dissimilarity computation in an RSA searchlight, or in the cosmo_target_dsm_corr_measure. Is there any way to do such a thing in COSMOMVPA? If it is possible through some programming, which function should be changed?

Thanks a lot!

Michael Peer

[raffaele tucciarelli]

Hi Michael,

I don't think that is implemented in CoSMo, but let's see what Nick says. If it helps, I know that Tsantani et al. (see preprint) implemented it and will make it available after publication. 

Cheers

Raffaele

--
You received this message because you are subscribed to the Google Groups "CoSMoMVPA" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cosmomvpa+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Nick Oosterhof]

Hello,

On Sat, 27 Apr 2019 at 23:27, Michael Peer <michae...@gmail.com> wrote:

first of all, thanks for this amazing tool - it is very useful, well documented and easy to use.

Thanks, great to hear that. 

I have a question: for computation of a neural dissimilarity matrix for RSA, I want to compute a cross-validated distance between patterns. That is, if I have two runs, I want to measure the distance between each pattern (condition) in run 1 and each pattern in run 2, to obtain the dissimilarity matrix. Alternatively if there are more than two runs, this can be done in a leave-one-out cross-validation way. This method appears to have the advantage of getting a measure of pattern consistency across runs (in the matrix diagonal), and removing a positive bias in distance values (Walther and Diedrichsen 2016).

I cannot seem to find any way to implement this kind of neural dissimilarity computation in an RSA searchlight, or in the cosmo_target_dsm_corr_measure. Is there any way to do such a thing in COSMOMVPA? If it is possible through some programming, which function should be changed?

This is not implemented in CoSMoMVPA at the moment, except for the split-half correlation measure which returns a difference between on- and off-diagonal values of split-half correlations (Haxby 2001-like).

If you want anything more complicated you would have to wrote some code yourself. Probably by implementing your own 'measure' function, so that it can also be used with a searchlight. The crossvalidation measure used with classifiers is probably a good starting point. 

If you need more help with this, please let us know.

best,

Nick

 

 

[David Wisniewski]

Hi Nick, Micheal,

I know this post is relatively old, but I have the same issue. I want to run a ROI based cross-validated euclidean distance RSA. Did you solve this issue in the end?

Best

David

[raffaele tucciarelli]

Hi David,

not sure this helps, but I have been (slowly) working on integrating this feature in CoSMoMVPA (but I haven't discussed this with Nick yet). If you are interested, maybe we can join efforts, I would need someone to test if it works, at least And obviously also some help from Nick...

The idea is to use the rsatoolbox that already has the cross-validated (Mahalanobis) distance implemented. 

Let me know

Cheers

Raffaele

--

You received this message because you are subscribed to the Google Groups "CoSMoMVPA" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cosmomvpa+...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/cosmomvpa/61f9792e-aee8-4657-a183-7ee731a39dfcn%40googlegroups.com.

[raffaele tucciarelli]

@Nick unless you have already been working on this, in that case, it would be even better :D

[Moritz Wurm]

I have implemented this a while ago (see attached, lines 126-145).

If partitions are included as arguments, then it will automatically use crossvalidation. 

I hope this helps!

Best,

Moritz

@Nick: feel free to integrate this feature :)

[raffaele tucciarelli]

That's great! Thanks Moritz!

To view this discussion on the web, visit https://groups.google.com/d/msgid/cosmomvpa/CAFfcD9TzA7c9zt-o9U4ipOQ26X%3DVs2sLYqYS3p5X5Wuor5EZLQ%40mail.gmail.com.

[David Wisniewski]

That is wonderful, thank you! I will take a look at the script and try to adapt it to my analysis. You just saved me so much work :)

Cheers

David

--

-----

David Wisniewski

[Nick Oosterhof]

Hi all,

nice to see the helpful replies from both Raffaele and Moritz! I'd be happy to discuss or consider contributions for such functionality. 

My first impression would be that such cross-validated distance functionality could go in a separate function. This is a personal preference, as cosmo_target_dsm_corr_measure is already quite long and through its evaluation has had features added I did not foresee in the earlier versions (such as linear regression as well as partial correlation support). I can also see the usefulness of supporting dissimilarity target matrices as well though. For inclusion in CoSMo some (at least basic) unit testing would also be high on my wish list. 

best,

Nick

To view this discussion on the web, visit https://groups.google.com/d/msgid/cosmomvpa/CAGRO-VNN5kvT009eL-FaEE-yx8aV_%2B%3DYBc09SGjGayiKub6hsA%40mail.gmail.com.

[David Wisniewski]

Hi Nick,

I agree, having a separate function for this would be great. In my case for instance, I am not using cosmo_target_dsm_corr_measure, but rather cosmo_dissimilarity_matrix_measure (I just need to estimate data RDMs without relating them to any theoretical models). So if we write a function that plugs into both, that would be super helpful. I saw that Moritz's solution uses the cosmo_correlation_measure, handing it over the partitions argument to run the cross validation. Maybe a function on that level (cosmo_cvdist_measure?) would work? It could be based off of cosmo_pdist, which cannot handle partitions if I get that right. I am pretty new to the toolbox still, so no idea if that is even possible. But I would be up for testing out potential solutions.

Best,

David

--

-----

David Wisniewski

[lyam.m...@gmail.com]

Hi all,

Was this ever integrated into CoSMo? If not, am I correct in thinking that the code supplied by Moritz requires a dataset containing as many instances of each target as there are partitions? So, something like this:

ds.sa.targets':

1 2 3 1 2 3 1 2 3

ds.sa.partitions':

1 1 1 2 2 2 3 3 3

Thanks!