multi-class classification using libsvm

[LA]

Hi Nick,

So I'm getting this error when using libsvm for a multi-class classification problem, and I have no idea why. Here's what is happening:

- I have 25 subjects that performed two scan sessions. I want to classify subject identity across the two different time points (i.e., training on scan 1, testing on scan 2, and vice versa) so I define targets and chunks as:

targets chunks
1 1
2 1
3 1
4 1
5 1
6 1
7 1
8 1
9 1
10 1
11 1
12 1
13 1
14 1
15 1
16 1
17 1
18 1
19 1
20 1
21 1
22 1
23 1
24 1
25 1
1 2
2 2
3 2
4 2
5 2
6 2
7 2
8 2
9 2
10 2
11 2
12 2
13 2
14 2
15 2
16 2
17 2
18 2
19 2
20 2
21 2
22 2
23 2
24 2
25 2

- This is the error that I get:

Error: label vector and instance matrix must be double
Error using cosmo_crossvalidate
In cosmo_searchlight, center id 1 caused an exception:
Unable to perform assignment because the size of the left side is 25-by-1 and the size of the right side is 0-by-0.

All partitions have non-empty train and test sets, so I'm not sure why am I getting this error message. I also ensured that samples and targets are of type double. I know libsvm supports multi-class, so again, I'm not sure what's causing this.

Any idea on what I could be missing here?

Thank you in advance for your help!

[Nick Oosterhof]

Greetings,

On Feb 15, 2024, at 00:39, LA <la...@georgetown.edu> wrote:

So I'm getting this error when using libsvm for a multi-class classification problem, and I have no idea why. Here's what is happening:

- I have 25 subjects that performed two scan sessions. I want to classify subject identity across the two different time points (i.e., training on scan 1, testing on scan 2, and vice versa) so I define targets and chunks as:

targets chunks
1 1
2 1
[…]

24 1
25 1
1 2
2 2

[…]
24 2
25 2

- This is the error that I get:

Error: label vector and instance matrix must be double
Error using cosmo_crossvalidate
In cosmo_searchlight, center id 1 caused an exception:
Unable to perform assignment because the size of the left side is 25-by-1 and the size of the right side is 0-by-0.

All partitions have non-empty train and test sets, so I'm not sure why am I getting this error message. I also ensured that samples and targets are of type double. I know libsvm supports multi-class, so again, I'm not sure what's causing this.

Any idea on what I could be missing here?

There is only one sample per target (participant) in the training set. However for most classification scenario’s one would want more than one — and generally quite a few more than that. An LDA classifier with just one sample per target doesn’t work at all because the covariance cannot be estimated, and something similar may apply to SVMs as well. 

You may want to consider the following ideas:

1. Fit multiple beta estimates for each target in each run. This may or may not be feasible and/or lead to noisy estimates depending on the design / timing of conditions.

2. Use a nearest neighbor classifier. This may just work, although fMRI signals are quite noisy and in my experience nearest neighbor classifiers don’t work so well. 

3. Use a split-half correlation measure (see cosmo_correlation_measure). Implement a Monte Carlo / bootstrap analysis with randomization of the targets (cosmo_randomize_targets) to assess statistical significance. 

My personal advice would be to follow idea 3.