LIBSVM and LIBLINEAR train/test functions

[Ryan Mruczek]

Here is some code I wrote for train/test functions using the libsvm
(http://www.csie.ntu.edu.tw/~cjlin/libsvm/) and liblinear (http://
www.csie.ntu.edu.tw/~cjlin/liblinear/). Each file is in a separate
post for ease of viewing on the google group. Please note that some
of the functionality of libsvm (optimizing cost param) is pretty raw
and should be carefully checked if you'd like to use it. Also, if you
find any problems or have any suggestions for improving the code, I'd
love to hear about them.

Cheers,
Ryan

[Ryan Mruczek]

TEST_LIBLINEAR

function [acts scratchpad] = test_liblinear
(testpats,testtargs,scratchpad)

% Testing function for LIBLINEAR (linear svm)
%
% [ACTS SCRATCH] = TEST_LIBLINEAR(TESTPATS,TESTTARGS,SCRATCH)
%
% This is the testing function that fits with TRAIN_LIBLINEAR. See
that
% for more info
%
% License:
%=====================================================================
%
% This is part of the Princeton MVPA toolbox, released under
% the GPL. See http://www.csbmb.princeton.edu/mvpa for more
% information.
%
% The Princeton MVPA toolbox is available free and
% unsupported to those who might find it useful. We do not
% take any responsibility whatsoever for any problems that
% you have related to the use of the MVPA toolbox.
%
%
======================================================================

% 02.27.09 - REHBM - created specific LIBSVM train function to
% distinguish from SVMLIGHT functions
% Based on train/test_svm.m that came with MVPA
toolbox.


%% validate arguments
if ~exist('scratchpad','var')
scratchpad = [];
end

sanity_check(testpats,testtargs,scratchpad);


%% Test Labels
% Labels will be 1:K for K conditions
[test_max_val testlabs] = max(testtargs); % testlabs = testing
labels, max index of max val

% Now test generalization performance on the test data
% Classify the test set using svmpredict
[scratchpad.predicted_label scratchpad.accuracy
scratchpad.decision_values] = ...
predict(testlabs', sparse(testpats)', scratchpad.model, '-b 0');


%% trying to make LIBLINEAR compatible with default perfmat functions.
%
% ACTS = is an nOuts x nTestTimepoints matrix that contains the
activations of the output units at test.
% We'll fill acts using winner-take-all activation based on
scratchpad.predicted_labels
acts = zeros(size(testtargs)); % initilize to all zeros
for i = 1:size(acts,1)
acts(i,scratchpad.predicted_label==i) = 1; % otherwise it remains
zero from initilization
end


%% sanity_check
function [] = sanity_check(testpats,testtargs,scratchpad)


if size(testpats,2) ~= size(testtargs,2)
error('Different number of testing pats and targs timepoints');
end

On Sep 23, 10:43 am, Ryan Mruczek <rmruc...@princeton.edu> wrote:
> Here is some code I wrote for train/test functions using the libsvm

> (http://www.csie.ntu.edu.tw/~cjlin/libsvm/) and liblinear (http://www.csie.ntu.edu.tw/~cjlin/liblinear/).  Each file is in a separate

[Ryan Mruczek]

TRAIN_LIBLINEAR

function [scratch] = train_liblinear
(trainpats,traintargs,in_args,cv_args)

% USAGE :
% [SCRATCH] = TRAIN_LIBLINEAR(TRAINPATS,TRAINTARGS,IN_ARGS,CV_ARGS)
%
% This is a linear support vector machine training function using the
% liblinear library. It train the classifier and makes is ready for
testing.
%
% You need to call TEST_LIBLINEAR afterwards to assess how well this
% generalizes to the test data.


% PATS = nFeatures x nTimepoints
% TARGS = nOuts x nTimepoints
%
% SCRATCH contains all the other information that you might need when
% analysing the network's output, most of which is specific to
% backprop. Some of this information is redundantly stored in multiple
% places. This gets referred to as SCRATCHPAD outside this function
%
% The classifier functions use a IN_ARGS structure to store possible
% arguments (rather than a varargin and property/value pairs). This
% tends to be easier to manage when lots of arguments are
% involved.
%
% IN_ARGS are the various arguments that can be passed in for type
% of kernels used and the learning parameters.
%
% IN_ARGS:
% in_args.training_options = (optional, default is ''). String of
options to
% pass to svmtrain of liblinear library.
See liblinear
% docs for more details. (example, '-s 1 -
c 1' for
% L2-loss SVM with LINEAR kernel and a
cost of 1).

%
% License:
%=====================================================================
%
% This is part of the Princeton MVPA toolbox, released under
% the GPL. See http://www.csbmb.princeton.edu/mvpa for more
% information.
%
% The Princeton MVPA toolbox is available free and
% unsupported to those who might find it useful. We do not
% take any responsibility whatsoever for any problems that
% you have related to the use of the MVPA toolbox.
%
%
======================================================================

% 03.01.09 - REHBM - created specific LIBLINEAR train function to
% distinguish from SVMLIGHT/LIBSVM functions

% Based on train/test_svm.m that came with MVPA
toolbox.


%% validate argument

defaults.train_funct_name = 'train_liblinear';
defaults.test_funct_name = 'test_liblinear';
defaults.training_options = '';
defaults.ignore_1ofn = 'false';

% Args contains the default args, unless the user has over-ridden them
args = propval(in_args,defaults);
scratch.class_args = args;
args = sanity_check(trainpats,traintargs,args);



%% Training Labels

% Labels will be 1:K for K conditions

[train_max_val trainlabs] = max(traintargs); % trainlabs = training

labels, max index of max val

%% *** TRAINING THE CLASSIFIER... ***
[scratch.model] = train(trainlabs',sparse
(trainpats)',args.training_options);


%% Local Functions
function [args] = sanity_check(trainpats,traintargs,args)

if size(trainpats,2)==1
error('Can''t classify a single timepoint');
end

if size(trainpats,2) ~= size(traintargs,2)
error('Different number of training pats and targs timepoints');
end

[isbool isrest isoveractive] = check_1ofn_regressors(traintargs);
if ~isbool || isrest || isoveractive
if ~args.ignore_1ofn
warning('Not 1-of-n regressors');
end

end


On Sep 23, 10:43 am, Ryan Mruczek <rmruc...@princeton.edu> wrote:

> Here is some code I wrote for train/test functions using the libsvm

> (http://www.csie.ntu.edu.tw/~cjlin/libsvm/) and liblinear (http://www.csie.ntu.edu.tw/~cjlin/liblinear/).  Each file is in a separate

[Ryan Mruczek]

TEST_LIBSVM

function [acts scratchpad] = test_libsvm
(testpats,testtargs,scratchpad)

% Testing function for LIBSVM
%
% [ACTS SCRATCH] = TEST_LIBSVM(TESTPATS,TESTTARGS,SCRATCH)
%
% This is the testing function that fits with TRAIN_LIBSVM. See that
% file for more info.

%
% License:
%=====================================================================
%
% This is part of the Princeton MVPA toolbox, released under
% the GPL. See http://www.csbmb.princeton.edu/mvpa for more
% information.
%
% The Princeton MVPA toolbox is available free and
% unsupported to those who might find it useful. We do not
% take any responsibility whatsoever for any problems that
% you have related to the use of the MVPA toolbox.
%
%
======================================================================

% 02.27.09 - REHBM - created specific LIBSVM train function to
% distinguish from SVMLIGHT functions

% Based on train/test_svm.m that came with MVPA
toolbox.

%% validate arguments
if ~exist('scratchpad','var')
scratchpad = [];
end

sanity_check(testpats,testtargs,scratchpad);

%% Test Labels

% Labels will be 1:K for K conditions

[test_max_val testlabs] = max(testtargs); % testlabs = testing

labels, max index of max val

% Now test generalization performance on the test data
% Classify the test set using svmpredict
[scratchpad.predicted_label scratchpad.accuracy
scratchpad.decision_values] = ...

svmpredict(testlabs', testpats', scratchpad.model, '-b 0');


%% trying to make LIBSVM compatible with default perfmat functions.

%
% ACTS = is an nOuts x nTestTimepoints matrix that contains the
activations of the output units at test.
% We'll fill acts using winner-take-all activation based on
scratchpad.predicted_labels
acts = zeros(size(testtargs)); % initilize to all zeros
for i = 1:size(acts,1)
acts(i,scratchpad.predicted_label==i) = 1; % otherwise it remains
zero from initilization
end


%% sanity_check
function [] = sanity_check(testpats,testtargs,scratchpad)


if size(testpats,2) ~= size(testtargs,2)

error('Different number of testing pats and targs timepoints');
end

On Sep 23, 10:43 am, Ryan Mruczek <rmruc...@princeton.edu> wrote:

> Here is some code I wrote for train/test functions using the libsvm

> (http://www.csie.ntu.edu.tw/~cjlin/libsvm/) and liblinear (http://www.csie.ntu.edu.tw/~cjlin/liblinear/).  Each file is in a separate

[Ryan Mruczek]

TRAIN_LIBSVM

function [scratch] = train_libsvm
(trainpats,traintargs,in_args,cv_args)
% USAGE :
% [SCRATCH] = TRAIN_LIBSVM(TRAINPATS,TRAINTARGS,IN_ARGS,CV_ARGS)
%
% This is a support vector machine training function using the libsvm
% library. It train the classifier and makes is ready for testing.
%
% You need to call TEST_LIBSVM afterwards to assess how well this

% generalizes to the test data.

% PATS = nFeatures x nTimepoints
% TARGS = nOuts x nTimepoints
%
% SCRATCH contains all the other information that you might need when
% analysing the network's output, most of which is specific to
% backprop. Some of this information is redundantly stored in multiple
% places. This gets referred to as SCRATCHPAD outside this function
%
% The classifier functions use a IN_ARGS structure to store possible
% arguments (rather than a varargin and property/value pairs). This
% tends to be easier to manage when lots of arguments are
% involved.
%
% IN_ARGS are the various arguments that can be passed in for type
% of kernels used and the learning parameters.
%
% IN_ARGS:
% in_args.training_options = (optional, default is ''). String of
options to

% pass to svmtrain of libsm library. See
libsvm
% docs for more details. (example, '-s 0 -
t 0 -c 1'
% for C-SVC with LINEAR kernel and a cost
of 1).
%
% in_args.search_for_c = (optional, default is 0). Boolean, perform
cross-validation
% within the training to find a -c value that
gives best
% generalization within training set. In this
case, you cannot
% provide a -c or -v option in
training_options and must also
% supply a k_fold_xval (see below).
% *****N.B. THIS OPTION NEEDS SOME WORKS. xxx
%
% in_args.k_fold_xval = (must be supplied if search_for_c is true,
default 0). The number
% of k-folds to use for within-training_set
cross-validation for
% optimizing c parameter. See libsvm library
doc for more info.

%
% License:
%=====================================================================
%
% This is part of the Princeton MVPA toolbox, released under
% the GPL. See http://www.csbmb.princeton.edu/mvpa for more
% information.
%
% The Princeton MVPA toolbox is available free and
% unsupported to those who might find it useful. We do not
% take any responsibility whatsoever for any problems that
% you have related to the use of the MVPA toolbox.
%
%
======================================================================

% 02.27.09 - REHBM - created specific LIBSVM train function to
% distinguish from SVMLIGHT functions
% Based on train/test_svm.m that came with MVPA
toolbox.

% 03.05.09 - REHBM - added support for c-parameter search using k-fold
cross-validation

% TODO: args.training_options should be split into each possible
argument with error checking?
% k-fold cross validation should be done by splitting each run
of the training set out. This
% could be a recursive method by redefining the training set
and training and testing.
% But it requires that we extract run info from
trainingtargs.

%% SORT ARGUMENTS
defaults.train_funct_name = 'train_libsvm';
defaults.test_funct_name = 'test_libsvm';
defaults.training_options = '';
defaults.search_for_c = 0;
defaults.k_fold_xval = 0;

defaults.ignore_1ofn = 'false';

% Args contains the default args, unless the user has over-ridden them
args = propval(in_args,defaults);
scratch.class_args = args;
args = sanity_check(trainpats,traintargs,args);

%% Training Labels

% Labels will be 1:K for K conditions

[train_max_val trainlabs] = max(traintargs); % trainlabs = training

labels, max index of max val

%% Parameter search?
% should we search for the best c-parameter by cross-
% validation within the training set.
% N.B. This probbaly needs to account for the non-
% independence across consecutive TRs, which are
% also most likely of the same class for a block
% design.

if args.search_for_c
% make sure user didn't try to set a specific c with -c in the
args.training_options
if ~isempty(regexp(args.training_options,'.*-[cv].*','once'))
error('train_libsvm.m - cannot supply -c or -v option when
search_for_c is true (1)')
end
if args.k_fold_xval < 1
error('train_libsvm.m - k_fold_xval (%d) must be supplied when
search_for_c is true (1)',args.k_fold_xval)
end
starttime = tic;
csearch.c = [2^-7:-6];%2.^(-7:1);
csearch.accuracy = repmat(NaN,[1 length(csearch.c)]);
csearch.loop_time = repmat(NaN,[1 length(csearch.c)]);
for i = 1:length(csearch.c)
looptime = tic;
c = num2str(csearch.c(i));
argstring = [args.training_options ' -v ' num2str
(args.k_fold_xval) ' -c ' c];
csearch.accuracy(i) = svmtrain
(trainlabs',trainpats',argstring);
csearch.loop_time(i) = toc(looptime);
end
csearch.total_time = toc(starttime);

% get c with best performance
csearch.best_c = csearch.c(find(csearch.accuracy(1,:)==max
(csearch.accuracy(1,:)),1));

% update args.training_options
args.training_options = [args.training_options ' -c ' num2str
(csearch.best_c)];
end


%% Train the classifier
[scratch.model] = svmtrain
(trainlabs',trainpats',args.training_options);

%% Local Functions
function [args] = sanity_check(trainpats,traintargs,args)

if size(trainpats,2)==1
error('Can''t classify a single timepoint');
end

if size(trainpats,2) ~= size(traintargs,2)

error('Different number of training pats and targs timepoints');
end

[isbool isrest isoveractive] = check_1ofn_regressors(traintargs);
if ~isbool || isrest || isoveractive
if ~args.ignore_1ofn
warning('Not 1-of-n regressors');
end

end



On Sep 23, 10:43 am, Ryan Mruczek <rmruc...@princeton.edu> wrote:

> Here is some code I wrote for train/test functions using the libsvm

> (http://www.csie.ntu.edu.tw/~cjlin/libsvm/) and liblinear (http://www.csie.ntu.edu.tw/~cjlin/liblinear/).  Each file is in a separate

[Ryan Mruczek]

Hi

nHidden is not a valid option for libsvm.  It is an option for the default backprop network in the tutorial.  Comment out that line before running with train/test_libsvm.

% class_args.nHidden = 0;

The valid options for libsvm are outlined in the comments of the train and test functions (try 'help train_libsvm' at the Matlab prompt).  But it should at least run using all of the default values.

Cheers,

Ryan

On Monday, October 8, 2012 8:52:54 AM UTC-4, jayran...@gmail.com wrote:

Hello

I am using the train_libsvm for MVPA Princeton toolbox.

I just replaced the class_args in the tutorial scripts into "train_libsvm" and 'test_libsvm". However when I run the program, it gives me the following error:

{

"Error using propval (line 197)

Property 'nHidden' has no default value.

Error in train_libsvm (line 94)

args = propval(in_args,defaults);

Error in cross_validation (line 222)

  scratchpad = train_funct_hand(trainpats,traintargs,class_args,cv_args);

Error in tutorial_hard_spm_tfmri_libsvm (line 152)

[subj results] = cross_validation(subj,'epi_z','conds_convt','runs_norest_xval','wholebrain',class_args);"

}

can you helo me to set a proper value for the "nHidden" ?

[Ryan Mruczek]

The function 'train' should refer to the ones in the libsvm toolbox.  See the first message in this thread for the website where that code can be found.  you'll need to install the libsvm package, along with the Matlab wrapper functions (included in the libsvm release).  Then, make sure that your path points to libsvm 'train' and not the NN 'train'.

Cheers,

Ryan

On Wednesday, October 10, 2012 1:13:00 AM UTC-4, jayran...@gmail.com wrote:

Hello

Thank you for your response

I also have another problem.

When it comes to this command;

{

%% *** TRAINING THE CLASSIFIER... *** 
[scratch.model] = train(trainlabs',sparse 
(trainpats)',args.training_options); 

}

it says, there is no train function for type 'double'.

when I change trainpats to double, it still have this problem.

I was thinking maybe the train(), which is referring to NN toolbox in matlab should be changes to svmtrain().

am I right?

Thank you

Jeiran

[sore...@gmail.com]

Hi,

Thanks for writing and posting these!  Think there's a typo in your code though. In the libsvm train function, you have have the variable 'trainlabs' as the training set label vector. It may be a function of the data set I'm using (I'm employing leave-one-out cv), but when I run your code this turns out to be a single value

 [scratch.model] = libsvmtrain(trainlabs',trainpats',args.training_options);  % original, though svm name chnaged to avoid confusion w/ other fun

I think this would be correct:

[scratch.model] = libsvmtrain(traintargs',trainpats',args.training_options);

I also noticed that your testing function give some wonky results too (again at least for my data). This time the problem seems to be that instead of passing svmpredict 

 the training labels, you're passing them an index of the labels themselves, or rather, the maximum label... 

%% Test Labels 

% Labels will be 1:K for K conditions 

[test_max_val testlabs] = max(testtargs); % testlabs = testing labels, max index of max val 

% Now test generalization performance on the test data 
% Classify the test set using svmpredict 

[scratchpad.predicted_label scratchpad.accuracy scratchpad.decision_values] = ... 

libsvmpredict(testlabs', testpats', scratchpad.model, '-b 0'); 

should be: 

[scratchpad.predicted_label scratchpad.accuracy scratchpad.decision_values] = ... 

libsvmpredict(testtargs', testpats', scratchpad.model, '-b 0'); 

the ACTs hack is off for me as well, not sure how to fix it.  Thanks for putting this up!!!

Jim