Second level classification

[Alireza]

Dear MVPA experts,

My attempt to use MVPA on FMRI data to dissociate between faces,
objects, scenes and words was succeeded with a vey good performance of
the classifier. Now my question is whether I can use the output of
this level to further categorize each category (faces, objects, scenes
and words) into two different categories A, and B? What is the best
way of doing that?

Is there any paper dealing with such a questions?

Any idea will be appreciated!

Thanks,
/Alireza

[Francisco Pereira]

A good place to look for an overview and references on classifier
combination would be Paul Bennett's PhD thesis:
http://reports-archive.adm.cs.cmu.edu/anon/2006/CMU-CS-06-121.pdf

cheers,
Francisco

[Yaroslav Halchenko]

if I got your right you wanted to provide further subcategorical
classification ... ie like a decision tree.. is that right?

in PyMVPA (0.4.3) we have a TreeClassifier which was destined for such
use:

class TreeClassifier(ProxyClassifier):
"""`TreeClassifier` which allows to create hierarchy of classifiers

Functions by grouping some labels into a single "meta-label" and training
classifier first to separate between meta-labels. Then
each group further proceeds with classification within each group.

Possible scenarios::

TreeClassifier(SVM(),
{'animate': ((1,2,3,4),
TreeClassifier(SVM(),
{'human': (('male', 'female'), SVM()),
'animals': (('monkey', 'dog'), SMLR())})),
'inanimate': ((5,6,7,8), SMLR())})

would create classifier which would first do binary classification
to separate animate from inanimate, then for animate result it
would separate to classify human vs animal and so on::

SVM
/ \
animate inanimate
/ \
SVM SMLR
/ \ / | \ \
human animal 5 6 7 8
| |
SVM SVM
/ \ / \
male female monkey dog
1 2 3 4

> Dear MVPA experts,

> Thanks,
> /Alireza
--
.-.
=------------------------------ /v\ ----------------------------=
Keep in touch // \\ (yoh@|www.)onerussian.com
Yaroslav Halchenko /( )\ ICQ#: 60653192
Linux User ^^-^^ [175555]

[Greg Detre]

is there any reason why you can't just train a new classifier on A vs B?

g

--


---
Greg Detre
cell: 617 642 3902
email: gr...@gregdetre.co.uk
web: http://www.gregdetre.co.uk

[Alireza]

Dear Yaroslav,

Thank you for your nice description. That was exactly what I want to
do.So in the first step the classifier is suppose to
seprate face,scene,object and word,then it suppose to elucidate for
each category whether it is trinsient or sustain:
the schematic example is as fiollows:
/ \ \ \
Face object scene word
/ \ /\ / \ / \
sustain transient sustain trnsient
sustain

but pyMVPA is not written in matlab. Can one plug the result of MVPA
classification (sepration between face, object, scene and word) into
this decision tree or one should do everything from beginning in
pyMVPA?

Thanks alot
/Alireza


On Nov 3, 6:11 pm, Yaroslav Halchenko

[Alireza]

Hello Greg
but is it possible to train the classifier on the result of fisrt
part. what I want to do is:

Face object scene word
/ \ /\ / \ / \
sustain transient sustain trnsient sustain

As I mentioned before,the classifiers performance was very good
(total_perfs=0.61) in seprating between face,object,scene and word.but
now I don't know how can I go further to subcategorize each category
to sustain and transient.Do you mean I have to train a new classifier
on results.iterations.act or ....? if yes,I mean how would you define
your pattern and your regressor onset for this second level?


here are some further information:

class_args: [1x1 struct]
nOut: 4
net: [1x1 struct]
errorfun: 4.8563
cur_iteration: 1


Thanks for any help!
/Alireza

> email: g...@gregdetre.co.uk
> web:http://www.gregdetre.co.uk- Hide quoted text -
>
> - Show quoted text -

[Yaroslav Halchenko]

On Tue, 03 Nov 2009, Alireza wrote:
> but pyMVPA is not written in matlab. Can one plug the result of MVPA
> classification (sepration between face, object, scene and word) into
> this decision tree or one should do everything from beginning in
> pyMVPA?

yeah -- from the beginning in PyMVPA but beginning might be quite short.

smth like this
http://www.pymvpa.org/examples/start_easy.html
just replace SMLR() there with a TreeClassifier I've mentioned before ;)

also you could read your .mat files from within Python...

[Greg Detre]

my question is this: do you really need to turn this into a 2-stage
classification?

could you instead simply run 4 separate fresh classifications?

- scoped to just face trials, 2 output units (sustain vs transient)
- scoped to just object trials, 2 output units (sustain vs transient)
- scoped to just scene trials, 2 output units (sustain vs transient)
- scoped to just word trials, 2 output units (sustain vs transient)

an alternative: take your trained 4-way classifier, and simply test it
(using apply_trained_classifier) on the sustained and transient
sub-categories to see whether its activations are systematically higher
for one than the other.

i don't know anything about your task or your aims, so i might be
completely off-base with both these suggestions.

g

email: gr...@gregdetre.co.uk
web: http://www.princeton.edu/~gdetre/

[Alireza]

Hello Greg!

I have tried to implement your second suggestion using
apply_trained_classifier to tease apart sustained and transient
condition for each category.The following error occured:


1??? Error using ==> perfmet_maxclass>sanity_check at 115
Can't calculate performance if acts and targs are different sizes

Error in ==> perfmet_maxclass at 66
sanity_check(acts,targs,scratchpad,args);

Error in ==> apply_trained_classifier at 124
pm = perfmet_fh(acts, testtargs, scratchpad, args.perfmet_args);

Regarding the design(you asked before),we had 4 different categories
(face,object,word and scene). Each category was presented both as a
block (10 items of the same category in each block) and transient (one
item of category A (e.g. transient face) during the block presentation
of category B(sustained scene)). What I have done was first trying to
classify the data into 4 lables standing for each category regardless
of sustained and transient.So in the first classification framework,
the regressor was defined with 4 rows standing for the four labels
(regardless of sustained/transient) and 500 columns standing for the
scans.Four interations were selected by create_xvalid_indices to
create the group of selectors. The accuracy of the classifier was
perfect with perfmet=.62 (p< 0.0000) delineating a nice dissociation
between faces,scenes,objects and words regardless of transient/
sustained. When I took a look at results.iterations.acts it seems that
classifier was capable of teasing the sustain and transient apart, as
well, though it hadn't been traind for that

Thereby,I applied apply_trained_classifier to new regressors with 2
rows standing for transient and sustain for example only for faces(the
row associated to faces (regressor) on the previous section is now
divided into two parts accounting for transient face and suatained
face).I shifted the new regressor to accommodate hrf shift,removed
rest times, and created a new selstem. The error prevents me to go
further!

I had also tried to make 8 categories from the begining(category X
transient/sustained) by making a regressor with 8 rows but the
accuracy of the classsifications for all the transient categories was
almost zero on the confusion matrix.

Any idea would be highly appreciated!
Thanks
/Alireza

> >> web:http://www.gregdetre.co.uk-Hide quoted text -

>
> >> - Show quoted text -
>
> --
>
> ---
> Greg Detre
> cell: 617 642 3902

> email: g...@gregdetre.co.uk
> web:http://www.princeton.edu/~gdetre/- Hide quoted text -

[Greg Detre]

hmmm. it's hard to guess what the issue is.

i haven't tried apply_trained_classifier recently, though i'm pretty
sure it was working when i wrote it.

that line of the code:

if ~compare_size(acts,targs)
error('Can''t calculate performance if acts and targs are different
sizes');
end

makes me think that the problem relates to what inputs you're feeding
in. this function gets called by the test_[classifier].m script.

it makes sense that you'd need the same number of timepoints in both
ACTS and TARGS, since you need one label per timepoint. it also makes
sense that you'd need the same number of rows, so that you can compare
each output of the classifier against the labels.

so have a careful look at the variables you're feeding into
apply_trained_classifier.m. remember: 'keyboard', 'dbstack' and 'dbup'
are your friends!

if you still can't figure things out, post back with the output of
running 'whos' and 'dbstack', and we'll go from there.

g

email: gr...@gregdetre.co.uk
web: http://www.gregdetre.co.uk