Erratic Classification

[Juan Manuel]

Hello everyone,

I've trained a neural network classifier (no hidden layer) on data
from two conditions and found that, across classification performances
of 20 participants, classification ranges from .0081 to .9762 with an
average classification of .4387 (SD = .3962). So the classifier is
accurate at rates of 90 percent for some participants and is
inaccurate at rates of less than 10 percent for others. Could this
wild variability be due to some systematic error I'm making in
training my classifier? Perhaps the data is somehow being overfitted?

Any thoughts or reports of similar results are appreciated.

Sincerely,
Juan Manuel Contreras

[Chris Day]

Sorry Juan, but I cannot help you.

I have a similar finding, and would also appreciate any help on the
matter.

Regards,

Chris

[Greg Detre]

Have a look at the motion parameters and behavioral performance of your
subjects. Perhaps you'll find that the low classifier performance is for
inattentive or fidgety subjects?

If that doesn't reveal anything, the best debugging approach I know is
to create some 'fake' (i.e. simulated/synthetic) data for an individual
subject, and feed it through your analysis. If the 'fake' data is nearly
perfect, you should see nearly perfect performance. If you feed in
nonsense or noisy fake data, you should see performance degrade towards
baseline. Creating fake data can be tricky, especially if your
experiment/analysis are complex, but 100% worth the effort. See e.g.:

http://code.google.com/p/princeton-mvpa-toolbox/source/browse/trunk/core/synth/noisify_regressors.m

If your analysis works as expected on fake data, you can start to
breathe more easily that it's not a straightforward bug. At that point,
start to think about ways to visualize what's going on.

g

[MS Al-Rawi]

In addition to Greg's comments 

> Perhaps the data is somehow being overfitted?

In NNs, if the training error goes very low, you might have overfitting, you may try another classifier, e.g. Logistic Regression. Have you done detrending and z-scoring to your data? 

Regards,
Rawi