Followed the convolutional neural network tutorial, the end result seems to have a high error rate.

[ZachTM]

Hi guys.  I have spent days trying to find a library like this, EBLearn looks amazing.  So I compiled it and did everything in this tutorial.  It ran fine with no errors, but the last few lines of output of the training are as follows:

i=20 name=train [245241]  sz=10000 energy=0.0571127 (class-normalized) errors=34.6084% uerrors=1.95% rejects=0% (class-normalized) correct=65.3916% ucorrect=98.05%
errors per class: 3_samples=6131 3_errors=1.28853% 9_samples=3869 9_errors=2.53295%

testing: 0 / 1000, elapsed: 0s, ETA: 0s
testing: 400 / 1000, elapsed: 1s, ETA: 1s
testing: 800 / 1000, elapsed: 2s, ETA: 0s
i=20 name=val [245241]  sz=1000 test_energy=0.0329366 (class-normalized) test_errors=50.6329% test_uerrors=6.4% test_rejects=0% (class-normalized) test_correct=49.3671% test_ucorrect=93.6%
errors per class: test_3_samples=1000 test_3_errors=1.2%

testing_time=38s
Displaying training...
iteration_minutes=2.53333
iteration_time=2m 32s
timestamp=20121120.215213

This looks like the training has a ton of errors, for example test errors = 50%.  But then again there is also a part of this that says test_3_errors=1.2%

Can somone tell me if this didnt work right for me?  If so, I saw a thread where people mentioned making sure libpthreads was installed, and pointed to correctly in the cmake files.  I have verified that it is all correct.  So does anyone have any suggestions?  I would REALLY appreciate it as this library was EXACTLY what i have been looking for.

 

[Pierre Sermanet]

Hi,

It looks like there's a problem with the dataset, you have only 3 and 9 in the training set and only 3 in the validation set. Can you copy the beginning of your outputs? There should be some description of each dataset with the number of classes.

Pierre

 

--
 
 

[soumith]

Hey Zach,

Glad you like EBLearn.

To give a little explanation on the output,

uerrors are unnormalized errors, i.e. [no. of wrongly classified samples / total number of samples]

(class-normalized) errors is a metric that takes the average uerror of each class, i.e.

[100 - [sum of ucorrect of all classes/ number of classes]]

.

In your test set, you have 1000 samples of class "3" and 0 samples of class "9"

Your testing concluded that 

test_3_samples=1000 test_3_errors=1.2% 

Hence, test_3_correct = 100 - 1.2 = 98.8%

and since "9" is a class that doesn't exist in the test set, the test_9_errors is taken as 100%

Hence, test_9_correct = 100 - 100 = 0%

Now, (class-normalized) error = 100 - ((0+98.8) / 2) = 50.6%

This as you see doesn't reflect the actual performance. To see the actual performance, you can look at the "uerrors"

--
 
 

[soumith]

Also, in the tutorial, I limited the training and validation sets to a certain number of samples, to make the tutorial faster, which is why you only have class "3" in your validation set.

To do a full set of training, you can comment the lines by putting a # before them

# train_size      = 10000

# val_size        = 1000      

The training will take a little longer, but you can limit the number of iterations to 1 or 2, to just see the rough performance using the "iterations" variable

iterations      = 2     # number of training iterations

[ZachTM]

Well first of all its nice to see a pretty active user group!  Thank you all for your responses, commenting out the train and val size seems to have fixed the problem.  I didnt notice there were only samples from 3 and 9 at first.  I will be sure to post back if it doesn't work.  Thanks again!

[ZachTM]

I didnt think i should start a new thread for this but when I comment out those val_size and train_size and make iterations = 40, it will run for a while but the ucorrect will bounce back between around 89 and 91 percent.  Every epoch it goes through all the training examples so I dont know what the problem is.  The thing is the output window of some samples is classified correctly, is showing some very (in my opinion) difficult validation samples that it got correct, samples that my old 3 layer fully connected neural network would have never classified correctly.  So should I just try using the network as It is trained by 40 epochs?  Or is there something else I should do to increase accuracy.  If it is only incorrectly classifying some of those difficult digit samples that is fine with me.  Thanks for any suggestions.

[ZachTM]

Sorry last post I fixed the problem.  I just used a higher initial learning rate with a decay value and I am getting much better results.