Accuracy measured in python falls far short of accuracy reported by caffe command line tool

[David Bau]

Trying to use a caffe model from python, but apparently failing.  Has anybody seen a problem like this?

When I use "caffe train" to train a lenet network on my own training set of 100,000 images (and testing on a smaller test set of just 500), I can get caffe to report > 82% accuracy on the test set after just a few minutes (20,000 iterations, batch size 100), but when I take the last caffemodel and load it into python, then use the python api to classify each image in my test set (taking argmax of the softmax output), I get correct classification on only 290/500 = 58% of the images.

Where is the difference coming from?  Also, according to python, my classification errors are disproportionately bad when trying to classify a class of blank images.  My network is 0% correct on telling that a blank image is blank, and it thinks it has 1 or two rectangles on it.  A summary of what I see from python:

This is very different from what "caffe train" reports when finishing training:

I1007 19:16:17.015362  4003 solver.cpp:753] Snapshotting solver state to binary proto file soln_iter_20000.solverstate

I1007 19:16:17.023927  4003 solver.cpp:327] Iteration 20000, loss = 0.288211

I1007 19:16:17.023952  4003 solver.cpp:347] Iteration 20000, Testing net (#0)

I1007 19:16:17.485422  4003 solver.cpp:415]     Test net output #0: accuracy = 0.828

I1007 19:16:17.485445  4003 solver.cpp:415]     Test net output #1: loss = 0.381967 (* 1 = 0.381967 loss)

I1007 19:16:17.485452  4003 solver.cpp:332] Optimization Done.

I1007 19:16:17.485458  4003 caffe.cpp:215] Optimization Done.

A few questions -

(1) there a command-line way to ask caffe to classify a single image?  I fear it's some mistake in the way I'm using the python API.

(2) I previously found that if I am not careful about treating my mean image as grayscale, the accuracy is even worse, at 41%, but after I get the mean image loaded as a grayscale image correctly, it is still only 58% as above.  What else might I be missing?

Here is my python script for reading and using my caffeemodel.

https://github.com/davidbau/stickcount/blob/063be8a099ffdaca50f8b8b40378c6ce0a765fda/status.py

If it helps to know exact settings, the whole thing including test case generation is on github here - if you set CAFFE_HOME, you can just "make" to do the whole thing from beginning to end.

https://github.com/davidbau/stickcount/tree/063be8a099ffdaca50f8b8b40378c6ce0a765fda

Thanks for any tips or words of wisdom,

David

[David Bau]

Lost a bunch of sleep over this one.  Stumbled on the solution just about when I was ready to give up on Caffe.

The issue was a confusion over scaling input values.  My train_test.prototext specified:  transform_param { scale: 0.00390625 }, scaling the input bitmaps range from 0...256 to 0...1.  And my python caffe.Classifier constructor included what I had _thought_ was a matching specification raw_scale=256.

However, that's not right.  For the classifier constructor, the right setting to match scale: 0.00390625 is raw_scale=1, which is the default.  With this setting, in python I'm now measuring 86% on accuracy on my toy example, right in line with the accuracy reported while training.