Outputting class probabilities for MNIST example using pycaffe

[Nitin Iyer]

There's a similar topic to this one that I found, but the issue was closed so I thought I would start a new one. Sorry if this isn't the correct protocol, I'm very new around here. I'm trying to familiarize myself with pycaffe and have been trying to write a python script to spit out the class probabilities for each image in the test db. Below is the code I have written so far:

import sys
import caffe
import cv2
import Image
import matplotlib
matplotlib.rcParams['backend'] = "Qt4Agg"
import numpy as np
import lmdb
caffe_root = '../'

MODEL_FILE = './examples/mnist/lenet.prototxt'
PRETRAINED = './examples/mnist/lenet_iter_10000.caffemodel'

net = caffe.Net(MODEL_FILE, PRETRAINED,caffe.TEST)
caffe.set_mode_cpu()
db_path = './examples/mnist/mnist_test_lmdb'
lmdb_env = lmdb.open(db_path)
lmdb_txn = lmdb_env.begin()
lmdb_cursor = lmdb_txn.cursor()
count = 0
correct = 0
for key, value in lmdb_cursor:
    print "Count:"
    print count
    count = count + 1
    datum = caffe.proto.caffe_pb2.Datum()
    datum.ParseFromString(value)
    label = int(datum.label)
    image = caffe.io.datum_to_array(datum)
    image = image.astype(np.uint8)
    out = net.forward_all(data=np.asarray([image]))
    predicted_label = out['prob'][0].argmax(axis=0)
    if label == predicted_label[0][0]:
        correct = correct + 1
    print("Label is class " + str(label) + ", predicted class is " + str(predicted_label[0][0]))
    if count == 3:
        break
print(str(correct) + " out of " + str(count) + " were classified correctly")

The issue I am having is that out['prob][0] only has 0. or 1., not actual probabilties. Additionally I have tried the following code below, but in this case, what happens is that all the probabilities for the batch are the same:

MODEL_FILE = './examples/mnist/lenet_train_test.prototxt'
PRETRAINED = './examples/mnist/lenet_iter_10000.caffemodel'

net = caffe.Net(MODEL_FILE, PRETRAINED,caffe.TEST)
caffe.set_mode_cpu()
out = net.forward()
print out['prob']

The difference here being that I use lenet_train_test.prototxt as opposed to just lenet.txt.

Any help on this issue would be greatly appreciated!

[Nitin Iyer]

Is it possible that I am in fact seeing the class probabilities but that the probabilities are so close to 0 and 1 that python is making a rounding error. Additionally I should mention that when I run make runtest, I fail the following tests:

src/caffe/test/test_math_functions.cpp:79: Failure
Value of: caffe_cpu_hamming_distance<TypeParam>(n, x, y)
  Actual: 717829
Expected: this->ReferenceHammingDistance(n, x, y)
Which is: 1422821
[  FAILED  ] MathFunctionsTest/1.TestHammingDistanceCPU, where TypeParam = double (18 ms)
[----------] 8 tests from MathFunctionsTest/1 (117 ms total)

[----------] Global test environment tear-down
[==========] 581 tests from 111 test cases ran. (22080 ms total)
[  PASSED  ] 576 tests.
[  FAILED  ] 5 tests, listed below:
[  FAILED  ] EltwiseLayerTest/0.TestSum, where TypeParam = caffe::FloatCPU
[  FAILED  ] EltwiseLayerTest/0.TestProd, where TypeParam = caffe::FloatCPU
[  FAILED  ] EltwiseLayerTest/1.TestSum, where TypeParam = caffe::DoubleCPU
[  FAILED  ] EltwiseLayerTest/1.TestProd, where TypeParam = caffe::DoubleCPU
[  FAILED  ] MathFunctionsTest/1.TestHammingDistanceCPU, where TypeParam = double

[Axel Angel]

I have tested your code and I have not reproduced your problem. I trained LeNet on MNIST according to this tutorial http://caffe.berkeleyvision.org/gathered/examples/mnist.html for 5000 iterations (~99% accuracy). All tests passed so that's probably a problem with your platform: libraries, environment or hardware. I don't know sorry, I'm quite new here.

Beside this, The prediction accuracy is surprisingly low, around 8% (lower than random!). I have noticed most of the output are nearly of the same class: 2 or 3, sometimes the other ones. That's why I wanted to try with Classifier, set raw_scale = 255 instead but I have an other problem yet* (that I'm actually trying to solve).

*Traceback (most recent call last):

  File "classify_mnist.py", line 26, in <module>

    out = net.predict([ image ])

  File "/mnt/Softwares/caffe/python/caffe/classifier.py", line 86, in predict

    out = self.forward_all(**{self.inputs[0]: caffe_in})

  File "/mnt/Softwares/caffe/python/caffe/pycaffe.py", line 150, in _Net_forward_all

    outs = self.forward(blobs=blobs, **batch)

  File "/mnt/Softwares/caffe/python/caffe/pycaffe.py", line 80, in _Net_forward

    self.blobs[in_].data[...] = blob

ValueError: could not broadcast input array from shape (64,28,28,28) into shape (64,1,28,28)

[Nitin Iyer]

Thanks for looking into it! I was having those accuracy issues before, but with the code above that is no longer a problem. The classifier is around 99% accurate. I tried generating my own 28x28 image of a digit and supplied it to the classifier, and in that case, I got class probabilities that were not 1 or 0, however, the classifier performed quite poorly on my created example (I did not follow all preprocessing steps so thats probably why). Can anyone else confirm that the class probabilities for the MNIST data set are very close to 0 or 1?

[Axel Angel]

I was able to fix my problem too: I used the definition coming from the git repo of caffe (LeNet) and the training version was mismatching the deploying one(!)*. Eventually I am able to get 98.5% accuracy on the testset using the Python program (similar to yours) and indeed I can confirm the output are always either 1.0 or 0.0 exactly. I don't really know, maybe the network is sure enough such that the floating point cannot express the probabilities so it's rounded, maybe? It is definitely weird and it doesn't look normal. I trained other classifiers and the probabilities were always non-integers. Anyone?

*I used the lenet train_test prototxt in the mnist directory. It uses some old strange notation, instead of layers it's layer; instead of upper case with underscores in the type, it's using Camel case inside quotes, weird and that was rejected by my python code but not yours, so the architecture was not the "same", I had the same output for every images, weird. So finally I abandoned and took the deploy version, adapted it using the online tutorial for MNIST which described the correct format. It seems the examples directory is out of date and should be fixed.

[Nitin Iyer]

Thanks again for trying to replicate my work! It still stands then that the outstanding question is "Does the net for the MNIST data set work so well, that the class probabilities essentially become 0 or 1?" If anyone could answer this definitively that would be great. I think that this is the case, because at an accuracy of around 98.5, it would seem that the classifier would be very confident in its predictions.

[Axel Angel]

My intuition is that the MNIST dataset is pre-processed in a way that makes the network output probabilities with certainty every time on this dataset. I have trained on images drawn by myself in the same format and I have observed the results is very accurate (correct classifications) and non-integers probabilities!

Examples: http://paste.gnugen.ch/raw/zkbL and http://paste.gnugen.ch/raw/Yszn (this one was incorrect). I used GIMP to draw them with my mouse by hand. Hope this helps.

[Axel Angel]

Hum sorry I didn't mean "train" but "classify" here: 

On Wednesday, February 25, 2015 at 1:42:47 PM UTC+1, Axel Angel wrote:

I have *tested* on images drawn by myself …

[Chris Maxey]

Why is "predicted_label[0][0]" written with indexing?  When I run the code, "predicted_label" is an integer and not a nested array...

On Monday, February 23, 2015 at 3:20:33 PM UTC-5, Nitin Iyer wrote:

[Toru Hironaka]

Your code works with MNIST example. I modify your code to classify my image with imagenet example but it did not work. The prediction is almost always the same label. Do you have any idea?

On Monday, February 23, 2015 at 3:20:33 PM UTC-5, Nitin Iyer wrote:

[Adarsh Chauhan]

Nitin,

I agree with Chris. Can you please clarify that?

Thanks. :)

[Felipe Cicero]

I reproduce your problem with another model, I'm training a residual net, tested using caffe.io.load_image(), but i think its too slow and kind a stupid to test image by image, so I tried via LMDB and the result was the same as yours, I used your code with my transformation and the result was the same, all probabilities 0 or 1 and many test errors, and the network is working as it should. I think to be some Datum setting, but I'm not sure, I'm still learning.

I will continue using my dumb method of loop and picture by picture, if anyone has a solution I thank you