Understanding test accuracy usage over mini-batches

[Isha Garg]

Hi, 

I've hit a snag for the first time that I can't resolve reading the previous threads. I probably am missing something simple, so I'd be glad if someone could point me in the right direction.

I'm fine-tuning a network, and have written the code for measuring accuracy as per the Learning LeNet notebook tutorial on the Caffe site. As an afterthought, I introduced a top-1 accuracy layer in my train_val.prototxt as well, just to be doubly sure. The numbers however are not matching up.

My solver.prototxt has a test_iterations=200 and my train_val.prototxt has a test batch_size of 50. (TinyImageNet data, 50 validation samples, 200 classes)

The code snippet is this, and it gives me test_acc[-1]=0.443 whereas top1 reports at 0.5

correct = 0
for test_it in range(200):
  solver.test_nets[0].forward()
  correct += sum(solver.test_nets[0].blobs['fc8'].data.argmax(1)
                 == solver.test_nets[0].blobs['label'].data)
test_acc[it // test_interval] = correct / 1e4
print solver.test_nets[0].blobs['accuracy_top1'].data

There is data pre-processing happening in the train_val.prototxt, but that will be common for both methods of calculating accuracy.

 

From Jan's answer on https://groups.google.com/forum/#!topic/caffe-users/_-QYWJmaVPs question, I understand that any multiple of test_interval (in my case, 500), the top-1 accuracy reported by solver.test_nets[0].blobs['accuracy_top1'].data would be the average of all 200 test_iters. Which should match what I'm doing with the code snippet above. Even if print solver.test_nets[0].blobs['accuracy_top1'].data just reports the accuracy for one minibatch of 50, then I should see the accuracy matching the code printed on the console at that testing run, which I don't.

I'd really appreciate some help on this. Thanks in advance!

Thanks,

Isha Garg