How is accuracy calculated in Caffe

[krishhtof]

Hello,

I'm new to Caffe and I'm willing to know, how is accuracy for any network calculated by Caffe at the end of training. I know that its calculated on the validation set, but is there any particular formula here as to how its calculated?

Thanks :)

krishhtof

[Jan C Peters]

There is a slight difference for TRAIN and TEST phase here (as you can have an accuracy layer in the TRAIN phase as well):

The accuracy layer itself calculates the accurace of the current minibatch _only_, since that is all the data it sees at a time. The accuracy is simple the relative number of correctly classified samples in the current minibatch. The inputs must be provided in the following format: the first bottom blob yields the prediction in the form of a k-element vector of values (persample, that is actually a (k,N)-blobsize for a minibatch size of N). The accuracy layer effectively computes an argmax over these values (per sample of course) which is the predicted class index, and compares that with the ground-truth class label given through the second bottom blob, which then should naturally a single number per sample in the range 0..k-1.

In TRAIN phase the accuracy layers outputs are just printed to the console, i.e. the values are only for the current batch, which is not a very informative value in general. In TEST mode (which happens at every test_interval training iterations), the solver accumulates the accuracy layer's outputs and averages them over all test iterations (actually this averaging happens for all output blobs then, for the loss as well), which gives a more representative performance indicator of your current network.

Jan

[Hossein Hasanpour]

Jan, would you please, elaborate more on this? I am getting confused . 
I have used this script to calculate the confusion matrix, this however, reports the accuracy as well, the problem is the accuracy reported by this script is way different thatn the one reported by caffe! . 

for example caffe reports accuracy lets say for CIFAR10, as 92.34%, while, when the model is fed to the script to calculate confusion matrix and its accuracy, it results in for example something like 86.5% .!

which one of these accuracies are the correct one, and can be reported in papers or compared with the results of other papers such as those here ?

I also saw something weird again, I trained two identical models, with only one difference, that being one used xavier, and the other used msra for initialization. 

the first one reports an accuracy of 94.25 and the other reports 94.26 in caffe. when these models are fed to the script I linked above, for confusion matrix computations. their accuracies were 89.2% and 87.4% respectivly. !

Is this normal? what is the cause for this? msra? 

I really dont know if the accuracies reported by caffe are truth-worthy. I'd be grateful if you could shed some light on the matter, 

Best regards 

Hossein 

[Hossein Hasanpour]

Problems solved. The reason and the solution is posted here : 

the reason was because of not taking mean-subtraction into account. 

[Nathan Ing]

Hi, I am resurrecting this thread to ask a follow up to this nice answer.

The accuracy layer also has per-class accuracy, if requested by specifying >1 top blob.

Would you happen to know if this accuracy is also averaged out over the TEST iterations?

The reason I'm asking is I see strange numbers like:

I0706 16:32:57.164111   975 solver.cpp:399]     Test net output #0: accuracy = 0.808811
I0706 16:32:57.164347   975 solver.cpp:399]     Test net output #1: loss = 0.516258 (* 1 = 0.516258 loss)
I0706 16:32:57.164367   975 solver.cpp:399]     Test net output #2: per_class_accuracy = 0.270592
I0706 16:32:57.164377   975 solver.cpp:399]     Test net output #3: per_class_accuracy = 0.353006
I0706 16:32:57.164387   975 solver.cpp:399]     Test net output #4: per_class_accuracy = 0.160097
I0706 16:32:57.164397   975 solver.cpp:399]     Test net output #5: per_class_accuracy = 0.696964

Thanks!