Output the test accuracy per category

[Gilad Sharir]

Hi,

I would like the Accuracy layer to output a separate test accuracy value for each category.
Is this feature implemented in Caffe, or does it require some code changes?

Thanks

[Ran Manor]

I didn't see a way to currently do this with Caffe so I implemented it on my own.

Basically I outputted from the AccuracyLayer some more outputs, i.e. the number of samples per class and the number of correct classifications per class.

Then in the solver I summed it up and divided the number of corrects by the total number per class.

I don't have time to make this into a good PR, but hopefully soon.

[Gilad Sharir]

Thanks,  I did something similar to what you described..

[Diego Rueda]

Can you guys show your process? it will be really helpful

[Ran Manor]

I started a PR here: https://github.com/BVLC/caffe/pull/1442#issuecomment-63729702
I have some more work on it, hopefully I'll get to it soon.

--
You received this message because you are subscribed to a topic in the Google Groups "Caffe Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/caffe-users/Y2JKRSt-yAI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to caffe-users...@googlegroups.com.
To post to this group, send email to caffe...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/caffe-users/d9ce2616-9bf2-4abd-ac16-7f10353aaa0d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Zheng Shou]

Hi Ran,

This functionality is really cool.

But I tried it and it didn't output per class accuracy as expected. I am wondering is there any thing to do besides adding

accuracy_param {

    output_per_class: true

  }

to the accuracy layer? Is there anythings else I need to do or any example?

Thanks so much!

在 2014年11月25日星期二 UTC-5下午4:40:24，Ran Manor写道：

[Ran Manor]

Hi Zheng,
The PR changed, it doesn't require the parameter now but another top blob in the layer definition.
Check it out after the changes.
I hope it will be merged soon..

To view this discussion on the web visit https://groups.google.com/d/msgid/caffe-users/714cb4d2-52d9-4b11-bc42-ff0c2a699c4d%40googlegroups.com.

[Zheng Shou]

Thanks. So what's the name of that another top blob?

Thanks, and All the best - Zheng

[Ran Manor]

The name doesn't matter, it just uses the second one.
Just make sure you use the updated PR.

[Cena]

Hi Ran

    When I use your PR to print per class accuracy.The output of the per class accuracy is -nan.Could you please tell me the possible reason？Thanks very much.

在 2015年2月17日星期二 UTC+8下午7:33:34，Ran Manor写道：

[Ran Manor]

Maybe there were no samples for this class in the test batch, so the denominator was zero.

To view this discussion on the web visit https://groups.google.com/d/msgid/caffe-users/c468fcdc-0db3-4aa0-900a-c347a857b80e%40googlegroups.com.

[Paul Delamusica]

What is a PR and how could one use it?

I had a similar problem here.
https://groups.google.com/forum/?utm_source=digest&utm_medium=email#!topic/caffe-users/fMxBRpdEIAk

[vanm...@stolaf.edu]

Hey Ran,

I hear that your PR is awesome, and I would love to use it. The problem is I am new to Caffe, and I am running into problems trying to do what you have stated here. I went https://github.com/BVLC/caffe/pull/2935 and see that chensiqin sort of has an example, but it is not in a format that really works well for me. My train_val.prototxt for both data layers uses lmdbs of my images, and after trying to use the image_data_param with source: it just doesn't seem to know how to open my images!

name: "LeNet"
layer {
  name: "mnist"
  type: "Data"
  top: "data"
  top: "label"
  include {
    phase: TRAIN
  }
  transform_param {
    scale: 0.00390625
  }
  data_param {
    source: "examples/otternet30px/otter_train_lmdb"
    batch_size: 100
    backend: LMDB
  }
}
layer {
  name: "mnist"
  type: "Data"
  top: "data"
  top: "label"
  include {
    phase: TEST
  }
  transform_param {
    scale: 0.00390625
  }
  data_param {
    source: "examples/otternet30px/otter_val_lmdb"
    batch_size: 100
    backend: LMDB
  }
}

...

layer {
  name: "accuracy"
  type: "Accuracy"
  bottom: "ip2"
  bottom: "label"
  top: "accuracy"
  include {
    phase: TEST
  }
}
layer {
  name: "loss"
  type: "SoftmaxWithLoss"
  bottom: "ip2"
  bottom: "label"
  top: "loss"
}

When I tried doing chensiqin's, I run into problems due to source: ...  I can have it open my val.txt but then it fails to open my correct images that correspond with the val.txt
How do I set this up?

Thank you,


layer {
name: "data"
type: "ImageData"
top: "data"
top: "label"
include {
phase: TEST
}
transform_param {
mirror: true
crop_size: 224
mean_file: "animal_mean.binaryproto"
}
image_data_param {
source: "/str/users/chensq/experiments/network/caffe/data/39/val_animal_0-37.txt"
batch_size: 50
shuffle: true
new_height: 256
new_width: 256

[Jose Carranza]

Hi Ran

Sorry for writing after so long. I wanted to know if your PR is already in the main trunk of Caffe. If so, how do you activate the accuracy-per-class option? what is the syntax?

Thanks in advance for the help

Jose

[Ran Manor]

Sorry, I haven't used caffe for a long time , I don't remember.

To view this discussion on the web visit https://groups.google.com/d/msgid/caffe-users/2ba2e88d-94c9-4978-878e-3c61412b5977%40googlegroups.com.

[Przemek D]

Just in case anyone stumbled upon this question sometime in the future, it has already been asked and answered here: https://groups.google.com/forum/#!topic/caffe-users/vCWeq6KSaDc