Validation accuracy using "Test a list of images" in custom network is REALLY wrong

[신승원]

Hi. Have a good day.

I'm training a CNN classifier and the learning curve looks like this.

However, when I manually put val.txt into "Test a list of images", the result looks like this.

As there are 281 classes, the top-1 accuracy is a little better than random guessing (0.36%).

This makes no sense when you take the graph above into consideration; as it uses the SAME dataset (training and validation) created by DIGITS, and none of the image files were modified at all, the validation accuracy must be at least above 50%.

I guess it is related to last layers in the network, where the classifying is held; But I'm not sure of the exact reason. Or it might be related to transform_param's scale parameter. But I'm not sure of any.

The network's original prototxt can be read here. The first two conv layers are transferred from autoencoder; the zero learning rate coefficients of them are intended.

Any help or suggestions are appreciated. Thanks.

[Greg Heinrich]

Hello, I don't see anything particularly shocking in your model. Are you seeing the correct result when you test individual images?
Can I ask which versions of DIGITS, Caffe, CUDA and CuDNN you are using?

Thanks.

[신승원]

I'm using:

DIGITS / 4.1-dev
Caffe / 0.15.13
CUDA Toolkit / 7.5.17
CuDNN / 5.0.5

Is transform_param and softmax or loss layers all right? I'm confused...

2016년 9월 30일 금요일 오후 8시 53분 18초 UTC+9, Greg Heinrich 님의 말:

[Greg Heinrich]

Good catch! If you are scaling the data during training/validation with a transform_param you need to apply the same transformation in the deploy network by using a Power layer. See this example: https://github.com/NVIDIA/DIGITS/blob/digits-4.0/digits/standard-networks/caffe/lenet.prototxt#L31-L41

--
You received this message because you are subscribed to the Google Groups "DIGITS Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to digits-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/digits-users/aee3b584-87d0-4a23-b6fa-fd0843b9a53a%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[신승원]

Thanks a lot!
Well, I'm a little confused here, can you help me?
The network I designed had phase: TRAIN and phase: TEST layers for data.
But the example LeNet uses stage: "train", stage: "val" and stage: "deploy" layers.
So I just modified the input layers to match the LeNet one, like HERE. (same layers were mostly skipped)

But this error came out:

ERROR: Layer 'flatdata' references bottom 'scaled' at the TRAIN stage however this blob is not included at that stage. Please consider using an include directive to limit the scope of this layer.

I'm not sure of the difference between phase and stage, can you help me out here again? Thanks!

2016년 9월 30일 금요일 오후 11시 7분 59초 UTC+9, Greg Heinrich 님의 말:

To unsubscribe from this group and stop receiving emails from it, send an email to digits-users...@googlegroups.com.

[Greg Heinrich]

The TEST *phase* applies to both the validation set during training and test images during inference. You can click the question mark in the custom network tab for more information on this. Note that the network description for LeNet changed between DIGITS 4.0 and the tip of the master branch. If you wish to replicate what's being done on the master branch then you need to remove your "transform_param" from the data layers and introduce a "scale" layer (without any include/exclude directive, so it applies to all stages).

[신승원]

Oh, thanks. It works smoothly.

Is there any way to mark this thread closed?

2016년 10월 1일 토요일 오전 12시 13분 46초 UTC+9, Greg Heinrich 님의 말: