Discrepancy at test phase: pycaffe vs caffe test

[pwj]

Dear Caffe Users,

my (pre-trained) caffe model yields different results depending on whether it is tested via pycaffe or from the terminal.
This is even though both methods use exactly the same model and copy the same weights. Here is how I do the calls:

(1) via pycaffe:

model      = '/home/user/caffe/models/modelFolder/network.prototxt'
weights    = '/home/user/caffe/models/modelFolder/pretrainedWeights.caffemodel'
net        = caffe.Net(model, weights, caffe.TEST)

outCaffe1 = net.forward()
outCaffe2 = net.forward()
outCaffe3 = net.forward()


(2) via terminal:

caffe test -model /home/user/caffe/models/modelFolder/network.prototxt -weights /home/user/caffe/models/modelFolder/pretrainedWeights.caffemodel -iterations 3

The differences are considerable, for method (1) testing accuracy is at about 0.5  (with a loss of circa 1.2 for batches of 32, this performance is expected!), while for (2) the accuracy is smaller than 0.05 (with a loss of circa 90 for batches of 32). Moreover, in the network.txt's ImageData layer I have set shuffle: false (which is also the default), in order to ensure both methods run on identical data. I can't find an explanation for this. Are the copied weights somehow overwritten in method (2)? How could I check this? Is there any other test I could conduct to find the cause of this difference?

Thank you very much for any insights!

[pwj]

Some further diagnostics w.r.t.:

Are the copied weights somehow overwritten in method (2)? How could I check this?

When calling

 caffe train -solver /home/user/caffe/models/modelFolder/network_solver.prototxt -weights /home/user/caffe/models/modelFolder/pretrainedWeights.caffemodel


the reported network performances during the validation phase are corresponding to those when calling caffe test, i.e. method (2) described above. After 1 iteration at zero learning rate I Ctrl+C the process and load the saved snapshot as the weights for method (1) in pycaffe. The reported performance values are back to those previously measured via method (1) when loading the pretrainedWeights.caffemodel.


[pwj]

Some further diagnostics:

I'm currently comparing results on the following minimal network:

name: "minimalNetwork"

layer {
    name: "img-nat"
    type: "ImageData"            
    top: "img-nat"        
    top: "labels"

    image_data_param {
        source: "/home/user/caffe/labels/nat_val.txt"        
        is_color: true
        batch_size: 32            
        new_height: 227
        new_width: 227
        shuffle: false
    }
    transform_param {
        crop_size: 227
    }
}

layer {
    name: "conv1-nat"
    type: "Convolution"
    bottom: "img-nat"
    top: "conv1-nat"
    param {
        lr_mult: 0
        decay_mult: 1
    }
    param {
        lr_mult: 0
        decay_mult: 0
    }
    convolution_param {
            num_output: 96
            kernel_size: 11
            stride: 4
    }
}

When running method (1) the output information is

outCaffe1["conv1-nat"][0,0,0,:]
Out[1]:
array([  7.64552259,   4.03639174,   8.78586674,  11.27703667,
         0.13313448,   ...], dtype=float32)

When running method (2) the output information is

I0406 13:09:02.739850 28307 caffe.cpp:313] Batch 0, conv1-nat = 7.05141
I0406 13:09:02.739914 28307 caffe.cpp:313] Batch 0, conv1-nat = 3.44813
I0406 13:09:02.739928 28307 caffe.cpp:313] Batch 0, conv1-nat = 8.17712
I0406 13:09:02.739939 28307 caffe.cpp:313] Batch 0, conv1-nat = 10.6507
I0406 13:09:02.739949 28307 caffe.cpp:313] Batch 0, conv1-nat = -0.496098
...

So there seems to be a nearby constant difference between the "conv1-nat" blobs of both methods. This difference is even though the blobs "img-nat" are identical (checked this before).

Is there something wrong in my caffe build? Even if so, pycaffe is only a wrapper so where does this difference come from and how to resolve it?

[Przemek D]

And you are absolutely sure that img-nat blob is identical in both cases?

Does your build pass make runtest?

[pwj]

Hello,

thanks for bearing with me here! The problem is now solved:

The input blobs were in fact identical but there seemed to be a problem with the compilation.
Previously, caffe was installed via the conda package manager and now I re-compiled it manually,
with the current properly working build passing make runtest.