Behaviour of C++ and Python code differs while predicting

[Christian Bartz]

Hi,

we do have a very interesting problem regarding predictions made with Caffe. We've created a C++ tool that can use trained caffe nets to make predictions. We thought it is working fine until we started to use models containing the Batch-Normalization Layer (https://github.com/BVLC/caffe/pull/3229). The predictions made by the C++ tool do not make any sense compared to the accuracy results during testing. While debugging we used the python interface to make predictions using the same network description and same model. We determined that predictions do work flawlessly there. During further investigation we also determined that prediction differences not only occur while using networks with Batch Normalization but also while using other networks. The difference there is that the python code delivers better results on the same model and the same input data.

In order to get rid of that, we tried to adapt our code as much as possible to behave like the python code, but the differences during prediction stay the same.

Is there anyone who might know what's happening there and why?

Below you can find the following code information:

C++ Code for prediction: http://hastebin.com/wikisohubo.cpp

Python Code for prediction: http://hastebin.com/hihakimeve.py

Thank you for having a look at the problem!

[Jan C Peters]

Very strange. For debugging I'd advise to use some very simple test images (completely black, completely white etc.) and look at the results, also the features computed by every layer, compare the results between python and C++ and try to find out where things go wrong. That brings you probably one step closer to WHY things go wrong. I can honestly only think of something that has to do with the preprocesing of the data, data layout in the input/output blobs and/or interpretation of the output blobs. Since everything inbetween should really be the same for pycaffe and C++ API.

Jan

[Christian Bartz]

Thanks for your reply.

I've created a black and a white image and tried different networks with these images. The interesting thing is that the python code and the C++ tool always return the same result for the black image, but the results differ while using the white image.

I'll try to compare the activations of each layer. Do you have a tip on how to do this easily in C++? For now I would try to get the data of each blob and put in a file or something and do the same with the python data and then compare both with a tool like meld or s.th... do you have any better ideas?

We also thought that the preprocessing of the data might be the key. But we are actually doing nothing more than I have shown in the code above. In C++ the data gets loaded before it is passed to the prediction function with

cv::imread(imagePath, cv::IMREAD_GRAYSCALE), and we are using PIL (pillow) in python for loading the image.

Christian

[Felix Abecassis]

In C++, try something like that:
for (auto& blob_name : net_->blob_names())
{
    auto blob = net_->blob_by_name(blob_name);

    int num = blob->num();
    int width = blob->width();
    int height = blob->height();
    int channels = blob->channels();
    const float* data = blob->cpu_data();
    std::cout << "Blob " << blob_name << ": "
                  << "(" << channels << "x" << width << "x" << height << "x" << num << ")"
                  << std::endl;
    [dump to file...]
}

I would also make sure that the input data after preprocessing is actually the same in both cases.

[Jan C Peters]

For black images it is probably the same because there are just zeros propagated through the network, which no matter what filtering, pooling, nonlinearity or fully connected layer is applied will stay zero.

Maybe it is a scaling issue related to value ranges of [0,255] instead of [0,1] or vice versa?

Felix method is a good way to do it, as you can do something very similar with respective output in python. Only take care at dumping the blob's data: If you are using a text format (as is suggested by Felix's post) to print float values, and you want to compare it in an automated manner (e.g. diff), make sure you use a consistent output format for those float values (e.g. fixed number of decimals).

My bet is still that the strange things are happening at the input step, not inside the network. As Felix said, inspect the input blobs in both cases, if they match and the rest does not that would be really strange...

Jan

[Christian Bartz]

Thank you for your suggestions!

I think I found the error in my C++ Code. As you suggested I dumped all the data and compared the data. While doing so I determined that the input data while using the C++ code looked very suspicious and I found out that the opencv function for normalizing the image did not work as I expected. I changed the normalization to behave as it would in python. Furthermore I found out that I was not correctly converting the datatypes of my Image from uchar to float32. Now everything works fine.

There is still a small difference between both approaches, but this is due to the different resizing functions I use (opencv resize in C++, and Image.resize from PIL in python).

Thanks for your help guys!

[Saman Sarraf]

Hi Christian, 

I have pretty the same issue. I am wondering how you re-scaled and normalized data in the C++ interface? For example, in MNIST example we use the scaling factor = 0.00390625 in training and testing phase. But in Depolyment, I'm not sure how to do it . In classification.cpp I multiplied the input by this value but still not working. Would you please share your ideas/solution with us?

Thanks

[Christian Bartz]

Hi Saman,

I am basically doing the following right now:

1. load the image:

   cv::Mat image = cv::imread(config.at("image_path"), cv::IMREAD_GRAYSCALE)

2. resize image to desired size:
   cv::Mat resizedImg;
   

   cv::resize(image, resizedImg, m_imageSize, 0, 0, cv::INTER_AREA);
   image = resizedImg;

3. normalize Image. I'm doing this in a dynamical faashion which is definitely not the best idea as it is not really subtracting the average over all training image but rather a normalisation on each image according to its values:

   image.convertTo(image, CV_32FC1);

   normalizeImage(image);

   This is the code for normalizing the images:

   void normalizeImage(cv::Mat & image)
   {

       double minVal;

       double maxVal;

       cv::minMaxLoc(image, &minVal, &maxVal);

       if (minVal != maxVal)

       {

           image -= minVal;

           image /= maxVal - minVal;

       }

       else

       {

           image /= maxVal;

       }

   }

   
   
4. Put the image into the Caffe Net and retrieve the prediction

So this is my current solution. I'm not sure whether this is s.th. which really helps you.

What do you mean by saying it does not work?

Christian

[Sirim Bak]

Hey,

I know thats this post is marked as completed. I'm having similar issue. The link to the prediction code is dead end, can you please post it again

[Christian Bartz]

Hi,

I'm afraid I can not help you with this. I don't have the state of the code I've posted here anymore.

Christian