transform the data with Caffe::dataTransformer using C++ API

[Jianyu Lin]

Hi,

Now I plan to deploy the trained caffemodel via C++ API, to use it in my C++ project, but I met a problem of transforming the input data for the forward pass.

The training prototxt defines a scaling factor as follows:

layer {

  name: "data"

  type: "Data"

  top: "data"

  include {

    phase: TRAIN

  }

  transform_param {

    scale: 0.00390625

  }

  data_param {

    source: "examples/SL/fcn_train_image_2_lmdb"

    batch_size: 1

    backend: LMDB

  }

}

but in the deploy prototxt the only thing that can be defined is the input blob size, as follows:

input: "data"

input_dim: 1

input_dim: 3

input_dim: 384

input_dim: 512

And I think the data transform of the input data for the forward pass can only be done via caffe::dataTransformer (http://stackoverflow.com/questions/32208193/how-to-modify-caffe-network-input-for-c-api)

So does anyone have the experience of transforming the input datum/blob in C++ API? Thanks. a lot.

Jianyu

[Jan C Peters]

I have no experience with the transformer, but I can tell you that of course you can do the preprocessing yourself. This magic scaling factor there is nothing but 1/256 and used to convert the pixel values from the [0,255] range to the [0,1] range. To implement that yourself is child's play. If you use OpenCV for loading the image you may even directly use their facilities to multiply the complete image data with that number.

Most things the transformer does are very simple preprocessing steps that you could do yourself, but this facility is there to not require any other tools or scripts from you, so you can possibly go directly from a folder of images to trained network, or predictions.

Jan

[Jianyu Lin]

Thanks Jan.

I'm quite new to Caffe, and I used the CVMatToDatum function to convert Mat to Datum, then to blob. But CVMatToDatum only handles unsigned char Mat.

If I convert my Mat to float (due to scaling), how shall I convert the Mat? 

Jianyu

[Jan C Peters]

To do it that way seems awfully inefficient, you can load the image into a cv::Mat, map the memory of the target blob to another cv::Mat and do the conversion in one statement, similar to this:

cv::Mat tmp;
cv::imread("myimage.png", CV_LOAD_IMAGE_GRAYSCALE).convertTo(tmp, CV_32FC1);

// get your blob from somewhere:
Blob<float>* b = net.input_blobs()[0]

// map blob's memory to mat
// make sure the dimensions of the loaded image and target blob match! (omitted here)
cv::Mat blob_mat(b->height(), b->width(), CV_32FC1, b->mutable_cpu_data());

// scale by 256 and move to blob
blob_mat = tmp / 256.0f;

// now the blob contains the image and you can call net.forward() or something like that.

Jan

[Jan C Peters]

Actually, for color images or 3-channel blobs, respectively, you should not do the LOAD_IMAGE_GRAYSCALE and rather use the CV_32FC3 data type for all the matrices, but you'll figure that out yourself. You might need to transpose the color data array from [H,W,3] to [3,H,W].

Jan

[Jianyu Lin]

Thank you. I tried your method, found that scaling blob_mat actually doesn't modify b->mutable_cpu_data()?
 any other way to convert Mat to b->mutable_cpu_data?

[Jianyu Lin]

apologize- it works! thanks a lot!

Jianyu

[Jianyu Lin]

Thank you again for helping me with both of my questions.

I followed your advise to use blob->mutable_cpu_data() to get access to the blob data. What I did is to transform the data one by one:

  float* input_data = blob->mutable_cpu_data();

  for (int i = 0; i<blob->channels()*blob->height()*blob->width();i++){

    input_data[i] = input_data[i]/255;

  }

Thanks again for your help.

Best wishes,

Jianyu

[ans...@silklabs.com]

The code below doesn't seem to be working fine for me. Caffe is reporting incorrect prediction with the code. I ended up using a modified CVMatToDatum to achieve the result. 

[Jianyu Lin]

This is my code to convert the data into blob deploy a forward pass:
Please suggest if there is a better way.
-------------------------------------------------------------------------------------------
    /// fcn segmentation
    /// step 1: prepare data for the neural networks
    //get datum
    caffe::Datum datum;
    CVMatToDatum(src, &datum);
    //get the blob
    Blob<float>* blob = new Blob<float>(1, datum.channels(), datum.height(), datum.width());

    //get the blobproto
    caffe::BlobProto blob_proto;
    blob_proto.set_num(1);
    blob_proto.set_channels(datum.channels());
    blob_proto.set_height(datum.height());
    blob_proto.set_width(datum.width());
    int size_in_datum = std::max<int>(datum.data().size(),
                                      datum.float_data_size());

    for (int i = 0; i < size_in_datum; ++i) {
        blob_proto.add_data(0.);
    }
    const string& data = datum.data();
    if (data.size() != 0) {
        for (int i = 0; i < size_in_datum; ++i) {
            blob_proto.set_data(i, blob_proto.data(i) + (uint8_t)data[i]);
        }
    }
    //set data into blob
    blob->FromProto(blob_proto);  // the blob is always in float!!
    //fill the vector
    src.convertTo(src, CV_32FC3, 1/255.0); //for scaling

    float* input_data = blob->mutable_cpu_data();
    for (int i = 0; i<blob->channels()*blob->height()*blob->width();i++){
        input_data[i] = input_data[i]/255;
    }

    vector<Blob<float>*> bottom_vec;
    bottom_vec.push_back(blob);

    float iter_loss;

    /// step 2: forward pass

    const vector<Blob<float>*>& result =
            caffe_net.Forward(bottom_vec, &iter_loss);
    const float* result_vec = result[0]->cpu_data();

-------------------------------------------------------------------------
All the best
Jianyu

[wolge...@gmail.com]

Hi, sorry I can't help you now since I am on the same road but can't access the caffe lib yet. Can you please give me a direction how do you call the caffe methods and which ones do I have to wrap as managed code for .Net. If I catch up with you I would like to share our knowledge if you like.

br

Wolfgang