Loading batch and labels directly from python in finetuning

[Rafael Padilha]

Hi everybody,

I am having some problems when I try to load a batch of images + labels within python and try use it to train a network.

I am working with pair of images, that I transform into one and then I feed it to the network. As the number of pairs is too big (combination of the individual images two by two) to store all in memory, I am creating each batch in each iteration and I would like to give it to the network.

I am instantiating the network this way:

solver = caffe.get_solver(path_to_solver_file)

where the input network layers are defined with:

layer {

  name: "data"

  type: "ImageData"

  top: "data"

  top: "label"

  include {

    phase: TRAIN

  }

  image_data_param { 

    batch_size: 1

    new_height: 227

    new_width: 227

  }

}

layer {

  name: "data"

  type: "ImageData"

  top: "data"

  top: "label"

  include {

    phase: TEST

  }

  image_data_param {

    batch_size: 1

    new_height: 227

    new_width: 227

  }

}

In my earlier experiments, I had a source parameter inside image_data_param, where I would pass a file with my images and labels for both training and test.

However, as I want to load them within python, I removed the source parameter, but got the following error:

I0830 17:01:49.014819 1967923200 layer_factory.hpp:77] Creating layer data

I0830 17:01:49.014858 1967923200 net.cpp:91] Creating Layer data

I0830 17:01:49.014868 1967923200 net.cpp:399] data -> data

I0830 17:01:49.014890 1967923200 net.cpp:399] data -> label

I0830 17:01:49.014910 1967923200 image_data_layer.cpp:38] Opening file 

I0830 17:01:49.014935 1967923200 image_data_layer.cpp:53] A total of 0 images.

Segmentation fault: 11

I didn't got to this point yet, but after I'm able to instantiate the network, I was going to load the batch and perform one step of the SGD solver using:

net.blobs["data"].data[...] = images

net.blobs["label"].data[...] = labels

net.step(1)

Do you guys have any hint on what I'm doing wrong?

Thanks in advance!

[Evan Shelhamer]

I didn't got to this point yet, but after I'm able to instantiate the network, I was going to load the batch and perform one step of the SGD solver using:

 

net.blobs["data"].data[...] = images
net.blobs["label"].data[...] = labels
net.step(1)

You cannot assign the tops of data layers like this. Instead, define an `Input` layer and assign the tops as you like, or implement your own Python data layer to encapsulate your data processing code. Here's one example of a Python data layer: http://nbviewer.jupyter.org/github/BVLC/caffe/blob/master/examples/pascal-multilabel-with-datalayer.ipynb

Evan Shelhamer

--
You received this message because you are subscribed to the Google Groups "Caffe Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to caffe-users+unsubscribe@googlegroups.com.
To post to this group, send email to caffe...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/caffe-users/bce15fdb-6e5c-4803-9d98-b2532d2caed5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.