Caffe failed with separate data and label layer for siamese network

[Daoyuan Jia]

Hi. I met a problem when running a revised siamese network (Only revised input layer). 

I am trying to use float label, so I separated the data_layer, one for data and another for float-label (float label to measure the similarity). The net prototxt for this part is:

layer {
  name: "pair_data"
  type: "Data"
  top: "pair_data"
  include {
    phase: TRAIN
  }
  transform_param {
    scale: 0.00390625
  }
  data_param {
    source: "../train_leveldb"
    batch_size: 64
  }
}


layer {
  name: "dist"
  type: "Data"
  top: "dist"
  include {
    phase: TRAIN
  }
  data_param {
    source: "../train_leveldb_label"
    batch_size: 64
  }
}

layer {
  name: "slice_pair"
  type: "Slice"
  bottom: "pair_data"
  top: "data"
  top: "data_p"
  slice_param {
    slice_dim: 1
    slice_point: 3
  }
}

The ../train_leveldb and train_leveldb_label was built with python, and they are aligned (no shuffle, corresponding data-entry and label-entry shared the same 'key' in two leveldb). Each entry train_leveldb_label was a float, and each entry in ../train_leveldb was composed of two image of 120x160. Two image matrices were transposed and vertically stacked. From the caffe console output (below), I think the data matrices were correctly read and sliced. 'dist' (label) was only used in Contrastive Loss layer.

I0930 15:34:05.866102 30811 layer_factory.hpp:74] Creating layer pair_data

I0930 15:34:05.866116 30811 net.cpp:84] Creating Layer pair_data

I0930 15:34:05.866122 30811 net.cpp:338] pair_data -> pair_data

I0930 15:34:05.866132 30811 net.cpp:113] Setting up pair_data

I0930 15:34:05.991760 30811 db.cpp:20] Opened leveldb ../test_leveldb2

I0930 15:34:05.992758 30811 data_layer.cpp:67] output data size: 64,6,120,160  <--- Two image vertically stacked together.

I0930 15:34:06.022660 30811 net.cpp:120] Top shape: 64 6 120 160 (7372800)   <---- TWO IMAGE VERTICALLY STACKED TOGETHER

I0930 15:34:06.022702 30811 layer_factory.hpp:74] Creating layer distance

I0930 15:34:06.022733 30811 net.cpp:84] Creating Layer distance

I0930 15:34:06.022742 30811 net.cpp:338] distance -> dist

I0930 15:34:06.022757 30811 net.cpp:113] Setting up distance

I0930 15:34:06.145051 30811 db.cpp:20] Opened leveldb ../test_leveldb2_label

I0930 15:34:06.145248 30811 data_layer.cpp:67] output data size: 64,1,1,1     <----- FLOAT USED AS LABEL 

I0930 15:34:06.145292 30811 net.cpp:120] Top shape: 64 1 1 1 (64)

I0930 15:34:06.145303 30811 layer_factory.hpp:74] Creating layer slice_pair

I0930 15:34:06.145319 30811 net.cpp:84] Creating Layer slice_pair

I0930 15:34:06.145341 30811 net.cpp:380] slice_pair <- pair_data

I0930 15:34:06.145352 30811 net.cpp:338] slice_pair -> data

I0930 15:34:06.145365 30811 net.cpp:338] slice_pair -> data_p

I0930 15:34:06.145375 30811 net.cpp:113] Setting up slice_pair

I0930 15:34:06.145385 30811 net.cpp:120] Top shape: 64 3 120 160 (3686400)    <--- FIRST IMAGE AFTER SLICE

I0930 15:34:06.145393 30811 net.cpp:120] Top shape: 64 3 120 160 (3686400)    <--- SECOND IMAGE AFTER SLICE

I0930 15:34:06.145401 30811 layer_factory.hpp:74] Creating layer conv1

I0930 15:34:06.145414 30811 net.cpp:84] Creating Layer conv1

I0930 15:34:06.145421 30811 net.cpp:380] conv1 <- data

I0930 15:34:06.145431 30811 net.cpp:338] conv1 -> conv1

I0930 15:34:06.145440 30811 net.cpp:113] Setting up conv1

However, caffe failed with error message like:

I0930 15:34:06.580920 30811 net.cpp:167] ip1 needs backward computation.

I0930 15:34:06.580926 30811 net.cpp:167] pool2 needs backward computation.

I0930 15:34:06.580932 30811 net.cpp:167] conv2 needs backward computation.

I0930 15:34:06.580938 30811 net.cpp:167] pool1 needs backward computation.

I0930 15:34:06.580945 30811 net.cpp:167] conv1 needs backward computation.

I0930 15:34:06.580950 30811 net.cpp:169] slice_pair does not need backward computation.

I0930 15:34:06.580956 30811 net.cpp:169] distance does not need backward computation.

I0930 15:34:06.580962 30811 net.cpp:169] pair_data does not need backward computation.

I0930 15:34:06.580968 30811 net.cpp:205] This network produces output loss

I0930 15:34:06.580981 30811 net.cpp:447] Collecting Learning Rate and Weight Decay.

I0930 15:34:06.580991 30811 net.cpp:217] Network initialization done.

I0930 15:34:06.580996 30811 net.cpp:218] Memory required for data: 419001604

I0930 15:34:06.581076 30811 solver.cpp:42] Solver scaffolding done.

I0930 15:34:06.581123 30811 solver.cpp:222] Solving siamese_dist

I0930 15:34:06.581130 30811 solver.cpp:223] Learning Rate Policy: inv

I0930 15:34:06.581138 30811 solver.cpp:266] Iteration 0, Testing net (#0)

F0930 15:34:06.599118 30828 data_transformer.cpp:137] Check failed: channels == datum_channels (6 vs. 1) 

*** Check failure stack trace: ***

    @     0x7fd41509cdaa  (unknown)

    @     0x7fd41509cce4  (unknown)

    @     0x7fd41509c6e6  (unknown)

    @     0x7fd41509f687  (unknown)

    @     0x7fd41540ce11  caffe::DataTransformer<>::Transform()

    @     0x7fd41547dba8  caffe::DataLayer<>::InternalThreadEntry()

    @     0x7fd40cdbda4a  (unknown)

    @     0x7fd40b9c8182  start_thread

    @     0x7fd41468747d  (unknown)

    @              (nil)  (unknown)

Is this because the transform mistake label_layer (dist) as the same as data_layer(pair_data)? 

How do I solve them? 

Is there any thing wrong in my prototxt / code / method?

Thanks for your help.

For reference, my python code to build the leveldb is attached below:

leftimg = caffe.io.load_image(left) #load first image
rightimg=  caffe.io.load_image(right) #load second image

leftimg = leftimg.transpose((2,0,1)) * 255
rightimg = rightimg.transpose((2,0,1)) * 255
image = np.vstack((leftimg, rightimg))
image = image.astype(np.uint8, copy = False)

#build image-data datum
datumData = caffe_pb2.Datum()
datumData.channels, datumData.height, datumData.width = image.shape
datumData.data = image.tostring()


#build label-data datum (float label)
datumlabel = caffe_pb2.Datum()
datumlabel.channels, datumlabel.height, datumlabel.width = (1,1,1)
datumlabel.float_data.extend(np.array([float(dist)]).flat) # dist is a float, measuring the distance of 2 image

#write to db
name = '%s_%s' % (path.basename(left), path.basename(right))
wb.put('%08d_%s' % (count, name), datumData.SerializeToString())
wblabel.put('%08d_%s' % (count, name), datumlabel.SerializeToString())

 

[De Wei]

Hi did you manage to solve this problem? i am facing similar issue