Questions using matcaffe for multi input classification

[Tim]

Hi there, 

I'm quite new to caffe and managed to train a network on my own data set, which is based on 4 grayscale images per sample (2 cameras, 1 depth and 1 intensity each) . For training I followed the imagenet example: converted my train and validation images to lmdb, resized to 96x96, put the gray option, but no shuffling, computed the mean and setup a simple architecture to see if it works. 

i) After the Data layer I call the Concat layer to combine the 4 images. Here starts my first question: How would the output of the Concat layer look like and how are all the operations performed on it? 

ii) Training is done and I would like to test my net with a testing dataset. Therefore I would like to use matlab and checked the demo classification. The testing images are stored in lmdb format as well as links to the images in a text file. Because of the lacking support of lmdb in matlab I was going to read images from the text file. Might be slow but the amount is just a few 1000 images and time is not a hard constraint at this point. But in the example the image_mean is subtracted from each testing image with a precomputed .mat file. Here comes my second question: From which images should I compute the mean, training or validation? Or is it the mean of each individual image, but this would not make any sense, does it? 

iii) So far the pre-processing in matlab. Starting training from matlab with importing the solver.prototxt and using lmdb works without a problem. I had problems running it -in contrast to above stated- image_data_layer and a text file with absolute path to the files. It never found the images although running it from script like the imagenet example worked fine. Anyhow, with the lmdb it some how works, so third question: After training and loading the test images I load the deploy.prototxt, .caffemodel file and set the phase to "test". After the network initialization is done and the required memory is shown, I run into this 

error: Check failed: ShapeEquals(proto) shape mismatch (reshape not set). 

iv) Assuming it would work fine, last question: How could I feed the images to the net? In particular sending four images at once straight to the net input from matlab. Something like net.forward(input_data, input_data2..) maybe?

The beginning of my deploy.prototxt:

name: "TestNet"

input: "intensity1"

input: "intensity2"

input: "distance1"

input: "distance2"

input_dim: 1

input_dim: 1

input_dim: 96

input_dim: 96

input_dim: 1

input_dim: 1

input_dim: 96

input_dim: 96

input_dim: 1

input_dim: 1

input_dim: 96

input_dim: 96

input_dim: 1

input_dim: 1

input_dim: 96

input_dim: 96

layer {

name: "concat"

type: "Concat"

bottom: "intensity1"

bottom: "intensity2"

bottom: "distance1"

bottom: "distance2"

top: "data"

concat_param {

concat_dim: 1

}

}

and so on.... 

the beginning of my train_val.prototxt: 

name: "TestNet"

# Define first data layer

layer {

  name: "dataIntensity1"

  type: "Data"

  top: "dataIntensity1"

  top: "labelIntensity1"

  include {

    phase: TRAIN

  }

  transform_param {

    mirror: true

#    crop_size: 85

scale: 0.00390625

    mean_file: "/path/to/train_lmdb_intensity1_fold1_mean.binaryproto"

  }

  data_param {

    source: "/path/to/database/train_lmdb_intensity1_fold1"

    batch_size: 50

backend: LMDB

#    is_color: false

  }

}

# Define second data layer

layer {

  name: "dataIntensity2"

  type: "Data"

  top: "dataIntensity2"

  top: "labelIntensity2"

  include {

    phase: TRAIN

  }

  transform_param {

    mirror: true

#    crop_size: 85

scale: 0.00390625

    mean_file: "/path/to/train_lmdb_intensity2_fold1_mean.binaryproto"

  }

  data_param {

    source: "/path/to/database/train_lmdb_intensity2_fold1"

    batch_size: 50

backend: LMDB

# is_color: false

  }

}

# Define third data layer

layer {

  name: "dataDistance1"

  type: "Data"

  top: "dataDistance1"

  top: "labelDistance1"

  include {

    phase: TRAIN

  }

  transform_param {

    mirror: true

#    crop_size: 85

scale: 0.00390625

    mean_file: "/path/to/train_lmdb_distance1_fold1_mean.binaryproto"

  }

  data_param {

    source: "/path/to/database/train_lmdb_distance1_fold1"

    batch_size: 50

backend: LMDB

#    is_color: false

  }

}

# # Define fourth data layer

layer {

  name: "dataDistance2"

  type: "Data"

  top: "dataDistance2"

  top: "labelDistance2"

  include {

    phase: TRAIN

  }

  transform_param {

    mirror: true

#    crop_size: 85

scale: 0.00390625

    mean_file: "/path/to/train_lmdb_distance2_fold1_mean.binaryproto"

  }

  data_param {

    source: "/path/to/database/train_lmdb_distance2_fold1"

    batch_size: 50

backend: LMDB

#    is_color: false

  }

}

layer {

name: "silence"

type: "Silence"

bottom: "labelIntensity2"

bottom: "labelDistance1"

bottom: "labelDistance2"

}

layer {

name: "concat"

type: "Concat"

bottom: "dataIntensity1"

bottom: "dataIntensity2"

bottom: "dataDistance1"

bottom: "dataDistance2"

top: "data"

concat_param {

concat_dim: 1

}

}

and so on.... 

 I appreciate any help and thank you very much in advance!!! 

[Tim]

Fixed iii). Problem was a mismatch of outputs in a fully connected layer.

Other question still up. especially the concat layer and feeding images from matlab to the net gives me headaches.