Multi-task learning - how to define the prototxt files?

4,134 views
Skip to first unread message

Gil Levi

unread,
Nov 25, 2014, 12:56:56 PM11/25/14
to caffe...@googlegroups.com
Hi,

I'm training to train a net to performs classification of two properties from the same images. For example - age and gender. I have 8 classes of age and 2 classes of gender.

I would like that the net would predict both properties from the input image.

I'm getting the following error when trying to train: "Duplicate blobs produced by multiple sources." 

I would really appreciate some advice, as I'm quite stuck and not sure how to solve it. 

I have tried to define the val_train_prototxt file as follows:

layers {
  name: "data"
  type: DATA
  top: "data"
  top: "label_age"
  data_param {
    source: "age_train_leveldb"
    mean_file: "mean.binaryproto"
    batch_size: 50
    crop_size: 227
    mirror: true
  }
  include: { phase: TRAIN }
}
layers {
  name: "data"
  type: DATA
  top: "data"
  top: "label_age"
  data_param {
    source: "age_val_leveldb"
    mean_file: "mean.binaryproto"
    batch_size: 50
    crop_size: 227
    mirror: false
  }
  include: { phase: TEST }
}

layers {
  name: "data"
  type: DATA
  top: "data"
  top: "label_gender"
  data_param {
    source: "gender_train_leveldb"
    mean_file: "mean.binaryproto"
    batch_size: 50
    crop_size: 227
    mirror: true
  }
  include: { phase: TRAIN }
}
layers {
  name: "data"
  type: DATA
  top: "data"
  top: "label_gender"
  data_param {
    source: "gender_val_leveldb"
    mean_file: "mean.binaryproto"
    batch_size: 50
    crop_size: 227
    mirror: false
  }
  include: { phase: TEST }
}
layers {
  name: "conv1"
  type: CONVOLUTION
  bottom: "data"
  top: "conv1"
  blobs_lr: 1
  blobs_lr: 2
  weight_decay: 1
  weight_decay: 0
  convolution_param {
    num_output: 96
    kernel_size: 11
    stride: 4
    weight_filler {
      type: "gaussian"
      std: 0.01
    }
    bias_filler {
      type: "constant"
      value: 0
    }
  }
}
A LOT OF CONV, POOL and NORM LAYERS



layers {
  name: "fc8_age"
  type: INNER_PRODUCT
  bottom: "fc7"
  top: "fc8_age"
  blobs_lr: 10
  blobs_lr: 20
  weight_decay: 1
  weight_decay: 0
  inner_product_param {
    num_output: 8
    weight_filler {
      type: "gaussian"
      std: 0.01
    }
    bias_filler {
      type: "constant"
      value: 0
    }
  }
}
layers {
  name: "fc8_gender"
  type: INNER_PRODUCT
  bottom: "fc7"
  top: "fc8_gender"
  blobs_lr: 10
  blobs_lr: 20
  weight_decay: 1
  weight_decay: 0
  inner_product_param {
    num_output: 2
    weight_filler {
      type: "gaussian"
      std: 0.01
    }
    bias_filler {
      type: "constant"
      value: 0
    }
  }
}
layers {
  name: "loss"
  type: SOFTMAX_LOSS
  bottom: "fc8_age"
  bottom: "label_age"
}
layers {
  name: "accuracy"
  type: ACCURACY
  bottom: "fc8_age"
  bottom: "label_age"
  top: "accuracy"
  include: { phase: TEST }
}

layers {
  name: "loss"
  type: SOFTMAX_LOSS
  bottom: "fc8_gender"
  bottom: "label_gender"
}
layers {
  name: "accuracy"
  type: ACCURACY
  bottom: "fc8_gender"
  bottom: "label_gender"
  top: "accuracy"
  include: { phase: TEST }
}

Alex Bewley

unread,
Nov 26, 2014, 10:16:47 PM11/26/14
to caffe...@googlegroups.com
Hi Gil

You are getting this error because of your data layers.
I.e. at TRAIN phase you have two layers both with:
layers {
  name: "data"
  type: DATA
  top: "data"
...
}
and same again for test.

You should give your age and gender data layers different names and top blobs.
Something like:


layers {
  name: "age_data"
  type: DATA
  top: "data_age"
  top: "label_age"
  data_param {
    source: "age_train_leveldb"
    mean_file: "mean.binaryproto"
    batch_size: 50
    crop_size: 227
    mirror: false
  }
  include: { phase: TRAIN }
}

layers {
  name: "gender_data"
  type: DATA
  top: "data_gender"
  top: "label_gender"
  data_param {
    source: "gender_train_leveldb"
    mean_file: "mean.binaryproto"
    batch_size: 50
    crop_size: 227
    mirror: true
  }
  include: { phase: TRAIN }
}

Also do the same for the TEST data.
If the existing top "data" blob from both age and gender data layers are identical you could just ignore or use a SILENCE layer.

Alex

Gil Levi

unread,
Nov 27, 2014, 8:47:20 AM11/27/14
to caffe...@googlegroups.com
Hi Alex,

Thanks for your answer. 

I got it to train (the loss and accuracy looks ok), but when I try to test it using PyCaffe, I get an excpetion.

I would be very thankful if someone would take a look at the prototxt files and tell me what I'm doing wrong.

Thanks!!!


Here is my val_train.txt file:

name: "CaffeNet"
layers {
  name: "data"
  type: DATA
  top: "data_age"
  top: "label_age"
  data_param {
    source: "age_train_leveldb"
    mean_file: "mean.binaryproto"
    batch_size: 50
    crop_size: 227
    mirror: true
  }
  include: { phase: TRAIN }
}
layers {
  name: "data"
  type: DATA
  top: "data_age"
  top: "label_age"
  data_param {
    source: "age_val_leveldb"
    mean_file: "mean.binaryproto"
    batch_size: 50
    crop_size: 227
    mirror: false
  }
  include: { phase: TEST }
}
layers {
  name: "data"
  type: DATA
  top: "data_gender"
  top: "label_gender"
  data_param {
    source: "gender_train_leveldb"
    mean_file: "mean.binaryproto"
    batch_size: 50
    crop_size: 227
    mirror: true
  }
  include: { phase: TRAIN }
}
layers {
  name: "data"
  type: DATA
  top: "data_gender"
  top: "label_gender"
  data_param {
    source: "gender_val_leveldb"
    mean_file: "mean.binaryproto"
    batch_size: 50
    crop_size: 227
    mirror: false
  }
  include: { phase: TEST }
}
layers {
  name: "conv1"
  type: CONVOLUTION
  bottom: "data_age"
  bottom: "data_gender"
  top: "conv1_age"
  top: "conv1_gender"
  blobs_lr: 1
  blobs_lr: 2
  weight_decay: 1
  weight_decay: 0
  convolution_param {
    num_output: 96
    kernel_size: 11
    stride: 4
    weight_filler {
      type: "gaussian"
      std: 0.01
    }
    bias_filler {
      type: "constant"
      value: 0
    }
  }
}
layers {
  name: "relu1_age"
  type: RELU
  bottom: "conv1_age"
  top: "conv1_age"
}
layers {
  name: "relu1_gender"
  type: RELU
  bottom: "conv1_gender"
  top: "conv1_gender"
}
layers {
  name: "pool1_age"
  type: POOLING
  bottom: "conv1_age"
  top: "pool1_age"
  pooling_param {
    pool: MAX
    kernel_size: 3
    stride: 2
  }
}
layers {
  name: "pool1_gender"
  type: POOLING
  bottom: "conv1_gender"
  top: "pool1_gender"
  pooling_param {
    pool: MAX
    kernel_size: 3
    stride: 2
  }
}
layers {
  name: "norm1_age"
  type: LRN
  bottom: "pool1_age"
  top: "norm1_age"
  lrn_param {
    local_size: 5
    alpha: 0.0001
    beta: 0.75
  }
}
layers {
  name: "norm1_gender"
  type: LRN
  bottom: "pool1_gender"
  top: "norm1_gender"
  lrn_param {
    local_size: 5
    alpha: 0.0001
    beta: 0.75
  }
}
layers {
  name: "conv2"
  type: CONVOLUTION
  bottom: "norm1_age"
  bottom: "norm1_gender"
  top: "conv2_age"
  top: "conv2_gender"
  blobs_lr: 1
  blobs_lr: 2
  weight_decay: 1
  weight_decay: 0
  convolution_param {
    num_output: 256
    pad: 2
    kernel_size: 5
    group: 2
    weight_filler {
      type: "gaussian"
      std: 0.01
    }
    bias_filler {
      type: "constant"
      value: 1
    }
  }
}
layers {
  name: "relu2_age"
  type: RELU
  bottom: "conv2_age"
  top: "conv2_age"
}
layers {
  name: "relu2_gender"
  type: RELU
  bottom: "conv2_gender"
  top: "conv2_gender"
}
layers {
  name: "pool2_age"
  type: POOLING
  bottom: "conv2_age"
  top: "pool2_age"
  pooling_param {
    pool: MAX
    kernel_size: 3
    stride: 2
  }
}
layers {
  name: "pool2_gender"
  type: POOLING
  bottom: "conv2_gender"
  top: "pool2_gender"
  pooling_param {
    pool: MAX
    kernel_size: 3
    stride: 2
  }
}
layers {
  name: "norm2_age"
  type: LRN
  bottom: "pool2_age"
  top: "norm2_age"
  lrn_param {
    local_size: 5
    alpha: 0.0001
    beta: 0.75
  }
}
layers {
  name: "norm2_gender"
  type: LRN
  bottom: "pool2_gender"
  top: "norm2_gender"
  lrn_param {
    local_size: 5
    alpha: 0.0001
    beta: 0.75
  }
}
layers {
  name: "conv3"
  type: CONVOLUTION
  bottom: "norm2_age"
  bottom: "norm2_gender"
  top: "conv3_age"
  top: "conv3_gender"
  blobs_lr: 1
  blobs_lr: 2
  weight_decay: 1
  weight_decay: 0
  convolution_param {
    num_output: 384
    pad: 1
    kernel_size: 3
    weight_filler {
      type: "gaussian"
      std: 0.01
    }
    bias_filler {
      type: "constant"
      value: 0
    }
  }
}
layers {
  name: "relu3_age"
  type: RELU
  bottom: "conv3_age"
  top: "conv3_age"
}
layers {
  name: "relu3_gender"
  type: RELU
  bottom: "conv3_gender"
  top: "conv3_gender"
}
layers {
  name: "conv4"
  type: CONVOLUTION
  bottom: "conv3_age"
  bottom: "conv3_gender"
  top: "conv4_age"
  top: "conv4_gender"
  blobs_lr: 1
  blobs_lr: 2
  weight_decay: 1
  weight_decay: 0
  convolution_param {
    num_output: 384
    pad: 1
    kernel_size: 3
    group: 2
    weight_filler {
      type: "gaussian"
      std: 0.01
    }
    bias_filler {
      type: "constant"
      value: 1
    }
  }
}
layers {
  name: "relu4_age"
  type: RELU
  bottom: "conv4_age"
  top: "conv4_age"
}
layers {
  name: "relu4_gender"
  type: RELU
  bottom: "conv4_gender"
  top: "conv4_gender"
}
layers {
  name: "conv5"
  type: CONVOLUTION
  bottom: "conv4_age"
  bottom: "conv4_gender"
  top: "conv5_age"
  top: "conv5_gender"
  blobs_lr: 1
  blobs_lr: 2
  weight_decay: 1
  weight_decay: 0
  convolution_param {
    num_output: 256
    pad: 1
    kernel_size: 3
    group: 2
    weight_filler {
      type: "gaussian"
      std: 0.01
    }
    bias_filler {
      type: "constant"
      value: 1
    }
  }
}
layers {
  name: "relu5_age"
  type: RELU
  bottom: "conv5_age"
  top: "conv5_age"
}
layers {
  name: "relu5_gender"
  type: RELU
  bottom: "conv5_gender"
  top: "conv5_gender"
}
layers {
  name: "pool5_age"
  type: POOLING
  bottom: "conv5_age"
  top: "pool5_age"
  pooling_param {
    pool: MAX
    kernel_size: 3
    stride: 2
  }
}
layers {
  name: "pool5_gender"
  type: POOLING
  bottom: "conv5_gender"
  top: "pool5_gender"
  pooling_param {
    pool: MAX
    kernel_size: 3
    stride: 2
  }
}
layers {
  name: "fc6_age"
  type: INNER_PRODUCT
  bottom: "pool5_age"
  top: "fc6_age"
  blobs_lr: 1
  blobs_lr: 2
  weight_decay: 1
  weight_decay: 0
  inner_product_param {
    num_output: 4096
    weight_filler {
      type: "gaussian"
      std: 0.005
    }
    bias_filler {
      type: "constant"
      value: 1
    }
  }
}
layers {
  name: "fc6_gender"
  type: INNER_PRODUCT
  bottom: "pool5_gender"
  top: "fc6_gender"
  blobs_lr: 1
  blobs_lr: 2
  weight_decay: 1
  weight_decay: 0
  inner_product_param {
    num_output: 4096
    weight_filler {
      type: "gaussian"
      std: 0.005
    }
    bias_filler {
      type: "constant"
      value: 1
    }
  }
}
layers {
  name: "relu6_age"
  type: RELU
  bottom: "fc6_age"
  top: "fc6_age"
}
layers {
  name: "relu6_gender"
  type: RELU
  bottom: "fc6_age"
  top: "fc6_age"
}
layers {
  name: "drop6_age"
  type: DROPOUT
  bottom: "fc6_age"
  top: "fc6_age"
  dropout_param {
    dropout_ratio: 0.5
  }
}
layers {
  name: "drop6_gender"
  type: DROPOUT
  bottom: "fc6_age"
  top: "fc6_age"
  dropout_param {
    dropout_ratio: 0.5
  }
}
layers {
  name: "fc7_age"
  type: INNER_PRODUCT
  bottom: "fc6_age"
  top: "fc7_age"
  # Note that blobs_lr can be set to 0 to disable any fine-tuning of this, and any other, layer
  blobs_lr: 1
  blobs_lr: 2
  weight_decay: 1
  weight_decay: 0
  inner_product_param {
    num_output: 4096
    weight_filler {
      type: "gaussian"
      std: 0.005
    }
    bias_filler {
      type: "constant"
      value: 1
    }
  }
}
layers {
  name: "fc7_gender"
  type: INNER_PRODUCT
  bottom: "fc6_gender"
  top: "fc7_gender"
  # Note that blobs_lr can be set to 0 to disable any fine-tuning of this, and any other, layer
  blobs_lr: 1
  blobs_lr: 2
  weight_decay: 1
  weight_decay: 0
  inner_product_param {
    num_output: 4096
    weight_filler {
      type: "gaussian"
      std: 0.005
    }
    bias_filler {
      type: "constant"
      value: 1
    }
  }
}
layers {
  name: "relu7_age"
  type: RELU
  bottom: "fc7_age"
  top: "fc7_age"
}
layers {
  name: "relu7_gender"
  type: RELU
  bottom: "fc7_gender"
  top: "fc7_gender"
}
layers {
  name: "drop7_age"
  type: DROPOUT
  bottom: "fc7_age"
  top: "fc7_age"
  dropout_param {
    dropout_ratio: 0.5
  }
}
layers {
  name: "drop7_gender"
  type: DROPOUT
  bottom: "fc7_gender"
  top: "fc7_gender"
  dropout_param {
    dropout_ratio: 0.5
  }
}
layers {
  name: "fc8_age"
  type: INNER_PRODUCT
  bottom: "fc7_age"
  top: "fc8_age"
  # blobs_lr is set to higher than for other layers, because this layer is starting from random while the others are already trained
  blobs_lr: 10
  blobs_lr: 20
  weight_decay: 1
  weight_decay: 0
  inner_product_param {
    num_output: 8
    weight_filler {
      type: "gaussian"
      std: 0.01
    }
    bias_filler {
      type: "constant"
      value: 0
    }
  }
}
layers {
  name: "fc8_gender"
  type: INNER_PRODUCT
  bottom: "fc7_gender"
  top: "fc8_gender"
  # blobs_lr is set to higher than for other layers, because this layer is starting from random while the others are already trained
  blobs_lr: 10
  blobs_lr: 20
  weight_decay: 1
  weight_decay: 0
  inner_product_param {
    num_output: 2
    weight_filler {
      type: "gaussian"
      std: 0.01
    }
    bias_filler {
      type: "constant"
      value: 0
    }
  }
}
layers {
  name: "loss_age"
  type: SOFTMAX_LOSS
  bottom: "fc8_age"
  bottom: "label_age"
}
layers {
  name: "loss_gender"
  type: SOFTMAX_LOSS
  bottom: "fc8_gender"
  bottom: "label_gender"
}
layers {
  name: "accuracy_age"
  type: ACCURACY
  bottom: "fc8_age"
  bottom: "label_age"
  top: "accuracy_age"
  include: { phase: TEST }
}
layers {
  name: "accuracy_gender"
  type: ACCURACY
  bottom: "fc8_gender"
  bottom: "label_gender"
  top: "accuracy_gender"
  include: { phase: TEST }
}




And here is my deploy.ptototxt file:


name: "CaffeNet"
input: "data_age"
input: "data_gender"
input_dim: 1
input_dim: 3
input_dim: 227
input_dim: 227
layers {
  name: "conv1"
  type: CONVOLUTION
  bottom: "data_age"
  bottom: "data_gender"
  top: "conv1_age"
  top: "conv1_gender"
  convolution_param {
    num_output: 96
    kernel_size: 11
    stride: 4
  }
}
layers {
  name: "relu1_age"
  type: RELU
  bottom: "conv1_age"
  top: "conv1_age"
}
layers {
  name: "relu1_gender"
  type: RELU
  bottom: "conv1_gender"
  top: "conv1_gender"
}
layers {
  name: "pool1_age"
  type: POOLING
  bottom: "conv1_age"
  top: "pool1_age"
  pooling_param {
    pool: MAX
    kernel_size: 3
    stride: 2
  }
}
layers {
  name: "pool1_gender"
  type: POOLING
  bottom: "conv1_gender"
  top: "pool1_gender"
  pooling_param {
    pool: MAX
    kernel_size: 3
    stride: 2
  }
}
layers {
  name: "norm1_age"
  type: LRN
  bottom: "pool1_age"
  top: "norm1_age"
  lrn_param {
    local_size: 5
    alpha: 0.0001
    beta: 0.75
  }
}
layers {
  name: "norm1_gender"
  type: LRN
  bottom: "pool1_gender"
  top: "norm1_gender"
  lrn_param {
    local_size: 5
    alpha: 0.0001
    beta: 0.75
  }
}
layers {
  name: "conv2"
  type: CONVOLUTION
  bottom: "norm1_age"
  bottom: "norm1_gender"
  top: "conv2_age"
  top: "conv2_gender"
  blobs_lr: 1
  blobs_lr: 2
  weight_decay: 1
  weight_decay: 0
  convolution_param {
    num_output: 256
    pad: 2
    kernel_size: 5
    group: 2
  }
}
layers {
  name: "relu2_age"
  type: RELU
  bottom: "conv2_age"
  top: "conv2_age"
}
layers {
  name: "relu2_gender"
  type: RELU
  bottom: "conv2_gender"
  top: "conv2_gender"
}
layers {
  name: "pool2_age"
  type: POOLING
  bottom: "conv2_age"
  top: "pool2_age"
  pooling_param {
    pool: MAX
    kernel_size: 3
    stride: 2
  }
}
layers {
  name: "pool2_gender"
  type: POOLING
  bottom: "conv2_gender"
  top: "pool2_gender"
  pooling_param {
    pool: MAX
    kernel_size: 3
    stride: 2
  }
}
layers {
  name: "norm2_age"
  type: LRN
  bottom: "pool2_age"
  top: "norm2_age"
  lrn_param {
    local_size: 5
    alpha: 0.0001
    beta: 0.75
  }
}
layers {
  name: "norm2_gender"
  type: LRN
  bottom: "pool2_gender"
  top: "norm2_gender"
  lrn_param {
    local_size: 5
    alpha: 0.0001
    beta: 0.75
  }
}
layers {
  name: "conv3"
  type: CONVOLUTION
  bottom: "norm2_age"
  bottom: "norm2_gender"
  top: "conv3_age"
  top: "conv3_gender"
  convolution_param {
    num_output: 384
    pad: 1
    kernel_size: 3
  }
}
layers {
  name: "relu3_age"
  type: RELU
  bottom: "conv3_age"
  top: "conv3_age"
}
layers {
  name: "relu3_gender"
  type: RELU
  bottom: "conv3_gender"
  top: "conv3_gender"
}
layers {
  name: "conv4"
  type: CONVOLUTION
  bottom: "conv3_age"
  bottom: "conv3_gender"
  top: "conv4_age"
  top: "conv4_gender"
  convolution_param {
    num_output: 384
    pad: 1
    kernel_size: 3
    group: 2
  }
}
layers {
  name: "relu4_age"
  type: RELU
  bottom: "conv4_age"
  top: "conv4_age"
}
layers {
  name: "relu4_gender"
  type: RELU
  bottom: "conv4_gender"
  top: "conv4_gender"
}
layers {
  name: "conv5"
  type: CONVOLUTION
  bottom: "conv4_age"
  bottom: "conv4_gender"
  top: "conv5_age"
  top: "conv5_gender"
  convolution_param {
    num_output: 256
    pad: 1
    kernel_size: 3
    group: 2
  }
}
layers {
  name: "relu5_age"
  type: RELU
  bottom: "conv5_age"
  top: "conv5_age"
}
layers {
  name: "relu5_gender"
  type: RELU
  bottom: "conv5_gender"
  top: "conv5_gender"
}
layers {
  name: "pool5_age"
  type: POOLING
  bottom: "conv5_age"
  top: "pool5_age"
  pooling_param {
    pool: MAX
    kernel_size: 3
    stride: 2
  }
}
layers {
  name: "pool5_gender"
  type: POOLING
  bottom: "conv5_gender"
  top: "pool5_gender"
  pooling_param {
    pool: MAX
    kernel_size: 3
    stride: 2
  }
}
layers {
  name: "fc6_age"
  type: INNER_PRODUCT
  bottom: "pool5_age"
  top: "fc6_age"
  inner_product_param {
    num_output: 4096
  }
}
layers {
  name: "fc6_gender"
  type: INNER_PRODUCT
  bottom: "pool5_gender"
  top: "fc6_gender"
  inner_product_param {
    num_output: 4096
  }
}
layers {
  name: "relu6_age"
  type: RELU
  bottom: "fc6_age"
  top: "fc6_age"
}
layers {
  name: "relu6_gender"
  type: RELU
  bottom: "fc6_age"
  top: "fc6_age"
}
layers {
  name: "drop6_age"
  type: DROPOUT
  bottom: "fc6_age"
  top: "fc6_age"
  dropout_param {
    dropout_ratio: 0.5
  }
}
layers {
  name: "drop6_gender"
  type: DROPOUT
  bottom: "fc6_age"
  top: "fc6_age"
  dropout_param {
    dropout_ratio: 0.5
  }
}
layers {
  name: "fc7_age"
  type: INNER_PRODUCT
  bottom: "fc6_age"
  top: "fc7_age"
  inner_product_param {
    num_output: 4096
  }
}
layers {
  name: "fc7_gender"
  type: INNER_PRODUCT
  bottom: "fc6_gender"
  top: "fc7_gender"
  inner_product_param {
    num_output: 4096
  }
}
layers {
  name: "relu7_age"
  type: RELU
  bottom: "fc7_age"
  top: "fc7_age"
}
layers {
  name: "relu7_gender"
  type: RELU
  bottom: "fc7_gender"
  top: "fc7_gender"
}
layers {
  name: "drop7_age"
  type: DROPOUT
  bottom: "fc7_age"
  top: "fc7_age"
  dropout_param {
    dropout_ratio: 0.5
  }
}
layers {
  name: "drop7_gender"
  type: DROPOUT
  bottom: "fc7_gender"
  top: "fc7_gender"
  dropout_param {
    dropout_ratio: 0.5
  }
}
layers {
  name: "fc8_age"
  type: INNER_PRODUCT
  bottom: "fc7_age"
  top: "fc8_age"
  inner_product_param {
    num_output: 8
  }
}
layers {
  name: "fc8_gender"
  type: INNER_PRODUCT
  bottom: "fc7_gender"
  top: "fc8_gender"
  inner_product_param {
    num_output: 2
  }
}
layers {
  name: "prob_age"
  type: SOFTMAX
  bottom: "fc8_age"
  bottom: "prob_age"
}
layers {
  name: "prob_gender"
  type: SOFTMAX
  bottom: "fc8_gender"
  bottom: "prob_gender"
}
Message has been deleted
Message has been deleted

Mohamed Omran

unread,
Nov 27, 2014, 10:50:47 AM11/27/14
to caffe...@googlegroups.com
You need to do (at least) two more things to get this to work:

1. Since you have two input blobs, you need two sets of input_dims. The first few lines of your deploy.prototxt should look like this:

input: "data_age"
input: "data_gender"
input_dim: 1
input_dim: 3
input_dim: 227
input_dim: 227
input_dim: 1
input_dim: 3
input_dim: 227
input_dim: 227

Otherwise the following check in ${CAFFE_ROOT}/src/net.cpp will fail: 
CHECK_EQ(param.input_size() * 4, param.input_dim_size()) << "Incorrect input blob dimension specifications.";

2. Also, you need to slightly customise ${CAFFE_ROOT}/python/caffe/classifier.py, which expects just a single input blob (typically "data")
You could change predict(...) as follows:

Instead of:
        out = self.forward_all(**{self.inputs[0]: caffe_in})
use: 
        input_blobs = np.zeros(tuple([len(self.inputs)] + list(caffe_in.shape)))
        for i in range(len(self.inputs)):
            input_blobs[i] = caffe_in
        out = self.forward_all(**dict(zip(self.inputs, input_blobs)))

This is supposed to duplicate the blob containing the input image, and then assign one copy each to "data_age" and "data_gender". If I'm not mistaken, this change should also preserve the existing behaviour of the function. (Disclaimer: Not fully tested, but I've had to do something similar before.)

Cheers,
Mohamed

Gil Levi

unread,
Nov 27, 2014, 12:05:00 PM11/27/14
to caffe...@googlegroups.com
Hi Mohamed,

Thanks for your help. I made the changes you in the prototxt and code as you instructed, but I'm still getting an exception. Any other tips?

Thanks!

Here is the updated deploy.prototxt:



name: "CaffeNet"
input: "data_age"
input: "data_gender"
input_dim: 10
input_dim: 3
input_dim: 227
input_dim: 227
input_dim: 10
input_dim: 3
input_dim: 227
input_dim: 227
layers {
  name: "conv1"
  type: CONVOLUTION
  bottom: "data_age"
  bottom: "data_gender"
  top: "conv1_age"
  top: "conv1_gender"
  convolution_param {
    num_output: 96
    kernel_size: 11
    stride: 4
  }
}
layers {
  name: "fc8_age"
  type: INNER_PRODUCT
  bottom: "fc7_age"
  top: "fc8_age"
  inner_product_param {
    num_output: 8
  }
}
layers {
  name: "fc8_gender"
  type: INNER_PRODUCT
  bottom: "fc7_gender"
  top: "fc8_gender"
  inner_product_param {
    num_output: 2
  }
}
layers {
  name: "prob_age"
  type: SOFTMAX
  bottom: "fc8_age"
  bottom: "prob_age"
}
layers {
  name: "prob_gender"
  type: SOFTMAX
  bottom: "fc8_gender"
  bottom: "prob_gender"
}




Mohamed Omran

unread,
Nov 27, 2014, 12:20:01 PM11/27/14
to caffe...@googlegroups.com
I suspect it's because you feed both data blobs (data_age and data_gender) to subsequent layers (e.g. conv1). I assume these are identical, so you only need to feed one to the network as is, and the other to a new "SILENCE" layer as suggested by Alex, otherwise it will become a network output.

e.g.
[...]
layers {
  name: "conv1"
  type: CONVOLUTION
  bottom: "data_age"
  top: "conv1_age"
  convolution_param {
    num_output: 96
    kernel_size: 11
    stride: 4
  }
}
layers{
  name: "silence"
  type: SILENCE
  bottom: "data_gender"
}
[...]

Gil Levi

unread,
Nov 27, 2014, 1:20:45 PM11/27/14
to caffe...@googlegroups.com
Thanks again Mohamed.

In my version of Caffe, I don't have a silence layer, but I don't mind that the input image will become a network output if I can ignore it and use the predictions instead.

I corrected the deploy.prototxt file according to your comments, but I'm still getting an exception. I removed all the "gender" blobs in all conv layers and also removed the RELU, POOL and NORM layers that use the gender blobs. 

I want to get two output prediction vectors, so I wrote inner product and two soft-max layers.

Any other comments or hints on how I can get it to run?

Thanks


Here is the updated deploy.prototxt file:

name: "CaffeNet"
input: "data_age"
input: "data_gender"
input_dim: 1
input_dim: 3
input_dim: 227
input_dim: 227
input_dim: 1
input_dim: 3
input_dim: 227
input_dim: 227
layers {
  name: "conv1"
  type: CONVOLUTION
  bottom: "data_age"
  top: "conv1_age"
  convolution_param {
    num_output: 96
    kernel_size: 11
    stride: 4
  }
}
layers {
  name: "relu1_age"
  type: RELU
  bottom: "conv1_age"
  top: "conv1_age"
}
layers {
  name: "pool1_age"
  type: POOLING
  bottom: "conv1_age"
  top: "pool1_age"
  pooling_param {
    pool: MAX
    kernel_size: 3
    stride: 2
  }
}
layers {
  name: "norm1_age"
  type: LRN
  bottom: "pool1_age"
  top: "norm1_age"
  lrn_param {
    local_size: 5
    alpha: 0.0001
    beta: 0.75
  }
}
layers {
  name: "conv2"
  type: CONVOLUTION
  bottom: "norm1_age"
  top: "conv2_age"
  convolution_param {
    num_output: 256
    pad: 2
    kernel_size: 5
    group: 2
  }
}
layers {
  name: "relu2_age"
  type: RELU
  bottom: "conv2_age"
  top: "conv2_age"
}
layers {
  name: "pool2_age"
  type: POOLING
  bottom: "conv2_age"
  top: "pool2_age"
  pooling_param {
    pool: MAX
    kernel_size: 3
    stride: 2
  }
}
layers {
  name: "norm2_age"
  type: LRN
  bottom: "pool2_age"
  top: "norm2_age"
  lrn_param {
    local_size: 5
    alpha: 0.0001
    beta: 0.75
  }
}
layers {
  name: "conv3"
  type: CONVOLUTION
  bottom: "norm2_age"
  top: "conv3_age"
  convolution_param {
    num_output: 384
    pad: 1
    kernel_size: 3
  }
}
layers {
  name: "relu3_age"
  type: RELU
  bottom: "conv3_age"
  top: "conv3_age"
}
layers {
  name: "conv4"
  type: CONVOLUTION
  bottom: "conv3_age"
  top: "conv4_age"
  convolution_param {
    num_output: 384
    pad: 1
    kernel_size: 3
    group: 2
  }
}
layers {
  name: "relu4_age"
  type: RELU
  bottom: "conv4_age"
  top: "conv4_age"
}
layers {
  name: "conv5"
  type: CONVOLUTION
  bottom: "conv4_age"
  top: "conv5_age"
  convolution_param {
    num_output: 256
    pad: 1
    kernel_size: 3
    group: 2
  }
}
layers {
  name: "relu5_age"
  type: RELU
  bottom: "conv5_age"
  top: "conv5_age"
}
layers {
  name: "pool5_age"
  type: POOLING
  bottom: "conv5_age"
  top: "pool5_age"
  pooling_param {
    pool: MAX
    kernel_size: 3
    stride: 2
  }
}
layers {
  name: "fc6_age"
  type: INNER_PRODUCT
  bottom: "pool5_age"
  top: "fc6_age"
  inner_product_param {
    num_output: 4096
  }
}
layers {
  name: "relu6_age"
  type: RELU
  bottom: "fc6_age"
  top: "fc6_age"
}
layers {
  name: "drop6_age"
  type: DROPOUT
  bottom: "fc6_age"
  top: "fc6_age"
  dropout_param {
    dropout_ratio: 0.5
  }
}
layers {
  name: "fc7_age"
  type: INNER_PRODUCT
  bottom: "fc6_age"
  top: "fc7_age"
  inner_product_param {
    num_output: 4096
  }
}
layers {
  name: "relu7_age"
  type: RELU
  bottom: "fc7_age"
  top: "fc7_age"
}
layers {
  name: "drop7_age"
  type: DROPOUT
  bottom: "fc7_age"
  top: "fc7_age"
  dropout_param {
    dropout_ratio: 0.5
  }
}
layers {
  name: "fc8_age"
  type: INNER_PRODUCT
  bottom: "fc7_age"
  top: "fc8_age"
  inner_product_param {
    num_output: 8
  }
}
layers {
  name: "fc8_gender"
  type: INNER_PRODUCT
  bottom: "fc7_age"

Gil Levi

unread,
Nov 30, 2014, 12:49:22 PM11/30/14
to caffe...@googlegroups.com
OK, I managed to solve it. Thanks for the huge help!

Gil

Mohamed Omran

unread,
Dec 3, 2014, 2:39:26 AM12/3/14
to caffe...@googlegroups.com
Don't mention it. What turned out to be the problem?

Gil Levi

unread,
Dec 3, 2014, 12:51:34 PM12/3/14
to caffe...@googlegroups.com
Hi Mohamed,

In testing - I actually used two different prototxt files - one for training and one for testing. It did the work:)

When I loaded the nets, I found out that they share the same conv layers, but have different fully-connected layers.

The train_val.prototxt and the deploy prototxt files for age and gender are attached.



Can you please explain how multi-task learning is performed? I have a loss layer for age and a loss layer for gender. Does Caffe tries to minimize the sum of those functions? Does it iterated between the age and gender "paths" when backpropogating? 

Thanks,
Gil
finetune_deploy_one_image_age.prototxt
finetune_deploy_one_image_gender.prototxt
finetune_train_val_prototxt.txt
Reply all
Reply to author
Forward
0 new messages