Use MemoryData layer with train and validation data

189 views
Skip to first unread message

Enes Deumić

unread,
Feb 4, 2016, 4:30:22 PM2/4/16
to Caffe Users
I want to use data from memory using python interface. Here is the relevant part of the train_val.prototxt file:

name: "caffe_cif100"
layer {
  name: "data"
  type: "MemoryData"
  top: "data"
  top: "label"
  memory_data_param {
    batch_size: 200
    channels: 3
    height: 32
    width: 32
  }
  include {
    phase: TRAIN
  }
  transform_param {
    mirror: true
    mean_value: 129.74
    mean_value: 124.285
    mean_value: 112.695
  }
  }

layer {
  name: "data"
  type: "MemoryData"
  top: "data"
  top: "label"
  memory_data_param {
    batch_size: 200
    channels: 3
    height: 32
    width: 32
  }
  include {
    phase: TEST
  }
  transform_param {
    mean_value: 129.74
    mean_value: 124.285
    mean_value: 112.695
  }
  }

layer {
  name: "conv1"
  type: "Convolution"
  bottom: "data"
  top: "conv1"
  param {
    lr_mult: 1
    decay_mult: 1
  }

  convolution_param {
    num_output: 96
    kernel_size: 3
    stride: 1
    pad: 1
    weight_filler {
      type: "xavier"
    }
    bias_filler {
      type: "constant"
      value: 0
    }
  }
} ...

 My solver.txt looks like this:

net: "/mnt/cifar100/bvlc_reference_caffenet/train_val.prototxt"
test_iter: 1000
test_interval: 1000
base_lr: 0.005
lr_policy: "step"
gamma: 0.75
stepsize: 50000
display: 20
max_iter: 450000
momentum: 0.9
snapshot: 50000
snapshot_prefix: "/mnt/cifar100/bvlc_reference_caffenet/snapshots/"
solver_mode: GPU
solver_type: ADAM

I have 4 data arrays, train (50000, 3, 32, 32), valid(10000, 3, 32, 32), y_train(50000,), y_valid( 10000,), loaded in memory, where I noted the dimensions of the arrays.

In some other post, I've seen an example that looks like this

solver = caffe.get_solver("/path2solver/solver.prototxt")
solver.net.set_input_arrays(data, labels)

But it's not clear to me how should I put train and valid data to solver, I've tried couple of combinations where I specified all 4 arrays, but none of them work. Assume you have my data, how would you do it?

If anyone could help me resolve this, I would greatly appreciate this, using the data layer from python is not well documented. Also if my dimensions are incorrect please correct me.

Enes Deumić

unread,
Feb 5, 2016, 4:51:27 AM2/5/16
to Caffe Users
Ok, I've got it.
solver = caffe.get_solver("path_to_solver/solver.prototxt")
solver.net.set_input_arrays(train, y_train)
solver.test_nets[0].set_input_arrays(test, y_test)

It seems that labels should be of dimension (n, 1, 1, 1) for memory layer as well.
Reply all
Reply to author
Forward
0 new messages