Hi all,
As far as I've understand, users are able to load desired portion of data to GPU memory through MemoryData layer. However, training takes longer with MemoryData with comparison to LMDB Data layer. My experiments are based on LeNet MNIST example distributed with Caffe and I am using PyCaffe also. I provide my additions below.
Related section of the Python script:
caffe.set_mode_gpu()
caffe.set_device(0)
solver = caffe.SGDSolver('lenet_solver.prototxt')
solver.net.set_input_arrays(xtrn, ytrn) # dtype=np.float32
solver.solve()
Replacements of Data layers in examples/mnist/lenet_train_test.prototxt:
layer {
type: "MemoryData"
top: "data"
top: "label"
include {
phase: TRAIN
}
memory_data_param {
batch_size: 100
channels: 1
height: 28
width: 28
}
}
# same for TEST data, but I am not using it though.
Finally, this is my lenet_solver.prototxt:
net: "lenet_train_test.prototxt"
base_lr: 0.1
lr_policy: "fixed"
display: 0
random_seed: 1
max_iter: 600
snapshot_after_train: false
solver_mode: GPU
Orijinal example with above solver configuration takes 3.55 sec and with MemoryData layer it takes 3.62 sec. Where is the mistake? How can I speed up the training process? Or is just because of using Python instead of C++ (I will try that soon also)?
Thanks in advance.