HDF5 data with different size of input

[Zhuotun Zhu]

Hello,

I am training a network with HDF5 data layer which has different blob size as input for each batch. My task is semantic segmentation like the FCN. I am using the SoftmaxWithLoss Layer. I got errors when I trained the networks. Note that my hardware configurations are Ubuntu 14.04, CUDA 7.5, CuDNN v5.1, TITAN X GPU.

The bt in GDB debugger looks like the following:

Program received signal SIGSEGV, Segmentation fault.

0x00007ffff71a4478 in boost::shared_ptrcaffe::SyncedMemory::operator!() const () at /usr/include/boost/smart_ptr/detail/operator_bool.hpp:62

62 return px == 0;

(cuda-gdb) bt

#0 0x00007ffff71a4478 in boost::shared_ptrcaffe::SyncedMemory::operator!() const () at /usr/include/boost/smart_ptr/detail/operator_bool.hpp:62

#1 0x00007ffff719a28f in caffe::Blob::gpu_data() const () at src/caffe/blob.cpp:93

#2 0x00007ffff71ce28a in caffe::SoftmaxWithLossLayer::Forward_gpu (this=0x22b3c970, bottom=..., top=...) at src/caffe/layers/softmax_loss_layer.cu:39

#3 0x00000000004167a2 in caffe::Layer::Forward(std::vector<caffe::Blob, std::allocator<caffe::Blob> > const&, std::vector<caffe::Blob, std::allocator<caffe::Blob> > const&) () at ./include/caffe/layer.hpp:440

#4 0x00007ffff7166881 in caffe::Net::ForwardFromTo(int, int) () at src/caffe/net.cpp:509

#5 0x00007ffff7166649 in caffe::Net::ForwardPrefilled(float*) () at src/caffe/net.cpp:529

#6 0x00007ffff7166a14 in caffe::Net::Forward(std::vector<caffe::Blob, std::allocator<caffe::Blob> > const&, float*) () at src/caffe/net.cpp:543

#7 0x00007ffff718bd13 in caffe::Solver::Test(int) () at src/caffe/solver.cpp:306

#8 0x00007ffff718bb41 in caffe::Solver::TestAll() () at src/caffe/solver.cpp:288

#9 0x00007ffff718abac in caffe::Solver::Step(int) () at src/caffe/solver.cpp:192

#10 0x00007ffff718a69b in caffe::Solver::Solve(char const*) () at src/caffe/solver.cpp:260

#11 0x00000000004120a8 in train() () at tools/caffe.cpp:132

#12 0x00000000004140d6 in main () at tools/caffe.cpp:310

Could anyone have any hints on this?

Thank you!

Zhuotun