protobuf warning/error while training on large images

32 views
Skip to first unread message

Philip Meier

unread,
Oct 21, 2017, 5:24:05 AM10/21/17
to Caffe Users
I'm trying to train on large color images (2048x4096x3) and I keep getting the following error message over and over during calculations

[libprotobuf ERROR google/protobuf/io/coded_stream.cc:180] A protocol message was rejected because it was too big (more than 67108864 bytes).  To increase the limit (or to disable these warnings), see CodedInputStream::SetTotalBytesLimit() in google/protobuf/io/coded_stream.h.

I found this was in issue in 2014 but is supposed to be fixed. As mentioned in the post I also manually adjusted the 

src/caffe/util/io.cpp#L58

to

coded_input->SetTotalBytesLimit(2147483648, 536870912) // 2^31 (compilation fails if one increases this even further), 2^29

but the message is still there.

For test purposes I trained with a simple (and rather silly) net consisting of single 1x1 convolution, ReLU-activation, max pooling, inner product and softmax layer with the following *.prototxt:

layer {
  name
: "data"
  type
: "Data"
  top
: "data"
  top
: "label"
  data_param
{
    source
: "/home/lmdb/train"
    batch_size
: 1
    backend
: LMDB
 
}
}
layer
{
  name
: "convolution_1"
  type
: "Convolution"
  bottom
: "data"
  top
: "convolution_1"
  convolution_param
{
    num_output
: 1
    kernel_size
: 1
    stride
: 1
    weight_filler
{
      type
: "gaussian"
      std
: 0.004999999888241291
   
}
    bias_filler
{
      type
: "constant"
      value
: 0.10000000149011612
   
}
 
}
}
layer
{
  name
: "activation_1"
  type
: "ReLU"
  bottom
: "convolution_1"
  top
: "convolution_1"
}
layer
{
  name
: "pooling_1"
  type
: "Pooling"
  bottom
: "convolution_1"
  top
: "pooling_1"
  pooling_param
{
    pool
: MAX
    kernel_size
: 128
    stride
: 128
 
}
}
layer
{
  name
: "score"
  type
: "InnerProduct"
  bottom
: "pooling_1"
  top
: "score"
  inner_product_param
{
    num_output
: 3
    weight_filler
{
      type
: "xavier"
   
}
 
}
}
layer
{
  name
: "loss"
  type
: "SoftmaxWithLoss"
  bottom
: "score"
  bottom
: "label"
  top
: "loss"
}

Correct me if I'm wrong but the input blob with 2048x4096x3 should be reduced to 16x32x1 after the pooling. So in total the net has to learn a single weight in the convolutional layer and (16*32*1)*3= 1536 weights in the inner product layer, which feels like a pretty low number. For this reason I suspect the error message is not caused by a too large net but rather from th input data.

So my questions are:

1. Can someone explain me what this message means in detail?
2. Is this message 'harmful' for the training process or can it be ignored? Note that I get values for loss and accuracy but I can't tell if they are reasonable.
3. If it can be ignored can someone tell me how to supress it?
4. If it can't be ignored can someone answer if Caffe is even able to handle large images like this at all?
Reply all
Reply to author
Forward
0 new messages