Memory exploded when group parameter is assigned

[Qian Yang]

Hi,

if I set group=num_output in layer conv5_3 in VGG16, the memory keeps increasing and explodes finally when I fine tune the model. Otherwise, everything is normal.

So I am sure it is the convolution_param "group" that caused the memory overload. Could anyone give me some advice?

layer {
  name: "conv5_3"
  type: "Convolution"
  bottom: "conv5_2"
  top: "conv5_3"
  param {
    lr_mult: 1.0
    decay_mult: 1.0
  }
  param {
    lr_mult: 2.0
    decay_mult: 0
  }
  convolution_param {
    num_output: 512
    pad: 0
    group: 512
    kernel_size: 1
    weight_filler {
      type: "xavier"
    }
    bias_filler {
      type: "constant"
    }
  }
}

Thanks.

[Przemek D]

What does it mean that "memory keeps increasing and finally explodes"? Consider attaching relevant logs to help us understand the problem.

[Qian Yang]

Through the system monitor tool in ubuntu, I found the memory occupancy rate kept increasing and finally reached up to  almost 100% since the training process began. And then the training process is killed. Sorry for no relevant logs.

在 2017年10月19日星期四 UTC+8下午3:45:42，Przemek D写道：