Some questions about Deconvolution Layer

[Lu Weiting]

Hi guys, I want to know what is the difference between Convolution and Deconvolution Layer. In matlab code, the deconvolution is the same as convolution with rot180 filter and full padding.

If the filters in Deconvolution and Convolution layer all need to be trained, that means they don't have the same filter parameters with 180 degree rotation relation. What's the difference between

Convolution  and Deconvolution Layer in Caffe ?  I extract the trained filter parameters, and i don't find any relationship  between Convolution and Deconvolution Layer. Can anyone help me?

Please and thank you .

[Jan]

Afaik the deconvolution can be seen as a backward convolution, similar to how the backpropagation works in convolution layers. Yes, the kernels can be viewed as transposed in comparison to the ordinary convolution layer, and the valid region is larger than the input (instead of smaller, as in conv layers). Maybe you can imagine deconvolution as single pixels being "expanded" into (usually overlapping) regions with the size of the filter kernels, with values of the original pixel weighted by the filter kernel values. Afaik in caffe the filter kernels are saved in a way that allows to share parameters between convolutional and deconvolutional layers meaningfully (to create a convolutional autoencoder for instance).

Jan

[Lu Weiting]

Thanks for your answer. If I want to share parameters between convolutional and deconvolutional layers, should I declare anything? I create a four layers architecture with 2 convolutional layers and 2 deconvolutional layers. But there trained filters didn't seem to have any relation (like transpose). If they don't share the parameters  between convolutional and deconvolutional layers, the four layers architecture I mentioned above is just the same as 4 convolutional layers essentially. So I want to know how to  share parameters between convolutional and deconvolutional layers in Caffe. Would you please help me? Thank you. 

Jan於 2016年4月27日星期三 UTC+8下午5時36分14秒寫道：

[Jan]

To share parameters you need to associate a name with the layers parameters, and use the same name in layers that should share these parameters. Look at https://github.com/BVLC/caffe/blob/master/src/caffe/proto/caffe.proto#L324-L326 and https://github.com/BVLC/caffe/blob/master/src/caffe/proto/caffe.proto#L283-L304. More specifically, in the prototxt:

layer {
  type: "Convolution"
  name: "conv1"
  # ...
 
  param {
    name: "conv1params"
  }
}

# and later

layer {
  type: "Deconvolution"
  name: "deconv1"
  # ...

  param {
    name: "conv1params"
  }
}

Of course the two layer configs have to match otherwise (e.g. number of filter kernels, kernel size, etc.), such that the parameter blobs have the same size (actually only one set of parameters exist then, used by both layers in computations and updates; hence "sharing").

Jan

[Lu Weiting]

Thanks a lot.  I still  have one more question. In signal process, the output signal is produced by the convolution of input signal and the filter. Then the input signal can be recovered by the deconvolution of output signal

and another filters. I want to know if caffe have this kind of relationship? For example, in a four layers(2 conv and 2 deconv) architecture, when training the filters in the last deconvolutional layer, the corresponding filters in the first convolutional layer should be tuned simultaneously. Or Caffe just treat deconvolutional layer as convolutional layer for expanding input ? 
Jan於 2016年4月27日星期三 UTC+8下午7時19分47秒寫道：

[Jan]

Somebody correct me if I'm wrong, but I think this statements is true, i.e. when you train a conv layer and a deconv layer with shared weights, then the deconv-layer tends to do the reverse operation to the conv layer.

"Or Caffe just treat deconvolutional layer as convolutional layer for expanding input ?" -- it is not completely wrong, but it's a bad way to put it. A statement in the similar spirit would be to say that "ReLUs just clip away negative values". It's somehow true, but not helpful in any way.

Jan

[Lu Weiting]

Maybe I misunderstand something. Does the ""share parameters"" between conv and deconv layers in Caffe means they are the same? Or they have some change like transposed relationship? Because in Auto-Enctoder, the filters in decoder is transposed filters in encoder. I wonder if Caffe can do the same thing in the deconvolutional layer according to its corresponding convolutional layer ? Thanks for your help.

Jan於 2016年4月27日星期三 UTC+8下午11時44分00秒寫道：

[Jan]

Sharing means they are the same, indeed both layers use the same parameter blob, i.e. there is only one instance of all the shared  parameters.

I was saying that the deconv layer treats the parameters in a transposed way, but I might be wrong about that, I haven't used the deconv layer myself, somebody with more experience with deconv layers needs to answer that. When I think about it, it is very possible that they are also saved in a transposed way. In this case you can still initialize them by hand with the transposed filters of the corresponding (pretrained) conv layers and train a convolutional autoencoder this way (no sharing).

Jan

[Lu Weiting]

I understand. Thanks for your help!

Jan於 2016年4月28日星期四 UTC+8下午4時04分40秒寫道：

[Ximing Xi]

Hi, I want to know how to let Deconvolution and Convolution layer  have the same filter parameters with 180 degree rotation relation in Caffe. In my case, I have already trained Convolution layer and want to 
give Deconvolution layer the  transposed weights as in [1311.2901] Visualizing and Understanding Convolutional Networks.

Can anyone help me? Thank you.

[Vadzim Piatrou]

For VGG16 net I try this method:

net.params["conv1_1t"][0].data[...] = net.params["conv1_1"][0].data.transpose(0,1,3,2)

I also add copy of biases.

But reconstruction for Convolution-Deconvolution net was bad: very high values on output layer. I don't solve this task yet :(

Original python notebook you can find here:
https://github.com/mariolew/caffe-unpooling/blob/master/examples/deconv-vgg.ipynb


пятница, 1 июля 2016 г., 12:50:52 UTC+3 пользователь Ximing Xi написал:

[mohsen zarrindel]

Hello guys, I want to run a model with upsampling layer:

layer {

  name: "upsample4"

  type: "Upsample"

  bottom: "conv5_1_D"

  bottom: "pool4_mask"

  top: "upsample4"

  upsample_param {

    scale: 2

  }

}

but I had Error. so, can I use deconvolution layer instead of it? would you please guide me to to it

در چهارشنبه 27 آوریل 2016، ساعت 7:00:43 (UTC+4:30)، Lu Weiting نوشته:

[Vadzim Piatrou]

Look at this conversation: https://github.com/BVLC/caffe/pull/3601#issuecomment-295461657
It should help

четверг, 7 сентября 2017 г., 16:29:30 UTC+3 пользователь mohsen zarrindel написал:

[mohsen zarrindel]

thank you. but I could not find my answer. would you please  write answer here?

در جمعه 8 سپتامبر 2017، ساعت 11:02:18 (UTC+4:30)، Vadzim Piatrou نوشته:

[Vadzim Piatrou]

You should add unpooling layer to caffe from https://github.com/BVLC/caffe/pull/3601

Prototxt layer will be the following:

layer {

  name: "upsample4"

  type: "Unpooling"

  bottom: "conv5_1_D"

  bottom: "pool4_mask"

  top: "upsample4"

  pooling_param {
    kernel_size: 2
    stride: 2
  }

}

среда, 13 сентября 2017 г., 13:21:31 UTC+3 пользователь mohsen zarrindel написал:

[Vadzim Piatrou]

You can use my fork of Caffe with Unpooling layer: https://github.com/belgraviton/caffe

среда, 13 сентября 2017 г., 13:21:31 UTC+3 пользователь mohsen zarrindel написал: