Convolution with multiple input layers

[Alex Ter-Sarkisov]

Is it possible in Caffe to take multiple layers (e.g. two 100x100 conv. layers, 1x1 filter and 100x100 output conv layer) as inputs? Not single layer with two maps, but two layers with single map each? I know a workaround (eltwise with sum operator followed by conv layer), but this would be more preferable if possible. 

[Przemek D]

First, I will assume that by "take multiple layers as inputs" you actually mean "multiple blobs as inputs". A less experienced user reading this answer is referred to the tutorial, especially the part on Nets, Layers and Blobs.

Caffe convolution layers allow you to take multiple inputs, but this does not work the way you want it to, I'm afraid. This is a use case when you want to convolve two independent blobs with the same filter bank (kind of as an alternative to weights sharing) - you make a single layer with two bottoms and two tops, and they are processed independently as if by two separate layers with the same filters.

What exactly do you want to do? In what way do you want the two inputs merged?

[Alex Ter-Sarkisov]

What I meant is 

# layer for good predictions

layer {

  name: "score_convolution_badpred"

  type: "Convolution"

  input: some input

  output: "score_convolution_badpred"

  #layer parameters go here

  # sinle map output, same size as score_convolution_goodpred

}

layer {

  name: "score_convolution_goodpred"

  type: "Convolution"

  input: some input

  output: "score_convolution_goodpred"

  #layer parameters go here

  # single map output, same size as score_convolution_badpred

}

layer {

  name: "fused_layer"

  type: "Convolution"

  input1: score_convolution_goodpred

  input2: score_convolution_badpred

  output: "fused output"

  #layer parameters go here

  # output same size as score_convolution_goodpred, score_convolution_badpred

}

I solved by creating an eltwise layer that sums the inputs pixelwise and then used a conv layer. But I'm still interested if it is possible to directly convolve with two inputs.

cheers,

Alex

[Przemek D]

Convolution only takes one input (assuming we consider weights as function parameters) and produces one output. If you have more inputs and want to only have one output, you have to fuse the inputs together somehow. One can think of several ways of doing that: element-wise add them or multiply them, concatenate along the channel dimension, max() etc.

I wrote a paper on feature fusion which you might find interesting (PM me for full text).

[Alex Ter-Sarkisov]

Thanks I used the pixelwise sum to fuse the layers, and then a conv layer with 3x3 filter. I'm not too happy with the results though.

Alex

[Clay Spence]

I may be misunderstanding, but does a concat layer do what you want? It turns multiple bottom blobs into a single blob with more channels. So in your case you might do this:

layer {

  name: "thegoodthebad_layer"

  # The bottoms have to have the same width and height. See the docs for constraints

  bottom: "score_convolution_goodpred"

  bottom: "score_convolution_badpred"

  top: "thegoodthebad_output"

  type: "Concat"

  concat_param: {

    axis: 1 # The channel axis

  }

}

layer {

  name: "fused_layer"

  type: "Convolution"

  input: "thegoodthebad_output"

  output: "fused output"

  #layer parameters go here

  # output same size as score_convolution_goodpred, score_convolution_badpred

}

The convolution layer learns how to mix the two inputs.

Clay

[Alex Ter-Sarkisov]

OK, so we have two input layers, (1,250,250) and (1,250,250) that concat turns into (2,250,250), is that right? 

Alex

[Clay Spence]

Yes, although that's axis 0. Usually the blobs would have dimensions something like (10,1,250,250) and (10,1,250,250), where the '10' is the batch size, and the '1' is the number of channels. The example I gave had axis: 1 to concatenate along the channels, i.e., the second axis. The result would be a blob with dimensions (10,2,250,250).

To nit-pick my previous message: I wrote that you can concatenate into a blob with more channels. You can concatenate along any axis, not just the channel axis. The other dimensions of the two input blobs have to be consistent.

Clay