Fusion of 2 feature maps

[chrert]

Hi,

I'm currently working on a network combining RGB and depth images. The depth data is pre-processed to fill up 3 channels which makes it possible to fine-tune a pre-trained model.
The network starts with some convolutional layers for both modalities. At some point, I need to fuse the two feature maps without changing the dimensions. My current approach is to simply concatenate the feature maps and apply another convolutional layer to reduce the number of features to the original one:

# ... some layers resulting in conv5_1 and conv5_2 (each 256x39x64)

layer {
  name: 'conv5_concat'
  type: 'Concat'
  bottom: 'conv5_1'
  bottom: 'conv5_2'
  top: 'conv5_concat'
  concat_param {
    axis: 1
  }
}

# conv_concat is now 512x39x64

layer {
  name: 'conv5_fusion'
  type: 'Convolution'
  bottom: 'conv5_concat'
  top: 'conv5_fusion'
  convolution_param {
    num_output: 256
    kernel_size: 3 pad: 1 stride: 1
  }
}

# conv fusion is againg 256x39x64

While it basically works, I think it would be beneficial to apply some kind of normalization to the concatenated features. What's the best way to achieve this? My first guess would be to use a LRN layer with mode ACROSS_CHANNELS but it would only normalize nearby channels, whereas the channels from the different modalities aren't nearby. Is there another way to do this?

[zss...@gmail.com]

Hi, 

This question confused me these days, I concatenate RGB features and Depth features, but I get  really a bad results which is  much worse than only using one modality. Have you solved this issue now, I'm wondering if you can do me favour? 

在 2016年3月21日星期一 UTC+8下午9:15:21，chrert写道：

[p.Paul]

@zss...@gmail.com

Were you able to find any pretrained model fro depth feature extraction?

[Fateh Singh]

Hi
Did anyone find a good way of doing this ?

[Jonathan R. Williford]

For the normalization, I would use Batch Normalization. Read the Caffe.proto details on the parameters:

http://caffe.help/manual/layers/batchnorm.html

You could just apply Batch Normalization after concatenation, although you might want to add Batch Normalization elsewhere also. Make sure you include the Bias and Scaling.

Just curious, how well aligned is your depth estimation and images?

Best,

Jonathan

--
You received this message because you are subscribed to the Google Groups "Caffe Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to caffe-users+unsubscribe@googlegroups.com.
To post to this group, send email to caffe...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/caffe-users/c68ceabd-3fe8-47ca-a0ab-27e8cb86da20%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.