Implement crop layer as in Caffe for semantic segmentation

[Feng Gu]

I've been trying to implement fully convolutional networks (fcn8s), however I had some difficulty in thecrop layer after deconvolution. I tried to use SliceLayer to perform center, upper or lowerslicing of each axis (width and height), and the results were not satisfactory.

After checking the source of Caffe as here, the crop layer takes two bottom layers (input andconvolution or deconvolution) and outputs a single layer that matches the output shape of input. In addition, it performs differently for convolution and deconvolution based on their parameters, e.g.,kernel size, stride and pad.

Can anyone give some suggestions on how to implement this in Lasagne/Theano? Many thanks in advance.

This was originally posted in https://github.com/Lasagne/Lasagne/issues/660.

[Feng Gu]

It seems for `deconvolution`, the `crop` layer perform upscaling by a factor determined by `stride` and shift the coordinates by a `offset=(kernel_size-1)/2-pad`. My question now is if I can pass a `slice` that are not successive (i.e. [0, 4, 8, 12, ...]) to the `SliceLayer` or I should try to use `ElemwiseMergeLayer` instead. 

[goo...@jan-schlueter.de]

It seems for `deconvolution`, the `crop` layer perform upscaling by a factor determined by `stride` and shift the coordinates by a `offset=(kernel_size-1)/2-pad`.

Wait, the crop layer does not just crop the tensors to match, it also scales them? If so, neither the SliceLayer nor the ElemwiseMergeLayer can do what you want. Can you give a brief explanation of the exact use case? E.g., a stack of two or three layers and the intended operation of the crop layer for that stack?

Cheers, Jan

[goo...@jan-schlueter.de]

My question now is if I can pass a `slice` that are not successive (i.e. [0, 4, 8, 12, ...]) to the `SliceLayer`

If you want to take every 4th element, this would be a SliceLayer(layer, slice(None, None, 4), axis=...). This is equivalent to x[::4] on a Python list (or numpy or Theano tensor).

[Feng Gu]

Hi Jan,

That's definitely good know, but is it possible to take a step further and pass a random set of indices (e.g. [1, 2, 7, 20, 50, ...]) to "slice"?

Many thanks.

[goo...@jan-schlueter.de]

Hey,

That's definitely good know, but is it possible to take a step further and pass a random set of indices (e.g. [1, 2, 7, 20, 50, ...]) to "slice"?

In theory yes, Theano supports that, but the SliceLayer currently doesn't. Seems this use case hasn't come up yet. Would you be willing to file a pull request?
Otherwise, for a quick solution, you could use ExpressionLayer(layer, lambda X: X[:, [1, 2, 7, 20, 50]]) to grab elements 1, 2, 7, 20, 50 from the second axis.

But why would you need to extract a nonregular set of indices?

Best, Jan

[emolson]

I did that in order to implement fractional pooling. Unfortunately it was pretty slow.

[goo...@jan-schlueter.de]

 But why would you need to extract a nonregular set of indices?

 

I did that in order to implement fractional pooling.

So there are uses cases for it, but I meant: What does Feng Gu need this for in implementing the fully-convolutional net fcn8s?

[Feng Gu]

Thanks very much for all the replies. After checking the Caffe outputs thoroughly, I found their implementation of cropping is centre crop with some offset. The value of offset is calculated based the type of layers (convolution and deconvolution) and their parameters, e.g. kernel size, stride and pad. I have resolved my issue by using existing SliceLayer with some offset. 

[Florian]

Hello,

I am trying to implement a fcn with lasagne however I got some difficulty with the CropLayer too. Can you share your code or maybe push it to Lasagne ?

Best,
Florian

[Feng Gu]

The following is a crop layer implementation using the SliceLayer:

    net = {}

    net['crop1'] = SliceLayer(input_layer,

                              indices=slice(index_1[0], index_1[1]),

                              axis=-1)

    net['output'] = SliceLayer(net['crop1'],

                               indices=slice(index_2[0], index_2[1]),

                               axis=-2)

You need to compute the indices first, and then pass them to the SliceLayer.  Hope this helps.

[Feng Gu]

This is inspired by "build_inception_module" in https://github.com/Lasagne/Recipes/blob/master/modelzoo/googlenet.py.