clarification about caffe batch norm

[Evan Shelhamer]

Hey all,

As there has been some confusion about the Caffe batch norm layer, I have tried to address it in PR #4704 now merged to master. In summary:

1. Caffe's batch norm layer only handles the mean/variance standardization. For the scale and shift a further `ScaleLayer` with `bias_term: true` is needed.

2. The layer parameters are not learnable parameters; they are the statistics estimated by batch norm. For this reason they should not be exposed to the solver since gradient descent on these parameters is an error and it will thrash training. This is now automatically handled by Caffe.

3. Before (2) it was necessary to manually mark the batch norm parameters as fixed by `param { lr_mult: 0 }` declarations. Since this is now handled by (2), old definitions are automatically upgraded to strip these now unnecessary declarations.

Here is an example batch norm layer definition in the latest master.

layer {

  name: "bn1"

  type: "BatchNorm"

  bottom: "conv1"

  top: "conv1"

}

Happy brewing,

Evan Shelhamer

[rkludge]

2. The layer parameters are not learnable parameters; they are the statistics estimated by batch norm. 

What are the layer parameters actually? There are 3 of them, for each BN layer.

[Evan Shelhamer]

​As of this PR, they are now documented in the layer API documentation: http://caffe.berkeleyvision.org/doxygen/classcaffe_1_1BatchNormLayer.html#details

 (0) mean, (1) variance, and (2) moving average factor​

Evan Shelhamer

On Fri, Sep 16, 2016 at 2:46 PM, rkludge <ramka...@gmail.com> wrote:

2. The layer parameters are not learnable parameters; they are the statistics estimated by batch norm. 

What are the layer parameters actually? There are 3 of them, for each BN layer.

--
You received this message because you are subscribed to the Google Groups "Caffe Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to caffe-users+unsubscribe@googlegroups.com.
To post to this group, send email to caffe...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/caffe-users/82e176b5-6899-4cba-b350-6abf5a6109a1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.