What is the meaning of lr_mult and decay_mult?

[Yin Zheng]

Hi guys,

I am learning caffe now and I am not sure what is the meaning of "lr_mult" and "decay_mult". I did not find any reference about them. Could anyone explain the meaning of them and It would be much appreciated if could provide some document to explain the meaning of the parameters of caffe?

layer {

  name: "conv1"

  type: "Convolution"

  bottom: "data"

  top: "conv1"

  param {

    lr_mult: 1

    decay_mult: 1

  }

  param {

    lr_mult: 2

    decay_mult: 0

  }

  convolution_param {

    num_output: 96

    kernel_size: 11

    stride: 4

    weight_filler {

      type: "gaussian"

      std: 0.01

    }

    bias_filler {

      type: "constant"

      value: 0

    }

  }

}

Best wishes,

Yin

[Cyndi]

In your solver you likely have a learning rate set as well as weight decay.  lr_mult indicates what to multiply the learning rate by for a particular layer.  This is useful if you want to update some layers with a smaller learning rate (e.g. when finetuning some layers while training others from scratch) or if you do not want to update the weights for one layer (perhaps you keep all the conv layers the same and just retrain fully connected layers).  decay_mult is the same, just for weight decay.

[Yin Zheng]

Hi Cyndi,

Got it. Thanks you very much!

Best regards!

Yin

--
You received this message because you are subscribed to a topic in the Google Groups "Caffe Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/caffe-users/8J_J8tc1ZHc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to caffe-users...@googlegroups.com.
To post to this group, send email to caffe...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/caffe-users/bb6ceea2-c579-459d-b1d5-b66071bfdd21%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

------------------------------------------------------------------------------------------------------

Wir mussen wissen, Wir werden wissen.

                                  ——David Hilbert(1862～1943)

 

 

[Yin Zheng]

Hi 

I think one is for W and the other is for b.

Yin

2016-01-04 4:15 GMT+08:00 zzz <mr.zizh...@gmail.com>:

Hi

I have an following question: why there are two pairs of param in Convolution layer.

  param {

    lr_mult: 1

    decay_mult: 1

  }

  param {

    lr_mult: 2

    decay_mult: 0

  }

Thanks very much!

--

You received this message because you are subscribed to a topic in the Google Groups "Caffe Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/caffe-users/8J_J8tc1ZHc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to caffe-users...@googlegroups.com.
To post to this group, send email to caffe...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/caffe-users/d1d29c5e-04be-4b99-9e11-a6503586b99e%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Evan Shelhamer]

Right, the first param { } is for the weights and the second is for the biases.

Evan Shelhamer

You received this message because you are subscribed to the Google Groups "Caffe Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to caffe-users...@googlegroups.com.

To post to this group, send email to caffe...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/caffe-users/CAG9%3D31Nj-eMKNZ0zUW3uzUHjk8wdAGbhcNaqGRcmOh4yxSOjFA%40mail.gmail.com.

[Qiang Guo]

Why the lr for weight is 1 while is 2 for the bias?