Biasing the outputs of a layer

[Alessandro Di Martino]

Hello,

My question is related to the following one which I asked yesterday:

https://groups.google.com/forum/#!topic/caffe-users/zhs3xOe44m4

Since I did not receive any answer, I'm starting to think that probably I didn't formulate my question clearly, as what I'm trying to achieve should be quite simple, and it's probably straightforward to achieve with Caffe.

What I need to do is to bias the outputs of a particular layer. Specifically, with "bias" I mean adding an actual constant bias term to each output, or alternatively multiplying by a constant should work as well.

So, say that I have the following fully connected layer that produces 5 outputs:

layer {

  name: "ip7"

  type: "InnerProduct"

  bottom: "ip6"

  top: "out"

  inner_product_param {

    num_output: 5

  }

}

How can I add (or multiply by) a different constant each one of the 5 outputs?

Thank you.

[Evan Shelhamer]

The InnerProduct layer and Convolution layer both have bias terms by default; see https://github.com/BVLC/caffe/blob/master/src/caffe/proto/caffe.proto#L616.

Normally this bias is learned, but you can set it manually as shown by the editing model parameters example: http://nbviewer.ipython.org/github/BVLC/caffe/blob/master/examples/net_surgery.ipynb

You can fix the bias params to keep your manual values by setting `lr_mult: 0` for that parameter:

layer {
  name: "fc6"
  type: "InnerProduct"
  bottom: "pool5"
  top: "fc6"
  param {  # first param is for weights
    lr_mult: 1
    decay_mult: 1
  }
  param {  # second param is for biases
    lr_mult: 0  # setting this to 0 fixes the bias values
    decay_mult: 0
  }
  inner_product_param {
    num_output: 4096
    weight_filler {
      type: "gaussian"
      std: 0.005
    }
    bias_filler {
      type: "constant"
      value: 1
    }
  }
}

Hope that helps,

Evan Shelhamer

--
You received this message because you are subscribed to the Google Groups "Caffe Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to caffe-users...@googlegroups.com.
To post to this group, send email to caffe...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/caffe-users/158c2950-d7b3-43d2-8752-2650802775b6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Alessandro Di Martino]

Thank you Evan for the reply.

I have adapted your example to the one in my question:

layer {

  name: "ip7"

  type: "InnerProduct"

  bottom: "ip6"

  top: "out"

  param {  # first param is for weights

    lr_mult: 1

    decay_mult: 1

  }

  param {  # second param is for biases

    lr_mult: 0  # setting this to 0 fixes the bias values

    decay_mult: 0

  }

  inner_product_param {

    num_output: 5

    weight_filler {

      type: "gaussian"

      std: 0.005

    }

    bias_filler {

      type: "constant"

      value: 1

    }

  }

}

In this way, however, I guess that the same constant bias will be added to each one of the 5 output values.  Which is different to what I would like to achieve:

On Wed, Jul 29, 2015 at 4:45 AM, Alessandro Di Martino <aled...@gmail.com> wrote:

How can I add (or multiply by) a different constant to each one of the 5 outputs?

Thank you.

Am I right? 

[Evan Shelhamer]

No, the bias is a vector with length equal to the number of outputs. See the editing model parameters example for more details.

Evan Shelhamer

[Alessandro Di Martino]

Thank you. Your suggestion was very helpful.