Is it a bug in the codes with the initialization of the weights ?

[Guoliang Kang]

template <typename Dtype>

class XavierFiller : public Filler<Dtype> {

 public:

  explicit XavierFiller(const FillerParameter& param)

      : Filler<Dtype>(param) {}

  virtual void Fill(Blob<Dtype>* blob) {

    CHECK(blob->count());

    int fan_in = blob->count() / blob->num();

    Dtype scale = sqrt(Dtype(3) / fan_in);

    caffe_rng_uniform<Dtype>(blob->count(), -scale, scale,

        blob->mutable_cpu_data());

    CHECK_EQ(this->filler_param_.sparse(), -1)

         << "Sparsity not supported by this Filler.";

  }

};

All the hiddent layers use the weights blob as the parameter of the XavierFiller.Fill（...), like this:

weight_filler->Fill(this->blobs_[0].get());

Are you sure it is right? I do not think it is the fan-in of a unit related to one specific connection. For inner product layer, the fan-in should be the width of the weight blob. And for the convolutional layer, it should be the channel number of the bottom blob multiplies the width and height of the weight blob. Do I have some misunderstanding? Can the author of the caffe answer my questions? Thank you.

[Evan Shelhamer]

The Xavier filler code is right. `blobs_[0]` is the whole weight matrix while `blobs_[1]` is the bias vector. There was a subtlety with the fan-in of Convolution and InnerProduct layers but it was resolved by the switch to N-D blobs in #1970. See this thread and comment by Sean Bell for details.

Evan Shelhamer

--
You received this message because you are subscribed to the Google Groups "Caffe Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to caffe-users...@googlegroups.com.
To post to this group, send email to caffe...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/caffe-users/0590d2d3-12fd-473a-b93a-a11580e4cb55%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.