NN Custom Weight Initialization?

[alessandro lusci]

Hi all,
I would like to initialize the weights of a 5 layer NN as follows:
Weights in first layer initialized from N(0, 0.1) 
Weights in last layer initialized from N(0, 0.001)
Weights in other layers initialized from N(0, 1/sqrt(num_inputs))
(N stands for normal distribution)

I know that torch initializes the weights with a uniform distribution that takes into account the fanin/fanout of the model.

It looks like that the only option to perform a custom initialization is to assign the desired values to the weights.
For example in the following model

model = nn.Sequential()
model:add(nn.Linear(ninputs,nhiddens))
model:add(nn.Tanh())  

we can change the weights like this

model.modules[1].weights = torch.rand(ninputs,nhiddens)

Although this approach works fine when we want to initialize from N(0,1),  a slight modification is needed when we want to initialize from N(0,A).
In this case we can do:

model.modules[1].weights = torch.rand(ninputs,nhiddens):mul(A)

 
The results I obtained using this approach look correct, but I would like to know if there is a cleaner/smarter way to do the same thing or if I am missing something from the libraries.

Thanks in advance,
Alessandro

   

[soumith]

You can overwrite the reset method for the modules you want to specialize the resets, or derive another class and cleanly overwrite the method.

Method 1!

--------------------------------------------------------------------------------------

require 'nn'

-- overwrite nn.Linear.reset with your own method.

-- rest of your code

--------------------------------------------------------------------------------------

Method 2!

require 'nn'

do
   local Linear, parent = torch.class('nn.CustomLinear', 'nn.Linear') 

   -- override the constructor to have the additional range of initialization

   -- override the :reset method to use your new weight initialization.

end

-- rest of your code using nn.CustomLinear

--
You received this message because you are subscribed to the Google Groups "torch7" group.
To unsubscribe from this group and stop receiving emails from it, send an email to torch7+un...@googlegroups.com.
To post to this group, send email to tor...@googlegroups.com.
Visit this group at http://groups.google.com/group/torch7.
For more options, visit https://groups.google.com/d/optout.

[alessandro lusci]

Thank you very much,
I find the second method great. I'm going to implement it and post my solution.

[alessandro lusci]

Hi again,
I have implemented and tested a solution based on method 2. It seems it works pretty well.

Here is my implementation of CustomLinear.lua

require 'nn'

do

    local Linear, parent = torch.class('nn.CustomLinear', 'nn.Linear')
   
    -- override

the constructor to have the additional range of initialization
    function Linear:__init(inputSize, outputSize, mean, std)
        parent.__init(self,inputSize,outputSize)
               
        self:reset(mean,std)
    end
   
    -- override the :reset method to use custom weight initialization.        
    function Linear:reset(mean,stdv)
       
        if mean and stdv then
            self.weight:normal(mean,stdv)
            self.bias:normal(mean,stdv)
        else
            self.weight:normal(0,1)
            self.bias:normal(0,1)
        end
    end

end

and her is my nn model

model = nn.Sequential()
model:add(nn.CustomLinear(ninputs,nhiddens,0,0.1))
model:add(nn.ReLU())
model:add(nn.CustomLinear(nhiddens,nhiddens2,0,1/math.sqrt(nhiddens)))
model:add(nn.ReLU())
model:add(nn.CustomLinear(nhiddens2,nhiddens3,0,1/math.sqrt(nhiddens2)))
model:add(nn.ReLU())
model:add(nn.CustomLinear(nhiddens3,nhiddens4,0,1/math.sqrt(nhiddens3)))
model:add(nn.ReLU())
model:add(nn.CustomLinear(nhiddens4,nhiddens5,0,1/math.sqrt(nhiddens4)))
model:add(nn.ReLU())
model:add(nn.CustomLinear(nhiddens5,noutputs,0,0.01))
model:add(nn.Sigmoid())

Thanks again for your precious help smth chntla.

[Chen Chen]

Thanks for your posted solution.

在 2014年8月15日星期五 UTC+8上午9:51:35，alessandro lusci写道：