Initialising Forget gate bias in Lstm

[ama...@gmail.com]

Hi,

Is there some way to set only the forget gate bias to a specific value. I've been trying to do so, but could not figure a way out. Actually is that even possible? or do we have to write our own Lstm layer to do this?Could someone kindly provide a quick example how this can be done? Thank you so much!

[Tasty Minerals]

If you look into LSTM.lua code, concretely LSTM:buildGate method, you can find how the gate is implemented:

local input2gate = nn.Sequential()
 :add(nn.Dropout(self.p,false,false,true,self.mono))
 :add(nn.Linear(self.inputSize, self.outputSize))

And if you look into nn.Linear class code, you'll find that it accepts "bias" parameter upon initialization:

function Linear:__init(inputSize, outputSize, bias)
 parent.__init(self)
 local bias = ((bias == nil) and true) or bias
 self.weight = torch.Tensor(outputSize, inputSize)
 self.gradWeight = torch.Tensor(outputSize, inputSize)
 if bias then
 self.bias = torch.Tensor(outputSize)
 self.gradBias = torch.Tensor(outputSize)
 end
 self:reset()
end

So, just do 

nn.Linear(self.inputSize, self.outputSize, 1)

Read this wonderful paper: http://proceedings.mlr.press/v37/jozefowicz15.pdf which describes why positive bias for LSTM is good.