Issue during inference in a batch normalized network

[Kshiteej Sheth]

Hi,

I am trying to train a CNN with batch normalization. I am saving a clone of the model which has cloned the weights, biases, running mean and std. After training it I load the saved model and irrespective of the input I pass through the net I get the same output which turns out to be the bias of the last conv layer. Here is the network. While inference I am doing net:evaluate(). I am also using optim package for updating the parameters

-- Network Definition

net = nn.Sequential()
net_ = nn.Parallel(1,2)
for i = 1,3 do
    model1 = nn.Sequential()
    model1:add(nn.SpatialConvolution(ch_in,ch_1,ker_1,ker_1,str_1,str_1,pad_1,pad_1));
    model1:add(nn.SpatialBatchNormalization(ch_1))
    model1:add(nn.ReLU())
    model1:add(nn.SpatialConvolution(ch_1,ch_out,ker_2,ker_2,str_2,str_2,pad_2,pad_2));
    net_:add(model1)
    end
net:add(net_)

lightModel = net:clone('weight','bias','running_mean','running_std')

-- here goes the code for training the network---

--after completing the training
torch.save(<Path of the network>,lightModel)


---code for testing the network---
net  = torch.load(<Path of the network>)
net:training()
temp = torch.rand(3,1,5,3,3)
temp = temp:cuda()
output = net:forward(temp)
print(output)

[zagor...@gmail.com]

Instead of clone you should use clearState() to save lighter network:

torch.save('model.t7', net:clearState())

--
You received this message because you are subscribed to the Google Groups "torch7" group.
To unsubscribe from this group and stop receiving emails from it, send an email to torch7+un...@googlegroups.com.
To post to this group, send email to tor...@googlegroups.com.
Visit this group at https://groups.google.com/group/torch7.
For more options, visit https://groups.google.com/d/optout.

[Arthita Ghosh]

Hi,

I am stuck with a similar problem. I copied weights from a trained nngraph model to a new one with almost similar structure.
I ran both models on the same input and the new model produced the bias values of last layer as output(The output of some intermediate layer basically goes to all zero). I did some backtracking and checked outputs of intermediate layer. I figured out that at the first BatchNorm layer in the network is the point where the layer outputs start getting different from each other.
I checked the weights,biases, running_means,std of the BN layers and they are the same.

Did you happen to figure out why the BN layer was producing wrong outputs in your case?



Arthita

[Farley Lai]

One more. I recently trained an autoencoder with SpatialBatchNormalization. 

During inference, the final output is nearly fixed when model:evaluate() is called. 

It turns out the the model should be in the training mode to generate the expected output but it is weird.

Another model trained without SpatialBatchNormalization does not suffer this issue. 

This feels very like a bug. 

 

What if a dropout is also added in this case?

Either model:training() or model:evaluate() seems likely to produce unexpected output.

In need of clarifications in 2017.