FCN-8 trained from zero point saturates and gives all zero output. How to overcome?

[Ilya Zhenin]

Trying to train(even overfitting will do) FCN using FCN-8 architecture on VOC2012. All the time - zero output, so loss ocsillates around big value. I looked one by one output of all convolutional and deconvolutional layers - through layers conv1_1 to conv3_2(about 6 convolutional layers) I can see that net learns some logical filters, they give quite understandable result, though for deeper layers there is more filters that give zero or low output. 

Starting from layer conv 3_3 network, I think, saturates(ReLU activations according to architecture), and all convolution result in zero, and after that layer there is no learning happening at all.

I've initialized all convolutional layers with Gaussian with std 0.01. Deconvolutional layers according to solve.py script initialized with surgery to weights all equal to 1.0.

So... Is here a way to overcome saturation without using weight initialization from other trained model?

[Evan Shelhamer]

VGG16 is too deep to train in this way. If you refer to the original paper it is fine-tuned in stages to workaround the vanishing actions in the forward pass​. As an alternative, you could try a different initialization like MSRA which was demonstrated to train deeper networks from random initialization. Batch norm is a common tactic to fix this too, but it's not immediately applicable to FCNs with batch size 1.

Evan Shelhamer

--
You received this message because you are subscribed to the Google Groups "Caffe Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to caffe-users+unsubscribe@googlegroups.com.
To post to this group, send email to caffe...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/caffe-users/075a94b9-0ab4-4ea6-aad0-9585dfca5402%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.