Softmax layer gives warning that sum of pixel wise loss weights is zero

[Erika Ek]

Hi! I'm stuck trying to debugg an issue where I get this warning:

[I0307 12:27:11.251971     1 softmax_loss_layer.cu:93] SoftmaxWithLoss warning: sum of pixel wise loss weights is zero!
[I0307 12:27:11.252085     1 softmax_loss_layer.cu:207] SoftmaxWithLoss warning: sum of pixel wise loss weights is zero!

This occurs for a few iterations during training, but not all, and it naturally results in loss=0 for those iterations. This does NOT happen consistently for certain data sets and I cannot see a pattern in when it happens.

When I train the network the loss consistently decrease so it appears to be training something but I am worried that this issue is something that I should be worried about. Does anyone know how to resolve this issue??

I am using the source code for 3D u-net and I've only made minor changes (such as entering my own data, changing augmentation parameters accordingly and I added an accuracy layer). The data is 3D CT images in HDF5-format and I've checked so that I have no all-zero softmax-weights for any set. There is very little foreground compared to background so I am using a weight matrix to weight up the foreground to make it as important as the background. I also have an ignore_label where I exclude padding added to make the image sets have consistent shape. Some data sets are not fully annotated, in those cases I use this ignore_label to exclude a probable region for the missing organ(s).

Grateful for any help!

Erika

[Przemek D]

Hard to give you any hints because the master branch of Caffe does not throw such warnings - your softmax_loss_layer.cu is probably very different to the one in Caffe master. Can you point to the repository that contains said code?

[Erika Ek]

Yes of course! I used the code provided by: https://lmb.informatik.uni-freiburg.de/resources/opensource/unet.en.html, where you can find the training prototxt-file also (I'm currently using the one without batch normalization due to GPU memory limitations). They have, among other things, implemented their own augmentation layers and added the possibility of an input weight matrix to the softmax_loss layer using the patch file applied according to the README in caffe_unet_3D_v1.0.tar.gz. To be able to debugg this problem I've copied the caffe git project to my own repository and applied the patch so you can clone this project instead if you want: https://github.com/erikaek/caffe.

Erika

[Przemek D]

From a brief look at the code, this can only happen when either 1) every pixel in the batch has been assigned an ignored label, or 2) you're using the weighted version and all weights were set to zero. Is it possible that due to some augmentation you sometimes end up with a batch of images all containing ignored regions? I don't want to jump to conclusions but since you say this does not happen consistently, this is one of the possibilities.

[Shima Masoumi]

Dear Erika, 

I am facing with exactly the same problem with my 3D data that i was trying to train with 3dUnet. 

Could you solve your problem? 

I have the augmentation layer in original 3D-Unet model and I have weight matrix for classes in which sum of weights of all classes is equal to one.

I would really appreciate if you post your solution here.

Thank you