Hi! I'm stuck trying to debugg an issue where I get this warning:
[I0307 12:27:11.251971 1
softmax_loss_layer.cu:93] SoftmaxWithLoss warning: sum of pixel wise loss weights is zero!
[I0307 12:27:11.252085 1
softmax_loss_layer.cu:207] SoftmaxWithLoss warning: sum of pixel wise loss weights is zero!
This occurs for a few iterations during training, but not all, and it naturally results in loss=0 for those iterations. This does NOT happen consistently for certain data sets and I cannot see a pattern in when it happens.
When I train the network the loss consistently decrease so it appears to be training something but I am worried that this issue is something that I should be worried about. Does anyone know how to resolve this issue??
I am using the source code for 3D u-net and I've only made minor changes (such as entering my own data, changing augmentation parameters accordingly and I added an accuracy layer). The data is 3D CT images in HDF5-format and I've checked so that I have no all-zero softmax-weights for any set. There is very little foreground compared to background so I am using a weight matrix to weight up the foreground to make it as important as the background. I also have an ignore_label where I exclude padding added to make the image sets have consistent shape. Some data sets are not fully annotated, in those cases I use this ignore_label to exclude a probable region for the missing organ(s).
Grateful for any help!
Erika