Local minimum with ~50% 1-bits in a semantic segmentation problem

[Alex Ter-Sarkisov]

I am training a number of models for binary pixel-level semantic segmentation based on FCN and CRF-RNN. Some of them do well, but some converge to the same strange solution: ~50% of bits in the output mask are 1 and the rest are 0. Training error can be quite low, so I assume this is some sort of a local minimum. I tried resolving the situation by decreasing the starting learning rate and increasing the momentum, but the situation keeps repeating. The problem is that models that get stuck in it are very different, in terms of starting weights, architecture, etc. 

The training/validation database I've built myself, Class 1 is well presented, and I didn't observe this situation with other datasets. 

Therefore, I wonder if there could be any intuition behind this local minimum and maybe suggestions on how to avoid it.