loss shocks sharply when training with a multi-domain dataset

[haoch...@gmail.com]

When I training a domain classifier with a fused multi-domain dataset based on VGG-19 model, the loss can not decrease. The fused domains vary a lot in distribution.
I guess this task is very hard to realize, I also try with batch normalization but nothing changes. Can somebody give me some advice for training such model.

Thanks ahead.