Tackling Class Imbalance: scaling contribution to loss and sgd.

[Maarten Bamelis]

Hello

I am a graduate student at the university of Ghent, Belgium; my research is about emotion recognition with deep convolutional neural networks.

Recently I've run into a problem concerning class imbalance. I'm using 9216 training samples, approx. 5% is labeled positively (1), the remaining samples are labeled negatively (0).

I'm using the SigmoidCrossEntropyLoss layer to calculate the loss. When training, the loss decreases and the accuracy is extremely high after even a few epochs. This is due to the imbalance: the network simply always predicts negative (0).

To solve this problem, I would like to scale the contribution to the loss depending on the prediction-truth combination (punish false negatives severely). My mentor/coach has also advised me to use a scale factor when backpropagating through sgd: the factor would be correlated to the imbalance in the batch. A batch containing only negative samples would not update the weights at all.

I have only added one custom-made layer to Caffe: to report other metrics such as precision and recall. My experience with Caffe code is limited but I have a lot of expertise writing C++ code.

Could anyone help me or point me in the right direction on how to adjust the SigmoidCrossEntropyLoss and Sigmoid layers to accomodate the following changes:

1. adjust the contribution of a sample to the total loss depending on the prediction-truth combination (true positive, false positive, true negative, false negative).

2. scale the weight update performed by sgd depending on the imbalance in the batch (negatives vs. positives).

Thanks in advance!

[Keith Monaghan]

I dealing with a similar problem with the dataset that I am working with.  Please let me know if you make any progress.

[Maarten Bamelis]

More information can be found on StackOverflow.

I haven't fully solved the imbalance problem but the InfogainLossLayer has proven to be a very valuable tool in battling class imbalance. However, the imbalance I'm dealing with is quite extreme and the amount of samples I have available for training is quite small (10k'ish).

These are my best results so far:

1. On the validation set (1024 samples, only used to score the network every 10 epochs):
1. Accuracy: 92%
2. Precision: 45%
3. Recall: 73%
2. On the test set (944 samples, only used to objectively score the network after training):
1. Accuracy: 92%
2. Precision: 20%
3. Recall: 29%

I'm training a network to predict 12 binary labels (either 1 or 0), the imbalance ranges from 10:1 up to 50:1 imbalance in favor of 0-labels. The metrics can be interpreted as follows:

• Accuracy: how many prediction were right
• Precision: how many of the 1-labels were predicted correctly ( = tp / (tp + fp) )
• Recall: how many of the 1-labels were found ( = tp / (tp + fn) )

I'm aiming to get recall to an acceptable level, once that's done, I'll try to increase precision. It worries me the final scores are quite low in comparison to the validation scores. I'm meeting my mentor/coach tomorrow and will ask his opinion on the matter.

Feel free to ask for more info, I'll try to help as much as possible!

[Abhishek Sharma]

Hey,

Can you pls tell me how did you get the tp, fp, tn, fn values in caffe? It would solve a huge problem for me.

Thanks 

[Axel Angel]

tp = true positive, fp = false positive, tn = true negative, fn = false negative. It's the confusion matrix for 2 classes. I suggest reading: https://en.wikipedia.org/wiki/Confusion_matrix and https://en.wikipedia.org/wiki/Sensitivity_and_specificity . To compute it is just a matter of summing if correct/incorrect then averaging for each 4 cases.

[Nanne van Noord]

As you mentioned, it is worrisome that the performance on your validation and training set differ so much. Your test set is also somewhat on the small side, granted you don't have a whole lot of data to work with, but with such big difference between the validation and the test set performance the results on the test set might not be very indicative. How did you split the data into a train/validation/test set? A poor distribution across these three sets might also impact your performance.

It might actually be worth it to do some kind of cross validation scheme, given that your model should be quite fast to train.

As for the imbalance (though I think InfoGainLoss is the right direction), it might be worth it to take a look at the discussion in https://github.com/BVLC/caffe/pull/523 and the changes made (if you haven't already).

[arikp]

Maarten, did you ever managed to solve this problem?

I also have a multiple binary classification problem I'm trying to solve using SigmoidCrossEntropyLoss, which suffers from class imbalance.

Do you mind sharing the code of your solution?