Class balance with pixel-wise semantic segmentation

[Corentin Jemine]

Hello,

I've been doing binary semantic segmentation on 144x96x3 (rgb) images, with as output either 144x96x1 binary images when using a regression loss or 144x96x2 when using the softmax loss. The final layer of my current network is currently a simple softmax:

layer {
  name: "loss"
  type: "SoftmaxWithLoss"
  bottom: "convc2"
  bottom: "labelScaled"
  top: "loss"
  exclude {
    stage: "deploy"
  }
}

where the output dimensions of convc2 and labelScaled are respectively BATCHx144x96x2 and BATCHx144x96x1.

However, there is a strong class imbalance in my images: 94.5% of the pixels in the entire dataset are from the negative class. I wish to weight the softmax loss in order to even it out. I've been following the instructions from here and I've tried changing my network head to the following:

layer {
  name: "prob"
  type: "Softmax"
  bottom: "convc2"
  top: "prob"
  exclude {
    stage: "deploy"
  }
}
layer {
  bottom: "prob"
  bottom: "labelScaled"
  top: "infoGainLoss"
  name: "infoGainLoss"
  type: "InfogainLoss"
  infogain_loss_param {
    source: "/path/to/infogainH.binaryproto"
  }
  exclude {
    stage: "deploy"
  }
}

but of course I must be doing something wrong since I get the following error:

Creating layer scale2
Creating Layer scale2
scale2 <- label_label_0_split_0
scale2 -> labelScaled
Setting up scale2
Top shape: 16 1 144 96 (221184)
Memory required for data: 364511232
Creating layer prob
Creating Layer prob
prob <- convc2
prob -> prob
Setting up prob
Top shape: 16 2 144 96 (442368)
Memory required for data: 366280704
Creating layer infoGainLoss
Creating Layer infoGainLoss
infoGainLoss <- prob
infoGainLoss <- label_label_0_split_1
infoGainLoss -> infoGainLoss
Check failed: bottom[1]->height() == 1 (144 vs. 1)

What am I doing wrong? Is it possible to weight the classes for semantic segmentation in caffe? I also cannot change my caffe installation since I am at work and I do not own the server it is installed on.

[Jonathan R. Williford]

The output dimensions of convc2 and labelScaled should be BATCHx2x144x96 and BATCHx1x144x96, no? I'm not sure why it is expecting "bottom[1]->height()" to be equal to 1, the height should be 144.

Are you sure "/path/to/infogainH.binaryproto" has the correct size?

Best,

Jonathan

--
You received this message because you are subscribed to the Google Groups "Caffe Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to caffe-users+unsubscribe@googlegroups.com.
To post to this group, send email to caffe...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/caffe-users/c0375339-2ca2-461a-a026-4d40218aa1b4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Corentin Jemine]

Yes I've used the BATCHxHEIGHTxWIDTHxCHANNEL format in my post when my data is of course in the caffe format BATCHxCHANNELxHEIGHTxWIDTH, your corrections have the right dimensions. My infogain matrix is only a 2 by 2 matrix, should it be scaled to the size of the image?

To unsubscribe from this group and stop receiving emails from it, send an email to caffe-users...@googlegroups.com.

[Jonathan R. Williford]

It looks like you are using an older version of Caffe? I think all of the references to "height" have been deprecated for quite some time.

I would suggest looking at your "src/caffe/layers/infogain_loss_layer.cpp" and try to understand why it is expecting a 1.  The error should be thrown by a CHECK_EQ or CHECK macro. Your infogain matrix is actually unlikely to be causing the issue.

Best,

Jonathan

To unsubscribe from this group and stop receiving emails from it, send an email to caffe-users+unsubscribe@googlegroups.com.

To post to this group, send email to caffe...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/caffe-users/606c1ff6-62be-4f1f-8787-9ddd917c4886%40googlegroups.com.

[Paul Delamusica]

Did you find the solution to your problem?

I have been looking into unbalanced data issue and found solutions evolve over the years. But I could not find a complete, updated working example.

Can some point/provide a working example?

Thanks!

[Corentin Jemine]

Nope. The best solution is probably to run away from caffe.