Object detection or classification with two softmax layers

[tuzand]

I have two type of datasets:

• images with concrete class labels (car types)
• images with labels only if the image has an object on it or not (car or not)

With these images, I would like to train a network with two different FCs+softmaxwithloss layers at the end after the last convolution layer. I think that I could gain performance also with the images having only a binary labels, and at test time I could also get class agnostic information from an image with unknown car type (so if it is a car or not).

My problem is at training time I should disable somehow one branch of my network (the FCs+softmaxwithloss) regarding to the label type of the input. 

Is there a way to do that in caffe?

Thank you!

ps.: I could organise one mini-batch e.g. first 50% with concrete class labels, the second 50% binary labels, and split the feature maps and the labels after the last conv layer. But in this case I can train / fine-tune my network only if I have images from both types.