How to use (a) sum of two loss functions and (b) multiply pre-softmax output with a constant

[Bharat Bhusan Sau]

I am facing following difficulties while implementing Dark Knowledge paper in Caffe:

1. How to divide/multiply  pre-softmax output(i.e. output of the last hidden layer before softmax layer) by some value T>1 ? (Its required to soften the softmax probabilistic output)

2. How to use sum of two loss functions in Caffe as my new loss function? (New Loss function= cross_entropy(hard_labels,predicted_labels) + lamda*euclidean_loss(soft_output,predicted_output) )

Section 2.1 in this paper describes the Dark Knowledge method in compact way.

Please comment if anybody knows any tricks to solve this issue.

[Nam Vo]

1) Write your own layer to do that, though I don't think that softening thing will affect the learning at all.
2) Just define 2 loss layers (and loss_weight accordingly), caffe will auto sum them

[Bharat Bhusan Sau]

Thanks Nam. Its really nice answer.

I now solved both issue 1 and 2.  Only thing is: Can I tune the loss_weight automatically ? (i.e. I want the network to learn loss_weight automatically).

Thanks again.

[Bharat Bhusan Sau]

Bdw, I used "Eltwise" layer with "PROD" to solve the first issue.

[uttisht]

Hi Bharat, have you implemented it? I would like to know the approach.

[Bharat Bhusan Sau]

Do you mean dark knowledge method or something else ?

--
You received this message because you are subscribed to a topic in the Google Groups "Caffe Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/caffe-users/gPhaNGY0kgU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to caffe-users...@googlegroups.com.
To post to this group, send email to caffe...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/caffe-users/9f22285b-d58a-499e-9e2f-62c783fcecb1%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[uttisht]

Hi Bharat, 

Thanks for your reply. Yes, Dark Knowledge. 

[Bharat Bhusan Sau]

I attached the training prototxt for reference. I did the tests on MNIST dataset.

To view this discussion on the web visit https://groups.google.com/d/msgid/caffe-users/69d33278-63d1-4b7f-98d1-b1c407b0a296%40googlegroups.com.

[uttisht]

Thank you. I will try this out.

[uttisht]

Hi Bharat,

I have a doubt. Do I need to train both models separately? I tried to train together using single training protxt file. But, didn't work. 

[Bharat Bhusan Sau]

The prototxt file I provided contains both teacher and student net. But learning rate of each layer of teacher net is zero implying during training there will be no update to the teacher net. So what you have to do is :

1. At first train a teacher net which will have good accuracy.

2. Once the teacher net is trained, take the teacher model by using '-weights' in the terminal execution command, as a result the learnt weights of the teacher model will be shared in the new model which contains both teacher and student. You can always extract the student model from this hybrid model later. The weights are shared between two layers when their name are same. You get output from the softmax layer of teacher net and feed it to student model for matching.

3. To test the model, only use the student architecture in the test.prototxt.

This method makes it easier to do train the  student model in case we have to change the temperature param many times. However, for more difficult datasets like CIFAR10, where the teacher model is cumbersome, it will take lot of time to train. In that case you may want to modify the method I suggested. For that case, I follow another way, which requires little bit of coding task, but is faster than the previous method:

1. Train Teacher model.

2. Get soft output of the training data from the teacher model.

3. Create hdf5 dataset which contains training image, ground truth label, soft outputs

4. Use this hdf5 dataset for training the student model.

You should be very careful while creating the hdf5 dataset. All the required preprocessing should be done properly. Then only it can be used for training.

[uttisht]

Thanks a lot Bharat. Everything is clear now.

[Bob Zigon]

Nam Vo

I would like to ask you a question about your response #2. I have a network that splits into 2 pieces.

Each piece uses the softmax loss to produce 8 possible outputs.

If I weight each layer with a coef of 0.5, are you saying that the output of the 2 softmax layers will

be identical because they will have been averaged internally and then the results are made available to

the 16 outputs?

[sjala...@gmail.com]

hello Sau ,Have you solved the problem that automatically tune loss_weight ?

在 2016年3月12日星期六 UTC+11上午2:14:43，Bharat Bhusan Sau写道：