About CaffeNet Training

[Patel Umang]

From Last Two Months I am Trying to train CaffeNet on my Own Data set which contain the image of Happy and neutral Face Emotions from scratch.

I am using Same the caffeNet model present in github caffe repository which is bvlc_reference_caffenet.

I am Creating LMDB and Mean Image from script present in imagenet to create lmdb and mean.

Still after 500 iteration the loss always play around 0.69.

Is any one have any idea whats going on or it will decrease on more iteration or something ?

[Tamas Nemes]

So my first question:

How did you manage to successfully train nets with Caffe? What hardware and OS are you using for training? When I try to train using Caffe, it aborts the training process while building the net without any error message.

Second, it depends whether you have a classification or a detection task (I could not figure out exactly by looking at what you provided). But either way: A loss of 0.69 is very good I think, as most call out a loss of 2.5-1.5 for their goal of training. To see exactly how good your net is, you should test it instead to get a mean AP. And in all cases, you could play with the hyperparameters (batch size, learning rate, lr decay...) to see how it affect your results.

[Patel Umang]

Thanks for Reply

I am using Google Colab as training platform which provide os as Ubuntu 18.04 and GPU one of Nvidia K80s, T4s, P4s and P100s. for more information here.

I am doing Classification task of Happy face and neutral face.

I found this error or something like that from this group only you can check here.This similar thing happens when I train the model for more iteration like 100K.

My goal is to get 1.5 or 2.5 as loss and accuracy approximately 80 to 75.

but here accuracy also stick to 0.5 on more iteration also.

and let me know if you need more info about training the model on caffe on googlecolab.

 

but my next question is 

please confirm me that this is not some kind of bug or error so I can proceed further?

Since my goal is to check what CNN learns which I can find using Deep Dream Algorithm, so this will affect on it or not?

[Patel Umang]

On Wednesday, July 8, 2020 at 9:24:10 AM UTC+5:30, Patel Umang wrote:

Thanks for Reply

I am using Google Colab as training platform which provide os as Ubuntu 18.04 and GPU one of Nvidia K80s, T4s, P4s and P100s. for more information here.

I am doing Classification task of Happy face and neutral face.

I found this error or something like that from this group only you can check here.This similar thing happens when I train the model for more iteration like 100K.

My goal is to get 1.5 or 2.5 as loss and accuracy approximately 80 to 75.

but here accuracy also stick to 0.5 on more iteration also.

and let me know if you need more info about training the model on caffe on googlecolab.

 

but my next question is 

please confirm me that this is not some kind of bug or error so I can proceed further?

Since my goal is to check what CNN learns which I can find using Deep Dream Algorithm, so this will affect on it or not?

 And when I load this model It gives me same probability why?

[Tamas Nemes]

So: If you aim to get 1.5-2.5 as loss, then you succeeded well, because lower loss is better. I don't know how exactly you tested the main accuracy, but assuming you meant 50%, then I got a theory for it sticking to this value.

You might have ran into the problem of overfitting. If 50% of your dataset is neutral faces and the other half happy faces, then after training the network for too long, it can happen it concludes to always predict whether happy or neutral, which gives an accuracy of 50% (but this number is of course not real because it always predicts the same thing). How long you should train depends on your dataset size. Training a classification network for 100k iterations is way too much as even a detection network with a middle sized dataset is trained for appr. 30k-60k iterations, so you should train less to avoid overfitting. This is most likely not a bug.