Caffe significance of Validation (test) loss and Train (loss)

[Suyog Trivedi]

I am trying to train caffe cifar 10 model for 3 custom classes. I have created the LMDB for training and validation. Data is shuffled before creating LMDB. I tried to plot the losses for training and testing for few iterations(4500). I do not understand what exactly is happening in training and whether the model is learning anything at all or not. Figure is attached below. 

Red line - Test Accuracy 

Green Line - Test loss

Blue Line - Train loss 

 Can anyone please look at the plot and explain to me the interpretation of the plot. 

[Ketil Malde]

> * Can anyone please look at the plot and explain to me the interpretation
> of the plot. *

My interpretation is that you have a bug in your plotting script. (The
oscillations look very strange, and my guess is that you are including
alternating numbers from two sources in the input to the plot.)

I have a script that I use to monitor progress, if anybody is
interested, I can post it here (or perhaps make a github repo for it?).

-k
--
If I haven't seen further, it is by standing in the footprints of giants

[Suyog Trivedi]

Thanks for your response. Posting the plotting script would be of great help. 

[Ketil Malde]

On 2016-10-12 13:05 (+0200), Suyog Trivedi <strive...@gmail.com> wrote:

> Thanks for your response. Posting the plotting script would be of great
> help.

It's a bit of a hack, but run the plot.sh script with caffe's INFO
output (found in /tmp) as parameter, and it will update the plot every
five seconds (easily tunable by modifying the scripts).

Let me know if it works out?

[Suyog Trivedi]

Thanks for your help. I ran the training for 15k iterations. I am getting the output curve as below. I am attaching the log file also. Is the network training at all? Why validation loss is not decreasing?

Red - validation accuracy 

Green - validation loss

Blue - training loss