Train Net output loss and Iteration loss are different in value

[Saumya Jetley]

A snippet of my log file is as below:

I0312 14:54:18.109833 18096 solver.cpp:291] Iteration 0, Testing net (#0)
I0312 14:54:34.441314 18096 solver.cpp:342]     Test net output #0: loss = 1.56061e+06 (* 1 = 1.56061e+06 loss)
I0312 14:54:44.045902 18096 solver.cpp:213] Iteration 0, loss = 1.81236e+06
I0312 14:54:44.046020 18096 solver.cpp:228]     Train net output #0: loss = 1.87957e+06 (* 1 = 1.87957e+06 loss)
I0312 14:54:44.046072 18096 solver.cpp:473] Iteration 0, lr = 1e-10
I0312 15:03:41.522418 18096 solver.cpp:213] Iteration 50, loss = 12807.8
I0312 15:03:41.522861 18096 solver.cpp:228]     Train net output #0: loss = 17813.5 (* 1 = 17813.5 loss)
I0312 15:03:41.522929 18096 solver.cpp:473] Iteration 50, lr = 1e-10
I0312 15:12:37.716862 18096 solver.cpp:213] Iteration 100, loss = 18774.9
I0312 15:12:37.717109 18096 solver.cpp:228]     Train net output #0: loss = 2264.65 (* 1 = 2264.65 loss)
I0312 15:12:37.717128 18096 solver.cpp:473] Iteration 100, lr = 1e-10
I0312 15:21:25.475277 18096 solver.cpp:213] Iteration 150, loss = 39028
I0312 15:21:25.475545 18096 solver.cpp:228]     Train net output #0: loss = 47999.4 (* 1 = 47999.4 loss)
I0312 15:21:25.475769 18096 solver.cpp:473] Iteration 150, lr = 1e-10
I0312 15:30:03.596294 18096 solver.cpp:213] Iteration 200, loss = 52798.1
I0312 15:30:03.596499 18096 solver.cpp:228]     Train net output #0: loss = 85300.1 (* 1 = 85300.1 loss)
I0312 15:30:03.596521 18096 solver.cpp:473] Iteration 200, lr = 1e-10
I0312 15:30:26.399509 18096 solver.cpp:362] Snapshotting to ./---------------.caffemodel
I0312 15:30:28.711554 18096 solver.cpp:370] Snapshotting solver state to ./models/----------------.solverstate
I0312 15:30:29.757135 18096 solver.cpp:291] Iteration 203, Testing net (#0)
I0312 15:30:46.402194 18096 solver.cpp:342]     Test net output #0: loss = 18269.9 (* 1 = 18269.9 loss)
I0312 15:40:22.855330 18096 solver.cpp:213] Iteration 250, loss = 43913.7

As you can see, the test loss is showing a definite reduction.
However, my query is that - "What is the difference between Iteration and Train Net output losses? Why don't they have the same value (when in majority of other log files the values are the same)? Possibly because I am training with 'Normalization: false' and that magnifies the difference!!"

Thanks in advance..
SJ

[Daniel Seita]

No, it's because they do have different meanings:

https://groups.google.com/forum/#!topic/caffe-users/WEhQ92s9Vus

As far as I can tell, train net output losses are the average losses for that particular batch (i.e., the average of the softmax losses for elements *in that batch*), whereas iteration loss is a *smoothed* version of the train net loss. (Or it might be the test loss smoothed ... but I'm pretty sure it's the train loss.)