Caffe Iteration loss versus Train Net loss

[Rohan]

I'm using caffe to train a CNN with a Euclidean loss layer at the bottom, and my solver.prototxt file configured to display every 100 iterations. I see something like this,

Iteration 4400, loss = 0

I0805 11:10:16.976716 1936085760 solver.cpp:229]     Train net output #0: loss = 2.92436 (* 1 = 2.92436 loss)

I'm confused as to what the difference between the Iteration loss and Train net loss is. Usually the iteration loss is very small (around 0) and the Train net output loss is a bit larger. Can somebody please clarify?

[Rohan]

Bump. Anyone have any ideas? 

[Evan Shelhamer]

The `net output #k` result is the output of the net for that particular iteration / batch while the `Iteration T, loss = X` output is smoothed across iterations according to the `average_loss` field. Here is the relevant solver code: https://github.com/BVLC/caffe/blob/master/src/caffe/solver.cpp#L190-L221

It's odd that your iteration loss should be 0... have you tried with the latest master? The line number for the logging suggests this is an older version.

Evan Shelhamer

--
You received this message because you are subscribed to the Google Groups "Caffe Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to caffe-users...@googlegroups.com.
To post to this group, send email to caffe...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/caffe-users/de10fc84-86cb-4579-988d-a090451ee430%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Rohan]

I followed Evan's suggestion to display the output at every iteration, but the problem seems to be occurring right off the bat. Does anyone have any ideas? 

I0808 13:29:05.183802 1936085760 solver.cpp:294] Iteration 0, Testing net (#0)

I0808 13:30:25.560981 1936085760 solver.cpp:343]     Test net output #0: accuracy = 9.61966

I0808 13:30:25.561022 1936085760 solver.cpp:343]     Test net output #1: loss = 9.61966 (* 1 = 9.61966 loss)

I0808 13:30:27.563423 1936085760 solver.cpp:214] Iteration 0, loss = 0

I0808 13:30:27.563458 1936085760 solver.cpp:229]     Train net output #0: loss = 10.8541 (* 1 = 10.8541 loss)

I0808 13:30:27.563467 1936085760 solver.cpp:486] Iteration 0, lr = 0.1

I0808 13:30:29.547422 1936085760 solver.cpp:214] Iteration 1, loss = -1.0842e-19

I0808 13:30:29.547458 1936085760 solver.cpp:229]     Train net output #0: loss = 10.8489 (* 1 = 10.8489 loss)

I0808 13:30:29.547466 1936085760 solver.cpp:486] Iteration 1, lr = 0.0999925

I0808 13:30:31.288096 1936085760 solver.cpp:214] Iteration 2, loss = 2

I0808 13:30:31.288131 1936085760 solver.cpp:229]     Train net output #0: loss = 8.46936 (* 1 = 8.46936 loss)

I0808 13:30:31.288139 1936085760 solver.cpp:486] Iteration 2, lr = 0.099985

I0808 13:30:33.067306 1936085760 solver.cpp:214] Iteration 3, loss = 0

I0808 13:30:33.067339 1936085760 solver.cpp:229]     Train net output #0: loss = 9.17068 (* 1 = 9.17068 loss)

I0808 13:30:33.067347 1936085760 solver.cpp:486] Iteration 3, lr = 0.0999775

I0808 13:30:34.928208 1936085760 solver.cpp:214] Iteration 4, loss = -1.0842e-19

I0808 13:30:34.928246 1936085760 solver.cpp:229]     Train net output #0: loss = 8.57569 (* 1 = 8.57569 loss)

I0808 13:30:34.928253 1936085760 solver.cpp:486] Iteration 4, lr = 0.09997

I0808 13:30:36.712865 1936085760 solver.cpp:214] Iteration 5, loss = -1.0842e-19

I0808 13:30:36.712895 1936085760 solver.cpp:229]     Train net output #0: loss = 8.197 (* 1 = 8.197 loss)

I0808 13:30:36.712903 1936085760 solver.cpp:486] Iteration 5, lr = 0.0999625

I0808 13:30:38.513576 1936085760 solver.cpp:214] Iteration 6, loss = 0

I0808 13:30:38.513610 1936085760 solver.cpp:229]     Train net output #0: loss = 8.1923 (* 1 = 8.1923 loss)

I0808 13:30:38.513618 1936085760 solver.cpp:486] Iteration 6, lr = 0.099955

I0808 13:30:40.467524 1936085760 solver.cpp:214] Iteration 7, loss = -1.0842e-19

I0808 13:30:40.467558 1936085760 solver.cpp:229]     Train net output #0: loss = 8.29589 (* 1 = 8.29589 loss)

I0808 13:30:40.467566 1936085760 solver.cpp:486] Iteration 7, lr = 0.0999475

I0808 13:30:42.230612 1936085760 solver.cpp:214] Iteration 8, loss = 0

I0808 13:30:42.230640 1936085760 solver.cpp:229]     Train net output #0: loss = 7.77678 (* 1 = 7.77678 loss)

I0808 13:30:42.230648 1936085760 solver.cpp:486] Iteration 8, lr = 0.09994

I0808 13:30:43.976712 1936085760 solver.cpp:214] Iteration 9, loss = 3.68935e+19

I0808 13:30:43.976749 1936085760 solver.cpp:229]     Train net output #0: loss = 7.617 (* 1 = 7.617 loss)

I0808 13:30:43.976758 1936085760 solver.cpp:486] Iteration 9, lr = 0.0999326

I0808 13:30:45.731000 1936085760 solver.cpp:214] Iteration 10, loss = 0

I0808 13:30:45.731034 1936085760 solver.cpp:229]     Train net output #0: loss = 7.6669 (* 1 = 7.6669 loss)