Loss stays a constant 0.69 while training the NIN ImageNet model

[Rose Perrone]

The only changes I made to the train_val.prototxt are the filenames for the data and image mean, as well as the number of outputs (changed from 1000 to 2). In the solver.prototxt, I reduced the stepsize and set the solver_mode to CPU. This is the output of training:

I1205 10:42:36.951148 2041447168 solver.cpp:298]     Test net output #0: accuracy = 0.0908989

I1205 10:42:51.826429 2041447168 solver.cpp:191] Iteration 0, loss = 0.699334

I1205 10:42:51.826479 2041447168 solver.cpp:403] Iteration 0, lr = 0.01

I1205 10:47:34.034021 2041447168 solver.cpp:191] Iteration 20, loss = 0.693147

I1205 10:47:34.034725 2041447168 solver.cpp:403] Iteration 20, lr = 0.01

I1205 10:52:22.601667 2041447168 solver.cpp:191] Iteration 40, loss = 0.693147

I1205 10:52:22.601980 2041447168 solver.cpp:403] Iteration 40, lr = 0.01

I1205 10:56:57.610334 2041447168 solver.cpp:191] Iteration 60, loss = 0.693147

I1205 10:56:57.610630 2041447168 solver.cpp:403] Iteration 60, lr = 0.01

I1205 11:01:21.185401 2041447168 solver.cpp:191] Iteration 80, loss = 0.693147

I1205 11:01:21.185777 2041447168 solver.cpp:403] Iteration 80, lr = 0.01

I1205 11:05:48.598832 2041447168 solver.cpp:191] Iteration 100, loss = 0.693147

I1205 11:05:48.599266 2041447168 solver.cpp:403] Iteration 100, lr = 0.01

I1205 11:10:21.061986 2041447168 solver.cpp:191] Iteration 120, loss = 0.693147

...

Perhaps it doesn't train properly on CPU?

The accuracy looks reasonable, because I'm testing on a 9:1 ratio of negative to positive images.

The NIN train_val.prototxt is here: https://gist.github.com/mavenlin/d802a5849de39225bcc6

[Mohamed Omran]

What happens if you decrease the learning rate by a factor of 5-10 or so?

--
You received this message because you are subscribed to the Google Groups "Caffe Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to caffe-users...@googlegroups.com.
To post to this group, send email to caffe...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/caffe-users/95c993d9-a6a3-4673-b770-0382da7baa83%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Rose Perrone]

Thank you! That worked. I posted the new training output below. Can you describe how a too-high learning rate caused the loss to remain constant?

I1205 17:28:00.763536 2041447168 solver.cpp:298]     Test net output #0: accuracy = 0.737865

I1205 17:28:19.864238 2041447168 solver.cpp:191] Iteration 0, loss = 0.690854

I1205 17:28:19.867792 2041447168 solver.cpp:403] Iteration 0, lr = 0.001

I1205 17:33:28.802063 2041447168 solver.cpp:191] Iteration 20, loss = 0.676342

I1205 17:33:28.809875 2041447168 solver.cpp:403] Iteration 20, lr = 0.001

I1205 17:38:10.105093 2041447168 solver.cpp:191] Iteration 40, loss = 0.653457

I1205 17:38:10.108834 2041447168 solver.cpp:403] Iteration 40, lr = 0.001

I1205 17:42:15.682371 2041447168 solver.cpp:191] Iteration 60, loss = 0.62827

I1205 17:42:15.689903 2041447168 solver.cpp:403] Iteration 60, lr = 0.001

I1205 17:46:37.104629 2041447168 solver.cpp:191] Iteration 80, loss = 0.589064

[Sai Hei Yeung]

Thanks for the suggestion.  I was having the same problem.