I am running a small example to test my understanding of the system. I am creating a small network (40 training files and 8 validation files with 4 classes) and while training the network the error I have ran into is my accuracy is 0 until the 120th iteration where it jumps to 1 then drops back to 0. I have dropped my lr to .0001 and my loss stays in the range of 3-5% does anyone know any solutions.
I0622 13:08:36.713556 9231 solver.cpp:280] Learning Rate Policy: step
I0622 13:08:36.717217 9231 solver.cpp:337] Iteration 0, Testing net (#0)
I0622 13:08:36.842509 9231 solver.cpp:404] Test net output #0: accuracy = 0
I0622 13:08:36.842541 9231 solver.cpp:404] Test net output #1: loss = 7.67709 (* 1 = 7.67709 loss)
I0622 13:08:37.158406 9231 solver.cpp:228] Iteration 0, loss = 6.35291
I0622 13:08:37.158449 9231 solver.cpp:244] Train net output #0: loss = 6.35291 (* 1 = 6.35291 loss)
I0622 13:08:37.158468 9231 sgd_solver.cpp:106] Iteration 0, lr = 0.0001
I0622 13:08:50.262631 9231 solver.cpp:454] Snapshotting to binary proto file /home/addisonbe/Desktop/sample/snapshot/caffenet_train_iter_40.caffemodel
I0622 13:08:50.324101 9231 sgd_solver.cpp:273] Snapshotting solver state to binary proto file /home/addisonbe/Desktop/sample/snapshot/caffenet_train_iter_40.solverstate
I0622 13:08:50.362587 9231 solver.cpp:337] Iteration 40, Testing net (#0)
I0622 13:08:50.486214 9231 solver.cpp:404] Test net output #0: accuracy = 0
I0622 13:08:50.486261 9231 solver.cpp:404] Test net output #1: loss = 5.00905 (* 1 = 5.00905 loss)
I0622 13:08:50.793370 9231 solver.cpp:228] Iteration 40, loss = 3.55094
I0622 13:08:50.793406 9231 solver.cpp:244] Train net output #0: loss = 3.55094 (* 1 = 3.55094 loss)
I0622 13:08:50.793411 9231 sgd_solver.cpp:106] Iteration 40, lr = 1e-12
I0622 13:09:03.913767 9231 solver.cpp:454] Snapshotting to binary proto file /home/addisonbe/Desktop/sample/snapshot/caffenet_train_iter_80.caffemodel
I0622 13:09:03.979215 9231 sgd_solver.cpp:273] Snapshotting solver state to binary proto file /home/addisonbe/Desktop/sample/snapshot/caffenet_train_iter_80.solverstate
I0622 13:09:04.017534 9231 solver.cpp:337] Iteration 80, Testing net (#0)
I0622 13:09:04.135341 9231 solver.cpp:404] Test net output #0: accuracy = 0
I0622 13:09:04.135375 9231 solver.cpp:404] Test net output #1: loss = 3.70063 (* 1 = 3.70063 loss)
I0622 13:09:04.444550 9231 solver.cpp:228] Iteration 80, loss = 3.54556
I0622 13:09:04.444594 9231 solver.cpp:244] Train net output #0: loss = 3.54556 (* 1 = 3.54556 loss)
I0622 13:09:04.444603 9231 sgd_solver.cpp:106] Iteration 80, lr = 1e-20
I0622 13:09:17.573283 9231 solver.cpp:454] Snapshotting to binary proto file /home/addisonbe/Desktop/sample/snapshot/caffenet_train_iter_120.caffemodel
I0622 13:09:17.639202 9231 sgd_solver.cpp:273] Snapshotting solver state to binary proto file /home/addisonbe/Desktop/sample/snapshot/caffenet_train_iter_120.solverstate
I0622 13:09:17.677817 9231 solver.cpp:337] Iteration 120, Testing net (#0)
I0622 13:09:17.796124 9231 solver.cpp:404] Test net output #0: accuracy = 1
I0622 13:09:17.796161 9231 solver.cpp:404] Test net output #1: loss = 0.213932 (* 1 = 0.213932 loss)
I0622 13:09:18.104405 9231 solver.cpp:228] Iteration 120, loss = 3.15777
I0622 13:09:18.104449 9231 solver.cpp:244] Train net output #0: loss = 3.15777 (* 1 = 3.15777 loss)
I0622 13:09:18.104467 9231 sgd_solver.cpp:106] Iteration 120, lr = 1e-28
I0622 13:09:32.082479 9231 solver.cpp:454] Snapshotting to binary proto file /home/addisonbe/Desktop/sample/snapshot/caffenet_train_iter_160.caffemodel
I0622 13:09:32.147368 9231 sgd_solver.cpp:273] Snapshotting solver state to binary proto file /home/addisonbe/Desktop/sample/snapshot/caffenet_train_iter_160.solverstate
I0622 13:09:32.186655 9231 solver.cpp:337] Iteration 160, Testing net (#0)
I0622 13:09:32.311929 9231 solver.cpp:404] Test net output #0: accuracy = 0
I0622 13:09:32.311976 9231 solver.cpp:404] Test net output #1: loss = 3.70125 (* 1 = 3.70125 loss)
I0622 13:09:32.618758 9231 solver.cpp:228] Iteration 160, loss = 5.15814
I0622 13:09:32.618805 9231 solver.cpp:244] Train net output #0: loss = 5.15814 (* 1 = 5.15814 loss)
I0622 13:09:32.618824 9231 sgd_solver.cpp:106] Iteration 160, lr = 1e-36
I0622 13:09:47.952828 9231 solver.cpp:454] Snapshotting to binary proto file /home/addisonbe/Desktop/sample/snapshot/caffenet_train_iter_200.caffemodel
I0622 13:09:48.017726 9231 sgd_solver.cpp:273] Snapshotting solver state to binary proto file /home/addisonbe/Desktop/sample/snapshot/caffenet_train_iter_200.solverstate
I0622 13:09:48.171784 9231 solver.cpp:317] Iteration 200, loss = 4.59124
I0622 13:09:48.171813 9231 solver.cpp:337] Iteration 200, Testing net (#0)
I0622 13:09:48.288655 9231 solver.cpp:404] Test net output #0: accuracy = 0
I0622 13:09:48.288714 9231 solver.cpp:404] Test net output #1: loss = 4.11947 (* 1 = 4.11947 loss)
I0622 13:09:48.288718 9231 solver.cpp:322] Optimization Done.
I0622 13:09:48.288722 9231 caffe.cpp:222] Optimization Done.