Why my testing error is not decreasing?

[Jinchao Lin]

Hi, I am new to Caffe and deep learning, so I would really appreciate if anyone can help with me: 

I am using a pre-trained OxfordNet (VGG team in ILSVRC-2014) to fine-tune on my own image data set. When I trained my network, I found that although my training loss decrease to very small number, my testing loss is not decreasing at all. 

-------------- At iteration 1000 --------------------: 

I1201 12:02:35.751077 58322 solver.cpp:294] Iteration 1000, Testing net (#0)

I1201 12:03:06.247112 58322 solver.cpp:343]     Test net output #0: accuracy@1 = 0.056

I1201 12:03:06.247381 58322 solver.cpp:343]     Test net output #1: accuracy@5 = 0.128

I1201 12:03:06.247411 58322 solver.cpp:343]     Test net output #2: loss/loss = 5.62865 (* 1 = 5.62865 loss)

I1201 12:03:06.761168 58322 solver.cpp:214] Iteration 1000, loss = 4.93189

I1201 12:03:06.761307 58322 solver.cpp:229]     Train net output #0: loss/loss = 4.93189 (* 1 = 4.93189 loss)

------------- iteration 10000 --------------------

I1201 16:58:24.017702 58322 solver.cpp:294] Iteration 10000, Testing net (#0)

I1201 16:58:53.020931 58322 solver.cpp:343]     Test net output #0: accuracy@1 = 0.099

I1201 16:58:53.021220 58322 solver.cpp:343]     Test net output #1: accuracy@5 = 0.192

I1201 16:58:53.021253 58322 solver.cpp:343]     Test net output #2: loss/loss = 7.48841 (* 1 = 7.48841 loss)

I1201 16:58:53.531373 58322 solver.cpp:214] Iteration 10000, loss = 0.0800702

I1201 16:58:53.531476 58322 solver.cpp:229]     Train net output #0: loss/loss = 0.080069 (* 1 = 0.080069 loss)

--------------iteration 20000 ----------------------

I1201 22:26:55.695014 58322 solver.cpp:294] Iteration 20000, Testing net (#0)

I1201 22:27:24.720124 58322 solver.cpp:343]     Test net output #0: accuracy@1 = 0.144

I1201 22:27:24.720396 58322 solver.cpp:343]     Test net output #1: accuracy@5 = 0.257

I1201 22:27:24.720427 58322 solver.cpp:343]     Test net output #2: loss/loss = 6.87847 (* 1 = 6.87847 loss)

I1201 22:27:25.231263 58322 solver.cpp:214] Iteration 20000, loss = 0.00149296

I1201 22:27:25.231372 58322 solver.cpp:229]     Train net output #0: loss/loss = 0.00149117 (* 1 = 0.00149117 loss)

-------------- iteration 50000 -----------------------

I1202 21:40:08.376410 52642 solver.cpp:294] Iteration 50000, Testing net (#0)

I1202 21:40:37.445066 52642 solver.cpp:343]     Test net output #0: accuracy@1 = 0.16

I1202 21:40:37.445420 52642 solver.cpp:343]     Test net output #1: accuracy@5 = 0.282

I1202 21:40:37.445453 52642 solver.cpp:343]     Test net output #2: loss/loss = 8.22018 (* 1 = 8.22018 loss)

I1202 21:40:37.956485 52642 solver.cpp:214] Iteration 50000, loss = 3.13867e-05

I1202 21:40:37.956634 52642 solver.cpp:229]     Train net output #0: loss/loss = 3.13329e-05 (* 1 = 3.13329e-05 loss)

-------------------------------------------------------------

I am using a K20 GPU with 5GB graphic memory. In order to fit the oxford-net, I reduce the training batch to be 20 images and test batch to 5 images. In order to compensate small testing batch, I increase the test_iter to be 200. 

 I have 4500 labeled image data with different sizes between 300~500. I use 4000 images for training and the rest for testing. Those images belongs to 447 asses and the classification is fairly difficult, at least by human's eye inspection.

It seems like the network is learning something, as the test prediction do increase from 0 to 16%. But the test loss is not decreasing at all. I am wondering what's wrong with my model? 

Thank you for any advice. 

Best.

Jin

[Kemal ÇİZMECİLER]

did you find a solution?

5 Aralık 2015 Cumartesi 22:40:45 UTC+2 tarihinde Jinchao Lin yazdı:

[Maryam Asadi]

Hi.

I have the same problem.

How could you solve your problem.

Thanks