I1124 17:58:37.106029 14422 solver.cpp:160] Solving CUBCaffeNet
I1124 17:58:37.106073 14422 solver.cpp:247] Iteration 0, Testing net (#0)
I1124 17:58:51.061630 14422 solver.cpp:298] Test net output #0: accuracy = 0.0056
I1124 17:58:51.137616 14422 solver.cpp:191] Iteration 0, loss = 3.81322
I1124 17:58:51.137655 14422 solver.cpp:403] Iteration 0, lr = 0.001
Segmentation fault (core dumped)
caffe test -model models/finetune_flickr_style/test.prototxt -weights data/flickr_style/finetune_flickr_style.caffemodel
I1124 15:38:45.287902 4025 caffe.cpp:169] Batch 49, prob = 0.051752
I1124 15:38:45.287912 4025 caffe.cpp:174] Loss: 0
I1124 15:38:45.287925 4025 caffe.cpp:186] prob = 0.0314738
I1124 15:38:45.287942 4025 caffe.cpp:186] prob = 0.0567422 (* 4.1037e-41 = 2.32896e-42 loss)
PC: @ 0x52eda1 test()
*** SIGSEGV (@0x16237a5d0) received by PID 4025 (TID 0x7f9060a15a40) from PID 1647814096; stack trace: ***
@ 0x7f905684ec30 (unknown)
@ 0x52eda1 test()
@ 0x53044c main
@ 0x7f9056839ec5 (unknown)
Segmentation fault (core dumped)
Just wondering in your case it says "Solving CUBCaffeNet" wheras I'm testing "FlickrStyleCaffeNet"
I get this problem for other tests as well:
Always after batch 49 and always following a loss of zero
/caffe/examples/cifar10$ caffe test -model ./cifar10_full.prototxt -weights cifar10_full_iter_70000.caffemodel -gpu 0
I1124 15:53:43.828536 4194 caffe.cpp:169] Batch 49, prob = 0.0551675
I1124 15:53:43.828547 4194 caffe.cpp:174] Loss: 0
I1124 15:53:43.828567 4194 caffe.cpp:186] prob = 0.0796905
I1124 15:53:43.828580 4194 caffe.cpp:186] prob = 0.0331636 (* 5.60519e-45 = 0 loss)
I1124 15:53:43.828598 4194 caffe.cpp:186] prob = 0.115024 (* -5.31795e+37 = -6.11692e+36 loss)
I1124 15:53:43.828611 4194 caffe.cpp:186] prob = 0.158395 (* 5.60519e-45 = 1.4013e-45 loss)
PC: @ 0x52eda1 test()
*** SIGSEGV (@0x9d19510) received by PID 4194 (TID 0x7f9f08c5ba40) from PID 164730128; stack trace: ***
@ 0x7f9efea8ec30 (unknown)
@ 0x52eda1 test()
@ 0x53044c main
@ 0x7f9efea79ec5 (unknown)
@ 0x52da09 (unknown)
Segmentation fault (core dumped)
I1125 09:38:07.639384 18102 solver.cpp:160] Solving CUBCaffeNet
I1125 09:38:07.639418 18102 solver.cpp:247] Iteration 0, Testing net (#0)
I1125 09:38:09.132885 18102 solver.cpp:298] Test net output #0: accuracy = 0.01
I1125 09:38:09.151484 18102 solver.cpp:191] Iteration 0, loss = 3.77259
I1125 09:38:09.151518 18102 solver.cpp:403] Iteration 0, lr = 0.001
I1125 09:38:09.863337 18102 solver.cpp:191] Iteration 20, loss = 0
I1125 09:38:09.863363 18102 solver.cpp:403] Iteration 20, lr = 0.001
I1125 09:38:10.564663 18102 solver.cpp:191] Iteration 40, loss = nan
I1125 09:38:10.564692 18102 solver.cpp:403] Iteration 40, lr = 0.001
I1125 09:38:11.266839 18102 solver.cpp:191] Iteration 60, loss = nan
I1125 09:38:11.266866 18102 solver.cpp:403] Iteration 60, lr = 0.001
I1125 09:38:11.968343 18102 solver.cpp:191] Iteration 80, loss = nan
I1125 09:38:11.968369 18102 solver.cpp:403] Iteration 80, lr = 0.001
I1125 09:38:12.669984 18102 solver.cpp:191] Iteration 100, loss = nan
I1125 09:38:12.670012 18102 solver.cpp:403] Iteration 100, lr = 0.001
I1125 09:38:13.371335 18102 solver.cpp:191] Iteration 120, loss = nan
I1125 09:38:13.371362 18102 solver.cpp:403] Iteration 120, lr = 0.001
Segmentation fault (core dumped)
I1125 09:57:15.725137 20973 solver.cpp:160] Solving CUBCaffeNet
I1125 09:57:15.725178 20973 solver.cpp:247] Iteration 0, Testing net (#0)
I1125 09:57:16.387609 20973 solver.cpp:298] Test net output #0: accuracy = 0.03
I1125 09:57:16.402441 20973 solver.cpp:191] Iteration 0, loss = 4.45611
I1125 09:57:16.402470 20973 solver.cpp:403] Iteration 0, lr = 0.001
I1125 09:57:16.952252 20973 solver.cpp:191] Iteration 20, loss = 0
I1125 09:57:16.952280 20973 solver.cpp:403] Iteration 20, lr = 0.001
I1125 09:57:17.499045 20973 solver.cpp:191] Iteration 40, loss = 0
I1125 09:57:17.499074 20973 solver.cpp:403] Iteration 40, lr = 0.001
I1125 09:57:18.040643 20973 solver.cpp:191] Iteration 60, loss = 87.3365
I1125 09:57:18.040671 20973 solver.cpp:403] Iteration 60, lr = 0.001
I1125 09:57:18.582733 20973 solver.cpp:191] Iteration 80, loss = 87.3365
I1125 09:57:18.582762 20973 solver.cpp:403] Iteration 80, lr = 0.001
I1125 09:57:19.124579 20973 solver.cpp:191] Iteration 100, loss = 87.3365
I1125 09:57:19.124608 20973 solver.cpp:403] Iteration 100, lr = 0.001
I1125 09:57:19.666509 20973 solver.cpp:191] Iteration 120, loss = 87.3365
I1125 09:57:19.666538 20973 solver.cpp:403] Iteration 120, lr = 0.001
I1125 09:57:20.208319 20973 solver.cpp:191] Iteration 140, loss = 87.3365
I1125 09:57:20.208346 20973 solver.cpp:403] Iteration 140, lr = 0.001
I1125 09:57:20.749861 20973 solver.cpp:191] Iteration 160, loss = 87.3365
I1125 09:57:20.749889 20973 solver.cpp:403] Iteration 160, lr = 0.001
I1125 09:57:21.291721 20973 solver.cpp:191] Iteration 180, loss = 87.3365
I1125 09:57:21.291751 20973 solver.cpp:403] Iteration 180, lr = 0.001
I1125 09:57:21.833292 20973 solver.cpp:191] Iteration 200, loss = 87.3365
I1125 09:57:21.833319 20973 solver.cpp:403] Iteration 200, lr = 0.001
I1125 09:57:22.374850 20973 solver.cpp:191] Iteration 220, loss = 70.6882
I1125 09:57:22.374876 20973 solver.cpp:403] Iteration 220, lr = 0.001
I1125 09:57:22.916332 20973 solver.cpp:191] Iteration 240, loss = 64.3139
I1125 09:57:22.916360 20973 solver.cpp:403] Iteration 240, lr = 0.001
I1125 09:57:23.457355 20973 solver.cpp:191] Iteration 260, loss = 87.3365
I1125 09:57:23.457381 20973 solver.cpp:403] Iteration 260, lr = 0.001
I1125 09:57:23.998904 20973 solver.cpp:191] Iteration 280, loss = 49.689
I1125 09:57:23.998937 20973 solver.cpp:403] Iteration 280, lr = 0.001
I1125 09:57:24.540182 20973 solver.cpp:191] Iteration 300, loss = 45.2114
I1125 09:57:24.540211 20973 solver.cpp:403] Iteration 300, lr = 0.001
I1125 09:57:25.081493 20973 solver.cpp:191] Iteration 320, loss = 41.3736
I1125 09:57:25.081521 20973 solver.cpp:403] Iteration 320, lr = 0.001
I1125 09:57:25.622854 20973 solver.cpp:191] Iteration 340, loss = 45.4241
I1125 09:57:25.622884 20973 solver.cpp:403] Iteration 340, lr = 0.001
Segmentation fault (core dumped)
...
make sure to test with the right model, for example:
caffe test -model ./cifar10_full_train_test.prototxt -weights cifar10_full_iter_10000.caffemodel -gpu 0
instead of
caffe test -model ./cifar10_full.prototxt -weights cifar10_full_iter_10000.caffemodel -gpu 0