Training killed with strange log when training caffenet

[Alykhan Tejani]

Hi,

I am trying to train the reference caffenet on my own data and have a very strange log message which killed the training after one iteration, does anyone know what this means?

I1222 11:35:18.429803  3466 net.cpp:219] Network initialization done.

I1222 11:35:18.429810  3466 net.cpp:220] Memory required for data: 343008808

I1222 11:35:18.429900  3466 solver.cpp:41] Solver scaffolding done.

I1222 11:35:18.430073  3466 solver.cpp:160] Solving CaffeNet

I1222 11:35:18.430104  3466 solver.cpp:247] Iteration 0, Testing net (#0)

I1222 11:37:02.970240  3466 solver.cpp:298]     Test net output #0: accuracy = 0.532

I1222 11:37:02.970716  3466 solver.cpp:298]     Test net output #1: loss = 0.727713 (* 1 = 0.727713 loss)

Killed

Thanks,

Aly

[Alykhan Tejani]

Ran it again and now it seg faults. 

It looks like it might be a memory issue, as the memory usage was >90%, however the segmentation fault and the arbitrary error seems strange

[Talat Shaikh]

Show your solver.prototxt & train.prototxt. This is mostly the problem of batch size.