caffe train crash

[Jeremy Rutman]

i hit the following segfault which appears to be an   internal caffe error, or maybe a hardware error - or maybe simply out of memory -  anyone have a clue?

This happened after a few hrs of training , using 

caffe train -solver solver.prototxt 

I0827 06:45:15.934933  5671 solver.cpp:454] Snapshotting to binary proto file snapshot/minimal_net_iter_5000.caffemodel
I0827 06:47:15.105124  5671 sgd_solver.cpp:273] Snapshotting solver state to binary proto file snapshot/minimal_net_iter_5000.solverstate
*** Aborted at 1472298435 (unix time) try "date -d @1472298435" if you are using GNU date ***
PC: @     0x7f884dd125e8 caffe::SolverState::SerializeWithCachedSizesToArray()
*** Error in `caffe': malloc(): memory corruption: 0x000000001eb38670 ***
Segmentation fault (core dumped)

[Hyo Jin Kim]

Hi, I am having same error. Have you solved the problem?

Thanks,

Hyo Jin

[Somayyeh Ataei Kachouei]

Something went wrong when you try to save your model in binary format and size of model is big, so try to save model as HDF5 and it probably work.