Issues loading pre-trained caffemodel in solver

[Stepan Andreenko]

I train a net via pycaffe using sdgsolver. At some point I completely stop the training and dump net parameters (not solver state)
to a .caffemodel file:

    solver.net.save("some.caffemodel")

Later I re-run training script to continue training, with exactly the same solver parameters and data, but loading
pre-trained net weights from the previous stage:

    solver = caffe.get_solver("solver.tmp")
    solver.net.copy_from("some.caffemodel")

If at this point I test the net using

    solver.net.forward()

I can make sure that this is exactly the same net that I got before saving.

However, when I try to continue training using the solver.step

    solver.step(10)

the loss value is many times more than it was at the end of the first training stage,
and after some iterations the net appears to be damaged or even completely destroyed.

It seems that after loading net to solver some solver data is left uninitialized or filled
with garbage. How should I initialize the solver so that it could work with a pre-trained net?

[Evan Shelhamer]

Resuming from a saved model (.caffemodel) instead of restoring from a saved solver state (.solverstate) discards history and state such as the momentum for SGD, and so can give different updates. You should call `solver.restore('snapshot.solverstate')` to resume learning with the same state.

Evan Shelhamer

--
You received this message because you are subscribed to the Google Groups "Caffe Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to caffe-users+unsubscribe@googlegroups.com.
To post to this group, send email to caffe...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/caffe-users/bbcc31c1-d3db-4a11-b451-abc51ddeaff5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.