Training from snapshot not qualitatively the same (in accuracy) as continous training
sarge
I am seeking help concerning training Caffe AlexNet. A summary of the problem is that reported accuracy of continuous training of Caffe network (AlexNet on a randomly selected 200,000 ILSVRC 2012 images) differs from that of accuracy reported from resuming training from a snapshot of solver state.
The picture attached shows that reported accuracy differs based on number of iterations before snapshot and do not match training without snapshot. X-axis on the plot is the number of iterations (or number of images processed if you multiply each point by mini batch size of 100). The expectation is that all lines should match up irrespective of how many iterations before snapshot.
Please note that all experiments are setup the same way with same training/validation data and parameters (i.e., same seed, same mini batch size, fixed learning rate policy, zero momentum etc).
Thanks for your help.
