Accuracy changes even when learning rate is set to zero

[Suraj Srinivas]

Hello all,

While finetuning a model, I noticed a curious thing - even when I set all the blobs_lr and weight_decay parameters to 0 for all layers, the loss and accuracy still change. After simply having run it for about 5k iterations, I noticed that the accuracy wasn't changing much from the initial value.

Intial acc = 64.2 %
After 1k iterations = 63.55%

After 2k iterations = 63.7%
After 3k = 64%

After 4k = 63.7% ...etc.

I even removed dropout and checked - no change in behaviour.
Has anyone encountered this kind of behaviour before? If so, can anyone tell me why this may be happening?

[Evan Shelhamer]

The momentum is independent of the learning rate. If you are resuming from a solverstate the old momentum will keep having an effect until it vanishes. This is a bit of a corner case, but that's what the standard update equation gives.

That or you're running on different subsets of your test data, which might be the case if it keeps oscillating.

--
You received this message because you are subscribed to the Google Groups "Caffe Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to caffe-users...@googlegroups.com.
To post to this group, send email to caffe...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/caffe-users/7a211391-9864-4145-b829-48f08e9e2be7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Suraj Srinivas]

Thanks for your response!
I am not entirely sure how it works. So if I delete the solverstate file then the momentum information would also be deleted, so if I finetune after that will it ensure that this effect is nullified?