Starting learning rate and weight_decay and momentum for finetuning.

[Vijay Daultani]

Hi,

I am fine tuning alexnet on imagenet, I took already pretrained weights of alexnet from model zoo. I made some changes in the weights and now want to fine tune again. In the solver.prototxt shipped with caffe have base_lr =0.01 which is quite large for me and distorts my weights by a large factor. 

My question is : If I want to finetune on such settings what should be the values of parameters in solver,prototxt like starting  base_lr? (obviously I should not be 0.01?) and weight_decay. 

Not to mention I have already searched caffe documentation and google but could not find any relevant answers.

Any help on the same will be appreciated

Regards,

Vijay

[Przemek D]

I also found 0.01 to be way too much for finetuning, but 0.001 did great and I didn't modify momentum nor decay rate. Experiment away, but remember that LR is multiplicative, so 0.01 and 0.02 will work almost the same even though one is twice as large as the other - search through orders of magnitude, 0.001, 0.0001 and so on until you find one that works for you.