load raw chain model and continue to chain

[李咏泽]

Hi,

   I runned recipe:aishell2/s5 to train a chain model with my own data(2900 hours). I trained this model with 6 epochs. Then I load this model and continue to train with additional 2 epochs. I draw train_objective in file accuracy.report.  Why there exists a breakpoint when I load raw model and start to train.

[Jiayu DU]

I guess you have non-smooth learning rate setup between pretraining and finetuning. here by "learning rate" I mean the combined effect of batch size & lrate & num_jobs etc.

imagine this is a non-parallel training, the curve seems like you increase learning rate suddenly and starting decaying the learning rate.

Just a guess, not quite sure about this.

[李咏泽]

Thanks for your response. I recheck the code:steps/nnet3/chain/train.py . The learning rate of finetuning changed even though I'v given the value of final effective learning rate in pretraining. As you say, "here by "learning rate" I mean the combined effect of batch size & lrate & num_jobs etc.". thanks for your help!

在 2020年7月31日星期五 UTC+8上午11:17:09，Jiayu DU写道：