Fine-tuning issue (loss explodes)

[Luke Yeager]

I've got a strange issue. I'm using the bvlc_alexnet pretrained model, and attempting to fine-tune it for a new dataset with ten categories.

At first, fine-tuning seems to go well. For the first 5-10 iterations, the accuracy improves and the loss decreases. But then things go bad and the loss explodes:

The model generated at ~0.01 epochs works great and all of the weights look reasonable.

But then by ~0.03 epochs the loss has exploded and the weights are all 0 or NaN.

Has anyone seen something like this before? It seems really weird that the fine-tuning would work for a little while before failing.

[li kai]

how did you get this figures？

在 2015年5月21日星期四 UTC+8上午3:46:32，Luke Yeager写道：

[Luke Yeager]

I'm using DIGITS:
https://github.com/NVIDIA/DIGITS

[Michele Pratusevich]

Your learning rate is too high - set it lower and the loss shouldn't explode.

--
You received this message because you are subscribed to the Google Groups "Caffe Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to caffe-users...@googlegroups.com.
To post to this group, send email to caffe...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/caffe-users/b26939fc-613f-44dc-82b2-92b9d8620c33%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.