Problems Fine-tune Flickr training imagenet model, Please Help!

[Hidden Dreamz]

Hi there, 

I am currently trying to Fine tune flickr train the imagenet deep dream model on 2.5k images in 50 classes but when i train it runs fine for the first few thousand iterations (on a batch of 20) then it goes from >1 loss to 100+ and after this change it just gives me black images as the output where before the 'crash' in loss it will give me normal deep dreamed images. what am i doing wrong?? i followed this guide https://generateme.wordpress.com/2015/08/12/training-own-dnn-for-deep-dream/ and read loads online but cannot see where i am going wrong. I can train as he does with each image being its own class and using the same train and val data without problem but when i reduce the classes to say 50 it breaks as i noted above. I have all my data labeled and i renamed the 3 classifier layers, input my own number of classes on the relevant lines but still not working. I have included my files for you to take a look at, 

Thanks for anyone who will help! Josh

[Przemek D]

What's your learning rate? I've observer similar behavior when setting it too high - try tuning it down a notch, see if it helps.

[Hidden Dreamz]

Hey thanks for the response, so learning rate was 0.0005 but the stepsize was 100k when i turned down the stepsize to 20k i was able to train and the model would work but i still get some layers giving black output and others giving a normal deepdream output. It must be something with overtraining breaking the weights or something. What learning rate and stepsize do you use for fine tune flicker training? could you post me your solver info so i can compare? i a currently trying to flickr train on 2.5 images in 6 categorys, thanks for the help!

[Przemek D]

So it was somewhat better when you decreased LR more often, yes?

How about starting from a lower base LR? Say, 1e-4 or 1e-5?

What do you mean by fine tune flicker training? I do fine-tuning all the time but solver params depend on a lot of things, it's not a "one rule to solve them all" kind of thing :P

[Hidden Dreamz]

its just what the guy called it in the blog i was following to fine tune imagenet deepdream caffe model on my own image set i see what your saying abour lr, i did not change the lr tat was still at 0,005 i changed the stepsize from 100k to 20k which is what they use on the fine tune example and that did give better results about still breaking some layers and not seeing the images from training. I will try with the Lr you suggest and see how it goes. thanks!