Training a Regression Model in Caffe

[Kwot Sin Lee]

Hi all,

I'm trying to train a regression model by changing the final FC layer of a BVLC CaffeNet to have an num_output of 1, and I've changed the loss layer from a SoftmaxWithLoss to a EuclideanLoss.

I have tried to train by:

1. Finetuning a pretrained CaffeNet for all layers

2. Finetuning a pretrained CaffeNet for only the last FC layer, while freezing the previous layers

3. Training a CaffeNet architecture from scratch.

However, I mostly gotten either an nan loss or a loss that hardly decreases and stagnates around ~0.2. No matter what, my validation/test accuracy is always 0. When I get a nan loss, I tried to reduce my learning rate, which will work but it gives me a stagnant loss and zero accuracy again.

My questions are:

1. Firstly, am I performing regression the right way in Caffe?

2. Does the stagnant loss mean that I have to vary my learning rate again? 

3. How do I find the optimal learning rate? Is the only way to do so is by experimentation and changing the train_val.prototxt?

4. If I were to decrease my batch_size from 256 to 192, do I have to compensate for my learning_rate by increasing by a factor of sqrt(3/4)? Since 192 is 3/4 of 256. 

5. How can I perform support vector regression in Caffe?

Thank you for your help.