How to write a solver.prototxt satisfy a given condition in CAFFE?

[john1...@gmail.com]

I am writing the solver.prototxt that follows the rule of a paper as

In the training phase, the learning rate was set as 0.001 initially and decreased by a factor of 10 when the loss stopped decreasing till 10−7. The discount weight was set as 1 initially and decreased by a factor of 10 every ten thousand iterations until a marginal value 10−3.

Note that, the discount weight is loss_weight in Caffe. Based on the information above, I wrote my solver as

    train_net: "train.prototxt"
    lr_policy: "step"
    gamma: 0.1
    stepsize: 10000
    base_lr: 0.001 #0.002

In train.prototxt, I also set

   

 layer {
      name: "loss"
      type: "SoftmaxWithLoss"
      bottom: "deconv"
      bottom: "label"
      top: "loss"
      loss_weight: 1
    }

However, I still don't know how to set solver to satisfy the rule "decreased by a factor of 10 when the loss stopped decreasing till 10−7" and "decreased by a factor of 10 every ten thousand iterations until a marginal value 10−3". I did not found any caffe's rule can do it:

    // The learning rate decay policy. The currently implemented learning rate

    // policies are as follows:

    //    - fixed: always return base_lr.

    //    - step: return base_lr * gamma ^ (floor(iter / step))

    //    - exp: return base_lr * gamma ^ iter

    //    - inv: return base_lr * (1 + gamma * iter) ^ (- power)

    //    - multistep: similar to step but it allows non uniform steps defined by

    //      stepvalue

    //    - poly: the effective learning rate follows a polynomial decay, to be

    //      zero by the max_iter. return base_lr (1 - iter/max_iter) ^ (power)

    //    - sigmoid: the effective learning rate follows a sigmod decay

    //      return base_lr ( 1/(1 + exp(-gamma * (iter - stepsize))))

    //

    // where base_lr, max_iter, gamma, step, stepvalue and power are defined

    // in the solver parameter protocol buffer, and iter is the current iteration.

If anyone knows it, please give me some guide to writing the solver.prototxt to satisfy above condition. 

[Przemek D]

Caffe has no tools to automatically control learning parameters. You can only proprogram them using various policies that only take time (iterations) into account. To modify the parameters basing on the current state of the network (for example loss), you will have to babysit the whole process, manually stopping and restarting it from a solverstate with new parameters.

[Jonathan R. Williford]

You can use the Python Caffe interface to automate it. If you are not concerned about precisely replicating the steps, you might want to choose a more modern learning policy, such as Adam. In case you are interested in doing this, you can see an example:

https://github.com/BVLC/caffe/blob/master/examples/mnist/lenet_solver_adam.prototxt

Jonathan

--
You received this message because you are subscribed to the Google Groups "Caffe Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to caffe-users+unsubscribe@googlegroups.com.
To post to this group, send email to caffe...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/caffe-users/8495d140-f257-4470-86a6-46c174b685da%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.