Can the learning rate be changed during model.fit?

[Jeremy Ellis]

Do model.fit callbacks have a way to dynamically alter the learning rate?

I saw this snippet of python code:

def lr_schedule(epoch):
    lrate = 0.001
    if epoch > 75:
        lrate = 0.0005
    elif epoch > 100:
        lrate = 0.0003       
    return lrate




I know that I can use a decay value with many optimizers, but can I do something like the above code

to manually alter the learning rate during model.fit?




I could probably pause model.fit at epoch 75, then run model.compile with the new learning rate and then restart model.fit,

but that seems quite onerous. Any suggestions?




[Shanqing Cai]

If you are using the sgd optimizer, you can change the its learning rate during a Model.fit() call using a callback like onEpochBegin or onEpochEnd.

The optimizer being used by the model for training is exposed via the model's public `optimizer` property.

The code would look something like:

```js

// ...

model.fit(xs, ys, {

  callbacks: {

    onEpochBegin: async (epoch) => {

      model.optimizer.setLearningRate(...);

   }

});

```

Unfortunately, this is currently only possible for sgd, as none of the other optimizer types expose a public method for setting the learning rate.

Please feel free to file a GitHub issue for that. 

--
You received this message because you are subscribed to the Google Groups "TensorFlow.js Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tfjs+uns...@tensorflow.org.
Visit this group at https://groups.google.com/a/tensorflow.org/group/tfjs/.
To view this discussion on the web visit https://groups.google.com/a/tensorflow.org/d/msgid/tfjs/156eba36-da29-413b-a16e-5f458590590b%40tensorflow.org.

--

---

Shanqing Cai

Software Engineer

Google

ca...@google.com

[Jeremy Ellis]

Thanks Shanqing Cai:

Works great. I made a bit of a silly CodePen to prove the learning rate changes.  At epoch 300 it changes the learning rate from 0.02 to 0.14 which sends the loss spiraling up.

https://codepen.io/rocksetta/pen/GXWyLw?editors=1010

[Jeremy Ellis]

Shanqing Cai a quick slightly connected question:

Can we freeze and unfreeze layers inside model.fit?

const layer0 = model.getLayer(null, 0);

const layer1 = model.getLayer(null, 1);

// Freeze the first layer.

layer0.trainable = false; 

Could I do the above code from within model.fit. Potentially changing what is being trained while it is being trained.

[Jeremy Ellis]

I tried a codepen at https://codepen.io/rocksetta/pen/WgjYEO?editors=1011

but freezing during model.fit does not seem to work. Works fine if set before model.fit

[Jeremy Ellis]

Working on freezing layers during model.fit.

I found this codepen works  https://codepen.io/rocksetta/pen/jvmdZW?editors=1011

Basically you stop the model.fit. Set your trainable layers, re-call model.compile and then re-start model.fit.

[Shanqing Cai]

Hi Jeremy,

Good question! To change the frozen/unfrozen status of weights during training, you can do something like the following:

```js

  // ...

  onEochEnd: async (epoch, logs) => {

     if (epoch === 3) {

       model.layers[1].trainable = false;

       model.compile({optimizer, loss});  // Do not forget to call compile() here!

     }

  }

```

The important thing to remember is to call model.compile() after you modify the trainable property of the layer(s) involved, because

the set of trainable weights is collected during the compile() call.

However, currently there seems to be a bug in tfjs which makes the above work only for non-sequential (a.k.a. "functional") models.

This doesn't work for sequential models yet. I've filed a GitHub issue to track this bug. Thanks for raising this question.

Best,

Shanqing

[Jeremy Ellis]

Sounds good Shanqing. Thank you.

[Ken Kahn]

While using public methods on optimizers safeguards against depending upon internals that may change I noticed that at least adam has a learningRate property one can set. 

model.optimizer.learningRate = (...)

seems to work. Is it safe to depend upon this?

Best,

-ken