separate learning rate (schedule) for parameter

[David Reichert]

Hi all,

Quick question: For SGD, I've got a parameter that needs to have a separate learning rate and schedule for it (e.g. increasing rather than decreasing like the learning rate for the rest of the parameters, or at least staying constant). Is there already way to do that?

(Essentially I'm trying to implement fast PCD for RBMs, and the fast weights use the same gradient as the weights but a different learning rate).

Cheers,
David

[Ian Goodfellow]

Separate learning rate per parameter:
Easy!
The get_lr_scalers() method of the Model class lets you scale the
learning rate differently on different parameters.

Separate learning rate *schedule* per parameter:

Some assembly required.

So far I've always had the same schedule for all learning rates, i.e.
the global learning rate decays according to some schedule and all
other learning rates are just some constant times that value.

If you need to have a different schedule for each parameter, probably
the best way to do that would be to return a shared variable rather
than a float in the get_lr_scalers() method. I'm pretty sure none of
the code cares whether the lr_scalers are symbolic or numeric so
putting a shared variable there should just work. It might not though
since I don't think anyone has been using it that way yet.

Once you have your learning rate scalers specified as shared variables
you'll then need to implement some sort of callback that changes the
values of these shared variables over time. If you want this callback
to run once each epoch, make it a TrainExtension subclass and pass it
to Train.__init__ in the list of extensions. If you want it to run
once each minibatch, make it a callable object and pass it to
SGD.__init__ in the list of update_callbacks.

> --
>
> ---
> You received this message because you are subscribed to the Google Groups
> "pylearn-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to pylearn-dev...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[David Reichert]

I see what you mean, I'll look into that. Thanks much!