custom optimizers?

[slavakung]

Hello,

I am a researcher in optimization and I am interested in writing my own custom optimization routines and testing them on DNNs. The easiest and most robust way for me to do this would be to find some other custom optimizer code written by a keras user floating around and adapt it to the algorithm I'm considering, but I've tried looking for some examples and wasn't successful. Does anyone know of any?

In addition, how would one adaptively update the minibatch size to calculate the gradient within the routine?

Thank you. I know these are very basic questions and I am not so familiar with keras, but it would be of great assistance if you can help me in this regard.

[Matias Valdenegro]

Hi, you can start by just looking how "built-in" keras optimizers are implemented:

 

https://github.com/fchollet/keras/blob/master/keras/optimizers.py

[slavakung]

Hi,
Yes I have looked at this code, but a custom optimizer would answer to me some basic things like how to include it as an option without having to change the all_classes structure in this particular file (or do I do this anyway and edit the optimizers.py file directly? but that seems counter to the modularity principle of keras....)

[Matias Valdenegro]

That is pretty easy, you just instantiate your Optimizer and pass it as parameter to model.fit:

 

opt = MyCustomOptimizer()

 

model.fit(loss = "foo", optimizer = opt)

 

And that's it.

[slavakung]

OK, I must be making a basic mistake here somewhere but I cannot see what it is. I tried:

----------------------------------------------------------

from __future__ import print_function
from __future__ import absolute_import
import numpy as np
from six.moves import zip
from keras import backend as K
np.random.seed(1337)  # for reproducibility

from keras.datasets import mnist
from keras.models import Sequential
from keras.layers.core import Dense, Dropout, Activation
from keras.optimizers import Optimizer
from keras.utils import np_utils


class testsgd(Optimizer):
..... [everything same as sgd] .....


myopt = testsgd()

....[define model]....

history = model.fit(X_train, Y_train,
                    batch_size=batch_size, nb_epoch=nb_epoch,
                    verbose=1, validation_data=(X_test, Y_test), optimizer = myopt)

-------------------------------------

and when I try it I get,
Traceback (most recent call last):
  File "test2.py", line 136, in <module>
    verbose=1, validation_data=(X_test, Y_test), optimizer = myopt)
  File "/usr/local/lib/python2.7/dist-packages/keras/models.py", line 661, in fit
    str(kwargs))
TypeError: Received unknown keyword arguments: {'optimizer': <__main__.testsgd object at 0x7f0e6196c750>}

[Matias Valdenegro]

Oh my mistake!

 

You pass the optimizer to model.compile, not model.fit

[slavakung]

OK, my test appears to work now. Thanks for your help

[swati.a...@gmail.com]

Hi Slavakung,

I am trying to do the same thing but I am encountering errors such as :

/usr/local/lib/python3.6/site-packages/keras/engine/training.py in _make_train_function(self)
    935                 self._collected_trainable_weights,
    936                 self.constraints,
--> 937                 self.total_loss)
    938             updates = self.updates + training_updates
    939             # Gets loss and metrics. Updates weights at each call.

TypeError: get_updates() takes 3 positional arguments but 4 were given

I Followed the same process as described below. Let me know if you have any insights.

Thanks

Swat

[wuyuan...@gmail.com]

Hi, would you mind giving some tony sample of your customized optimizer in keras? I want to call the loss function in the SGD function, but I don't know how to do it, if you could give me some help that might be appreciated. Thanks.

[hiteshtek...@gmail.com]

Hi. Have you implemented any new optimizer. If yes Please share for just a example