How to use Tensorflow Contrib optimizers in Keras using TFOptimizer ?

[Rohit Gupta]

I am trying to use the AdamW Optimizer from Tensorflow Contrib, however I can't find any official or unofficial examples of how to wrap an existing Tensorflow optimizer for use in Keras. Some help in this regard will be highly appreciated.

AdamW Optimizer in Tensorflow: https://www.tensorflow.org/api_docs/python/tf/contrib/opt/AdamWOptimizer

Code I have tried:

import numpy as np

from keras import optimizers, Input
from keras.models import Sequential, Model
from keras.layers.core import Dense, Activation, Lambda

from tensorflow.contrib.opt import MomentumWOptimizer, AdamWOptimizer


num_classes = 2

#tfoptimizer = AdamWOptimizer(weight_decay=0.1, learning_rate=0.001, momentum=0.9, use_nesterov=True)
options = {'weight_decay': 0.1, 'learning_rate': 0.001, 'momentum': 0.9, 'use_nesterov': True}


tfoptimizer = AdamWOptimizer(options)
optimizer = optimizers.TFOptimizer(tfoptimizer)

model = Sequential()
model.add(Dense(num_classes, activation="tanh", input_shape=(3,)))
model.compile(loss='mean_squared_error', optimizer=optimizer)
model.fit(np.random.random((5, 3)), np.random.random((5, num_classes)), epochs=5, batch_size=5)

This code results in the following error:

Using TensorFlow backend.
Traceback (most recent call last):
  File "AdamW_SGDW_Test.py", line 22, in <module>
    model.fit(np.random.random((5, 3)), np.random.random((5, num_classes)), epochs=5, batch_size=5)
  File "/home/deepromay/anaconda2/envs/tensorflow110/lib/python2.7/site-packages/keras/engine/training.py", line 1008, in fit
    self._make_train_function()
  File "/home/deepromay/anaconda2/envs/tensorflow110/lib/python2.7/site-packages/keras/engine/training.py", line 498, in _make_train_function
    loss=self.total_loss)
  File "/home/deepromay/anaconda2/envs/tensorflow110/lib/python2.7/site-packages/keras/legacy/interfaces.py", line 91, in wrapper
    return func(*args, **kwargs)
  File "/home/deepromay/anaconda2/envs/tensorflow110/lib/python2.7/site-packages/keras/optimizers.py", line 701, in get_updates
    grads, global_step=self.iterations)
  File "/home/deepromay/anaconda2/envs/tensorflow110/lib/python2.7/site-packages/tensorflow/contrib/opt/python/training/weight_decay_optimizers.py", line 144, in apply_gradients
    grads_and_vars, global_step=global_step, name=name)
  File "/home/deepromay/anaconda2/envs/tensorflow110/lib/python2.7/site-packages/tensorflow/python/training/optimizer.py", line 591, in apply_gradients
    self._prepare()
  File "/home/deepromay/anaconda2/envs/tensorflow110/lib/python2.7/site-packages/tensorflow/contrib/opt/python/training/weight_decay_optimizers.py", line 151, in _prepare
    weight_decay, name="weight_decay")
  File "/home/deepromay/anaconda2/envs/tensorflow110/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 998, in convert_to_tensor
    as_ref=False)
  File "/home/deepromay/anaconda2/envs/tensorflow110/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1094, in internal_convert_to_tensor
    ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
  File "/home/deepromay/anaconda2/envs/tensorflow110/lib/python2.7/site-packages/tensorflow/python/framework/constant_op.py", line 217, in _constant_tensor_conversion_function
    return constant(v, dtype=dtype, name=name)
  File "/home/deepromay/anaconda2/envs/tensorflow110/lib/python2.7/site-packages/tensorflow/python/framework/constant_op.py", line 196, in constant
    value, dtype=dtype, shape=shape, verify_shape=verify_shape))
  File "/home/deepromay/anaconda2/envs/tensorflow110/lib/python2.7/site-packages/tensorflow/python/framework/tensor_util.py", line 525, in make_tensor_proto
    "supported type." % (type(values), values))
TypeError: Failed to convert object of type <type 'dict'> to Tensor. Contents: {'use_nesterov': True, 'learning_rate': 0.001, 'weight_decay': 0.1, 'momentum': 0.9}. Consider casting elements to a supported type.

[Daπid]

I don't have much experience with this, but probably the easiest is to use Keras to build the model, and raw Tensorflow to train.

--
You received this message because you are subscribed to the Google Groups "Keras-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to keras-users...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/keras-users/98cf4cf7-12bb-422a-a514-db8c4acc4897%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Nad]

Check how to use optimizers and Adam optimizer in keras from this link https://keras.io/optimizers/