Is this Tensorflow.js model equivalent to this Keras model structure?

[micah price]

Hi all.

I am converting my Keras model to Tensorflow.js, but getting some strange differences. For one, the loss is not decreasing at the same rate, and secondly, the prediction results are all janky. I've deduced that the issue must be with the model itself, but I can't figure out how it differs from my Keras model structurally.

Here is my Keras model:

model = Sequential()
model.add(LSTM(128, return_sequences=True, input_shape=(maxlen, len(words))))
model.add(Dropout(0.2))
model.add(LSTM(128, return_sequences=False))
model.add(Dropout(0.2))
model.add(Dense(len(words)))
model.add(Activation('softmax'))
model.compile(loss='categorical_crossentropy', optimizer=RMSprop(lr=0.002))
model.fit(X, y, batch_size=32, nb_epoch=100)

and the corresponding Tensorflow.js model:

var model = tf.sequential();
model.add(tf.layers.lstm({ units: 128, returnSequences: true, inputShape: [maxlen, words.length] }));
model.add(tf.layers.dropout(0.2))
model.add(tf.layers.lstm({ units: 128, returnSequences: false }));
model.add(tf.layers.dropout(0.2))
model.add(tf.layers.dense({units: words.length, activation: 'softmax'}));
model.compile({loss: 'categoricalCrossentropy', optimizer: tf.train.rmsprop(0.002)});
await model.fit(x_tensor, y_tensor, { epochs: 100, batchSize: 32 })

Looking at the docs, I see that there may be a difference in the version of RMSprop used in Tensorflow.js, compared to Keras. I've tried tweaking the decay to match the default one in the Keras docs, but it had no effect. I don't know much about this optimisation technique though, so I wasn't sure what else I may need to fiddle with. As for the actual structure, I've been comparing the docs between Keras and Tensorflow.js for each layer, and checking the default parameters between each. As far as I can tell, they are all equal.

Is there anything I could have missed that differentiates these 2 models from one another?

[Shanqing Cai]

At first glance, the two Sequential models are equivalent. As you wrote, there might be some difference in the optimizer.

However, a more likely explanation for the mismatch you are seeing is that that both

Python Keras and TensorFlow.js initialize the weights of layers such LSTM and Dense randomly.

So by default, you'll get different results every time you run the program, even within the same language and library. 

But as an experiment, you can force Keras and TensorFlow.js to use deterministic initializes. This is not a good idea

for solving real ML problems, though. For example, in TensorFlow.js, you can do

  * For LSTM, do kernelInitializer: 'ones',  recrurrentInitializer: 'ones', biasInitializer: 'zeros'

  * For Dense, do kernelInitializer: 'ones', biasInitialize: 'zeros'

The syntax for Keras is similar, just replace the colons with equal signs and replace the camel case with snake case.

Shanqing