why fit() and fit_generator() return different results?

[Percival]

Hi, can I know what is going on? what I did is trying to replace fit() with fit_generator()

my training data is train_VGG16 with shape (1624, 7, 7, 512)

valid data is valid_VGG16 with shape (98, 7, 7, 512),

test data is test_VGG16 with shape (109, 7, 7 512)

the model is very simple one as I just wanna test out replacing it with fit_generator().

VGG16_model = Sequential()

VGG16_model.add(GlobalAveragePooling2D(input_shape=(7,7,512)))

VGG16_model.add(Dense(2, activation='softmax'))

VGG16_model.summary()

for fit():

VGG16_model.fit(train_VGG16, train_targets, 

          validation_data=(valid_VGG16, valid_targets),

          epochs=100, batch_size=2, callbacks=[checkpointer], verbose=1)

I understand python generator concept and made custom generator. Please correct me if I'm wrong.

I'm using the train_VGG16 and slice it with batch_size for example, 2, then each time, it will use the sliced train_VGG16 of shape (2, 7, 7, 512) to train the model.

and to run through the whole 1624 records, I've specified the steps_per_epoch as 812, so that 812 * 2 = 1624.

and here is the fit_generator():

VGG16a_model.fit_generator(generator=train_gen,

                                  validation_data=valid_gen,

                                  steps_per_epoch=812,

                                  validation_steps=49,

                                  max_q_size=1,

                                  verbose=1,

                                  shuffle=False,

                                  callbacks=[checkpointer1], 

                                  epochs=100)

I don't know why the outputs of the two are different? it is normal (which I don't think so)? 

I did the same for using batch_size of 1624 and step_per_epoch = 1, the results is still different to fit()

Please help! Keras experts!

I dont think it is a completely new problem and many experts should have solution on it or there are working code.  please advise.  thanks.

the use of generator is to solve the out of memory issue.

If it were possible to have 1Tb of RAM, I would not need to use generator method.

fit() just load everything to the memory and it is done.

anyway, Please kindly help.  I am running out of my methods. I just feel fit_generator() is totally different to fit() in results. but it should not be. 

the results is fit 96.3303%, while fit_generator(), 88.0734%

[Sergey O.]

I tried to see if I can reproduce your error with a simple model. Using either fit or fit_generator I get identical results!

The only thing that seems to be different (besides potentially different version of keras/tensorflow) is that you did not set shuffle=False for model.fit(), for model.fit_generator() shuffle=True/False does not make a difference (you get identical result).

Try running the follow code. Is the loss the same for both fit functions?

import numpy as np

np.random.seed(1)

import keras

from keras.layers import Input,Dense

from keras.models import Model

import tensorflow as tf

tf.set_random_seed(1)

data = np.random.randint(0,10,size=(100,1))

def gen():

  i = 0

  while True:

    yield data[i],data[i]

    i += 1

def gen_percival(batch_size):

  batch_features = np.zeros((batch_size, 1))

  batch_labels = np.zeros((batch_size, 1))    

  cursor = 0

  while True:        

      for i in range(batch_size):   

          batch_features[i] = data[cursor]

          batch_labels[i] = data[cursor]

          cursor = (cursor+1)%data.shape[0]

      yield (batch_features.copy(), batch_labels.copy())

for i in range(2):

  keras.backend.clear_session()

  I = Input(shape=(1,))

  O = Dense(1,kernel_initializer=keras.initializers.constant())(I)

  model = Model(I,O)

  model.compile("adam","mean_squared_error")

  batch_size = 2

  steps = int(data.shape[0]/2)

  

  if i == 0: model.fit(data,data,epochs=10,batch_size=batch_size,shuffle=False)

  if i == 1: model.fit_generator(gen_percival(batch_size=batch_size),steps_per_epoch=steps,epochs=10)

[Percival]

thanks for your reply.  Please find the log on running your code.

for fit(), the loss is 11.6004, and for fit_generator(), the loss is 11.6131. do you think it is similar or fine?

it is only for data (10,1), if it is (1624, 7, 7, 512), the loss would be much different in both cases.

so is fit_generator() can't perform the task it supposes to have? I doubt it now..

C:\Users\Percival\Anaconda3\lib\site-packages\h5py\__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
  from ._conv import register_converters as _register_converters
Using TensorFlow backend.

(100, 1)
Epoch 1/10
100/100 [==============================] - 0s 2ms/step - loss: 32.4251
Epoch 2/10
100/100 [==============================] - 0s 447us/step - loss: 29.2802
Epoch 3/10
100/100 [==============================] - 0s 320us/step - loss: 26.3563
Epoch 4/10
100/100 [==============================] - 0s 344us/step - loss: 23.6554
Epoch 5/10
100/100 [==============================] - 0s 320us/step - loss: 21.1694
Epoch 6/10
100/100 [==============================] - 0s 320us/step - loss: 18.8884
Epoch 7/10
100/100 [==============================] - 0s 320us/step - loss: 16.8019
Epoch 8/10
100/100 [==============================] - 0s 360us/step - loss: 14.8988
Epoch 9/10
100/100 [==============================] - 0s 280us/step - loss: 13.1684
Epoch 10/10
100/100 [==============================] - 0s 280us/step - loss: 11.6004
Epoch 1/10
50/50 [==============================] - 0s 4ms/step - loss: 32.4443
Epoch 2/10
50/50 [==============================] - 0s 2ms/step - loss: 29.3021
Epoch 3/10
50/50 [==============================] - 0s 2ms/step - loss: 26.3768
Epoch 4/10
50/50 [==============================] - 0s 2ms/step - loss: 23.6747
Epoch 5/10
50/50 [==============================] - 0s 1ms/step - loss: 21.1876
Epoch 6/10
50/50 [==============================] - 0s 1ms/step - loss: 18.9055
Epoch 7/10
50/50 [==============================] - 0s 2ms/step - loss: 16.8178
Epoch 8/10
50/50 [==============================] - 0s 1ms/step - loss: 14.9136
Epoch 9/10
50/50 [==============================] - 0s 2ms/step - loss: 13.1823
Epoch 10/10
50/50 [==============================] - 0s 2ms/step - loss: 11.6131

[xin...@gmail.com]

Hello, 

I am having the same problem as u now.

fit generator is giving slightly worse result than fit, no matter how much i try to debug it.

the input and model and every parameter i am using are the same.

Have you yet find the reason of this?

[wolf...@gmail.com]

same problem for me..

[Lance Norskog]

"shuffle=False,"

This is one problem :) You're supposed to shuffle input data.

--
You received this message because you are subscribed to the Google Groups "Keras-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to keras-users...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/keras-users/6111fcbf-7526-41c7-84a6-9b135135f6aa%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Lance Norskog
lance....@gmail.com
Redwood City, CA

[ntuc...@gmail.com]

shuffle is only useful if input is Sequence?

shuffle: Boolean. Whether to shuffle the order of the batches at the beginning of each epoch. Only used with instances of Sequence (keras.utils.Sequence). Has no effect when steps_per_epoch is not None.

https://keras.io/models/sequential/#fit_generator

To unsubscribe from this group and stop receiving emails from it, send an email to keras...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/keras-users/6111fcbf-7526-41c7-84a6-9b135135f6aa%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.