fit_generator and evaluate_generator

[Ernst]

Hello,

I have a strange phenomenon:

I have a LSTM - model. If I call it fit_generator and evaluate_generator with identical data, I get different results:


The code looks like this:

        losstest = model.evaluate_generator(lstm_generator(X, y, sequence_length, batch_size), samples_per_epoch)
        print("Evaluate : ",losstest)

        losstest = model.fit_generator(lstm_generator(X, y, sequence_length, batch_size), samples_per_epoch=samples_per_epoch,nb_epoch=1 )
        print("FIT: ",losstest.history)

If I fit the model to 98 % accuracy before, I get these results when I run the above code

Evaluate :  [0.52922583955585945, 0.69763257586623006]
FIT:  {'acc': [0.9859469670237917], 'loss': [0.072467219414477396]}

I am using Keras 1.0.2 (freshly cloned).

Am I wrong or shpuld evaulate_generator and fit_generator in this example produce similar results?

Thanks and kind regards
Ernst

[Daπid]

On 13 May 2016 at 01:12, Ernst <erns...@gmx.at> wrote:
> Am I wrong or shpuld evaulate_generator and fit_generator in this example
> produce similar results?

Evaluate is working on the model as it is, without modification. Fit
also trains the model, so you would definitely expect different
results.

If you now run evaluate after fit you should get 0.98 accuracy again.

[Ernst]

Hi David,

thank you for your reply. I tried what you suggested:

        losstest = model.evaluate_generator(lstm_generator(X_pred.loc[date_idx], y_pred.loc[date_idx], sequence_length, batch_size), samples_per_epoch)
        print("Evaluate 1: ",losstest)

        losstest = model.fit_generator(lstm_generator(X_pred.loc[date_idx], y_pred.loc[date_idx], sequence_length, batch_size), samples_per_epoch=samples_per_epoch,nb_epoch=1 )
        print("Fit: ",losstest.history)

        losstest = model.evaluate_generator(lstm_generator(X_pred.loc[date_idx], y_pred.loc[date_idx], sequence_length, batch_size), samples_per_epoch)
        print("Evaluate 2: ",losstest)

The output is

Evaluate 1:  [0.14203481459494099, 0.71821969604740543]
Fit:  {'loss': [0.10416310184402147], 'acc': [0.98418560425440471]}
Evaluate 2:  [0.087580072839798379, 0.71369318014123673]

So, the loss aver the 2nd Evaluate and is more or less in range, but the accuracy is completely different.

Thanks,
Ernst

[sophie.cr...@gmail.com]

I encountered the same problem which I posted as stack overflow question.

Is there a solution for this?

Best,

Sophie