Training using a generator that depends on the current state of the model being fitted

[bre...@limsi.fr]

I am trying to train a model using a generator that yields batches that depend on the current state of the model being fitted.

More precisely, the idea is to filter the output of a generator to only yield difficult cases:

model = Model()

# ... adding layers ...

model.compile(...)

def f(model):

    for x, y_true in batch_generator:

        y_pred = model.predict(x)

        difficult_x, difficult_y_true = keep_only_difficult_cases(x, y_pred, y_true)

        yield (difficult_x, difficult_y_true)

generator = f(model)

model.fit_generator(generator, ...)

My question is the following: are model updates (after each batch) atomic or -- since generator is run in parallel to the actual optimization -- is there a chance that model.predict(x) (called in function f) would rely on a model whose internal state is incoherent (e.g. while weights are being updated) and thus returns a completely random y_pred? 

If the latter situation arises, what would you suggest? 

Using a callback that copies the model after each batch and update the generator accordingly?

Anyway, thanks for this great library: I went from complete noob to trying my wildest deep ideas in less than a week thanks to Keras.

Hervé Bredin.