model.predict() always different for same image

[mhubri...@gmail.com]

Hello,

At the moment, I deal with a strange problem. I've trained a model for several epochs and saved its weights. Now, I want to load/restore this model and make some predictions, i.e. feed the network with some images. But every time I load my model and use its predict() function, the result is different. Here is an example:

model1 = load_model('/path/to/weights')
model1.compile(loss='categorical_crossentropy', optimizer='sgd', metrics=['accuracy'])

model2 = load_model('/path/to/weights')
model2.compile(loss='categorical_crossentropy', optimizer='sgd', metrics=['accuracy'])

img = Image.open('/path/to/image')
img = img.convert('RGB')
x = np.asarray(img, dtype='float32')
x = x.transpose(2, 0, 1)
x = np.expand_dims(x, axis=0)

out1 = model1.predict(x)
print(np.argmax(out1))
> 0

out2 = model2.predict(x)
print(np.argmax(out2))
> 9


Both models should be exactly the same since I've restored them using the same file (by the way, I'm using this VGG16 model). Indeed, I've checked that both models have the same weights.

So, why are the predictions so different for the same image? In my mind, they should be exactly the same.

[Bruno Guberfain do Amaral]

What the function "load_model" does? It has any state?

Another thing to try is run the program on a CPU instead of GPU.

--
You received this message because you are subscribed to the Google Groups "Keras-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to keras-users...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/keras-users/18adf4d1-3502-4f2b-91b4-4d0fbfcab8a7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[mhubri...@gmail.com]

Sorry, I forgot the mention, I tested this on CPU and GPU as well.

load_model() basically builds the VGG16 model and then loads the weights. Here is the code:

def load_model(weights_path, num_classes=10):
  model = Sequential()
  model.add(ZeroPadding2D((1,1), input_shape=(3,224,224)))
  model.add(Convolution2D(64, 3, 3, activation='relu'))
  model.add(ZeroPadding2D((1,1)))
  model.add(Convolution2D(64, 3, 3, activation='relu'))
  model.add(MaxPooling2D((2,2), strides=(2,2)))
 
  model.add(ZeroPadding2D((1,1)))
  model.add(Convolution2D(128, 3, 3, activation='relu'))
  model.add(ZeroPadding2D((1,1)))
  model.add(Convolution2D(128, 3, 3, activation='relu'))
  model.add(MaxPooling2D((2,2), strides=(2,2)))
 
  model.add(ZeroPadding2D((1,1)))
  model.add(Convolution2D(256, 3, 3, activation='relu'))
  model.add(ZeroPadding2D((1,1)))
  model.add(Convolution2D(256, 3, 3, activation='relu'))
  model.add(ZeroPadding2D((1,1)))
  model.add(Convolution2D(256, 3, 3, activation='relu'))
  model.add(MaxPooling2D((2,2), strides=(2,2)))
 
  model.add(ZeroPadding2D((1,1)))
  model.add(Convolution2D(512, 3, 3, activation='relu'))
  model.add(ZeroPadding2D((1,1)))
  model.add(Convolution2D(512, 3, 3, activation='relu'))
  model.add(ZeroPadding2D((1,1)))
  model.add(Convolution2D(512, 3, 3, activation='relu'))
  model.add(MaxPooling2D((2,2), strides=(2,2)))
 
  model.add(ZeroPadding2D((1,1)))
  model.add(Convolution2D(512, 3, 3, activation='relu'))
  model.add(ZeroPadding2D((1,1)))
  model.add(Convolution2D(512, 3, 3, activation='relu'))
  model.add(ZeroPadding2D((1,1)))
  model.add(Convolution2D(512, 3, 3, activation='relu'))
  model.add(MaxPooling2D((2,2), strides=(2,2)))
 
  model.add(Flatten())
  model.add(Dense(4096, activation='relu'))
  model.add(Dropout(0.5))
  model.add(Dense(4096, activation='relu'))
  model.add(Dropout(0.5))
 
  # VGG16 above
  # Now remove last FC layer and replace by your own output layer
  model.layers.pop()
  model.layers.pop()
  model.add(Dense(num_classes, activation='softmax'))
  model.load_weights(weights_path)
 
  for layer in model.layers[:10]:
    layer.trainable = False
 
  return model

[mhubri...@gmail.com]

I just set the dropout to 0.0 to test if that's causing the problem. But still the same odd results.

[riaanzo...@gmail.com]

Lets go over some trivial stuff:

Are your target values one hot vectors?

Is your learning rate about 0.001? (I had it at 0.1 as it comes as a default. VGG doesn't train that way!)

Op vrijdag 8 juli 2016 19:53:56 UTC+2 schreef mhubri...@gmail.com:

[mhubri...@gmail.com]

For training, I use a generator and its flow_from_directory() method, so something like:

generator = ImageDataGenerator()  # Where I specify some augmentation parameters
inputs = generator.flow_from_directory(
      '/path/to/source_dir/',
      target_size=(224, 224),
      color_mode='rgb',
      batch_size=16,
      class_mode='categorical',
      shuffle=True)

So, I guess because of the categorical class mode, the targets are one hot.

And yes, I start with a learning rate of 0.001.

But I think training isn't the problem here. After some epochs, my training loss is about 0.08 and the accuracy of the validation set is about 0.9. These results look great!

The model struggles with predicting. Like I showed, every time I reload the model using the same weights, the prediction is different. Even for images from the validation set. They basically all get predicted into two out of 10 classes. So, why is the accuracy of the validation set during training at 0.9, but the model is unable to predict the same images correctly?

[riaanzo...@gmail.com]

Is your preprocessing the same for all of these images? If you preprocess the train, test and val images differently, it will suck at predicting :)

Op zaterdag 9 juli 2016 12:01:07 UTC+2 schreef mhubri...@gmail.com:

[mhubri...@gmail.com]

I think you didn't understand the problem.

At the moment, it doesn't matter if the predictions are good or bad. That's not the problem here. The point is, that I get two different prediction values for the same image. So let's say my image is called img.jpg. Now, when I restore my model and predict img.jpg, the label is say 0. Once again, I restore the same model and make another prediction of img.jpg. But now the label is 9. But that shouldn't be possible. A model should always give the same prediction for the same image, because there is no randomness involved in predicting/evaluating.

[Bruno Guberfain do Amaral]

Could you show us the output of the network, before applying the argmax function?

I just like to know if class 0 and 9 are very far apart.

Another thing, could you call model1.predict(x) twice? Both return the same thing?

--

You received this message because you are subscribed to the Google Groups "Keras-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to keras-users...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/keras-users/939f1987-36db-4387-b844-347e9d5d317c%40googlegroups.com.

[mhubri...@gmail.com]

I already tested that, both classes are very far apart:

out1
> array([[  1.87165151e-05,   8.84232186e-17,   2.62140153e-34,
          2.05315212e-22,   6.29821518e-25,   2.11711216e-24,
          1.95507500e-32,   1.44736917e-30,   4.60744398e-13,
          9.99981284e-01]])
out2
> array([[  9.94905829e-01,   1.57839488e-11,   9.36652955e-35,
          7.26868101e-22,   1.22291198e-20,   3.31008678e-19,
          1.29003039e-23,   2.60989331e-30,   6.66653650e-05,
          5.02748974e-03]])

If I call model1.predict(x) a second time, I get the same probabilities as in the first call. The only time I get different results is when I reload the model. But if you take a look at my code for loading the model (for example here), there is non kind of randomness involved.

[mhubri...@gmail.com]

By the way, this also happens if I call model.evaluate() - the results are different for two models restored from the same weights.

On Saturday, July 9, 2016 at 3:00:51 PM UTC+2, Bruno Guberfain do Amaral wrote:

[Bruno Guberfain do Amaral]

Actually, I have no clue of what is causing this issue, but reviewing your code I'll suggest:

- Your X is in the correct range? Like, between 0 and 1 (from your code, you didn't divide the pixel values by 255).

- When creating the Sequence model, you replaced a Dense(4096) and a Dropout layer with a Dense(num_classes). Afterwards, you load the weighs. These weights are from the original VGG16 or they are your from your pre-trained network?

To view this discussion on the web, visit https://groups.google.com/d/msgid/keras-users/d85cb86f-b464-4511-978e-a99d5808ce46%40googlegroups.com.

[mhubri...@gmail.com]

I think the problem is solved!

Apparently, the order of loading weights and compiling is important. In my case, I loaded the weights and then compiled the model. To avoid the above problem, one should first compile the model and then load the weights.

Would be nice if that is mentioned in the documentation, too.

Anyhow, thanks a lot for your help and advices!

[nancyag...@gmail.com]

I am also facing the same problem.. Can you please tell me, how can I resolve it.

[Lance Norskog]

I think you do not need model.compile(). In fact, this may be wiping out the weights.

Lance

--

You received this message because you are subscribed to the Google Groups "Keras-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to keras-users...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/keras-users/8909a13a-380a-484f-b8bf-d39629d8b9b9%40googlegroups.com.

--

Lance Norskog
lance....@gmail.com
Redwood City, CA

[shubhamba...@gmail.com]

i am facing this problem too. Since i had done everything which is recommanded by this post, like load weights after compile , check for same architecture,still i don't understand what to do.... 

My classification model predicts person or background(0 to 1) but same outputs from the model increase my problem. also i predicts same probability for all classes as predicts person with 0.5 and bg with 0.5 . I have check model summary while creating model and after loading weights. both times the weights are different which shows the model weights  are loaded successfully. Plz help me to overcome from this problem.....

[shubhamba...@gmail.com]

I am still facing same prediction problem after loading pretrained weights..I tried precompile and after-compilation of model. but still i am getting same problem. Like, i have  classes named person,bus,background. now when i predict the classifier for every time i get [[0.33333334 0.33333334 0.33333334]] .which i think means it directly predicts initialized weights model. I also tried prediction without compile and direct loading pretraied weight. But still facing same problem. To plz provide your valuable information into this... 

On Sunday, 10 July 2016 15:36:00 UTC+5:30, mhubri...@gmail.com wrote: