How can i control fluctuating validation accuracy??

[Javad Mahdavi]

when i start fitting my model , my model acc going to 99% but validation_acc still fluctuating , how can i fix that.

i try Normalization layers before fit() , but didn't fix that.

[Dennis S]

I think you should ask yourself if this is something that you want to "fix". Of course you want high validation accuracy but I don't think you should be thinking of this as a code fix. You should be considering, on a conceptual level, what it means when high training accuracy isn't carried over into your validation score. Have you examined your training data? Is it balanced? Is it human validated? Does it accurately represent all of the labels in your population? Do you have sufficient hyper parameter testing to get you to the current model? 

I think that if you take the time to think about this in the greater context of model design - not just Keras - you will find many, many ways to improve your current model. 

Thanks,

Dennis

On Sat, Nov 27, 2021 at 3:04 PM Javad Mahdavi <amir.y...@gmail.com> wrote:

when i start fitting my model , my model acc going to 99% but validation_acc still fluctuating , how can i fix that.

i try Normalization layers before fit() , but didn't fix that.

--
You received this message because you are subscribed to the Google Groups "Keras-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to keras-users...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/keras-users/ea045d2a-385a-4dea-8b62-30f70a7027ben%40googlegroups.com.

--

Thanks,

Dennis

[Lance Norskog]

This is a common behavior of models- the training accuracy keeps going up, but the validation accuracy at some point stops increasing. This means that the model is 'overtrained'- it is just memorizing the actual training data. When both accuracies are still increasing, the model is building a generalizable understanding of the common "features" in the training data.

The current "best practice" is to make three subsets of the dataset: training, validation, and "test". When you are happy with the model, try it out on the "test" dataset. The resulting accuracy should be close to the validation dataset. If the two diverge, there is something basic wrong with the model or the data.

Cheers,

Lance Norskog

On Sat, Nov 27, 2021 at 3:04 PM Javad Mahdavi <amir.y...@gmail.com> wrote:

when i start fitting my model , my model acc going to 99% but validation_acc still fluctuating , how can i fix that.

i try Normalization layers before fit() , but didn't fix that.

--
You received this message because you are subscribed to the Google Groups "Keras-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to keras-users...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/keras-users/ea045d2a-385a-4dea-8b62-30f70a7027ben%40googlegroups.com.

--

Lance Norskog
lance....@gmail.com
Redwood City, CA

[Lance Norskog]

I think that you do not need to set 'steps_per_epoch' or 'validation_steps'.

RMSProp is an older optimizer, Adam or others might work better. You can try 'swish' or 'relu' instead of 'elu'.

It is common to use 80% of data for training. Under 500 images is a small training dataset.

Good luck!

On Mon, Nov 29, 2021 at 3:17 AM Javad Mahdavi <amir.y...@gmail.com> wrote:

Yes i have divided the data to Train set , validation set , test set and fitting the model with train data and validation data.

I have a total of 1800 person face data, which is divided into three classes, because I have a picture of three people
And I've divided this into training data, validation data, and test data.
Education: 900 pieces of data (300 pieces per class)
Validation: 450 pieces of data (150 pieces per class)
Test: 450 pieces of data (150 pieces per class)

this is my plot:

model architecture:

# Data Preproccessing:

train_datagen = ImageDataGenerator(

    rescale = 1./255,

    rotation_range = 40,

    width_shift_range = 0.2,

    height_shift_range = 0.2,

    shear_range = 0.2,

    zoom_range = 0.2,

    horizontal_flip = True,

    vertical_flip = True)

test_datagen = ImageDataGenerator(rescale=1./255)

train_generator = train_datagen.flow_from_directory(

    train_dir,

    target_size=(150,150),

    batch_size=32,

    class_mode="categorical",shuffle=True)

validation_generator = test_datagen.flow_from_directory(

    validation_dir,

    target_size=(150,150),

    batch_size=32,

    class_mode="categorical",shuffle=True)

# Netwrok Architecture:

conv_base = VGG16(pooling="max",include_top=False,weights="imagenet",input_shape=(150,150,3))

Network = models.Sequential()

Network.add(conv_base)

Network.add(layers.Dense(8500,activation="elu",kernel_regularizer=regularizers.l1(0.001)))

Network.add(layers.Dropout(0.4))

Network.add(layers.Dense(3,activation="softmax"))

Network.compile(optimizer=optimizers.RMSprop(learning_rate=0.0001),loss="categorical_crossentropy",metrics=['accuracy'])

Network.layers[0].trainable = False

result = Network.fit(train_generator,

    steps_per_epoch=28,

    epochs=38,

    validation_data=validation_generator,

    validation_steps=14)

This is my last epochs log:

Epoch 35/45 28/28 [==============================] - 33s 1s/step - loss: 0.4564 - accuracy: 0.9839 - val_loss: 0.5221 - val_accuracy: 0.9621 Epoch 36/45 28/28 [==============================] - 33s 1s/step - loss: 0.6338 - accuracy: 0.9551 - val_loss: 1.4283 - val_accuracy: 0.7857 Epoch 37/45 28/28 [==============================] - 33s 1s/step - loss: 0.8284 - accuracy: 0.9574 - val_loss: 1.1220 - val_accuracy: 0.8929 Epoch 38/45 28/28 [==============================] - 34s 1s/step - loss: 0.3888 - accuracy: 1.0000 - val_loss: 1.3512 - val_accuracy: 0.8571 Epoch 39/45 28/28 [==============================] - 33s 1s/step - loss: 1.3770 - accuracy: 0.9424 - val_loss: 0.6160 - val_accuracy: 0.9040 Epoch 40/45 28/28 [==============================] - 33s 1s/step - loss: 0.4311 - accuracy: 0.9781 - val_loss: 0.4648 - val_accuracy: 0.9554 Epoch 41/45 28/28 [==============================] - 33s 1s/step - loss: 0.3741 - accuracy: 0.9931 - val_loss: 0.6041 - val_accuracy: 0.9442 Epoch 42/45 28/28 [==============================] - 33s 1s/step - loss: 0.9070 - accuracy: 0.9551 - val_loss: 0.6701 - val_accuracy: 0.8549 Epoch 43/45 28/28 [==============================] - 33s 1s/step - loss: 0.3771 - accuracy: 0.9873 - val_loss: 2.2323 - val_accuracy: 0.6853 Epoch 44/45 28/28 [==============================] - 32s 1s/step - loss: 0.3730 - accuracy: 0.9896 - val_loss: 3.0664 - val_accuracy: 0.6763 Epoch 45/45 28/28 [==============================] - 33s 1s/step - loss: 0.8752 - accuracy: 0.9747 - val_loss: 4.6807 - val_accuracy: 0.6652

i glad you can help me!!!

[Lance Norskog]

This is human face data. It is possible that rotation, shear and flipping the face upside down are not productive for face data.

On Mon, Nov 29, 2021 at 10:28 AM Javad Mahdavi <amir.y...@gmail.com> wrote:

thank you !!

I will definitely try it !!!!

but i have augmented images.

[Madiouni Mohsen]

From what I see

I think that u should change the partition  and try to makr a shuffle when dividing also change the batch size 

To view this discussion on the web, visit https://groups.google.com/d/msgid/keras-users/CAP0ykh_P%2BF0qo0WZsnuxTJfYS79gjLUhZ4%3D8a-c_n5Ba8Q5QoQ%40mail.gmail.com.

[Madiouni Mohsen]

Please change the color of graphes