Trying to load a pre trained VGG-166 and adding a small net on top of it

[Marcelo Mota de Azevedo Junior]

hello everyone :) 

I am trying to adapt the code of fchollet (https://gist.github.com/fchollet/7eb39b44eb9e16e59632d25fb3119975) to load a small fully connected layer on top of a VGG-16 to train it in a 8 label class images. There is 2 problems I am facing..

1) in his code (in url above), he uses model.add(top_model) where it should add the new fully connected  layers to the vgg-16. But I get an error telling that the VGG-16 model has no add function.. how is this possible?

2) second problem. he recommends to load a pretrained fully connected layer on top of vgg-16 instead of random initialized fully connected model. Does it mean that I have to build a model like this: dense(7x7x512) -> dense(256) -> softmax and train this plain network (with no convolutions) on the images to get weights and then load this layers (and its weights) to the vgg-16?

thanks guys!

[Dodo]

Hi Marcelo again,

It's good that you are having good results. To answer your question:

1. It's weird that it did not work for you. You may want to try this instead:

from keras.applications import vgg16
from keras.Models import Sequential,Model
#..... import what you need here

base_model = base_model = vgg16.VGG16(include_top=False,weights='imagenet')
top_model = load_model(top_model_path)

full_model = Sequential()
for layer in base_model.layers:
    full_model.add(layer)
for layer in top_model.layers:
    full_model.add(layer)

print full_model.summary()

2. You are quite close. What he meant was to use the VGG16 with no top layer to do the feature extraction and use a simple top layer take that features and output target classes (8 in your case). You can do something similar to this (this is not training time optimal for top model as he already suggested in the blog - but easier to understand I suppose)

#replace connect_layer with the index of the connecting layer between your base_model and top_model
connect_layer = 19 #or so as I vaguely remember. You need to check the full_model.summary() to be sure
for layer in full_model.layers[:connect_layer]:
    layer.trainable= False #freeze the base model to train the top_model first
....
full_model.fit(....)

# for fine-tuning the last conv blocks
ft_layer= 15#again recheck this
for layer in full_model.layers[:ft_layer]:
    layer.trainable= False #freeze the base model to train the top_model first
....
full_model.fit(....)

Cheers,

[Marcelo Mota de Azevedo Junior]

Thanks Dodo for helping again :)

But, when I add layers of the vgg16 in another model (in this for loop), does it keep the vgg pretrained weights??

[Dodo]

 Hi,
 I did this before in one of my projects and It did keep the weights.
You can re-confirm it in your case by using layer.get_weights() and set_weights().

bests,

[Marcelo Mota de Azevedo Junior]

Hello Dodo!

Nice, will try it this morning.

Did you noticed that in fchollet's code, he first creates just the top of the fully connected layer as Sequential() starting with a Flatten layer and ending with a sigmoid, and then load weights.h5 just for this part of layers.

I was wondering how did he do it? I mean, what did he train to create this weights file for a network with few layers... Do you have any idea?

Have a nice day!

[Marcelo Mota de Azevedo Junior]

Just got a funny error... after adding layer of VGG in sequential mode (for loop), then I try to flatten it and thats what I get:

ValueError: The shape of the input to "Flatten" is not fully defined (got (None, None, 512).

What does it mean?

[Marcelo Mota de Azevedo Junior]

my bad... when I chosse include_top = False, I have to tell the input shape...

[Dodo]

Hi Marcelo,

About the question you asked before:

>>> Did you noticed that in fchollet's code, he first creates just the top of the fully connected layer as Sequential() starting with a Flatten layer and ending with a sigmoid, and then load weights.h5 just for this part of layers.

>>>I was wondering how did he do it? I mean, what did he train to create this weights file for a network with few layers... Do you have any idea?

I believe it is related to a blog posted by fchollet. In that blog he described several way to have a network that works on a new dataset (different from the dataset previous model was trained on).

1. New simple network on new dataset.

2. Use existing model (VGG16) on a new dataset: via fine-tuning of the top layers only. # This is where he gets the weights #

3. Finetuning last Conv block and the top layers for better result # This is where he loads the weights #

While you have to go through the whole blog post to fully understand the process. I will summary it as follows (your case falls to 2&3 step):

1. You use VGG16(include top = false) to transform all of your images to arrays of features, save it into a file. (as explained in the blog and I have mentioned it before, it will save more time)

2. Create a small top model (with the structure you have mentioned). The input is the features before, the output is the classes (8 classes in your case). After finishing training, save the model.

3. Load the VGG16(top = false), load your top model (with its weights). Stitch them together. Freeze the model up to the last Conv block (layer1 to layer 15 or so, re-check this). Re-run the training again (this time, inputs are original images). The stitched model is your final model and the model that you need. Save it.

Good luck,

[Marcelo Mota de Azevedo Junior]

Dodo, thank you so much for your patience and very good explanation.

Everything worked perfectly... after doing all of this, the training was stuck in accuracy 0.1250 (which is 1/8 classes). I was pretty sure that I loaded and labeled the images properly.. but when I left the job for the ImageDataGenerator, the training worked well and reached 0.99 accuracy. Don't know the reason... :(

Do you use to work with convnets and image recognition? Your explanations were very very good and helpful!

Is there a way to know what parts of images activate some parts of the last filters? Like those pictures in training videos where they show an "eye" or a "nose" being recognized in middle filters and "faces" in the last filters?

And what is a heatmaps of filters? Are they used for the question above?

Thanks again!

[Dodo]

Good to know it works! Cheers!

While my main expertise is natural language processing, I have been involved in some side projects working with images. My knowledge on the field is not really 'official'. However, keras is really an amazing and helpful tool which saved me a lot of time learning the rope with images and machine learning. That's why I want to return the favor and contribute to this community some way or another.

About what you asked about the filters, from my (limited) knowledge, the two keywords you might be interested are 'saliency map' (heat map which shows region of interest by the network) - and 'filter activation map' (showing which kind of feature activates which kind of filter).

Hope these will hint you in the right direction.

Bests,