Accuracy decreasing when finetuning a CNN and extracting features from it

[Moni Mo]

Hi,

I am using CNNs to extract features from the fully-connected layers (namely fc6, fc7, fc8). I am using caffenet (bvlc_reference_caffenet.caffemodel) that was also used in the feature extraction example on the caffe website. The model is not pretrained with the images I use. I am extracting the features and use an SVM classifier. This gives me around 80 % accuracy.

When fine-tuning (fc8 layer from scratch) the caffenet on my own data (173000 images in the training set) I get a network accuracy of 72 %. Then I use my fine-tuned model for feature extraction and again the SVM for classification. The accuracy drops for all fc-layers, it is around 78 % now.

I was expecting the accuracy to increase as the network has now seen similar kind of data as the one I'm extracting the features for. Has anyone faced a similar issue or can explain this behaviour?

Thanks for your answers!
Moni