Using an Over-fitted CNNs as a feature extractor

[Chris Parry]

I am working on a multiclass multilabel image classification problem. I am using a pre-trained CNN from model zoo to perform feature extraction, as here Caffe Docs

Despite trying a several different pre-trained CNNs, performance on the test data has maxed out.The pre-trained CNNs are trained on completely different sets of images than those that I am trying to classify. Therefore, I am wondering if the features that they are extracting are sufficient rich.

I have my own CNN that I trained on the specific training images of this multilabel problem. It has very good performance on the training set, but I never managed to get it to generalize to the validation set (so I decided not to use it directly). What about using this CNN as a feature extractor? Thanks!