Cross Validation is a bad idea?

[Antonio Paes]

Hi guys,

I'm trying do cross-validation in caffe because I've a small set of images 3K, I'm do it this way:

split my train images in 10 subsets, then I generate the lmdb files, after this I train in subsets 1 to 9 and validation in 10, after this I finetuning this trained network in subsets 2 to 10 an validate in subset 1, and subsequently until pass for all subsets.

The problem is which the network suffers of overfiting because when I do finetuning my subset of validation already exposed to training phase.

So what I'm doing wrong? Is this way which do cross-validation in caffe?

I read about this in some places like github and here in caffe user too, but isn't clear.

If anyone help me I really thanks, I've lost a week on this and the problem persist.

Thanks guys.