How to process data images by pairs instead of just one image

[Youyou]

Hi guys,
 I'm new in caffe and i made a tour of the different demos and tutorials. The Mnist one worked perfectly for me, same as Cifar 10, and imagenet.

Now that i'm a bit used of the different data and layer structures, i found a problem when i wanted to process my own data. The thing is that my data is a bit different, thus i'm not trying to recognize objects in one image but i want to give the network a set of pair images wich are true labeled if there is per exemple the same car in the image and false if it is not.

I have a dataset containing a lot of images but i have to feed the network a combination of two images. The goal of the network is to return 1 if there is the same object in the two images and 0 on the other side.

I looked at all the forums, and the caffe documentation but i still doesn't have a clue of how to process that problem.
Thanks in advance :)

[Samarth Brahmbhatt]

You could modify the ImageData layer to do this. Instead of a scalar label, it should output an image in the top[1]. One thing to look out for is that random cropping should be done simultaneously in both the images.

[Youyou]

Hi Samarth. Thanks for your answer, although i didn't get it all. Did you mean that my data layer should output my set of second images in the top[1]? In this case my labels should be in top[2].

I should then do my convolutions, pooling separately in the two images, but how to combine those results to see if the two images contains the same object then? As i said before, i'm a bit of a newbie so guys don't  get upset with my questions?

:) Thanks again.

[Aske Rasch Lejbølle]

Hi,
From information gathered from different sites, I made my dataset using this python script (modified a bit from my actual).
It loads all image paths to a list and then randomly selects two which are concatenated and added to the database along with a label (0 or 1). To use this, though, your images have to include the label in their filename.

Furthermore, you should also create a workaround such that you don't get tons of negative samples and only a few positive samples (I used 4:1 negative samples). When you made the dataset you can follow the 'siamese network' tutorial and use the "slice" layer to split up the images (slice point: 3).
Otherwise, this thread also covers this subject:
https://groups.google.com/forum/#!topic/caffe-users/SOgOkPYps1g

Hope that it helps

Best,
Aske

[Youyou]

Thanks a lot Aske, it's a bit clearer right now, i'll take a look at all of that. :)

[lax]

You may also want to look at this tutorial on Siamese networks (if you haven't already): http://caffe.berkeleyvision.org/gathered/examples/siamese.html

[Youyou]

Hi again Aske,
  I learned a lot from your code and it helped me a lot. Although i still have some questions. I realise that if i make a totally random selection, i'll end up with a lot of negative pairs compared to positive ones, so when you said you used 4:1 negative samples did you mean that you used one negative pair for 4 positive ones? and if so it is a part of your code or you just organised your cuhk01 file to be so? and by the way how did you organised your cuhk01 file. Also i wanted to know ( last question i promise :) )if you performed data augmentation before converting your dataset or after?.

Thanks in advance for your answer Aske, this is very kind of you. Best regards :)

[Aske Rasch Lejbølle]

Hi,
I haven't done any data augmentation yet. If you are to include that, I guess it should be done before creating the dataset (I think there's already a python function implemented for this, check io.py).
I uploaded my script (it's not pretty, I know). It basically checks the number of negative pairs compared to positives (1 positive for every 4 negative) and add a positive pair whenever three negative have been added. I didn't change anything in the cuhk01 files just extracted them from the zip file.
Hope that helps.

Best,
Aske