Basic queries for Object Detection using deep learning in MatConvNet

[Adnan Awan]

Hello

I have recently started working on the Object detection using deep learning.

I have understood the basic steps of object detection are:

Input Image -> Object proposals -> Feature Extraction -> Training.

I have successfully implemented my own code up to the Object Proposal extraction. Now I want to use deep learning here from that point. For the input to deep learning I have fixed number of object proposals (i.e. 800 proposals from one image) and I need a labeled data.
For example:
I have a one class dataset. 1) Aircraft,
Images number 1-90 are aircraft.
and then above 800 aircraft's
Q1: Here, I want to ask that how can I make the labels for the dataset with this configuration of class?

Q2: After we have the labels, How it will process for the deep learning. As I have seen some examples in the MatConvNet that the deep learning use batch size. So if the batch size is 50, then for the first training image have 50 proposals (from 800 proposals, comparing ground-truth objects) and it will train pass to the CNN layers for feature extraction and then at the end for fully connected layers?

Regards

Adnan

[moalt wisp]

For creating labels, you might want to look at cnn_cifar.m getCifarImdb functions in the examples folder for cifar dataset. 

all of your 800 proposals will run into the CNN model for every iteration (epoch) , but in 50 proposal per batch.

[Adnan Awan]

I have taken the labels in this way for the training: (Please correct me if its wrong)

800 proposals are extracted from one image. I have checked the overlap of those 800 proposals with the ground truth. If the overlap ratio is above 0.6, the label is set to '1'. Otherwise, It will be '0'.

I have checked the Minist data, there are two variables: 1) Images 2) Meta.

Do I need to make my data same as these two variables that before going for the training?

Regards

Adnan

[moalt wisp]

For this cases, you use are using two labels which I believe you are trying to do binary classification. 1 for positive and 0 for negative?

When you have 800 proposals( or patch) from one images, you can assign the labels in loop like this for example:

% Contain all your proposals that is above 0.6 in folder called positive

positives= dir([folder '/positive/*.jpg']);

% Count total of proposals that is above 0.6

Npos =  numel(positives);

    % Here we devide the dataset training,validation and testing. 

    Npos_train = 0.8*Npos; 

    Npos_val = 0.1*Npos;

    Npos_test = 0.1*Npos;

% iterate every proposals (or patch from one image) and assign its label to 1

for i=1:Npos

                % read your proposals (or patch from one image)

                im = single(imread([folder '/positive/', positives(i).name]));

                % Assign im  to 4D dimension

  images.data(:,:,:, i) = im;

                % Set proposals labels.

images.labels(i) = 1;

% You need to divide your dataset usually into three, 1 = training , 2 = validation or 3 = testing. Most examples only consider 1 and 2. 

% You can try 70% training and 30% for validation.

        [ ~, x_tn] = size(find(images.set == 1));

        [ ~, x_v] = size(find(images.set == 2));

if x_tn < NPos_val

images.set(i) = 1;

else 

                    if x_v < NPos_val

                    images.set(i) = 2;

                else

                    images.set(i) = 3;

                end

end

Then you do about the same for negative part. I hope this answer your question.

[Adnan Awan]

Thanks Moalt Wisp.

Yes at the moment regardless of results (good or bad). I just want to make for the one object. And the test image should show that which proposal is detected correctly which is not.

A total number of positive proposals are 4000 so how many negative proposals I should I add? is there any percentage?

I want to ask few things regarding the code 

Q1:  Why need to define the images.data as 4D?

Q2: Is it possible I can use 45% for training, 25% validation and 30% testing?

Q3: If I define the images.set tfor the training and validation only does it means I will not testing? 

I have modified the code which you have send me as:

for i=1:Npos

                % Assign im  to 4D dimension

  images.data(:,:,:, i) = p_regions{i,:};

                % Set proposals labels.

images.labels(i) = 1;

                images.set=[ ones( 1, Npos_train ) 2*ones( 1, Npos_val) 3*ones( 1, Npos_test)];

% You need to divide your dataset usually into three, 1 = training , 2 = validation or 3 = testing. Most examples only consider 1 and 2. 

% You can try 70% training and 30% for validation.

        [ ~, x_tn] = size(find(images.set == 1));

        [ ~, x_v] = size(find(images.set == 2));

        if x_tn < NPos_val

        images.set(i) = 1;

        elseif x_v < NPos_val

                    images.set(i) = 2;

                else

                    images.set(i) = 3;

                end

end

Q4: Also, I couldn't understand the last part of the code. What this part of code do?

[ ~, x_tn] = size(find(images.set == 1));

        [ ~, x_v] = size(find(images.set == 2));

        if x_tn < NPos_val

        images.set(i) = 1;

        elseif x_v < NPos_val

                    images.set(i) = 2;

                else

                    images.set(i) = 3;

                end

Regards

Adnan

[moalt wisp]

q1. so that you can have index for all your image to call for training later. you can see how its work, without using for loop.

% using Matlab built in image

test1=imread('pears.png');

test2=imread('peppers.png');

% resize since need to fit dimension

imageSize= [256, 256, 3];

test1 = imresize(test1, imageSize(1:2));

test2 = imresize(test2, imageSize(1:2));

imdb.images.data(:,:,:,1) = test1;

imdb.images.data(:,:,:,2) = test2;

q2.Yes its possible if your dataset is big enough. Only training and validation set is used when doing the training for the neural network. Then you evaluate the model accuracy with your test set.

example in mnist dataset that classify handwritten digit 0 to 9 it uses 60000 images for training and validation and 10000 for testing. 

q3. yes. you need separate script to test the trained model.example testing for 1 image

im_ = single(imdb.images.data(:,:,:,1)) ; 

%im_ = imresize(im_, net.meta.normalization.imageSize(1:2));

%im_ = im_ - net.meta.normalization.averageImage;

% run the CNN

net.eval({'input',im_}) ;

% obtain the CNN last output

scores = net.vars(net.getVarIndex('prediction')).value;

scores = squeeze(gather(scores));

        % Get the max prediction score and its class

        [~, class] = max(scores) ;

q4. That just a simple way to assign the set . Usually people add randomization when assigning train,validation and test set.

It strongly advise you to follow the example in the matconvnet folder. mnist or cifar is an easy start.

[Adnan Awan]

Thank you very much for answering my questions.

images.set=[ ones( 1, Npos_train ) 2*ones( 1, Npos_val) 3*ones( 1, Npos_test)];

Q1: This initialization will define inside the Npos loop or outside?

Q2: A total number of positive proposals are 4000 so how many negative proposals I should I add? is there any percentage?

After the data arranged in the required format. I will follow the MINST  or CIFAR example.

[moalt wisp]

q1 . if you do it like that you should put outside the loop.

q2. I do not have direct answer for this really, maybe if you found out the answer you can share it here. For starters you can try equal positive and negative and give it a run. good luck =)

[Adnan Awan]

Thanks. :)

To save the time I have reduced the number of proposals to 100 as it was taking so long time for the 800 proposals.

For the positive images I have set the code like that:

MATLAB Code:

    Npos_train = floor(0.25*Npos); 

    Npos_val = floor(0.25*Npos);

    Npos_test = floor(0.50*Npos);

images.set =[ ones( 1, Npos_train ) 2*ones( 1, Npos_val) 3*ones( 1, Npos_test)];

% iterate every proposals (or patch from one image) and assign its label to 1

for i=1:Npos

                % Assign im  to 4D dimension

        im= imresize (double(p_regions{i,:}),[50,50]);

  images.data(:,:,:, i) = im;

                % Set proposals labels.

images.labels(i) = 1;

% You need to divide your dataset usually into three, 1 = training , 2 = validation or 3 = testing. Most examples only consider 1 and 2. 

% You can try 70% training and 30% for validation.

        [ ~, x_tn] = size(find(images.set == 1));

        [ ~, x_v] = size(find(images.set == 2));

        [ ~, x_ts] = size(find(images.set == 3));

        

        if x_tn < Npos_val

            images.set(i) = 1;

        elseif x_v < Npos_val

            images.set(i) = 2;

        else

            images.set(i) = 3;

        end

end

If was confuse, if i used only training and validation in images.set then I don't have any positive proposals for the testing. 

Q1: So i decided to add testing also (please correct me if I am wrong).

After the loop is executed the variables x_tn, x_v, x_ts, Npos_train, Npos_val, and Npos_test are:

Now, the next step for the negative samples. Negative samples will be added like:

for i=1:Nneg

                % Assign im  to 4D dimension

        im= imresize (double(n_regions{i,:}),[50,50]);

  images.data(:,:,:, Npos+i) = im;

                % Set proposals labels.

images.labels(Npos+i) = 0;

end

Q2: Do I again need to do the images.set step?

MATLAB Code:

 [ ~, x_tn] = size(find(images.set == 1));

        [ ~, x_v] = size(find(images.set == 2));

        [ ~, x_ts] = size(find(images.set == 3));

        

        if x_tn < Npos_val

            images.set(i) = 1;

        elseif x_v < Npos_val

            images.set(i) = 2;

        else

            images.set(i) = 3;

        end

Regards

Adnan

[moalt wisp]

q1. 

what images.set does is pretty similar on how images.labels does. 

images.labels are assigned on range of images.data  to 0 or 1 in your case

images.set are assigning images that have labels you assign above to 1=training, 2=validation or 3=test. 

if you use this code which you yourself code it, not me:

images.set =[ ones( 1, Npos_train ) 2*ones( 1, Npos_val) 3*ones( 1, Npos_test)];

you do not need to use my simple code, just delete that assign set using for loop:

          [ ~, x_tn] = size(find(images.set == 1));

        [ ~, x_v] = size(find(images.set == 2));

        [ ~, x_ts] = size(find(images.set == 3));

        

        if x_tn < Npos_val

            images.set(i) = 1;

        else

            if x_v < Npos_val

              images.set(i) = 2;

            else

              images.set(i) = 3;

        end

q2.

if you do not use set in for loop then you need to adjust a bit your set code so that it also include range for the negative sample because:

images.set =[ ones( 1, Npos_train ) 2*ones( 1, Npos_val) 3*ones( 1, Npos_test)];

only cover range for positive. I hope this answer your question.

[Adnan Awan]

Thanks for the nice explanation.

What I have understood from your explanation is first i need to make image.labels and then make image.set of both positive and negative proposals.

Following is my code of what I have understood from your explanation. Please correct me if I miss understood anything.

MATLAB CODE:

Npos =  numel(p_regions);

Nneg =  numel(n_regions);

Npos_train = floor(0.25* (Npos+Nneg )); 

Npos_val = floor(0.25*(Npos+Nneg ));

Npos_test = floor(0.50*(Npos+Nneg ));

images.set =[ ones( 1, Npos_train ) 2*ones( 1, Npos_val) 3*ones( 1, Npos_test)];

for i=1:Npos

        im= imresize (double(p_regions{i,:}),[50,50]);

  images.data(:,:,:, i) = im;

images.labels(i) = 1;

end

% for negative samples

for i=1:Nneg

        im= imresize (double(n_regions{i,:}),[50,50]);

  images.data(:,:,:, i+Npos) = im;

images.labels(i+Npos) = 0;

  end

images.set =[ ones( 1, Npos_train ) 2*ones( 1, Npos_val) 3*ones( 1, Npos_test)];

Q1: If the above code is correct that there will all positive samples on one side and negative samples on another side. How the training, testing, and validation in image.set will get positive and negative samples?

Regards

Adnan

[moalt wisp]

q1 in your code above the last line is not required.

to find it corresponding labels from set, you can do it like this say for testing:

% assign test set

test_image = find(images.set==3) ;

for i = 1:numel(test_image)

% iterate each truth label on set 3

label(i) = images.labels(test_image(i));

end

% count positive label = 1 in test set 3

numel(find(label==1))

% count negative label = 0 in test set 3

numel(find(label==0))

[Adnan Awan]

Thanks :)

That means I just need to define images.set once at the top as in the below code.

Now after setting all the negatives and positives the images.data will be something look like this [All Positives All Negatives ]

and images.labels will look like [All 1's All 0's ] and images.set will be defined as images.set =[ ones( 1, Npos_train ) 2*ones( 1, Npos_val) 3*ones( 1, Npos_test)];

Q: The thing which makes me confuse here is:
if we want 500 samples of training. then how CNN will automatically take positive and negative samples?

MATLAB CODE:

Npos =  numel(p_regions);

Nneg =  numel(n_regions);

Npos_train = floor(0.25* (Npos+Nneg )); 

Npos_val = floor(0.25*(Npos+Nneg ));

Npos_test = floor(0.50*(Npos+Nneg ));

images.set =[ ones( 1, Npos_train ) 2*ones( 1, Npos_val) 3*ones( 1, Npos_test)];

for i=1:Npos

        im= imresize (double(p_regions{i,:}),[50,50]);

  images.data(:,:,:, i) = im;

images.labels(i) = 1;

end

% for negative samples

for i=1:Nneg

        im= imresize (double(n_regions{i,:}),[50,50]);

  images.data(:,:,:, i+Npos) = im;

images.labels(i+Npos) = 0;

  end

Adnan

[moalt wisp]

Ah now when i look back at your set code i think it is incorrect.  I have fix it for you.

MATLAB CODE:

Npos =  numel(p_regions);

Nneg =  numel(n_regions);

Npos_train = floor(0.25* Npos); 

Npos_val = floor(0.25*Npos);

Npos_test = floor(0.50*Npos);

Nneg_train = floor(0.25*Nneg); 

Nneg_val = floor(0.25*Nneg);

Nneg_test = floor(0.50*Nneg);

for i=1:Npos

        im= imresize (double(p_regions{i,:}),[50,50]);

  images.data(:,:,:, i) = im;

images.labels(i) = 1;

if i < Npos_train

images.set(i) = 1;

else

                        if i < Npos_val

  images.set(i) = 2;

                        else

                           images.set(i) = 3;

                       end

end

end

% for negative samples

for i=1:Nneg

        im= imresize (double(n_regions{i,:}),[50,50]);

  images.data(:,:,:, i+Npos) = im;

images.labels(i+Npos) = 0;

if i < Nneg_train

images.set(NPos+i) = 1;

else

                        if i < Nneg_val

  images.set(NPos+i) = 2;

                        else

                           images.set(NPos+i) = 3;

                       end

end

  end

[adnan farooq]

Thanks alot.

When i run the code which you have fixed for me, at the output of images.set: there is only 1(training) and 3(validation) 

there is no set for the 2 (testing). is there any reason for not having 2 (testing)?

Adnan

[moalt wisp]

Ah sorry about that, i have made the fix below

MATLAB CODE:

Npos =  numel(p_regions);

Nneg =  numel(n_regions);

Npos_train = floor(0.25* Npos); 

Npos_val = floor(0.25*Npos);

Npos_test = floor(0.50*Npos);

Nneg_train = floor(0.25*Nneg); 

Nneg_val = floor(0.25*Nneg);

Nneg_test = floor(0.50*Nneg);

for i=1:Npos

        im= imresize (double(p_regions{i,:}),[50,50]);

  images.data(:,:,:, i) = im;

images.labels(i) = 1;

if i <= Npos_train

images.set(i) = 1;

else

                        if i <= Npos_train+Npos_val

  images.set(i) = 2;

                        else

                           images.set(i) = 3;

                       end

end

end

% for negative samples

for i=1:Nneg

        im= imresize (double(n_regions{i,:}),[50,50]);

  images.data(:,:,:, i+Npos) = im;

images.labels(i+Npos) = 0;

if i <= Nneg_train

images.set(NPos+i) = 1;

else

                        if i <= Nneg_train+Nneg_val

  images.set(NPos+i) = 2;

                        else

                           images.set(NPos+i) = 3;

                       end

end

  end

[Adnan Awan]

Thanks Alot :)

For the Network, I have made the code which is similar to the MINST example. But it shows me this error.

MATLAB CODE:

net.meta.inputSize = [50 50 3] ;

net.meta.trainOpts.learningRate = 0.001 ;

net.meta.trainOpts.numEpochs = 20 ;

net.meta.trainOpts.batchSize = 10 ;

opts.train.gpus = [];

f=1/100 ;

net.layers = {} ;

net.layers{end+1} = struct('type', 'conv','weights', {{f*randn(5,5,1,20, 'single'), zeros(1, 20, 'single')}},'stride', 1,'pad', 0);

net.layers{end+1} = struct('type', 'pool','method', 'max', 'pool', [2 2], 'stride', 2, 'pad', 0) ;

net.layers{end+1} = struct('type', 'conv', 'weights', {{f*randn(5,5,20,50, 'single'),zeros(1,50,'single')}}, 'stride', 1, 'pad', 0) ;

net.layers{end+1} = struct('type', 'pool', 'method', 'max', 'pool', [2 2], 'stride', 2, 'pad', 0) ;

net.layers{end+1} = struct('type', 'conv', 'weights', {{f*randn(4,4,50,500, 'single'),  zeros(1,500,'single')}}, 'stride', 1, 'pad', 0) ;

net.layers{end+1} = struct('type', 'relu') ;

net.layers{end+1} = struct('type', 'conv', 'weights', {{f*randn(1,1,500,10, 'single'), zeros(1,10,'single')}}, 'stride', 1, 'pad', 0) ;

net.layers{end+1} = struct('type', 'softmaxloss') ;

net.meta.classes.name =arrayfun(@(x)sprintf('%d',x),0:1,'UniformOutput',false);

disp( 'Net is Ok.' );

[net, info] = cnn_train( net, imdb, @getBatch, net.meta.trainOpts, opts.train, 'val', find( imdb.images.set == 2 ) ) ;

ERROR

Regards

Adnan

[Adnan Awan]

Actually, the error is