Need explanation of Image Classification sample

[Jean-Francois Côté]

Hi!

I don't know if anyone here can help me understand the image classification sample that come with Accord.NET

I try it with the default images and everything is beautiful and perfect. Then, I modify the code a bit to load image that are not "dolphins" and put them in the test list for dolphins image.

I was very surprised that the multiple image of cats that I put in the test of dolphins turns out green ... The rate at the bottom was like 95% when almost half of the image were cats instead of dolphins...

Is it me that doesn't understand something or is this application very weak? I could understand that between a cats and a dog it could be difficult but between dolphins and cats, the difference seems obvious.

Anyone can help me understand what is going on?

Thanks a lot and have a good day!
J-F

[jfcote....@gmail.com]

Ok I discovered how it work, let me know if I'm wrong

If you give the algorithm 2 training classes, the testing data will be classified within these 2 classes. So if I put dolphin in the testing data but I only train for cats and sunflowers, it will need to decide if a dolphin looks more like a sunflowers or a cat.

So to have a system that is very precise, you must create a LOT of categories.

Hope it helps!
J-F

[César]

Hi Jean!

Well, the training and test sets must match. If the training set contains dolphins, the testing set for dolphins must contain dolphins as well. At the training phase, the classifiers know the labels of the training samples, and make use of those labels to learn the classification problem properly. At the testing phase, the classifiers ignore the class labels and attempt to give their own interpretation to the images. Then we compare and mark misses as red and hits as green.

Yeah, you are correct that they will always try to classify something among the classes they know. If you would like to reject images which may not belong to any of the classes learned by the machines, you can try to calibrate the machines using ProbabilisticOutputLearning. Then you can define some minimum threshold for rejecting the images.

Hope it helps!

Cesar

[jfcote....@gmail.com]

Hi Cesar!

Thanks for the clarification. I will try ProbalisticOutputLearning and see if I can modify it for my needs.

Do you know if it's possible to use another feature detector than SURF for this algorithm? I know that the SURF algo is patented in the US (I am in canada but there are possibilities that someone in the u.s buy our product) and I would like to avoid dealing with lawyers to know if we can use it or not.

Thanks!
J-F

[César]

Hi Jean,

Yes, it could be done, but better results are often found with SURF. Starting with 2.8.2 you can use any corners detector as well, such as FAST or Harris, but they only give spatial information and may not be able to explore other features such as colors, angles, etc.

Another way is to consider HOGs, but they are still experimental at this time. Also, if your images will always have the same dimensions, there are things which could be tried to perform classification.

Regards,

Cesar

[jfcote....@gmail.com]

Thanks!

But do you have an example of how to specify FAST? I can't find the place where it's done. I've also started your suggestion about the probalisticOutputLearning but I'm still too much of a newbie in Accord.NET to be able to do this modification right now. :) Will try it later when I'll be better!

[jfcote....@gmail.com]

Oh I just saw that I had 2.8.1! That's why I couldn't find how to use FAST instead of SURF.

[César]

In order to add the probabilistic output calibration, right after the lines 

    // Train the machines. It should take a while.
    double error = ml.Run();

you can add 

 

    ml = new MulticlassSupportVectorLearning(ksvm, inputs, outputs);

    // Configure the learning algorithm
    ml.Algorithm = (svm, classInputs, classOutputs, i, j) =>
    {
         return new ProbabilisticOutputLearning(svm, classInputs, classOutputs);
    };

    double likelihood = ml.Run();  

This will calibrate the machines to produce probabilistic outputs. Now, when you ask the machine to classify a new sample, you can also ask for the probability of this classification. In line 340, where it reads

   int actual = ksvm.Compute(input);

you can alter to

  double probability;
  int actual = ksvm.Compute(input, MulticlassComputeMethod.Voting, out probability);

After those changes, if you see probability is too low, then most likely the image is not similar to any of the other images and can possibly be rejected.

Hope it helps!

Cesar

[jfcote....@gmail.com]

Thanks you very much for this code, I will try it very soon.

In the code below, I don't understand how to change SURF for FAST(and it's version 2.8.2 so it should have the option). I've search the constructor and parameter but no luck...

// Create bag-of-words (BoW) with the given number of words
BagOfVisualWords bow = new BagOfVisualWords(bs);

if (cbExtended.Checked)
bow.Detector.ComputeDescriptors = SpeededUpRobustFeatureDescriptorType.Extended;

[César]

Sorry for not including much documentation for this new feature, but it will be added progressively. For example, suppose we would like to use the Moravec corners detector to form our bag-of-words. We can do:

      MoravecCornersDetector moravec = new MoravecCornersDetector();
      CornerFeaturesDetector detector = new CornerFeaturesDetector(moravec);

      var bow = new BagOfVisualWords<CornerFeaturePoint>(detector, numberOfWords: 10);

but please note that a bow based on corners will only take spatial (x,y) information in account, which may not be very adequate. An option is to use the CornerFeaturesDetector as a base and implement another feature detector in the future.

Best regards,

Cesar

[jfcote....@gmail.com]

Thank you very much!

Is FREAK detector available in Accord.NET? I know it has been implemented in OpenCV and compared to SURF it seems very good (see this blog: http://computer-vision-talks.com/2012/08/a-battle-of-three-descriptors-surf-freak-and-brisk/ )

I will try the code you posted. Keep up the good work, very nice framework!

[César]

Well, not yet. But I was looking forward to add another feature detector, preferably one what would not be encumbered by patents. I will take a look! 

By the way you can also feel free to register the addition of any other feature detector which could be possibly royalty-free in the project's issue tracker as an enhancement request!

Hope it helps,

Cesar

[jfcote....@gmail.com]

Thanks! I will.

By the way, I sent you a message on your blog on the Decision Tree article that you did with some correction of the code and question about something.

Have a good day

[Jfcote]

Hi César,

 

I'm bringing this post to life! I have finally try this (The probalistic output) and it doesn't work as expected. In fact, the probability is always returning 1. Is this normal? I have even put picture of VERY different class. For example, if I have a class looking for Panda and the other one is looking for Dolphin, if I put Airplanes in the testing group, I'm thinking that at least I would have a probability under 1, no?

 

My goal is that if I put a picture that doesn't belong in the testing group, it will detect it (in red). Right now, the probalistic output is not giving me this information and everything is classified in either one or the other group. Here is my code on the elimination click. It always go in the first "IF" since the probability is always 1.

double probability;
 int actual = ksvm.Compute(input, MulticlassComputeMethod.Voting, out probability);

 
 if (probability == 1)
 {
 item.BackColor = Color.LightGreen;
 if (item.Group.Name.EndsWith(".train"))
 trainingHits++;
 else trainingHits++;
 }
 else
 {
 /*
 if (expected == actual)
 {
 item.BackColor = Color.LightGreen;
 if (item.Group.Name.EndsWith(".train"))
 trainingHits++;
 else testingHits++;
 }
 else
 {*/
 item.BackColor = Color.Firebrick;
 if (item.Group.Name.EndsWith(".train"))
 trainingMiss++;
 else testingMiss++;
 //}
 }

Thanks a lot!

 

 

[César]

Hi Jean,

Please, can you try calling the 

int Compute(double[] inputs, MulticlassComputeMethod method, out double[] responses, out double output)

method overload instead? The probability for each class would be given on the "responses" vector. To get the probability of a class i, you should be able to get it on responses[i] instead. 

In order to get more probabilistic realistic results through the "output" out parameter, if I remember correctly, you would need to calibrate the machines first using ProbabilisticOutputLearning (please take a look at the link). The problem is that the standard SVM is not able to produce probabilities directly. If uncalibrated, what you should be getting in the "output" is a distance measure in the decision space. However, it should't have been 1 all the time as you mentioned, so I still have to take a look on that.

But please let me know if either using the double[] overload or Platt's Probabilistic Output Learning helps (the double[] approach is simpler to try first). For an example on how to use Probabilistic Output Learning, you may also take a look on the lines 268-263 of the Handwritting sample application.

Hope it helps!

Regards,

Cesar

[Jfcote]

Ok I understand the problem.

 

I had only 2 class. The probability was 100% for one and zero for the other class. When I use the same code with 3 classes, I get 66%, 33% and 0% in the 3rd one. So the probability works but it's kind of strange when there is only 2 class.

 

Thanks for your help!

[César]

Hi Jean,

I am glad you could solve it. But the behavior you are getting occurs only when the machines have not been calibrated using Probabilistic Output Learning. After the calibration, they should produce more "proper" probabilities.

[Matt]

Hi César,

I can confirm the issue of "improper" probabilities on 2-class data reported by Jfcote still exists. After some digging, I found that if the Compute call is instead using MulticlassComputeMethod.Elimination, the probabilities do appear as "proper." Even in your handwriting sample, it actually uses Elimination when drawing the graphs in the user test pane (since the parameter wasn't provided, it defaulted to Elimination).

Thanks,
Matt