Machine Learning for multiple input to multiple output

[Jfcote]

Hi César,

 

Long time no see, I hope you are doing well!

 

I'm currently in R&D on a new project and was wondering if you can point me to a specific algorithm in the Accord.NET for this task.

 

I'm kind of use to the KSVM now because I used it a lot to classify data in my previous project. Now it's a little bit different. Since I can't tell you exactly the goal of the project itself, I will go with general logic (for non-disclosure agreement reason).

 

So here we go:

 

I will receive multiple state of variable on a system. Let's call them A, B and C.

 

Depending on these variable value, we want the system to output X, Y and Z value.

 

To do this, I was hoping to put my system in lots of situation, let's say 100 situation. In each of this situation, I will record the variable A, B and C. Then, I will adjust the variable manually in the system to get the best result (at least, in my human perception). Then, when it's "perfect", I will record the new variable X, Y and Z. I would like to just give all this to a training and hope for the best.

 

My problem is that in KSVM, you give it a lot of info and it gives you only one result (one of the class). Is there an algorithm of machine learning that can take some X number of variables and return Y number of variables? For example, it can receive 20 inputs and return 10 output value.

 

I hope this question is not dumb or stupid lol. I'm a software engineer first so I'm better at using these machine Learning algorithm than to know them all and since the Library is huge, I was hoping to get the information from you!

 

Thanks in advance!

 

Jean-François Côté

[César]

Hi Jean,

This could possibly be achieved using multiple Support Vector Machines. You could, for example, create one SVM that accepts X, Y, Z and outputs A, another that accepts the same X, Y, Z and outputs B, and another that accepts the same X, Y, Z and outputs C. Since you have experience with SVMs, perhaps it would be nice to test this first as you could leverage your existing knowledge (and you could use it as a baseline for further experiments).

For multiple outputs, it is not rare to use neural networks instead of SVMs. The only problem is that there might be a lot of guesswork to be done (i.e. guess how many neurons will be used, etc). For initial experiments (and since it seems your network is going to be fairly small), I would suggest a few things: starting with a network containing an arbitrary high number of hidden neurons, use the Nguyen-Widrow to initialize your network, and use the Levenberg-Marquardt with Bayesian regularization enabled to learn your network. Hopefully this combination of techniques should reduce the effort in finding suitable parameters (i.e. the number of neurons) in order to achieve good results!

If you need further help, please let me know!

Best regards,

Cesar

[Jfcote]

Wow that was fast!

 

I will follow your 2 ideas and come back later to give you feedback!

 

Thank you very much!

Jean-François

[Jfcote]

A little question for you:

 

With KSVM, is it possible to give it for example 12 int, 12 double and one boolean as input and to get an output that is an integer between 0 and 99?

 

I have always done KSVM with an output that was a category (for example 6 class) but now I need a value. Maybe the 99 value are considered class but what if some class are not represented in the training data (for example, not having any training data for 44)?

 

Sorry for all the questions haha!

 

Thanks!

[César]

Hi Jean,

In this case, it seems there is a kind of "order" relationship between your output values. Is that correct? If that is the case, it seems it would be better to use a regression support vector machine instead of a classifier. Those machines can output continuous values between -1 and +1, which you can then map into the [0,99] interval. They should also be able to generate intermediary values that they haven't seen before, as you mentioned in your question. If you need, they can be built using the SequentialMinimalOptimizationRegression class, which can be found in the Accord.MachineLearning.GPL.dll assembly. 

Hope it helps!

Best regards,

Cesar

[Jfcote]

Hi César!

 

I'm trying to implement the SequentialMinimalOptimizationRegression but I have 2 questions:

 

1. You tell me that the output are in the range of -1 to 1 but when I look at a code sample that you gave in the comment at the bottom of this page (http://crsouza.blogspot.ca/2010/04/kernel-support-vector-machines-for.html), it seems to output double value. Can I train it with double value between 0 and 99 or this is old documentation and I should only work with -1 to 1?
2. In the multiclass ksvm that I'm used to work with, there is always a step of calibration after the initial "training". In other terms, I'm used to call the Run() function twice with ProbabilisticOutputLearning . In the same samples at the bottom of your blog, you seem to skip this part? Is it necessary?

Thanks a lot!

 

 

[César]

Hi Jean!

The code outputs a double value, but it will always be a value between -1 and +1. What you might need to do is to normalize your values so they fit into the -1 and +1 interval, learn the machine, and then when you ask the machine to produce an output, rescale it back to 0...99. 

For example, you can do something like that (using the Scale method):

double[] outputs = ...

// Convert your outputs into the [-1, +1] range
double[] scaledOutputs = Accord.Math.Tools.Scale(fromMin: 0, fromMax: 99, toMin: -1, toMax: +1, outputs);


// ...
// create and learn the machine
// ...


// Compute a test point
double y = svm.Compute(input);

// Rescale it back to a 0...99 interval:
double answer = Accord.Math.Tools.Scale(fromMin: -1, fromMax: +1, toMin: 0, toMax: 99, y);

Regarding the second question, the ProbabilisticOutputLearning is only necessary if you are interested in classification probability. I am afraid it won't work with regression machines at all, but it also shouldn't be necessary. You can skip that part if you wish!

Hope it helps!

Regards,

Cesar

[Jfcote]

Hey César,

 

It works great! We are currently trying multiple kernel but do you have a suggestion which would be best suitable to a regression case like mine?

 

We first try with Linear(1) with very poor result and we are currently testing others but is there some "common" kernel to use in this case?

 

Thanks!

[César]

Hi Jean!

I am glad it worked. Yeah, there are some heuristics you can try to start with some good values. You can try estimating a Gaussian kernel from the data using the Gaussian.Estimate method. This method will use an heuristic to compute a good Sigma value and return you a properly configured Gaussian kernel with this value. I would suggest starting with this one first!

Regarding the other kernels, there is also another thing that you can try to do. In the SequentialMinimalOptimization teacher, leave its Complexity parameter unconfigured - this way the algorithm will try to figure out a good starting value for the Complexity parameter C as well. This should work for the Linear, Polynomial and other kernels, but please note that for some choice of kernels (i.e. Gaussian or Laplace) it shouldn't make any difference.