Support Vector Machine in Torch7

[Mridul Kavidayal]

I have based my model upon the following tutorial:

https://github.com/torch/tutorials/tree/master/2_supervised

For the last stage a neural network is used upon the features extracted from the CNN. I want to use a SVM in the final layer. How can I add that to my existing model ?

It has been shown in some papers that SVM seem to function better than neural network as the final layer in the CNN and therefore I wanted to try them out to increase the accuracy of the model. Also SVM's can be use for one class classification which neural networks lack.I need a one class classifier in the end and so the need for adding an SVM to the CNN.

Kindly help

[Jonghoon Jin]

you can use linear layer (nn.Linear) combined with hinge loss functions.

for ex, https://github.com/torch/nn/blob/master/doc/criterion.md#nn.MarginCriterion

--
You received this message because you are subscribed to the Google Groups "torch7" group.
To unsubscribe from this group and stop receiving emails from it, send an email to torch7+un...@googlegroups.com.
To post to this group, send email to tor...@googlegroups.com.
Visit this group at http://groups.google.com/group/torch7.
For more options, visit https://groups.google.com/d/optout.

[Mridul Kavidayal]

DO you suggest using this as a one class SVM ( which can be used to detect anomalies ?) As in I would simply provide only the positve labels and hope to get a lower score ( say < 0.5) for anomalies ?
Or was this just a way to implement SVM ?

Thanks.

[Hugo Alberto Perlin]

Could you get any advance on this issue?

[Mridul Kavidayal]

I did that but the results were bad. It hardly seems that it was able to learn a boundary. I am relatively new to torch and DO not know how I could have visualized the boundary. Also Weren't we adding a linear layer in the end ( in place for the svm where we have the choice to choose a (nonlinear) kernel ? )

Kindly Help as I am still stuck.

--
You received this message because you are subscribed to a topic in the Google Groups "torch7" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/torch7/bEDxJgaDlj8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to torch7+un...@googlegroups.com.

[Remi Cadene]

The simplest solution I found was to save the output of my neural net to a csv then to use python, pandas, sklearn for classifying.

local net = nn.Sequential()
[...]
net:add(SpatialConvolutionMM(4096, 4096, 1, 1, 1, 1))

net:add(nn.Threshold(0, 1e-6))

net:add(nn.View(4096)) -- I added this just after the threshold

-- net:add(SpatialConvolutionMM(4096, 1000, 1, 1, 1, 1))

-- net:add(nn.View(1000))

-- net:add(nn.SoftMax())

net = net:float()

My code is not easy to read, but here is my repo : https://github.com/Tamazy/DeepRecipes
I just began to learn deep learning, so I will improve it and I will add a nice readme.

Hope it will help someone.

[Francisco Vitor Suzano Massa]

Why don't you use the torch-svm package https://github.com/koraykv/torch-svm ?
It has bindings to liblinear and libsvm.
They have an example usage in their repo.

I have also trained SVMs on top of fc7 features for object detection. Code is here https://github.com/fmassa/object-detection.torch
Check the SVMTrainer.lua file. It implements hard negative mining (as in SPP and RCNN code), I took it off from the main file, but it works.

[Mridul Kavidayal]

I did not quite understand how that works. How are the weights updated in that case ?

--

[Francisco Vitor Suzano Massa]

If you train a SVM on top of your fc7 features, then the network parameters are not updated, you only learn the final classifier.
I think that the solution proposed by Jonghoon Jin should be the way to go if you want to backpropagate through the network.
What papers do you have in mind that do that ?

Le mercredi 24 juin 2015 16:41:58 UTC+2, Mridul Kavidayal a écrit :

I did not quite understand how that works. How are the weights updated in that case ?

[Mridul Kavidayal]

I had gone through the following paper :

Fu Jie Huang and Yann LeCun. 2006. Large-scale Learning with SVM and Convolutional for Generic Object Categorization. In Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Volume 1 (CVPR '06), Vol. 1. IEEE Computer Society, Washington, DC, USA, 284-291. DOI=10.1109/CVPR.2006.164 http://dx.doi.org/10.1109/CVPR.2006.164

But I hadn't realized that the weights were not being updated simultaneously ..  SO Is it that the CNN is trained first and then the features of the last layer(before the output) are fed to the SVM ? SO The reason I was particularly interested was to build a one class classifier so that I do not have to sample images for the negative class. However if the case is that A CNN is first trained independent of the SVM, I do not think that this approach would allow me to  learn good features from the CNN.

Kindly help

On Wed, Jun 24, 2015 at 8:19 PM, Francisco Vitor Suzano Massa via torch7 <torch7+APn2wQekV-tKjwlno4xVwxuKi...@googlegroups.com> wrote:

If you train a SVM on top of your fc7 features, then the network parameters are not updated, you only learn the final classifier.
I think that the solution proposed by Jonghoon Jin should be the way to go if you want to backpropagate through the network.
What papers do you have in mind that do that ?

Le mercredi 24 juin 2015 16:41:58 UTC+2, Mridul Kavidayal a écrit :

I did not quite understand how that works. How are the weights updated in that case ?

[Francisco Vitor Suzano Massa]

The paper you showed first trains a CNN, then uses it to extract features and trains a non-linear SVM on top of those features, there is no backpropagation from the SVM to the CNN.
I don't get your goal. If you only feed the classifier with one class, it won't learn anything. You need at least 2 classes (like object and not object) to be able to do some classification.

Le mercredi 24 juin 2015 18:52:40 UTC+2, Mridul Kavidayal a écrit :

I had gone through the following paper :

Fu Jie Huang and Yann LeCun. 2006. Large-scale Learning with SVM and Convolutional for Generic Object Categorization. In Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Volume 1 (CVPR '06), Vol. 1. IEEE Computer Society, Washington, DC, USA, 284-291. DOI=10.1109/CVPR.2006.164 http://dx.doi.org/10.1109/CVPR.2006.164

But I hadn't realized that the weights were not being updated simultaneously ..  SO Is it that the CNN is trained first and then the features of the last layer(before the output) are fed to the SVM ? SO The reason I was particularly interested was to build a one class classifier so that I do not have to sample images for the negative class. However if the case is that A CNN is first trained independent of the SVM, I do not think that this approach would allow me to  learn good features from the CNN.

Kindly help

On Wed, Jun 24, 2015 at 8:19 PM, Francisco Vitor Suzano Massa via torch7 <torch7+APn2wQekV-tKjwlno4xVwxuKilcETsstPSyRRCgUkb8pMoLf7mCXxUJVy@googlegroups.com> wrote:

If you train a SVM on top of your fc7 features, then the network parameters are not updated, you only learn the final classifier.
I think that the solution proposed by Jonghoon Jin should be the way to go if you want to backpropagate through the network.
What papers do you have in mind that do that ?

Le mercredi 24 juin 2015 16:41:58 UTC+2, Mridul Kavidayal a écrit :

I did not quite understand how that works. How are the weights updated in that case ?

[Mridul Kavidayal]

Yes My bad. I had taken a very superficial glance at the paper and had missed out the most important aspect that the CNN is trained beforehand. SO in brief the training procedure remains the same as before only that a second stage is now added training an SVM from the features obtained from the second last layer in the CNN.

My aim is given ANY image classify it as a sunflower or not. CNNS work pretty well in terms of true positives. However given the vast domain of 'not sunflower' it is very difficult to train for all negative samples. How should I sample images for my negative class in that case ? Given this issue I wanted to try out my luck with a one class SVM ( anomaly detection ). But since the CNN has to be trained before the same issue of sampling negative images comes up.

On Wed, Jun 24, 2015 at 10:51 PM, Francisco Vitor Suzano Massa via torch7 <torch7+APn2wQekV-tKjwlno4xVwxuKi...@googlegroups.com> wrote:

The paper you showed first trains a CNN, then uses it to extract features and trains a non-linear SVM on top of those features, there is no backpropagation from the SVM to the CNN.
I don't get your goal. If you only feed the classifier with one class, it won't learn anything. You need at least 2 classes (like object and not object) to be able to do some classification.

Le mercredi 24 juin 2015 18:52:40 UTC+2, Mridul Kavidayal a écrit :

I had gone through the following paper :

Fu Jie Huang and Yann LeCun. 2006. Large-scale Learning with SVM and Convolutional for Generic Object Categorization. In Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Volume 1 (CVPR '06), Vol. 1. IEEE Computer Society, Washington, DC, USA, 284-291. DOI=10.1109/CVPR.2006.164 http://dx.doi.org/10.1109/CVPR.2006.164

But I hadn't realized that the weights were not being updated simultaneously ..  SO Is it that the CNN is trained first and then the features of the last layer(before the output) are fed to the SVM ? SO The reason I was particularly interested was to build a one class classifier so that I do not have to sample images for the negative class. However if the case is that A CNN is first trained independent of the SVM, I do not think that this approach would allow me to  learn good features from the CNN.

Kindly help