why don't we use unsupervised training as initial point for supervised finetuning

[Maruti Agarwal]

I just want to dicuss it from theoretical point of view. Won't it help to incorporate unsupervised training in caffe? Actually, I was working on 44-class classification problem and classification accuracy was stuck at 2%. I increased the step-size (assuming that I am stuck at a local minima) and then it worked out just fine.

So I thought why not use unsupervised training to a get good starting point for supervised learning. Please share your views & experience on this. Thanks !

[Evan Shelhamer]

Unsupervised learning is a whole conversation in itself, but for now I'll just point out that Caffe models need not be supervised convolutional neural nets although that's sometimes the impression and you can have an unsupervised good time now. See for instance the MNIST autoencoder example. Supervision is just a matter of the loss, and one can make reconstruction models of various kinds in Caffe.

Granted common unsupervised models like RBMs are not presently modeled by Caffe, but there is no framework opposition to this. A contrastive divergence Solver plus an RBM layer for shorthand would do nicely.

As ever, PRs are welcome!

(I will be excited for more attention to return to the unsupervised side of life and for the space of models types to keep growing in general and in Caffe.)

Evan Shelhamer

On Thu, Sep 18, 2014 at 3:41 AM, Maruti Agarwal <marutih...@gmail.com> wrote:

I just want to dicuss it from theoretical point of view. Won't it help to incorporate unsupervised training in caffe? Actually, I was working on 44-class classification problem and classification accuracy was stuck at 2%. I increased the step-size (assuming that I am stuck at a local minima) and then it worked out just fine.

So I thought why not use unsupervised training to a get good starting point for supervised learning. Please share your views & experience on this. Thanks !

--
You received this message because you are subscribed to the Google Groups "Caffe Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to caffe-users...@googlegroups.com.
To post to this group, send email to caffe...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/caffe-users/706f1151-d4f5-4f5d-9de5-7046d1d4440c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Khalid Ashraf]

I am going to write a RBM pre-training step in Caffe soon. Stay tuned. 

[Maruti Agarwal]

@Khalid - Thanks ! I m also planning to do the same. 

[Maruti Agarwal]

Shud the RBM training be in a top-down fashion? Considering that top 2 layers form a rbm?

Hows it we decide on a particular deepnet architecture and pre-train it using Theano and get the initial weights. Use these weights for further finetuning in caffe.

 

 

On Thursday, 18 September 2014 16:11:19 UTC+5:30, Maruti Agarwal wrote:

[zhe wang]

have you finished RBM in caffe toolbox?

在 2014年9月20日星期六 UTC+8上午6:57:32，Khalid Ashraf写道：

[Steven Du]

? Any updates , sounds great.


在 2014年9月18日星期四 UTC+8下午6:41:19，Maruti Agarwal写道：