Re: [gensim:680] Topic modeling w/ neural nets
Timmy Wilson
> I watched the talks, it was very interesting to see the approach and the
> results. But I am afraid a lot of it din't make sense to me.
Sounds like we're in the same boat -- i'm just learning this stuff myself
I recommend this wiki from Andrew Ng's group --
http://ufldl.stanford.edu/wiki/index.php/UFLDL_Tutorial
"
This tutorial will teach you the main ideas of Unsupervised Feature
Learning and Deep Learning. By working through it, you will also get
to implement several feature learning/deep learning algorithms, get to
see them work for yourself, and learn how to apply/adapt these ideas
to new problems.
"
I have a little list going here -- http://www.smarttypes.org/blog/deep_learning
I setup a mailing list where we can ask questions, and learn from each
other -- http://groups.google.com/group/smarttypes
Take care Senthil,
Timmy Wilson
On Thu, Nov 10, 2011 at 1:08 AM, Senthil <gsn.co...@gmail.com> wrote:
> Hi Tim,
> I watched the talks, it was very interesting to see the approach and the
> results. But I am afraid a lot of it din't make sense to me. Assume I know
> only about regular neural nets and a bunch of linear algebra - do you have
> any study material/tutorials/books/links to get me up to speed on these
> techniques?
> thanks,
> Senthil
> On Sun, Nov 6, 2011 at 9:42 AM, Timmy Wilson <tim...@smarttypes.org> wrote:
>>
>> Inspired by these two great talks:
>>
>> - Geoffrey Hinton -- The Next Generation of Neural Networks --
>> http://www.youtube.com/watch?v=AyzOUbkUf3M
>>
>> - Andrew Ng -- Unsupervised Feature Learning and Deep Learning --
>> http://www.youtube.com/watch?v=ZmNOAtZIgIk
>>
>> i'm interested in using deep learning to model latent topics
>>
>> i did some digging, and found Ruslan Salakhutdinov's -- Replicated
>> Softmax: an Undirected Topic Model --
>> http://www.mit.edu/~rsalakhu/papers/repsoft.pdf
>>
>> "
>> The model can be efficiently trained using Contrastive
>> Divergence, it has a better way of dealing with documents
>> of different lengths, and computing the posterior distribution
>> over the latent topic values is easy. We will also demonstrate
>> that the proposed model is able to generalize much better
>> compared to a popular Bayesian mixture model, Latent
>> Dirichlet Allocation (LDA) [2], in terms of both the
>> log-probability on previously unseen documents and the
>> retrieval accuracy.
>> "
>>
>> and
>>
>> "
>> The proposed model have several key advantages: the
>> learning is easy and stable, it can model documents of
>> different lengths, and computing the posterior distribution
>> over the latent topic values is easy. Furthermore, using
>> stochastic gradient descent, scaling up learning to billions
>> of documents would not be particularly difficult.
>> "
>>
>> i want to 'cobble together' a distributed python implementation --
>> she'll feel right at home in http://radimrehurek.com/gensim/ -- if
>> Radim will have her :]
>>
>> i figured i'd spam everyone that may be interested, and ask/plead for
>> help/existing code examples
>
>