meeting this week
20 views
Skip to first unread message
Leif Johnson
Apr 28, 2014, 5:32:30 PM4/28/14
to ut-f...@googlegroups.com
Hi all -
This week we'll meet on Friday at 3pm in GDC 4.518. Craig will lead
discussion of the following paper, which uses an interesting learning
technique called score matching to interpret what happens in a
regularized auto-encoder:
What Regularized Auto-Encoders Learn from the Data Generating Distribution
Alain, G and Bengio, Y
http://arxiv.org/pdf/1211.4246v4.pdf
What do auto-encoders learn about the underlying data generating
distribution? Recent work suggests that some auto-encoder variants do
a good job of capturing the local manifold structure of data. This
paper clarifies some of these previous observations by showing that
minimizing a particular form of regularized reconstruction error
yields a reconstruction function that locally characterizes the shape
of the data generating density. We show that the auto-encoder captures
the score (derivative of the log-density with respect to the input).
It contradicts previous interpretations of reconstruction error as an
energy function. Unlike previous results, the theorems provided here
are completely generic and do not depend on the parametrization of the
auto-encoder: they show what the auto-encoder would tend to if given
enough capacity and examples. These results are for a contractive
training criterion we show to be similar to the denoising auto-encoder
training criterion with small corruption noise, but with contraction
applied on the whole reconstruction function rather than just encoder.
Similarly to score matching, one can consider the proposed training
criterion as a convenient alternative to maximum likelihood because it
does not involve a partition function. Finally, we show how an
approximate Metropolis-Hastings MCMC can be setup to recover samples
from the estimated distribution, and this is confirmed in sampling
experiments.
This week we'll meet on Friday at 3pm in GDC 4.518. Craig will lead
discussion of the following paper, which uses an interesting learning
technique called score matching to interpret what happens in a
regularized auto-encoder:
What Regularized Auto-Encoders Learn from the Data Generating Distribution
Alain, G and Bengio, Y
http://arxiv.org/pdf/1211.4246v4.pdf
What do auto-encoders learn about the underlying data generating
distribution? Recent work suggests that some auto-encoder variants do
a good job of capturing the local manifold structure of data. This
paper clarifies some of these previous observations by showing that
minimizing a particular form of regularized reconstruction error
yields a reconstruction function that locally characterizes the shape
of the data generating density. We show that the auto-encoder captures
the score (derivative of the log-density with respect to the input).
It contradicts previous interpretations of reconstruction error as an
energy function. Unlike previous results, the theorems provided here
are completely generic and do not depend on the parametrization of the
auto-encoder: they show what the auto-encoder would tend to if given
enough capacity and examples. These results are for a contractive
training criterion we show to be similar to the denoising auto-encoder
training criterion with small corruption noise, but with contraction
applied on the whole reconstruction function rather than just encoder.
Similarly to score matching, one can consider the proposed training
criterion as a convenient alternative to maximum likelihood because it
does not involve a partition function. Finally, we show how an
approximate Metropolis-Hastings MCMC can be setup to recover samples
from the estimated distribution, and this is confirmed in sampling
experiments.
Leif Johnson
May 2, 2014, 1:19:12 PM5/2/14
to ut-f...@googlegroups.com
Aditya Rawal
May 12, 2014, 3:06:42 PM5/12/14
to Leif Johnson, ut-f...@googlegroups.com
Is there a plan to continue the reading group through the summer?
Aditya
--
You received this message because you are subscribed to the Google Groups "FLARE" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ut-flare+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Leif Johnson
May 14, 2014, 5:36:05 PM5/14/14
to Aditya Rawal, ut-f...@googlegroups.com
We will be on hiatus for the summer, since a number of folks are out.
Let's plan on re-convening in late August.
Of course, over the summer when you run across good papers in deep
models, feature learning, etc. please add them to the reading list!
lmj
--
http://www.cs.utexas.edu/~leif
Let's plan on re-convening in late August.
Of course, over the summer when you run across good papers in deep
models, feature learning, etc. please add them to the reading list!
lmj
http://www.cs.utexas.edu/~leif
Reply all
Reply to author
Forward
0 new messages