long paper for meeting this friday, 10am
17 views
Skip to first unread message
Leif Johnson
Nov 10, 2014, 5:23:38 PM11/10/14
to ut-f...@googlegroups.com
Hi all -
Our next meeting will be this Friday, the 14th of November, at 10am in
GDC 3.516. We'll have some bagels and discuss sections 3, 5, 6, and 7
of the following journal-length paper (warning, much longer than
typical conference paper!):
http://jmlr.org/papers/volume11/erhan10a/erhan10a.pdf
Why Does Unsupervised Pre-training Help Deep Learning?
Erhan et al, JMLR 2010
Much recent research has been devoted to learning algorithms for deep
architectures such as Deep
Belief Networks and stacks of auto-encoder variants, with impressive
results obtained in several
areas, mostly on vision and language data sets. The best results
obtained on supervised learning
tasks involve an unsupervised learning component, usually in an
unsupervised pre-training phase.
Even though these new algorithms have enabled training deep models,
many questions remain as to
the nature of this difficult learning problem. The main question
investigated here is the following:
how does unsupervised pre-training work? Answering this questions is
important if learning in
deep architectures is to be further improved. We propose several
explanatory hypotheses and test
them through extensive simulations. We empirically show the influence
of pre-training with respect
to architecture depth, model capacity, and number of training
examples. The experiments confirm
and clarify the advantage of unsupervised pre-training. The results
suggest that unsupervised pre-training guides the learning towards
basins of attraction of minima that support better generalization
from the training data set; the evidence from these results supports a
regularization explanation for
the effect of pre-training.
lmj
--
http://www.cs.utexas.edu/~leif
Our next meeting will be this Friday, the 14th of November, at 10am in
GDC 3.516. We'll have some bagels and discuss sections 3, 5, 6, and 7
of the following journal-length paper (warning, much longer than
typical conference paper!):
http://jmlr.org/papers/volume11/erhan10a/erhan10a.pdf
Why Does Unsupervised Pre-training Help Deep Learning?
Erhan et al, JMLR 2010
Much recent research has been devoted to learning algorithms for deep
architectures such as Deep
Belief Networks and stacks of auto-encoder variants, with impressive
results obtained in several
areas, mostly on vision and language data sets. The best results
obtained on supervised learning
tasks involve an unsupervised learning component, usually in an
unsupervised pre-training phase.
Even though these new algorithms have enabled training deep models,
many questions remain as to
the nature of this difficult learning problem. The main question
investigated here is the following:
how does unsupervised pre-training work? Answering this questions is
important if learning in
deep architectures is to be further improved. We propose several
explanatory hypotheses and test
them through extensive simulations. We empirically show the influence
of pre-training with respect
to architecture depth, model capacity, and number of training
examples. The experiments confirm
and clarify the advantage of unsupervised pre-training. The results
suggest that unsupervised pre-training guides the learning towards
basins of attraction of minima that support better generalization
from the training data set; the evidence from these results supports a
regularization explanation for
the effect of pre-training.
lmj
--
http://www.cs.utexas.edu/~leif
Karl Pichotta
Nov 10, 2014, 7:54:33 PM11/10/14
to Leif Johnson, ut-f...@googlegroups.com
All,
IN ADDITION TO THE REQUIRED READING, I declare Geoff Hinton's reddit AMA from today should be this week's Supplementary Material, to be discussed if desired:
IN ADDITION TO THE REQUIRED READING, I declare Geoff Hinton's reddit AMA from today should be this week's Supplementary Material, to be discussed if desired:
_k
--
You received this message because you are subscribed to the Google Groups "FLARE" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ut-flare+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply all
Reply to author
Forward
0 new messages