Auto-encoder loss gradient
36 views
Skip to first unread message
Adam Salvail-Bérard
Nov 3, 2012, 9:41:33 PM11/3/12
to ift-72...@googlegroups.com
Hi everyone !
%7D)%7D%20l(f(x%5E%7B(t)%7D)) "\nabla_{\hat{a}(x^{(t)})} l(f(x^{(t)}))")
%7D))%7D%20l(f(x%5E%7B(t)%7D)) "\nabla_{o(\hat{a}(x^{(t)}))} l(f(x^{(t)}))")
Adam Salvail-Bérard
If I'm not mistaken (and Hugo should confirm), the gradient given in the slides of the autoencoder shouldn't be
but
(get GmailTex, it's awesome :P)
since the sigmoid hasn't been derived here.
Have fun !
Adam Salvail-Bérard
Hugo Larochelle
Nov 3, 2012, 9:51:21 PM11/3/12
to ift-72...@googlegroups.com
Nope, the slides are correct.
Let me be more precise: whether you
1) use the cross entropy loss with sigmoid outputs, or
2) use the squared difference loss with linear outputs
you get the same gradient on the pre-activation function outputs (\hat(a)(x)).
For the assignment, you are using the cross entropy loss with sigmoid outputs, since the data is binary.
Hugo
--
Adam Salvail-Bérard
Vous recevez ce message, car vous êtes abonné au groupe Google Groupes IFT 725 A2012.
Pour envoyer un message à ce groupe, adressez un e-mail à ift-72...@googlegroups.com.
Pour vous désabonner de ce groupe, envoyez un e-mail à l'adresse ift-725-a201...@googlegroups.com.
Pour plus d'options, visitez le site https://groups.google.com/groups/opt_out .
Reply all
Reply to author
Forward
0 new messages