HW1 possible typo?

[gregory...@gmail.com]

Hi,

Does anyone know if problem 1 has a typo? I think it might make more
sense if we were asked to minimize KL(p(x)||q(x)) instead of KL(q(x)||
p(x)). Because of the asymmetry of the KL divergence, the optimal
updates are probably different depending on whuich one we want to
minimize. (Just to be clear, i'm assuming KL(a(x),b(x))=integral
a(y) log(a(y)/b(y)) )

tfl
-greg

[gregory...@gmail.com]

yea okay, nevermind. (Though I am under the impression that typically
the first term of KL divergence is the ugly distribution--why is this
the case?).

On Nov 26, 11:01 pm, "gregory.vali...@gmail.com"

[Percy Liang]

When you do maximum likelihood estimation, you are minimizing KL(\hat
p(x) || p_\theta(x)),
where \hat p(x) is your empirical distribution and p_\theta(x) is
your model, which you are trying to optimize.

In variational inference, we are minimizing KL(q(x) || p(x)), where
now the first argument q(x) is what you're trying to optimize.

-Percy