Problem 3 seems broken

[Will Chang]

Coming up with the EM update equations are easy enough, but the
problem seems broken in the sense that you are given essentially only
two pieces of information (ratio of number of both-boy births to
number of both-girl births, ratio of number of same-sex births to boy-
girl births) and you have to come up with three pieces of information
(p, p', theta). This is impossible. The EM algorithm might converge,
but the answer will be meaningless.

Am I missing something?

~ Will

[Percy Liang]

You're right...this is a problem with non-identifiability of the
model, which means that two settings of the parameters (p, p', theta)
could yield the exact same distribution over (m, f, b). So given (m,
f, b), there is no way to tell the difference between the two
settings of the parameters.

Non-identifiability generally arises with hidden-variable models and
discrete variables - as an example, note that a mixture of
multinomials is a multinomial distribution. Even in the good old HMM
model with discrete variables, the parameters are not identifiable in
general. For example, note that if you permute the identities of the
hidden states, you can get the same likelihood.

In general, the result that EM gives us won't be totally meaningless
because we are still maximizing the likelihood of the data, so for
the purpose of density estimation (where we only care about the
induced distribution over p(x)), we are doing our job (although in
the simple example of problem 3, a much more direct way to do density
estimation is to just use a single multinomial distribution).

One solution to fix this non-identifiability is to put a prior over
the parameters which will break ties between parameters that yield
the same likelihood. One can still use EM with a prior in this case.

-Percy