For a fixed entropy, which distirbution is closest (in KL sense) to any possible distribution
Golabi Doon
I was wondering if the answer to the following basic optimality
questions are known:
1. Consider the set P of all possible continious distributions with
"finite support in [a,b]" AND "equal entropy h". Which distribution q*
in the space of all possible distributions is closest, on average, in
terms of KL divergence, to all elements of the set, i.e.:
q*=argmin_{q} sum_{p in P} KL(p,q)
I feel the answer is uniform distribution in [a,b], but I am not sure.
2. Consider the set P of all possible continious distributions with
"given mean m" AND "equal entropy h". Which distribution in the space
of all possible distributions is closest, in a similar sense as
quesiton 1.
I guess the answer is normal distribution with mean m and isotropic
covariance I*s^2 (I is identity matrix, and s^2 is scalar) whose
entropy is equal to h, i.e. s is chosen such that d*log(s)+log(sqrt
(2*pi*e)^d)=h. Again this is my conjecture, and I am not sure if this
is true.
Your help would be greatly appreciated.
Golabi
Golabi Doon
> 2. Consider the set P of all possible continious distributions with
> "given mean m" AND "equal entropy h". Which distribution in the space
> of all possible distributions is closest, in a similar sense as
> quesiton 1.
I do apologize, but please ignore my question 1 and only consider
question 2.
Also I need to add some piece of information to question 2. The
covariance matrix C of the distributions in P is also given and is
fixed across all pdf's p in P. So restating P, it is the set of all
continous distributions in the world that have mean m, covariance C
and entropy h, for a given choice of m, C and h.
I would like to know what q* is:
q*=argmin_{q} sum_{p in P} KL(p,q)
where q is any possible continous distribution in the universe.
My conjecture is that, the answer to question 2 is normal distribution
with mean m and covariance C. However, I am not sure if that is
correct.
Thanks
Golabi