Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss

Claimed approximation for Kullback-Leibler distance D(p||q) when p and q are close, apparently based on Taylor series

21 views
Skip to first unread message

jdm

unread,
Jan 1, 2012, 6:36:33 PM1/1/12
to
The following claim featured in a research paper I've been studying -
however, no proof accompanied it beyond a statement that the
approximation could be obtained using Taylor series at order 2 - and
it wasn't clear what the variable was supposed to be or around which
point.

Let p and q be discrete probability distributions of random variables
taking on values from a set with M+1 elements;

p=(p_0, ..., p_M)

(p_i = P(random variable with distribution p is equal to i))

Likewise, q=(q_0, ..., q_M)

Where p and q are close - defined as |p_{i} - q_{i}| << q_{i} \forall
i - let e_i denote the value (p_i - q_i).

Then, according to the paper, D(p||q) \approx D(q||p) \approx sum_{i=0}
^{M}(e_{i}^{2}/q_{i})/2.

I haven't been able to verify this approximation for myself, as I
stated, and if anyone reading this can help (even by arguing that the
approximation isn't in fact valid) it would be much appreciated!

Many thanks,

James McLaughlin.

0 new messages