Entropy

[Frozz]

Hi. I am a little confused about the entropy problem.

How many bits of information are required to express 32 equally probable alternatives? Is it
2^(5*32) = 2^160
160 bits of information

Is the entropy 5?

Thanks!

[Achava Nakhash, the Loving Snake]

What I coincidence! I just started learning this stuff a couple of
days ago.
The entropy of a finite probality distribution whose simple events
have
probabilities p_1, ..., p_m is usually defined to be

sum(i=1, m) ( - (p_i * log(p_i) )

where the logarithm is base 2 which is always a positive number, since
all of the p_i are between 0 and 1. The case p_i = 0 by using 0
instead.

In your case, there are 32 simple events all having probability 1/32,
so the
entropy come out to - 32 * (1/32) * log(1/32) = log(32) = 5.

So the entropy is 5, as you thought, and the number of bits required
to
express this is 5, not 2^160 (!).

This is obvious without any information theory, as the binary number
system can express all whole numbers between 0 and 31 inclusive
using 5 bits, and this is a total of 32 distinct alternatives.

Regards,
Achava

[Frozz]

Thanks so much! This is really detailed.

So, if I want to count the maximum entropy, is it counted like this?

H = - ([0.2 (log 0.2)] * 5)
= 2.32

Thanks!!

[Achava Nakhash, the Loving Snake]

Maximum entropy implies that there is some set of options with
possibly different entropies over which you are finding the choice
whose entropy is the highest available. It is, in general, true that
if there are n points in your sample space that setting the
probability of all of them to be equal gives you the largest possible
entropy over all possible choices of the n probabilities that still
have to sum to 1, but without letting us know more detail about the
situation, I don't know how to proceed with finding the maximum
entropy for it.

Regards,
Achava

[kunzmilan]

If you have 32 objects (32 equally probable alternatives),
you need for their indexing by a binary decision tree
160 digits 0 and 1 (00000, 00001, ... till 11111).
The number necessary for one object is 160/32 = 5.
Construction of large binary decision tree were tedious, and therefore
it is replaced by binary logarithm.
If you have some information about objects, if they are already
indexed by symbols, as alphabet symbols are, the binary decision tree
disintegrates into a set of smaller trees.
The binary logarithm of the ratio (product of individual decision
trees/whole decision tree) is known as information entropy.
kunzmilan