How informative is the Maximum Entropy method?
hl...@delphi.com
I am working with expected values and am searching for methods to
measure the reliability of such values at they relate to information
and probability theories. Could someone comment on the maximum entropy
method as a tool to explore expected values in these related areas.
Thank you, Harry Edwards hl...@delphi.com
Radford Neal
>I would like to hear comments from users of the Maximum Entropy method.
>I am working with expected values and am searching for methods to
>measure the reliability of such values at they relate to information
>and probability theories. Could someone comment on the maximum entropy
>method as a tool to explore expected values in these related areas.
You don't say enough about what you are doing to allow any sensible
comment on specifics. I will however, make a few comments on Maximum
Entropy in general. (I would not say that I am a "user" of Maximum
Entropy, but I have read lots about it.)
First of all, there are two rather different things that go under this
general name. The oldest "Maximum Entropy" method is a way of
selecting a probability distribution when all you know is the expected
values of various functions under this distribution, the method being
to choose the distribution that produces these desired expectations
while maximizing the entropy of the distribution. (From your
comments, I can't see how this is of any use to you - since this
method starts by *assuming* values for these expectations, it isn't
going to be able to say how "reliable" these values are. But again,
your explanation is not adequate to determine what you are actually
trying to do.)
Jaynes originally developed the Maximum Entropy method in the context
of statistical physics. Here, it is possibly a reasonable approach,
since macroscopic observations of systems with enormous numbers of
particles can perhaps be regarded as giving something approaching
direct knowledge of expectations. In the statistical contexts to
which the method has since been applied, it is difficult to see how
one could ever be said to know the values of these expectations
without also knowing the full probability distribution. Explanations
by Jaynes and others on this point are unintelligible.
More recently, a more reasonable "Maximum Entropy" method (now
somewhat of a misnomer) has been advocated, in which one defines a
prior over probability distributions for use in Bayesian inference by
saying that the probability of a distribution, D, is proportional to
exp(aS(D)), where S(D) is the entropy of D, and a is a constant.
Various arguments are advanced for why such a prior is the only one
consistent with basic desiderata for rational behaviour. These
arguments are unconvincing, not least because their authors then
immediately proceed to make a an unknown, with given hyperprior, and
thereby end up using an effective prior outside the class that they
have just declared contains all reasonable priors.
Nevertheless, there is no particular reason why "Maximum entropy"
priors might not on occasion prove useful. Go ahead whenever it seems
right.
Radford Neal