Goodness of fit measures for a distribution
Unknown
I have a basic question regarding what are some common quantitative measures
for the goodness of (parametric) fit of a distribution.
Here is what I am trying to do. I have some sample data which lets say it
has a lognormal distribution. I can get some hints of how it is distributed
from the histogram. What I want to do, is fit a few distributions (e.g.
Lognromal, Beta, Gamma, inverse Gaussian etc) and find out which fits the
best.
I have carried out some fits uing maximum likelihood and I can plot the
pdfs, or cdfs over my data to see which fits the best. However, I need some
quantitative results (i.e. numbers). Just to point out that I am not really
interested at the parameters of the distributions, but only which fits the
best.
I could do Kolmogorov-Smirnov and chi-square tests but that's as far as I
know.
Would someone be able to tell me what sort of metrics I can use for my
problem?
Regards,
V.Z.
Reef Fish
Unknown wrote:
> Hi everyone,
> I have a basic question regarding what are some common quantitative
measures
> for the goodness of (parametric) fit of a distribution.
But this is a different question from what you're trying to do:
>
> Here is what I am trying to do. I have some sample data which lets
say it
> has a lognormal distribution. I can get some hints of how it is
distributed
> from the histogram.
A histogram is the WORST you can possibly do.
Here's my unpublished "Theorem" :-) :
You recognize (or best distinguish) a PERSON by his/her face
and BODY; you recognize a distribution by its TAIL.
> What I want to do, is fit a few distributions (e.g.
> Lognromal, Beta, Gamma, inverse Gaussian etc) and find out which fits
the
> best.
You get yourself immediately into the unnecssary complication of
"fitting" and what metric to use to judge "best" or departure from
the fit.
The good-ole PROBABILITY PAPER plot is the idea you should use.
in all probability papers, the accent is on the departure of the
TAIL of the empirical distribution from the cdf of the theoretical
distribution.
> I have carried out some fits uing maximum likelihood and I can plot
the
> pdfs, or cdfs over my data to see which fits the best.
Do a PP plot or QQ plot. Just LOOK.
> However, I need some
> quantitative results (i.e. numbers).
Why? Like the drunk who uses a lampost for support rather than light?
:-)
> Just to point out that I am not really
> interested at the parameters of the distributions, but only which
fits the
> best.
> I could do Kolmogorov-Smirnov and chi-square tests but that's as far
as I
> know.
Chi-square is based on histograms -- it's worthless.
Kolmogorov uses on ONE POINT in the difference between the empirical
and theoretical cdfs, the point of maximum departure.
Your EYEBALLS can do an infinitely better job than that, looking at
the plot of the entire cdfs.
>
> Would someone be able to tell me what sort of metrics I can use for
my
> problem?
>
> Regards,
> V.Z.
You're hung up on the traditional "confirmatory data analysis", which
sheds little or no light on your REAL problem. I am suggesting
something along the line of John Tukey's "exploratory data analysis"
without any of Tukey's cryptic acronyms, as the most suitable way
of addressing your problem.
-- Bob.
Unknown
> Why? Like the drunk who uses a lampost for support rather than light?
> :-)
Because there are quite a few distributions that will visually fit my data
and the PDF, CDF and probability plots will look good. In such cases
quantitative results might shed some light for which is the best fit.
Unknown
do.
I know what the distribution of my data looks like and I have a good idea of
what parametric model it follows.
The problem is, that there are many distributions that VISUALY fit the data
well.
I want to find which of these distributions fits my data BEST.
As you can understand I cannot rely only on visual results because they all
look pretty much the same.
That's why I need some NUMBERS!!!!
V.Z.
R. Martin
It may not be quite what you want, but there is a test you can use to
determine which distributions are not "right" (i.e. probably consistent
with the data). Martin, R. L., "A Statistic Useful for Characterizing
Probability Distributions, with Application to Rain Rate Data", J. Appl.
Meteor., 28, 354 (1989).
Cheers,
Russell
--
All too often the study of data requires care.
Unknown
I will take a look.
"R. Martin" <russell...@wdn.com> wrote in message
news:425464...@wdn.com...
Eric Bohlman
news:d31i6v$ld5$1...@news8.svr.pol.co.uk:
There's no single definition of "best" in this case; before you can even
start to consider metrics, *you* have to, based on subject-matter
knowledge and not statistical considerations, decide which forms of
departure between two distributions are of practical importance for your
work and which aren't. Different metrics will weight different aspects
of departure differently; you want one that will heavily weight the ones
that are important in your field and lightly weight those that aren't.
You'll need to consider the "costs" of a) assuming no difference in a
particular aspect when there really is one and b) assuming a difference
when there really isn't one.
Glen
> Here is what I am trying to do. I have some sample data which lets say it
> has a lognormal distribution. I can get some hints of how it is distributed
> from the histogram. What I want to do, is fit a few distributions (e.g.
> Lognromal, Beta, Gamma, inverse Gaussian etc) and find out which fits the
> best.
> I have carried out some fits uing maximum likelihood and I can plot the
> pdfs, or cdfs over my data to see which fits the best. However, I need some
> quantitative results (i.e. numbers). Just to point out that I am not really
> interested at the parameters of the distributions, but only which fits the
> best.
WHY? If there are several that fit well, why does one that fits "best"
matter?
What if I could suggest another distribution that would fit even
better? WOuld you use that instead?
> I could do Kolmogorov-Smirnov and chi-square tests but that's as far as I
> know.
>
> Would someone be able to tell me what sort of metrics I can use for my
> problem?
Goodness of fit test statistics such as the K-S (NB the statistic
itself, not the p-value) provide a pretty sensible way of measuring
"fit".
But if finding the "best" is actually important in any sense, then why
is what its actually measuring as fit so unimportant that you're happy
to take random suggestions from the gallery? (Since two different
metrics will give two different "bests", and you seem to have no a
priori way of deciding which metric fits your situation, the meaning
of "best" is utterly arbitrary)
You said you don't care about parameter estimates. Do you care about
number of parameters? Unimodality? Continuity?
I suspect that you actually have no need for a "best" fit at all.
Perhaps you should think more carefully about what it is you
ultimately wish to achieve.
Glen
Unknown
"Glen" <glenb...@geocities.com> wrote in message
> I suspect that you actually have no need for a "best" fit at all.
> Perhaps you should think more carefully about what it is you
> ultimately wish to achieve.
>
> Glen
Hi,
what I am trying to do is to determine which univariate model is the most
appropriate and valid for my data. Of course such a requirement can be
interpreted in many ways.
Since I am not really a statistician, I thought of using goodness of fit
measures and graphical interpretations to accomplish the above requirement.
I guess this comes from the analogous example of curve fitting, where given
some data you can fit a parametric model to it that minimises some error
metric and best describes the behaviour of the data.
If you know of a more appropriate way please let me know.
V.Z.
Reef Fish
Several of the respondents were telling you the same thing -- that this
is NOT a curve fitting problem, for different reasons.
Your analogy of "curve fitting to data that minimizes some error
metrics"
(say the least squares criterion) is seriously flawed. You didn't
recognize it because you admitted that you are "not really a
statistician",
which is actually quite obvious without fitting any curve. :-)
The key reasons why curve-fitting of an empirical cdf to a theoretical
cdf is embedded in my "Theorem" <G> that it's the TAIL of the
distributions
that are tell-tale about distributions, not the bulk of the body.
The other reason, mentioned by others, is that there is no reasonable
"criterion" of what "best fit" means, or should mean.
For two different Empiricalcdf vs Theoreticalcdf comparisons, the
correct interpretation may be COMLETELY different if they have the
IDENTICAL "goodness of fit number", whether it's least squares,
minimum absolute desviations, or anyting else, because it would be
the PATTERN of the deviations (rather than the total) that matters.
It's no different in the analysis of residuals in a regression problem.
If you have a pattern +++++++++-------------+++++++++ , arranged in
suitable order of course, then it matters not how small the + or -
deviations are, it would be a SERIOUS violation of the assumptions
about the errors.
> If you know of a more appropriate way please let me know.
Look at the ENTIRE PP or QQ plot. Forget about this NUMBER you're
looking for use as a crutch, as the drunk uses a lampost for support.
This goodness-of-fit number you're seeking is ALWAYS the WRONG number,
because it cannot possibly capture the departure that matters that
you can judge BY EYE.
Hope the above clarifies this data-analytic issue.
-- Bob.
Unknown
impossible to choose which is the best distribution to fit your data,
considering all you know about the data is that it comes from a random
process, since there are really no objective metrics and all you have to do
is rely on graphic plots (which by the way dont tell much since many models
will give you similar fits).
Would that be an accurate statement?
Reef Fish
No. :)
But close. :-))
It appears to be the recommendations of at least one or more of those
few who followed up on your question. But they can't speak for this
newsgroup, nor intended to.
The cliche "a picture is worth a thousand words" can be aptly extended
to
"a picture is worth a thousand summary statistics" provided the
picture-looker knows what and how to look!
That is more or less the foundation and guiding principle of the
Statistical Graphics Section of the ASA, and those think deeper
in applying statistics than routinely using methods, whether the
non-graphical, analytic methods were invented/discovered by others
or even themselves.
That's my take. YMMV.
-- Bob.
Anon.
G.E.P. Box quote: "All models are wrong, but some are useful". This
suggests that "best" is not a good criterion (because it'll never be
right anyway, and there are an infinite number of models you could try).
I suppose the criterion used in practice is "good enough". And we try
to discourage people from putting a number on that as well.
Bob
--
Bob O'Hara
Department of Mathematics and Statistics
P.O. Box 68 (Gustaf Hällströmin katu 2b)
FIN-00014 University of Helsinki
Finland
Telephone: +358-9-191 51479
Mobile: +358 50 599 0540
Fax: +358-9-191 51400
WWW: http://www.RNI.Helsinki.FI/~boh/
Journal of Negative Results - EEB: www.jnr-eeb.org
Richard Ulrich
wrote:
Not especially accurate.
There are a slew of objective metrics, already mentioned.
"Objectivity" is not the problem.
Your lack of an "objective" is a problem.
There is not a "best" numerical way to fit *your* data if you
can't define what the problem is -- and you (so far) can not,
or have not done so, given the multiplicity of *statistical*
alternatives that exist, which you were (are?) not alive to.
You wanted *one* answer -- There is not *one* answer.
In that circumstance, looking at graphs is a good way to
start checking a few assumptions, or possibilities.
The most-used and best method of 'fitting a distribution'
is to consider how it is generated. Very often, that leads
to a conclusion which leads to Normal, Poisson, or
another one of a fairly small, simple set.
There is your 'assumed' distribution, which perhaps should
be trusted - for the sake of (a) robustness, or (b) ease of
discussion - unless something casts doubt.
Then, if you want to pursue the method: for each simple
distribution, consider the points that are outliers: Is there
something special about them? "Contamination" from a
second source of points? Errors? - Is there reason to expect
that NO pure distribution will fit these points?
Then, you eventually can get to those questions that
folks described, concerning what criterion ought to matter,
in order to best suit your *purpose* in fitting a distribution.
Do you have hundreds or thousands of points?
--
Rich Ulrich, wpi...@pitt.edu
http://www.pitt.edu/~wpilib/index.html
Unknown
>>
> Well, it's one I would go for. It's time to trot out the well known
> G.E.P. Box quote: "All models are wrong, but some are useful". This
> suggests that "best" is not a good criterion (because it'll never be right
> anyway, and there are an infinite number of models you could try). I
> suppose the criterion used in practice is "good enough". And we try to
> discourage people from putting a number on that as well.
>
> Bob
>
Yes, and I guess that would explain the fact why there are over 100
distributions out there when one could have "good enough" results with the
Normal distribution right?
V.Z.
Unknown
From this rollercoaster of subjective information, off colour jokes,
suggestions that I lack of an objective and statistical knowledge I sort of
derived the following two conclusions:
1) Never ever ask any more questions on any newsgroup. Instead, go to the
library and read a book on statistics.
2)If I ever see any statistician in the street, I will run him over with my
car.
Regards,
V.Z.
Reef Fish
Unknown wrote:
> Well gentlemen, it has certainly been an eye opener.
> From this rollercoaster of subjective information, off colour jokes,
> suggestions that I lack of an objective and statistical knowledge I
sort of
> derived the following two conclusions:
>
> 1) Never ever ask any more questions on any newsgroup. Instead, go to
the
> library and read a book on statistics.
Never have so many <in a newsgroup> been so helpful to one needy soul.
>
> 2)If I ever see any statistician in the street, I will run him over
with my
> car.
Never have I seen such a thin-skinned, ungratful and hateful Clueless
Newbie.
-- Bob.
>
> Regards,
>
> V.Z.
Glen
> So basically, based on the recommendations from this newsgroup, it is
> impossible to choose which is the best distribution to fit your data,
Impossible? not at all.
But:
Meaningful? probably not
Providing an answer a question that you're actually interested in?
likely not
> considering all you know about the data is that it comes from a random
> process, since there are really no objective metrics and all you have to do
Since there are no objective metrics, what the heck does "best"
possibly mean, except something as arbitrary as picking a good one at
random?
Glen
Glen
> Yes, and I guess that would explain the fact why there are over 100
> distributions out there when one could have "good enough" results with the
> Normal distribution right?
A **lot** more than 100. You think it's a long way down to the corner
shop, but that's peanuts compared to the number of distributions. Not
all of them have names, though.
Glen
Glen
Actually, you received several quite important pieces of information,
and apart from one poster I don't see any kind of comment that a
reasonable person would take offense at.
The largest difficulty appears to be an unwillingness to accept that
it would be better to more clearly define your aim rather than receive
a meaningless answer, If you wanted a meaningless recipe rather than
actual statistical advice you should have stated that at the start.
If you went to a newsgroup frequented by doctors and said "I have
something wrong with me that I'm not going to talk about much, and I
want to know which kind of leech I should use." you shouldn't be
surprised if they wanted you to clarify your problem a little and at
the same time didn't actually nominate a particular species of leech
right away. If you got angry because they didn't come up with a leech,
would that really be their fault? Or does it just demonstrate that
perhaps when you seek expert help it's because you didn't understand
something?
If you took their natural response as something you should issue
threats over, then maybe you need to look carefully at your motives in
seeking help in the first place. What answer did you want us to give?
Glen
Data Matter
is visualize the data. The QQ plot is great for this purpose, as Bob
and others have said.
Given that distributions are called different names for the reason that
they are different, I find it hard to imagine that the QQ plots of your
empirical distribution against all these reference distributions "look
the same". Is this a case of having 3 points and saying you can fit a
straight line, or 2-degree polynomial, or 3-degree polynomial, etc.
etc.?
Anon.
them.
(OK, I apologise: it's just an old joke in a new setting)
Anon.
Reef Fish
Unknown wrote:
> Well gentlemen, it has certainly been an eye opener.
> From this rollercoaster of subjective information, off colour jokes,
It dawned on me just NOW that "Unknown" must have referred to this
part of my initial response as "off colour jokes":
RF> A histogram is the WORST you can possibly do.
That is a fact.
RF> Here's my unpublished "Theorem" :-) :
That is my joke (with the smiley as the hint that it was humor)
RF> You recognize (or best distinguish) a PERSON by his/her face
RF> and BODY; you recognize a distribution by its TAIL
That is a PUN, but hardly "off colour" because that's precisely
the REASON why this histogram is no good because it loses ALL of
the information in the TAILS <which is a proper technical term
that "Unknown" is probably unaware> and also the reason why
probability plot are much more effective because (a) it does not
compress any information by grouping; and (b) it is much more
sensitive to detecting TAIL departures.
That statement actually goes along with my serious unpublished
Lecture Notes on Data Analysis addressing precisely the topic of
how to tell if a set of data comes from a particular distribution,
such as the Normal/Gaussian.
>From the misspelling of "color" :0), I infer "Unknown" is a British
chap, who often have a dry sense of humor, but I didn't realize one
could be THAT stuffy and humorless.
>
> 1) Never ever ask any more questions on any newsgroup. >
> 2)If I ever see any statistician in the street, I will run him over
with my
> car.
But you would be driving on the wrong side of the road to try to run
over ME! LOL
Cheerio,
-- Bob.
Russell...@wdn.com
Unknown wrote:
> Well gentlemen, it has certainly been an eye opener.
> From this rollercoaster of subjective information, off colour jokes,
> suggestions that I lack of an objective and statistical knowledge I
sort of
> derived the following two conclusions:
>
> 1) Never ever ask any more questions on any newsgroup. Instead, go to
the
> library and read a book on statistics.
That's often a good idea. NGs are not a perfect medium for
getting answers, although I've found much good help and
information over the years, and I hope I've occasionally
provided answers, too.
>
> 2)If I ever see any statistician in the street, I will run him over
with my
> car.
>
> Regards,
>
> V.Z.
Fortunately I'm a meteorologist. :-)
Cheers,
Russell
Reef Fish
> > 2)If I ever see any statistician in the street, I will run him over
> > with my car.
>
> Fortunately I'm a meteorologist. :-)
Don't worry, Russell.
As resourceful as "Unknown" is, he'll carry enough rocks in his
car to stone you to death. :-)
-- Bob.
Herman Rubin
Unknown <no...@nowhere.net> wrote:
>> Bob
Whether the normal distribution gives good enough results
depends on the problem. For regression, under mild conditions,
assuming normality is of little importance. For deciding
what is an outlier, it is of great importance. For being
confident about the tails, or in setting up an IQ scale,
it is of very great importance.
--
This address is for information only. I do not claim that these views
are those of the Statistics Department or of Purdue University.
Herman Rubin, Department of Statistics, Purdue University
hru...@stat.purdue.edu Phone: (765)494-6054 FAX: (765)494-0558
Aleks Jakulin
> Chi-square is based on histograms -- it's worthless.
>
> Kolmogorov uses on ONE POINT in the difference between the empirical
> and theoretical cdfs, the point of maximum departure.
>
> Your EYEBALLS can do an infinitely better job than that, looking at
> the plot of the entire cdfs.
In principle yes. But what do you do when you have a few dozen or
hundred variables in a complex data mining task? Then you do want a
one-number summary: you have no time to manually inspect a few hundred
thousand QQ plots.
You can use the p-value or some VaR quantification as a summary of the
tail, use it order the fits, and let the eyes inspect the deviant ones
first.
--
mag. Aleks Jakulin
http://kt.ijs.si/aleks/
Department of Knowledge Technologies,
Jozef Stefan Institute, Ljubljana, Slovenia.
Reef Fish
Aleks Jakulin wrote:
> Reef Fish wrote:
> > Chi-square is based on histograms -- it's worthless.
> >
> > Kolmogorov uses on ONE POINT in the difference between the
empirical
> > and theoretical cdfs, the point of maximum departure.
> >
> > Your EYEBALLS can do an infinitely better job than that, looking at
> > the plot of the entire cdfs.
>
> In principle yes.
In principle AND in practice.
Here is a very simple PRINCIPLE. If something is INAPPROPRITE for
1 (ONE) <such as a histogram>, it's inappropriate for the other 999
of a thousand also.
> But what do you do when you have a few dozen or
> hundred variables in a complex data mining task?
You should mine more carefully and delicately than running a steam-
roller over all of them when more delicate tools are required.
When you have a few thousand variables, the first task is to
selectively consider ONLY a few dozen, if that many, that seem
most appropriate, for substantive reasons.
> Then you do want a one-number summary: you have no time to manually
> inspect a few hundred thousand QQ plots.
It's easy to do a one number summary. Just generate one RANDOM NUMBER,
and say "that's what the 'puter gave me!" And that number is probably
as meaningful <or meaningless> as your single number from GIGO
(Garbage In, Garbage Out).
What you have argued is in fact the WORST that has happened to the
application of statistics -- when computer programs and packages
are readily availble for any Tom, Dick, and Harry to throw data
into the bin to get some meanless and useless number(s) out.
Progress takes 10 steps backwards.
-- Bob.
Aleks Jakulin
> You should mine more carefully and delicately than running a steam-
> roller over all of them when more delicate tools are required.
I agree, this is a steam-roller approach. A hand-sewn suit can be of
higher quality than a machine-sewn suit. Why aren't we all wearing
hand-sewn suits?
> When you have a few thousand variables, the first task is to
> selectively consider ONLY a few dozen, if that many, that seem
> most appropriate, for substantive reasons.
On several occassions data mining yielded novel patterns that we were
unaware of, and that we would not include into the analysis. This is
especially valid of interactions: although one can judge the relevance
of variables, we are rarely able to reliably judge the abundance of
possible interactions. Automation helps in such cases.
>>Then you do want a one-number summary: you have no time to manually
>>inspect a few hundred thousand QQ plots.
>
> It's easy to do a one number summary. Just generate one RANDOM NUMBER,
> and say "that's what the 'puter gave me!" And that number is probably
> as meaningful <or meaningless> as your single number from GIGO
> (Garbage In, Garbage Out).
Which one number summary is less garbage than others? Goodness of fit
does a solid job for categorical data, and it can be appropriately
reformulated for numerical data. The agreement between summaries of fit
and expert opinion was good in the experiments we did, so it does
provide better summaries than generating random numbers.
> What you have argued is in fact the WORST that has happened to the
> application of statistics -- when computer programs and packages
> are readily availble for any Tom, Dick, and Harry to throw data
> into the bin to get some meanless and useless number(s) out.
Yes, good tailors too were quite dismissive of the quality of the early
machine-sewn dresses. They also pointed out how hurtful it is a look at
all those unsophisticated clothes. They were forgetting that it is still
better to see a person in a machine-sewn suit than in rags: that's what
one should be comparing.
--
Reef Fish
Aleks Jakulin wrote:
> Reef Fish wrote:
> > You should mine more carefully and delicately than running a steam-
> > roller over all of them when more delicate tools are required.
>
> I agree, this is a steam-roller approach. A hand-sewn suit can be of
> higher quality than a machine-sewn suit. Why aren't we all wearing
> hand-sewn suits?
Your hand-sewn vs machine-sewn suit is not quite the appropriate
analogy. The appropriate analogy would be comparing the IRONING
of a hand-sewn or machine-sewn suit against by a conventional iron
vs spreading them out on the road and run a steam-roller over them.
> > When you have a few thousand variables, the first task is to
> > selectively consider ONLY a few dozen, if that many, that seem
> > most appropriate, for substantive reasons.
>
> On several occassions data mining yielded novel patterns that we were
> unaware of, and that we would not include into the analysis. This is
> especially valid of interactions: although one can judge the
relevance
> of variables, we are rarely able to reliably judge the abundance of
> possible interactions. Automation helps in such cases.
The Caveat is of course that you would find many SPURIOUS relations
patterns by this kind of steam-roller approach. Statisticians indeed
have found many spurious correlations (nonsense from the point of
view of causal relationship) by measuring the correlations among a
large set of variables.
Try this. Generate several thousand random sets of data from various
distributions, and do YOUR steam-roller analysis on them. I BET
you'll find may significant correlations, abundance of <spurious>
interactions, and all the rest of the similar Garbage you found in your
"Automation". :-)
>
> >>Then you do want a one-number summary: you have no time to manually
> >>inspect a few hundred thousand QQ plots.
> >
> > It's easy to do a one number summary. Just generate one RANDOM
NUMBER,
> > and say "that's what the 'puter gave me!" And that number is
probably
> > as meaningful <or meaningless> as your single number from GIGO
> > (Garbage In, Garbage Out).
>
> Which one number summary is less garbage than others? Goodness of fit
> does a solid job for categorical data, and it can be appropriately
> reformulated for numerical data. The agreement between summaries of
fit
> and expert opinion was good in the experiments we did, so it does
> provide better summaries than generating random numbers.
When the goodness of fit number is NOT indicative of what one is
looking
for in "goodness of fit", then it is as good (or bad) as just a random
number.
For HISTOGRAM data slotted in bins where the TAILS are all lumped into
one bin not far from the center, you can hardly tell by using a chi-
square goodness of fit criterion whether a set of CAUCHY data is normal
or not. Or a T with 10 df or a T with 3 d.f.
That's what I mean you have to look at the OVERALL cdf as well as the
TAIL behavior for your assessment of fit to be reliable or useful.
> > What you have argued is in fact the WORST that has happened to the
> > application of statistics -- when computer programs and packages
> > are readily availble for any Tom, Dick, and Harry to throw data
> > into the bin to get some meanless and useless number(s) out.
>
> Yes, good tailors too were quite dismissive of the quality of the
early
> machine-sewn dresses.
Good laundry and dry-clean establishments do not take their business
out to the street to run the laundered or dry-cleaned suits, hand-sewn
or machine-made by a gignatic steam roller, capable of "ironing"
THOUSANDS of shirts in one steam-rolling session in a large parking
lot.
-- Bob.
Aleks Jakulin
> vs spreading them out on the road and run a steam-roller over them.
Thanks for entertaining analogies! Let me use a 'harvester' instead of a
'steam-roller' though. :)
> Try this. Generate several thousand random sets of data from various
> distributions, and do YOUR steam-roller analysis on them. I BET
> you'll find may significant correlations, abundance of <spurious>
> interactions, and all the rest of the similar Garbage you found in your
> "Automation".
Fortunately, true correlations still stand out in most bunches of
spurious correlations. And of course there is a human in the loop that
checks if the stuff that got caught in the harvester is grain or sand.
As long as the harvester isn't losing too much grain, you're better off
with the harvester than with a sickle.
> For HISTOGRAM data slotted in bins where the TAILS are all lumped into
> one bin not far from the center, you can hardly tell by using a chi-
> square goodness of fit criterion whether a set of CAUCHY data is normal
> or not. Or a T with 10 df or a T with 3 d.f.
If you have unordered categorical data, you don't have any notion of a
'tail' in there. If you have ordered or real-valued data, then you
shouldn't use a histogram, because it throws this kind of information
away. I think we agree here.
Aleks
Reef Fish
Aleks Jakulin wrote:
> Reef Fish wrote:
> > vs spreading them out on the road and run a steam-roller over
them.
>
> Thanks for entertaining analogies! Let me use a 'harvester' instead
of a
> 'steam-roller' though. :)
>
> > Try this. Generate several thousand random sets of data from
various
> > distributions, and do YOUR steam-roller analysis on them. I BET
> > you'll find may significant correlations, abundance of <spurious>
> > interactions, and all the rest of the similar Garbage you found in
your
> > "Automation".
>
> Fortunately, true correlations still stand out in most bunches of
> spurious correlations.
I think we have a misunderstanding of the terminology here.
By "true correlation" I think YOU mean those correlations that actually
relate to causal or other relations that can explain the correlations.
But "spurious correlations" are "true correlations" in the sense that
they are correlation numbers between two numerical variables, say.
They
may make sense from SOME explaination, but nonsense from another.
The example I like to use is the "spurious" correlation between the
number of storks observed over a number of years in some country and
the number of babies born in that country. Even the social scientists
dared not interpret that as suppot for the theory that babies are
brought by storks!
On the other hand, your TRUE correlations do NOT stand out among
spurious correlations as reailiy as you think.
Otherwise, I owuldn't have to have written my severest criticism
in a published book review in JASA about "Correlation and Causation"
in which the author claimed that correlation (in observational data,
in the ABSENCE of a designed experiment) can be used to ascertain
CAUSE and EFFECT, by the black-magic and quackery of "causal
diagrams" drawn by those in the cult of (mostly) social scientists.
> And of course there is a human in the loop that
> checks if the stuff that got caught in the harvester is grain or
sand.
> As long as the harvester isn't losing too much grain, you're better
off
> with the harvester than with a sickle.
Read my review of the book "Correlation and Causation", in an 1982
issue
of JASA, I think. There are plenty of humans (actually serious
academic scholars -- the author was a Harvard Professor -- obviously
not in Harvards's statistics department) who don't know how to tell
the grain from the chaff, whether they harvested from a sickle or
from a harvester.
So your new analogy is as useless as the former one. :-)
> > For HISTOGRAM data slotted in bins where the TAILS are all lumped
into
> > one bin not far from the center, you can hardly tell by using a
chi-
> > square goodness of fit criterion whether a set of CAUCHY data is
normal
> > or not. Or a T with 10 df or a T with 3 d.f.
>
> If you have unordered categorical data, you don't have any notion of
a
> 'tail' in there. If you have ordered or real-valued data, then you
> shouldn't use a histogram, because it throws this kind of information
> away. I think we agree here.
When I say histogram, I always assume it's the histogram of the
interval-variable type, more than just ordered. If you agree THERE<
then we would have had no argument in the first place.
Our suits, steamroller, sickles and harvesters would have been all
irrelevant insofar as the HISTOGRAMS go, but I am glad I had the
opportunity to unveil the naked emporer in your machine-made suit,
as well as the naked farmer riding your harvester. :)
-- Bob.
Reef Fish
Reef Fish wrote:
> Aleks Jakulin wrote:
> > Reef Fish wrote:
> > > vs spreading them out on the road and run a steam-roller over
> them.
> >
> > Thanks for entertaining analogies! Let me use a 'harvester' instead
> of a
> > 'steam-roller' though. :)
> >
> >
> > spurious correlations.
>
> I think we have a misunderstanding of the terminology here.
>
> By "true correlation" I think YOU mean those correlations that
actually
> relate to causal or other relations that can explain the
correlations.
>
> But "spurious correlations" are "true correlations" in the sense that
> they are correlation numbers between two numerical variables, say.
> They
> may make sense from SOME explaination, but nonsense from another.
I forgot to give the punch line (the make sense part) of this example:
>
> The example I like to use is the "spurious" correlation between the
> number of storks observed over a number of years in some country and
> the number of babies born in that country. Even the social
scientists
> dared not interpret that as suppot for the theory that babies are
> brought by storks!
The EXPLANATION is that over the period of years of observation, both
the HUMAN population and the STORK population grew!
Thus, over the period t1< t2 < ... < tn,
Both the reported stork sights and the numbers of babies born from
hospital records are monotonically increasing functions, and if the
annual increases are large, then you would observe a high POSITIVE
correlation, whether the annual CHANGES are positively or negatively
related.
Think of some time series increasing in TIME, they will have positive
correlations.
That was how cigarrette smoking was first SUSPECTED of lung-cancer
causing, from correlational data, until someone was so insensitive
to point out that over the same years, cigarette smoking was
NEGATIVELY correlated with stomach cancer, and implicitly asked
whether that was to be interpreted as smoking CURES stomach cancer
from the causal inference point of view.
-- Bob.
Aleks Jakulin
> By "true correlation" I think YOU mean those correlations that actually
> relate to causal or other relations that can explain the correlations.
In the context of the artificial example you suggested, there should be
no misunderstanding: we know what distributions we've generated our
samples from. Our assumptions are the cause.
> On the other hand, your TRUE correlations do NOT stand out among
> spurious correlations as reailiy as you think.
Agreed.
> Otherwise, I owuldn't have to have written my severest criticism
> in a published book review in JASA about "Correlation and Causation"
Aha, David Kenny's book. Unfortunately, can't access the full text of
your review online, just the citation.
I guess the main dilemma here is between two options:
a) Treat the topic of "causality" as something
unknowable-unless-you-assume-it, holy and untouchable.
b) Relax and treat "causality" as a useful metaphor, mind the Simpson's
paradox, and provide workable definitions, while realizing that a model
may say 'causal' to something that you know isn't, or later find
evidence against. But you forgive the model, because it's working with
far less evidence than you are (at least we think so ;).
I'm not sure what's best. Although I practice b), I do not refer to the
models of b) as "causal".
> When I say histogram, I always assume it's the histogram of the
> interval-variable type, more than just ordered. If you agree THERE<
> then we would have had no argument in the first place.
I guess our argument was a pro/contra on the topic of testing and
one-number summaries of fit.
For interval variables, I'd use a nicely-behaved discepancy measure
between distributions such as Kullback-Leibler divergence or Brier
score. Kolmogorov's is too brittle to be useful, unless you're from
Asymptotia.
One can then interpret this discrepancy measure as a test statistic or
as an "objective" loss function if no better one is given. But broadly
speaking, this gives you a one-number goodness-of-fit summary, and
harvesting is what this is good for.
Reef Fish
Aleks Jakulin wrote:
> Reef Fish wrote:
> > By "true correlation" I think YOU mean those correlations that
actually
> > relate to causal or other relations that can explain the
correlations.
>
> In the context of the artificial example you suggested, there should
be
> no misunderstanding: we know what distributions we've generated our
> samples from. Our assumptions are the cause.
But my point was that in that case we DO know the "correct" true state
of nature, that the thousands of datasets are all INDEPDENTLY generated
with zero pairwise correlation between each pair of the populations.
In practice, we don't have the luxury of playing the all-knowing god,
and
we have to "guess", "infer", or "deduce" from DATA from the
populations.
That's when the SPURIOUS relations are found and you have to have the
correct mathodology to sort out what's real from what's not.
>
> > On the other hand, your TRUE correlations do NOT stand out among
> > spurious correlations as reailiy as you think.
>
> Agreed.
>
> > Otherwise, I owuldn't have to have written my severest criticism
> > in a published book review in JASA about "Correlation and
Causation"
>
> Aha, David Kenny's book. Unfortunately, can't access the full text of
> your review online, just the citation.
I can't access it anywhere readily either. :)
> I guess the main dilemma here is between two options:
> a) Treat the topic of "causality" as something
> unknowable-unless-you-assume-it, holy and untouchable.
That is unacceptable. While some may consider statistics a religion,
I don't, or at least we belong to different churches of statistics. :)
>
> b) Relax and treat "causality" as a useful metaphor, mind the
Simpson's
> paradox, and provide workable definitions, while realizing that a
model
> may say 'causal' to something that you know isn't, or later find
> evidence against. But you forgive the model, because it's working
with
> far less evidence than you are (at least we think so ;).
This part is acceptable. George Box's type of philosophy and
pragmatism,
published in his articles on "Science and Statistics" that all models
are WRONG, but we need to distinguish what's IMPORTANTLY wrong, and
worry about the tigers and not the mice ... or something to that
effect.
That's SCIENCE. Everything is TENTATIVE, until we find better models
and explanations.
>
> I'm not sure what's best. Although I practice b), I do not refer to
the
> models of b) as "causal".
Practicing b) does NOT give one the license to ASSUME causality on the
basis of observational data WITHOUT a controlled experiment designed to
test the causality part.
That's where Kenny and many social scientists and economists made their
blunders in their use of correlations and regression models on
observational data as if they ascertain cause!
There is one advantage of the Bayesian approach -- that if X does not
cause Y, you can collect and analyze data from now to doomsday, you
will NOT be able to conclude, the Bayesian way, that X causes Y, as
Kenny did in his book. Or, as a Bayesian, the causal diagrams are
irrelevant to that fact that data cannot support a FALSE model, if
you analyze it properly the Bayesian way -- this fact is discussed
in length Edwards, Lindman, and Savage paper.
>
> > When I say histogram, I always assume it's the histogram of the
> > interval-variable type, more than just ordered. If you agree
THERE<
> > then we would have had no argument in the first place.
>
> I guess our argument was a pro/contra on the topic of testing and
> one-number summaries of fit.
Quit a bit more than that, from my point of view. What you can SEE
and grasp in a plot (if you know what and how to look) is better
than 1,000 summary statistics numbers.
>
> For interval variables, I'd use a nicely-behaved discepancy measure
> between distributions such as Kullback-Leibler divergence or Brier
> score. Kolmogorov's is too brittle to be useful, unless you're from
> Asymptotia.
You should read some of the published articles by statisticians in
the Statistical Graphics sections of the ASA, including Tukey and
others, on the efficacy of EYEBALLING the approproach plots and
nothing else.
>
> One can then interpret this discrepancy measure as a test statistic
or
> as an "objective" loss function if no better one is given. But
broadly
> speaking, this gives you a one-number goodness-of-fit summary, and
> harvesting is what this is good for.
Here's Herman Rubin's line --------------> step over there please.
:-)
-- Bob.
Anon.
claimed). The growth in the human population caused more houses to be
built, which lead to more chimneys being put up. Storks use chimneys to
build nests on, so this gave them more nesting space. Ergo, an increase
in population size.
Bob
P.S. I managed to find your JASA book review (I wanted to find out who
you were!), and I've passed it on to Aleks. I think you enjoyed writing it.
Aleks Jakulin
> That's SCIENCE. Everything is TENTATIVE, until we find better models
> and explanations.
Right. So when you hear Pearl, Spirtes, Glymour and others (Kenny?)
mention "causality", that's pretty much what they mean. Recently, Pearl
has been talking about the do() operator which marks a controlled
intervention. Say we have a variable A. If I control it, the variable
will be do(A), and then you can interpret anything that's correlated
with do(A) as caused by do(A) *IF* there is no Simpson's paradox (i.e.,
if there are no other causal influences).
> Practicing b) does NOT give one the license to ASSUME causality on the
> basis of observational data WITHOUT a controlled experiment designed to
> test the causality part.
As above, if you have control over relevant variables, and if you assume
no Simpson's paradox, then you can use the SEM/TETRAD approach to infer
causal connections.
The resulting causal connections thus assume:
1) the data and the statistical model are representative
2) probability theory is a foundation for causality
3) everything relevant has been controlled for (there is no Simpson's
paradox)
> There is one advantage of the Bayesian approach -- that if X does not
> cause Y, you can collect and analyze data from now to doomsday, you
> will NOT be able to conclude, the Bayesian way, that X causes Y, as
> Kenny did in his book. Or, as a Bayesian, the causal diagrams are
> irrelevant to that fact that data cannot support a FALSE model, if
> you analyze it properly the Bayesian way -- this fact is discussed
> in length Edwards, Lindman, and Savage paper.
The question is what Kenny meant (I haven't read the book). If I say: "A
causes B." You can ask me "What are your assumptions?" And I'll clearly
not say "no assumptions", I'll say "Points 1-3 of the previous paragraph."
Yes, agreed, in the epistemological Bayesian context everything is
uncertain, except for the uncertainty itself ;)
> others, on the efficacy of EYEBALLING the approproach plots and
> nothing else.
There is a problem with eyeballing, though. This is like sweeping the
problem under the carpet. Yes, it's good to eyeball, but where does the
experience come from? I bet that a significance test is better than at
deciding incidence fron coincidence than an eyeballing by newbie that
has seen 5 scatterplots in his life.
> Here's Herman Rubin's line --------------> step over there please.
Herman disagrees with my approach :) To save his time, I'll try to
summarize, hoping that I don't misrepresent his opinions (but he'll
correct me, I presume). He stresses that the utility/loss should be
something that makes sense, and not an "objective" loss function. I
agree, but I'm building universal harvesters. The second point of
difference is that he integrates out *all* the variables, trusting that
the model is right. Here, I prefer to compare/eyeball/whatever the
posterior distributions on some variables because I am curious about
what the posterior looks like - that's what many Bayesians do these days
of MCMC. The third point of difference *may* be that I follow the
following advice from I.J.Good:
"I regard it as mentally healthy to believe that credibilities (logical
probabilities) exist and as subjectively probable that physical
probabilities exist."
Reef Fish
Aleks Jakulin wrote:
> Reef Fish wrote:
> > That's SCIENCE. Everything is TENTATIVE, until we find better
models
> > and explanations.
>
> Right. So when you hear Pearl, Spirtes, Glymour and others (Kenny?)
> mention "causality", that's pretty much what they mean.
The keyword is "controlled experiment" to ascertain OR debunk a
causal hypothesis. I don't know about the other fellas, but Kenny
was definitely NOT one of them, to consider ANY kind of control, in
using correlational data, when he wrote his book.
> > Practicing b) does NOT give one the license to ASSUME causality on
the
> > basis of observational data WITHOUT a controlled experiment
designed to
> > test the causality part.
>
> As above, if you have control over relevant variables, and if you
assume
> no Simpson's paradox, then you can use the SEM/TETRAD approach to
infer
> causal connections.
Absolutely NOT! That's one of my points in the review. If X is
spuriously correlated with Y, say positively, but X is definitely
NOT the causal factor of Y, then no matter how much data you observe
(with ocntrol), you can never DISPROVE the (assumed) causal
relation from the voodoo skethces (path diagrams).
I have no time to give a course in Logic and valid statistical
inference here. Read the book by Copi (an introductory book on
LOGIC at the college level), and you'll better understand the
necessary and sufficient conditions to establish CAUSE as well
as the meaning of "proximity" versus "remote" cause.
http://froogle.google.com/froogle?q=introduction+to+logic+by+copi&hl=en&lr=&sa=N&tab=ff&oi=froogler
>
> The resulting causal connections thus assume:
> 3) everything relevant has been controlled for
How can that be done in an observation study WITHOUT a designed
control?
>
> > There is one advantage of the Bayesian approach -- that if X does
not
> > cause Y, you can collect and analyze data from now to doomsday, you
> > will NOT be able to conclude, the Bayesian way, that X causes Y, as
> > Kenny did in his book. Or, as a Bayesian, the causal diagrams are
> > irrelevant to that fact that data cannot support a FALSE model, if
> > you analyze it properly the Bayesian way -- this fact is discussed
> > in length Edwards, Lindman, and Savage paper.
>
> The question is what Kenny meant (I haven't read the book).
Ah well, that explains at least a good part of it. If you read
the book, his quackery should be quite apparent, even to you. :-)
The government and other gullible non-thinkers had been fooled
for nearly 30 years on the false causal "assumption" that lower
maximum speed on Interstate saves lives, as if that was the
DIRECT cause, and that RAISING back the speed from 55 to 70
(as the law was finally repealed) will CAUSE many lives to be lost.
http://www.motorists.com/issues/speed/u_turn.html
Empirical data and statistics proved otherwise! They should
have done the CONTROL study 30 years ago!!!
*> In California, where interstate speed limits are set at 70 mph,
*> the fatality rate declined 4 percent between 1995 and 1996 -
That's just a drop in the bucket of the flood of examples that the
legislators and politicians were full of bunk!
Here's another:
http://www.cato.org/pubs/pas/pa-346es.html
Here's a pdf version of the detailed report (long download time):
http://www.cato.org/pub/pas/pa346,pdf
Here's another one on the same theme:
http://www.motorists.com/ma/globestory.html
In 1973, when the 55 MPH law was enacted because of the oil crisis,
the highway fatality numbrs dropped drastically over the next decades
during which the 55 mph was a FARCE. But the REAL causes were --
people drove less (in 1973) becuase of high gas price AND lowered
speed; later there were better safety equipments on automobiles such
as seat belt, air bags (not the politicians <g>), and other NEW
safety equipments enacted into law or voluntarily put into effect
by the auto industry.
We are at another oil crisis now. I haven't heard any politician
lately about dropping the speed limit to save oil and safe lives. :-)
Drop the max speed to 10 mph nationwide, on interstate, state,
rural, and all roads, and I'll GUARANTEE you that the highway
fatality rate will drop to virtually ZERO. Would you attribute
the REAL cause to the "speed" or something else (lots of other
things). Conversely, if Bush remove ALL speed limits on
Interstate highways, do you think he'll wipe out the population
of the USA by highway fatalities?
It didn't work on autobahns in Europe. :-)
Drivers in the USA are already diving about as fast as their
cars are capable of doing, and beyond. :-)
To put SPEED as the SOLE CAUSE was exactly the kind of mindless
conclusion from UNCONTROLLED experiments and callous analysis of
DATA.
As a matter of fact, there WERE controlled data, but hardly anyone
looked or scrutized them, except for the sports car magazines and
those of us who kept getting penalized by speeding fines and
insurance increases!
When the NATIONAL limit was dropped from 75 to 55, there were SOME
states that were NOT affected by the change (those whose max
speed were ALREADY 55). The SAME drop in other states were
observed in those states as well!!! If SPEED were the sole cause,
then the fatality of those states shouldn't have changed at all
when the national limit was dropped but it remained the same in
THOSE states.
THAT is the problem with the simplistic view of those who look at
correlations, make assumptions, and think that it can establish
some causal inference without appropriate CONTROL of the relevant
factors.
So, what happened when the limit was raised to 70? Folks in
the Atlanta area and the Florida Interstates drive at speeds
somewhere between 75 and 100 mph. :-) I got TWO tickets
this year at speeds over 25 mph over the posted interstate limits
and it had NOTHING to do with safety whatsoever, when I car and
tires were equipped to cruise in excess of 100 mph indefinitely
and I've never had an accident that was my fault EVER (that's
over 40 years of driving). My car was hit at the rear when it
was stopped at red light traffic a couple of times, at 0 MPH. :-)
Don't get me started (well, you already did) on the stupidity of
all the laws and false conclusions folks got from IMPROPER
use of statistics and failure to understand the necessary
methodolgy to ascertain CAUSE is much more than just scratching
some arrows and diagrams on a little piece of paper, without
doing ANY necessary controlled experiments.
>
> > others, on the efficacy of EYEBALLING the approproach plots and
> > nothing else.
>
> There is a problem with eyeballing, though. This is like sweeping the
> problem under the carpet. Yes, it's good to eyeball, but where does
the
> experience come from? I bet that a significance test is better than
at
> deciding incidence fron coincidence than an eyeballing by newbie that
> has seen 5 scatterplots in his life.
I would repeat the same below:
>
> > Here's Herman Rubin's line --------------> step over there please.
>
> Herman disagrees with my approach :) To save his time,
Then let Herman tell you where to go. :-))
-- Bob.
Aleks Jakulin
> Absolutely NOT! That's one of my points in the review. If X is
> spuriously correlated with Y, say positively, but X is definitely
> NOT the causal factor of Y, then no matter how much data you observe
> (with ocntrol), you can never DISPROVE the (assumed) causal
> relation from the voodoo skethces (path diagrams).
That's the domain of priors and nulls.
Anyway, I've been meaning to post these three links:
http://singapore.cs.ucla.edu/LECTURE/lecture_sec1.htm
http://singapore.cs.ucla.edu/IJCAI99/
http://bayes.cs.ucla.edu/BOOK-2K/index.html
One can argue for or against, but it's nicely written, if one accepts
that this material is logic, not statistics (in spite of the "bayes"
word all over the place).
> The government and other gullible non-thinkers had been fooled
> for nearly 30 years on the false causal "assumption" that lower
> maximum speed on Interstate saves lives, as if that was the
> DIRECT cause, and that RAISING back the speed from 55 to 70
An interesting story.