Testing for Normality
Margaret
I would like to start a discussion on tests of Normality - in
particular, with respect to what you perceive to be best practice.
Personally, I feel that there is a lot to be said for having a raw
measure of Normality by means of histograms with Normal curves fitted
and then to compare the results obtained thereby with those of more
sophisticated procedures, such as the Shapiro-Wilk or
Kolmogorov-Smirnov test or in some cases, values of skewness and
kurtosis.
However, all in the garden is not always rosy. I have just witnessed a
case for a sample size of about 200 where the Shapiro-Wilk test results
contradict the Kolmogorov-Smirnov test results. I understand that these
tests test for Normality in different senses but this does not help me
greatly in deciding whether it is a good idea to use them at all. In
turn, the Shapiro-Wilk test results were not backed up by the
histograms or by Q_Q plots of residuals weights.
Clearly, if one is to rely on skewness and kurtosis they need to opt
for sensible lower and upper bounds as conditions for Normality but my
reading has shown me that recommendations differ a lot and this has
dissuaded me from using these measures of Normality (so as to avoid
being too subjective).
To add to my dilemma, whilst SPSS provide us with both the Shapiro-Wilk
test and the Kolmogorov-Smirnov test, in their publication "SPSS 13.0
Statistical Procedures Companion", they advise:
"Warning: If your sample size is large, even small deviations from
normality that won't affect your analysis may cause you to reject the
null hypothesis that the samples come from normal plots. Make your
decisions based on the diagnostic plots."
This leads to questions like, what does "large" mean?
Finally, from what I have read, there is some conflict over sample size
criteria for deciding whether the Shapiro-Wilk test or the
Kolmogorov-Smirnov test is the more appropriate one to choose. SPSS
does not immediately help us here.
Having researched the various rules and maxims and witnessed some
inconsistencies, I would be interested to learn if anyone has developed
an algorithm to help them to decide which procedure is best and when.
Sorry for a long message but perhaps the reader would kindly note that
whilst I appreciate that there are plenty of well-recognized possible
procedures, I am not so clear on best practice. Moreover, since the
results for these procedures don't always agree, in the absence of a
well-researched methodology for choosing the correct procedures, it
could be all too easy to choose the procedure which tells us what we
wish it to. Therefore the question of best practice is rather important
here.
Thank you very much for sharing your views
Regards
Margaret
Bland, M.
given: small samples will give non-significant results because they are
small and large samples give significant ones because they are big. If
the data look Normal, I doubt that any departure from the Normal present
will upset t tests, etc., because they are robust. If the sample is
large, small departures from the Normal, even if we know they are there,
are usually irrelevant. I think a Normal plot is much more useful.
Martin
Margaret wrote:
--
***************************************************
J. Martin Bland
Prof. of Health Statistics
Dept. of Health Sciences
Seebohm Rowntree Building Area 2
University of York
Heslington
York YO10 5DD
Email: mb...@york.ac.uk
Phone: 01904 321334
Fax: 01904 321382
Web site: http://www-users.york.ac.uk/~mb55/
***************************************************
Patrick Musonda
reliable in my view, sometimes tests like those mentioned by margaret can be
used to substantiate the eyeball test.
Patrick.
Patrick Musonda
reliable in my view, sometimes tests like those mentioned by margaret can be
used to substantiate the eyeball test.
Patrick.
>From: "Bland, M." <mb...@york.ac.uk>
>Reply-To: MedS...@googlegroups.com
>To: MedS...@googlegroups.com
>Subject: MEDSTATS: Re: Testing for Normality
>Date: Fri, 09 Dec 2005 13:43:09 +0000
>
>
Jeremy Miles
Hi Margaret,
Nice to meet you briefly at the SPSS conference the other day - it's
always good to put faces to email names.
A paper I rather like (at least in title) is: Micceri, T. (1989). The
unicorn, the normal curve, and other improbable creatures. Psychological
Bulletin, 105(1), 156-166.
If I were worried, I would use a robust estimator, or bootstrap.
It's worth playing around with sample sizes and distributions to see
what matters.
Here's a bit of SPSS syntax:
input program.
loop #case = 1 to 100 * 50.
compute vnorm =NORMAL(1).
compute vchisq1 = rv.chisq(1) - 1.
compute vchisq2 = rv.chisq(2) - 2.
end case.
end loop.
end file.
end input program.
compute group = mod($casenum, 100).
FREQUENCIES VARIABLES=vnorm vchisq1 /FORMAT=NOTABLE /HISTOGRAM .
GRAPH /ERRORBAR( CI 95 )=vnorm BY group .
GRAPH /ERRORBAR( CI 95 )=vchisq1 BY group .
GRAPH /ERRORBAR( CI 95 )=vchisq2 BY group .
It generates 100 sets of data, of sample size 50, from three
distributions: normal, chi-square with 1 df (very highly skewed,
skewness = approx 3), and chi-square with 2 df (highly skewed, skewness
= approx 2). The mean of each set is zero.
It then draws three graphs, one for each variable, of the mean, with 95%
confidence intervals. We would expect that in 95% of the samples, the
population mean (zero) would lie within the CIs. If fewer (or more,
even) do not lie within the CIs, then the assumption would seem to have
been violated sufficiently badly that it matters.
Change the sample sizes by changing the number 50, on the 2nd line.
Sample sizes of 50 don't do very well.
A sample size of 100 does fine, even with highly skewed data.
Because I don't have to do what I thought I was going to have to do this
afternoon, a couple more thoughts.
First, it's often not the data that are assumed to be normally
distributed, it's the residuals, so be careful about that.
Second, Wilcox and other proponents of robust estimation, say that it's
not skew that's the problem, it's mixture distributions. I think it's
in his 1997 Introduction to robust estimation and hypothesis testing
(London, Academic Press) that he gives an example of a mixture
distribution. 90% of the sample is sampled from a population where the
mean is 0 and the sd = 1, and 10% from a population where the mean is 0
and sd = 10. In this case, there is no skew, but the percentage of
times the 95% CIs include the population mean veers away from what we
would expect.
Here's another simulation (written this time in R, for the sake of
variety - obviously I'm recycling these, and R must have suited my needs
at the time). With the values described above, we tend to get about 97%
of the samples include the population mean - as this is too high, it
means we will have reduced power.
sampleSize <- 40 #Set size of samples
nSamples <- 100000 #set number of samples
lowerSD <- 1 #set SD for group with lower SD
upperSD <- 10 #setSD for group with higher SD
proportionInUpper = 0.1 #set proportio of people with higher SD
data <- rnorm(sampleSize * nSamples, 0, lowerSD) + rnorm(sampleSize *
nSamples, 0, (upperSD - lowerSD)) * rbinom(n = sampleSize * nSamples,
size = 1, p = proportionInUpper)
dim(data) <- c(nSamples, sampleSize)
sampleMeans <- rowSums(data) / sampleSize
xAxisLabels <- seq(min(sampleMeans), max(sampleMeans), length = 100)
xAxisLabels <- xAxisLabels - mean(sampleMeans) / sd(sampleMeans)
NormalHeights <- dnorm(xAxisLabels, mean = mean(sampleMeans), sd =
sd(sampleMeans))
#hist(sampleMeans, probability = TRUE)
heights <- table(round(sampleMeans, 2))
plot(table(round(sampleMeans,1 )))
lines(xAxisLabels, NormalHeights * (nSamples / 10), col = "red")
dataSquared <- data^2
dataRowSums <- rowSums(data)
dataRowSumsSquared <- rowSums(dataSquared)
sampleSD <- sqrt((sampleSize * dataRowSumsSquared - dataRowSums^2) /
(sampleSize * (sampleSize - 1)))
sampleSE <- sampleSD/sqrt(sampleSize)
lowerCIs <- sampleMeans - (sampleSE * (qt(0.975, sampleSize - 1)))
upperCIs <- sampleMeans + (sampleSE * (qt(0.975, sampleSize - 1)))
within95CIs <- lowerCIs < 0 & upperCIs > 0
table(within95CIs)
Jeremy
--
Jeremy Miles
mailto:jn...@york.ac.uk http://www-users.york.ac.uk/~jnvm1/
Dept of Health Sciences (Area 4), University of York, York, YO10 5DD
Phone: 01904 321375 Mobile: 07941 228018 Fax 01904 321320
Ted Harding
>
> Peronally, I prefer the eyeball test for the reasons you have
> already given: small samples will give non-significant results
> because they are small and large samples give significant ones
> because they are big. If the data look Normal, I doubt that any
> departure from the Normal present will upset t tests, etc.,
> because they are robust. If the sample is large, small departures
> from the Normal, even if we know they are there, are usually
> irrelevant. I think a Normal plot is much more useful.
>
> Martin
I can't disagree with the above general comments! As Marting hints,
the "bottom-line" question is: Even if non-Normal, does it matter?
You may have a large dataset where a test for non-Normality gives
an earth-shatteringly tiny P-value, but the extent of non-Normality
would change the P-value of a t-test from 0.0106 to 0.0107 if the
P-value were calculated from the "true" non-Normal distribution
rather than an exactly Normal distribution. Does this matter?
Whether it matters or not can be, in principle, evaluated by
carrying out a robustness study: For alternative distributions
exhibiting degrees of non-Normality compatible with the data,
what corresponding results emerge? If all these results are
within limits that you are prepared to ignore, then the Normality
issue is indeed irrelevant.
However, if more than a small proportion give rise to differences
that you feel you cannot ignore, then you face a dilemma! Basically,
if your true distribution is one of the ones that give non-ignorable
discrepancies, then you should not rely on results obtained by
assuming normality; but if it is one of those which give ignorable
discrepancies, then you could rely on them. But how do you know
which it is? Probably, you have no further evidence enabling you
to choose. So you can only remain uncertain. You cannot decide
this question.
Further comments interleaved below.
> Margaret wrote:
>
>>Dear all
>>
>>I would like to start a discussion on tests of Normality - in
>>particular, with respect to what you perceive to be best practice.
>>
>>Personally, I feel that there is a lot to be said for having a raw
>>measure of Normality by means of histograms with Normal curves fitted
>>and then to compare the results obtained thereby with those of more
>>sophisticated procedures, such as the Shapiro-Wilk or
>>Kolmogorov-Smirnov test or in some cases, values of skewness and
>>kurtosis.
Tere I'd like to remind (or inform) people about Tukey's "hanging
rootogram" here. Procedure: Fit a normal law to the data; Make
a histogram of the data; From the fitted normal law, evaluate
the expected counts in the histogram cells; Take square roots
of the observed and expected counts; Hang the tops of the
square-root-observed histogram bars off the tops of the
square-root-expected histogram bars; Draw horizontal lines at
+/- 0.5 either side of y=0; And then eyeball the bottoms of
the hanging bars relative to these two lines. This can be very
revealing of systematic departures from Normality, and is a
useful adjunct to other graphical methods such as Q-Q plots.
For this, and other very good ideas for graphical display, see:
John W. Tukey, Some Graphic and Semigraphic Displays.
In: T.A. Bancroft, ed., Statistical Papers in Honor of
George W. Snedecor, (Ames, Iowa 1972), pp. 293-316,
which can be downloaded from Edward Tufte's website:
http://www.edwardtufte.com/tufte/tukey
Tufte's whole website
http://www.edwardtufte.com/tufte/
is well worth a visit for ideas and exmaples about the presentation
of information.
>>However, all in the garden is not always rosy. I have just witnessed a
>>case for a sample size of about 200 where the Shapiro-Wilk test results
>>contradict the Kolmogorov-Smirnov test results.
I think "contradict" is too strong a word! They may give different
results but (as you note below) this need not be surprising since
they are sensitive to different kinds of departure from Normality.
The K-S test is based simply on the maximum discrepancy between
the empirical cumulative distribution function of the sample,
and the theoretical CDF of the fitted Normal distribution, at
the sample values. Therefore any variation within the empirical
CDF which does not change the maximum discrepancy will give the
same result (i.e. the K-S test is not sensitive to such variations).
The S-W test is based on
Sum[i = 1..n]{(a[i]*(x[n-i+1] - x[i]))^2}/s^2
where s is the sample SD, i.e. on the "standardised" differences
(x[n]-x[1])/s, (x[n-1]-x[2])/s, (x[n-2]-x[3])/s between the ordered
sample values x[1]<x[2]<...<x[n] pairing them off from the outside
inwards. In particular, any shifting of the x-values which keeps
all of these standardised differences comstant will have no effect
on the S-W test. Such variations do not coincide with the variations
which have no effect on the K-S test.
As a special case, you could move each pair around bodily, keeping
them a constant distance apart, subject to preserving the value of s.
In this way, you could introduce skewness up to a certain degree
which S-W would simply not be able to detect. On the other hand,
if skewness is the only "non-Normal" feature of the data, then
K-S would be influenced by it (but this could be out-ranked by
kurtosis if also present, in which case K-S would not be
sensitive to mild variations in kurtosis).
Conversely, there are variations in sample values which markedly
change K-S without much affecting S-W, and vice versa.
>>I understand that these
>>tests test for Normality in different senses but this does not help me
>>greatly in deciding whether it is a good idea to use them at all. In
>>turn, the Shapiro-Wilk test results were not backed up by the
>>histograms or by Q_Q plots of residuals weights.
As indicated above, a certain degree of skewness would not be picked
up by S-W, but would show up clearly on a Q-Q plot (not that skewness
is the only feature to which S-W, up to a point, would be insensitive).
>>Clearly, if one is to rely on skewness and kurtosis they need to opt
>>for sensible lower and upper bounds as conditions for Normality but my
>>reading has shown me that recommendations differ a lot and this has
>>dissuaded me from using these measures of Normality (so as to avoid
>>being too subjective).
Whenever you expose yourself to having to carry out a formal test,
you are inevitably being subjective, if only because the mere choice
of a P-value as threshold for "significance" is an entirely
subjective matter.
Asserting "significance" is a *decision" -- etymologically "cut-off",
in that you draw a line somewhere in a grey area. Where you draw the
line is up to you.
More generally, the choice of one kind of statistical test rather
than another is a matter of what features of possible data you
want the test to be sensitive to. Ideally, there will be evidence
based on information coming from outside the data which will give
you expectations as to what kind of departures from assumptions
are likely to be present (if present at all), and as to what kinds
of departures are not likely to be present. In that case you can
choose a test which, by design, is sensitive to those which are
likely to occur and is not sensitiver to those which are not likely
to occur. To that extent, the choice is not entirely subjective.
In practice, however, it is often the case that such external
evidence is absent of scanty. In such circumstances, people
often rely on (possibly informed) intuition).
There is another (amongst many) dimension of subjectivity,
as indicated above at the start. Does, for instance, the effect
of non-Normality really matter for the results you obtain?
This is a quantitative issue, and the question is whether the
amount of difference is important.
At the end of the day, such an appraisal is "political": does
an alleviation-rate of 90% vs 85% really matter, for the
symptoms of the common cold? Does it matter for the symptoms
of schizophrenia?
>>To add to my dilemma, whilst SPSS provide us with both the
>>Shapiro-Wilk test and the Kolmogorov-Smirnov test, in their
>>publication "SPSS 13.0 Statistical Procedures Companion",
>>they advise: "Warning: If your sample size is large, even
>>small deviations from normality that won't affect your
>>analysis may cause you to reject the null hypothesis that
>>the samples come from normal plots. Make your decisions
>>based on the diagnostic plots."
>>
>>This leads to questions like, what does "large" mean?
It means, relative to what "even small deviations" means,
large enough for a significant (e.g. subjegtively chosen
to 5% P-value) result to be likely for that small deviation.
As to what "small" means, well, that's popssibly subjective ...
>>Finally, from what I have read, there is some conflict over
>>sample size criteria for deciding whether the Shapiro-Wilk
>>test or the Kolmogorov-Smirnov test is the more appropriate
>>one to choose. SPSS does not immediately help us here.
Since the two tests are sensitive to different kinds of things,
while both will, for sufficiently large samples in each case,
ultimately detect non-normality of any kind, if you apply
K-S to a situation yielding non-Normality of a kind to
which K-S is sensitive but not S-W, you would have to use a
much larger sample size to get a good result with S-W; and
vice versa. So it's not surprising to find conflicting advice,
since what the advice should be depends on what is going on.
>>Having researched the various rules and maxims and witnessed
>>some inconsistencies, I would be interested to learn if anyone
>>has developed an algorithm to help them to decide which procedure
>>is best and when.
Probably it clear by now that I don't envisage a general
algorithm in the sense of something straightforwardly computed.
It all depends on how much you know about what might happen,
and what might not happen, in the situation you are looking at.
If you're coming to it with an open mind and no prior evidence,
then you should at least be looking at the data graphically
as suggested by Martin (and by Tukey, etc.).
Also, there's nothing wrong, in itself, in simply reporting
different features of the data (e.g. skewness, kurtosis,
bimodality, ... ) as individually "significant" relative to
what is likely under a Normal distribution -- all you are
doing is describing features of the data, i.e. representing
them, just as a histogram or Q-Q plot represents them.
You need to watch, however, that others do not then pick these
up and run with them as if they had an interpretation which
they do not really bear. But that's another "political"
question: Just watch any politician "interpreting" data ...
"There has been a significant increase in crime under the
current Government" -- errr, what does this mean?
>>Sorry for a long message but perhaps the reader would kindly
>>note that whilst I appreciate that there are plenty of
>>well-recognized possible procedures, I am not so clear on best
>>practice. Moreover, since the results for these procedures
>>don't always agree, in the absence of a well-researched methodology
>>for choosing the correct procedures, it could be all too easy to
>>choose the procedure which tells us what we wish it to. Therefore
>>the question of best practice is rather important here.
Best practice is: Know the form and act accordingly!
This is all rather general, of course, and a more pointed
commentary could be made with more detailed knowledge of the
context.
All best wishes,
Ted.
--------------------------------------------------------------------
E-Mail: (Ted Harding) <Ted.H...@nessie.mcc.ac.uk>
Fax-to-email: +44 (0)870 094 0861
Date: 09-Dec-05 Time: 15:38:51
------------------------------ XFMail ------------------------------
Robert Newcombe
Robert G. Newcombe PhD CStat FFPH
Professor of Medical Statistics
Wales College of Medicine
Cardiff University
Heath Park
Cardiff CF14 4XN
Phone 029 2074 2329
Fax 029 2074 2898
http://www.cardiff.ac.uk/medicine/epidemiology_statistics/research/statistics/newcombe.htm
>>> Ted.H...@nessie.mcc.ac.uk 09/12/05 15:38:55 >>>
Alexandre Santos Aguiar
> Peronally, I prefer the eyeball test for the reasons you have already
> given: small samples will give non-significant results because they are
> small and large samples give significant ones because they are big. If
Is the eyeball test good enough to prove normality from a simulation?
I mean, for example, extracting 10,000+ t statistics in a Monte Carlo
simulation with the explcit intention of proving normality for a certain
sample size of a tipically non-normally distributed variable (say, Glasgow
Coma Scale as in a 2003 paper). What is the best approach in this setting?
Thanks,
Alexandre
Swank, Paul R
Paul R. Swank, Ph.D.
Professor, Developmental Pediatrics
Director of Research, Center for Improving the Readiness of Children for Learning and Education (C.I.R.C.L.E.)
Medical School
UT Health Science Center at Houston
-----Original Message-----
From: MedS...@googlegroups.com [mailto:MedS...@googlegroups.com] On Behalf Of Alexandre Santos Aguiar
Sent: Monday, December 12, 2005 2:43 PM
To: MedS...@googlegroups.com
Subject: MEDSTATS: Re: Testing for Normality
Alexandre Santos Aguiar
> Are you talking about estimating the normality of a sampling distribution?
Yes.
> If the sampling distribution is relatively symetric and unimodal than using
> normal approximation will not be far off.
But how to state that in a paper? Without formal testing is enough?
Alexandre
Ted Harding
I would not accept Paul's "rule of thumb" without looking more closely.
A t distribution is indeed "relatively symmetric and unimodal",
but can be a long way of normal -- especially in the tails -- for
small degrees of freedom. For example, with d.f. = 2:
P t scaled t Normal
0.001 -22.327 -15.175 -3.090
0.005 -9.925 -6.745 -2.576
0.010 -6.965 -4.733 -2.326
0.050 -2.920 -1.985 -1.645
0.100 -1.886 -1.282 -1.282
0.250 -0.816 -0.555 -0.674
0.500 0.000 0.000 0.000
0.750 0.816 0.555 0.674
0.900 1.886 1.282 1.282
0.950 2.920 1.985 1.645
0.990 6.965 4.733 2.326
0.995 9.925 6.745 2.576
0.999 22.327 15.175 3.090
Here P is the probability, t is such that P(<=t) = P for a
t distribution with 2 d.f., the 4th column is the same for
a Normal dsistribution, and "scaled t" is "t" scaled so as
to coincide with "Normal" at P=0.1 and 0.9.
As you can see, even with the scaling they diverge greatly for
more extreme values of P, and it is much worse for unscaled t!
Best wishes,
Ted.
--------------------------------------------------------------------
E-Mail: (Ted Harding) <Ted.H...@nessie.mcc.ac.uk>
Fax-to-email: +44 (0)870 094 0861
Date: 13-Dec-05 Time: 02:45:32
------------------------------ XFMail ------------------------------
Swank, Paul R
distribution will be much closer to normal than will the original
distribution. It seems the issue here is whether the sampling
distribution of a nonnormally distributed variable will be normal enough
to use a normal approximation. The problem is that with a large number
of observations, even tiny deviations from normality will be
statistically significant. A better test would be to figure out how much
deviation from normality will be tolerated. The maximum value is the
effect size. Then draw a enough samples to insure that a bigger
difference than that will be significant 90 or 95% of the time. Then you
would have a reasonable test. The hard part, of course, is step number
1.
Paul R. Swank, Ph.D.
Professor, Developmental Pediatrics
Director of Research, Center for Improving the Readiness of Children for
Learning and Education (C.I.R.C.L.E.)
Medical School
UT Health Science Center at Houston
-----Original Message-----
From: MedS...@googlegroups.com [mailto:MedS...@googlegroups.com] On
Behalf Of Ted.H...@nessie.mcc.ac.uk
Sent: Monday, December 12, 2005 8:46 PM
To: MedS...@googlegroups.com
Subject: MEDSTATS: Re: Testing for Normality
Margaret
printed off the details of the discussion and plan to have a good read
over Christmas before replying to you individually.
I can see that there is lots here to give me food for thought!
Best wishes
Margaret