For nonrejection of H0, don't we want high signifance?

[Paul]

I'm perusing http://en.wikipedia.org/wiki/Anderson%E2%80%93Darling_test#Test_for_normality
for a statistical test of normality for my residuals. H0 is normality
of the residuals.

For typical hypothesis testing, we want small significance, which
means a small rejection region. Thus, and value of the statistic that
falls in the rejection region is less likely due to chance (in
combination with the truth of H0). In testing a drug for a medical
effect, that makes sense because we often want to demonstrate an
effect, and H0 is typically the absence of an effect. For values of
the statistic that fall in the small rejection region, we can say that
if H0 is true, it is highly unlikely for us to get this value for the
statistic. The smaller the significance, the smaller the rejection
region, and less we are able to attribute the chance any values in the
rejection region.

For normality, we often want the opposite. We want H0, which is
normality of the residuals. We can not accept H0 to any degree of
confidence using this setup of hypothesis testing, but at least we can
make it very easy to reject H0 so any non-rejection of H0 is seen to
be well founded. This implies large rejection region and high
significance. In fact, we might want to a 95% rejection region, the
counterpart of the wanting a 5% rejection region when the intent is to
demonstrate that rejection of H0 is not due to chance.

Is this reasonable? I ask because the table in the above link shows
significance values 1%, 2.5%, 5%, 10%, and 15%. These small values
seem more like the values that one might be interested in when wanting
to demonstrate valid rejection of H0.

[Rich Ulrich]

No, it is not reasonable in the context of testing.

"Nothing is really normal." Any real data that you collect (or
any residuals that you want to look at) are going to look
non-normal -- at any p-value that you want to consider -- if
you can take a sample N that is large enough.

That's part of why "tests of normality" are not used much by
practicing statisticians. There are usually more obvious clues
(source of the data; eyeballing the numbers) that a test is
going to be affected.

- The F-test is pretty darn robust, once you have a large
enough sample. If the sample is small enough that normality
really matters, it is too often too small for the test of normality
to have enough power to detect the non-normality.

- Furthermore, the nature of the non-normality will matter.
Hetergeneity across the range of prediction warns about the
validity of the whole model, and can be important than outliers;
but one or two very *extreme* outliers can sabotage the error
term, rendering testing useless.

Independent t-test.
S1 = (1,2,3,4,5); S2= (6,7,8,9,10).
- t-test, probably significant.
Replace the highest value, 10, in the higher group with
1000. t-test is no longer significant or even suggests a
difference.

--
Rich Ulrich

[Paul]

On Apr 23, 3:51 pm, Rich Ulrich <rich.ulr...@comcast.net> wrote:
> On Tue, 23 Apr 2013 12:15:20 -0700 (PDT), Paul
>
>
>
>
>

> <paul.domas...@gmail.com> wrote:
> >I'm perusinghttp://en.wikipedia.org/wiki/Anderson%E2%80%93Darling_test#Test_for_n...

OK, I get it. There are practical considerations at play. However,
from a conceptual standpoint, say H0 was some generic hypothesis
rather than residual normality. My question about whether the
significance should be small or large depending on whether I'm
interested resoundingly showing the resonableness of rejecting or not
rejecting H0 still seems to hold. Would you (or anyone else) be able
to chime in under this modified scenario?

[David Jones]

"Paul" wrote in message
news:41b75327-afe1-488e...@z10g2000yqd.googlegroups.com...

========================================================

You can't try to impose to impose a false symmetry between the null and
alternative hypotheses in anything like this way. For ALL reasonable tests,
for fixed sample size, lowering the probability of rejecting the null
hypothesis when it is true (changing the significance level) also reduces
the probability of accepting a fixed alternative hypothesis under the
assumption that hypothesis is true.

If you were dead keen on using "normal distribution" as the alternative
hypothesis .... so that there is only a small chance of reaching that
conclusion if a certain different null hypothesis is true, then you could
set up such a test , but this would depend heavily on the null hypothesis
distribution you choose . If you wanted to follow this up you could look up
"tests of separate families of hypotheses" to see one possible approach. The
basic sources for these illustrate that testing H0=HA against H1=HB is not
the opposite of testing H0=HB against H1=HA and you can get results where
both are accepted, both rejected, or only one accepted.

But there would still be radically different results from testing, for
example : H0=Cauchy vs H1=Normal, compared with H0=Laplace (double
exponential) vs H1=Normal.

None of such tests would be at all like an Anderson-Darling test. One basic
point here is that while the the null hypothesis distribution can be (and
often is) a special case of the alternative hypothesis, the alternative
hypothesis can't be a special case of the null hypothesis (as otherwise the
alternative hypothesis would always be true if the the null hypothesis is
true. So the Anderson-Darling test can test H0:Normal against H1:any
possible distribution, it can't be used (and there is no test that can be
used) to test H0=any possible distribution against H1=Normal.

In the above, H0 means the specifically designated null hypothesis for a
given test.


For the general version of your question, significance tests are not framed
in terms either of "showing the reasonableness of rejecting or not
rejecting H0" (whatever that might mean). They are based on controlling the
probability of rejecting the null hypothesis when it is true ... usually
with a fixed significance level, but it is possible to try to compromise
between using a higher false-rejection probability in order to get higher
power for some alternative. But you would have to have a good reason for
doing this and would still need to limit the significance level to a
reasonable level, otherwise you might just as well always accept the
alternative hypothesis and not bother with testing. For example, if you were
doing an initial screening of an extremely large number of different
treatments for some medical condition, in a situation where is is unlikely
that a randomly chosen treatment would be effective , you might want to
reduce the number for further testing to 10% and so choose in the initial
stage to test for effectiveness (against H0:noneffective) at the 10% level.
There would be little point in testing at a 90% significance level as that
wouldn't reduce the work-load at the second stage much. An alternative here
would be to specify the required power for a given size of effect
(probability of detecting an effect if the effect is larger than a given
value) and work back to a significance level to achieve this. This might
lead to a large proportion of treatments being taken forward for further
testing. In either approach, one could consider improving the initial
testing stage (for example by increasing the sample size) in order to
achieve a good power for a reasonably low rate of treatments being passed to
the next stage (low significance level).

So ... high significance levels might be reasonable in some cases but you
would need to have a good reason to use one ... at least they are not
theoretically impossible, but you would be close to needing to answer "why
do the test at all, instead of always rejecting the null hypothesis?". You
need to remember the essentially non-symmetric nature of the null and
alternative hypotheses, which is imbued in "accept the null hypothesis
unless there is sufficient evidence to reject it ". You are free to switch
the null and alternative around if that suits your line of thinking in a
given instance, leading to "accept the NEW null hypothesis unless there is
sufficient evidence to reject it ", but such a test would not necessarily be
logically valid and, even if valid, would not be the logical opposite of the
original test

David Jones

[Jeff Miller]

On Wednesday, April 24, 2013 11:31:11 AM UTC+12, Paul wrote:

> from a conceptual standpoint, say H0 was some generic hypothesis
> rather than residual normality. My question about whether the
> significance should be small or large depending on whether I'm
> interested resoundingly showing the resonableness of rejecting or not
> rejecting H0 still seems to hold.

I think your basic intuition is right: Other things being equal, a _larger_ p value tends to indicate that the data are _less discrepant_ from what Ho would predict.

But this view is so oversimplified that it can easily be misleading. For example, the p value might be large because there isn't much data or because there is lots of random error, and in neither case should you conclude that the data provide good support for Ho.

When the research goal is to show that Ho is true to a good approximation, Ho testing is just not a good method. The confidence interval approach is much more informative in most such situations, because they allow conclusions of the form "Ho is not wrong by more than X units". CIs are not easily adapted to your original question of checking for normality, though, AFAIK.

[David Jones]

"Jeff Miller" wrote in message
news:9de41597-61aa-4417...@googlegroups.com...

The confidence interval approach is much more informative in most such
situations, because they allow conclusions of the form "Ho is not wrong by
more than X units". CIs are not easily adapted to your original question of
checking for normality, though, AFAIK.

========================================================================

CIs are "easy" to apply here, in one of two ways. CIs are just things that
contain the set of all null hypotheses that are not rejected based on the
data. So either:

(i) form a discrete collection of possible families of distributions, test
each of these as the null hypothesis and form a list of all those that are
not rejected. This list would indicate how much non-normality is not
excluded by the data available.... but it clearly requires both wide-ranging
behaviours in the original list and an understanding of these behaviours if
the CI s to be useful.

(ii) embed the normal distribution in a 3 or 4 parameter family of
distributions, with the extra parameters representing departure from
normality. Then an "ordinary" confidence region can in indicate how much
non-normality is not excluded by the data available.

David Jones

[Herman Rubin]

This is what the research goal should be, but I do not know
of any work on this other than my paper, which leaves much to
be dessired. I considered the Bayesian testing of a slightly
spread out null against a much more spread out alternative.
What I found is that if the spread of the region in which one
would want to accept is small relative to the standard error
of estimate, one can treat it as a point null. If it is
large relative to the standard error, just look at the value.
In between, it depends on the nature of the spread of the
weight measure for acceptance, and I have not in the 40+
years since I go this result found a good way of approximating
the questions to the user to get robustness.


--
This address is for information only. I do not claim that these views
are those of the Statistics Department or of Purdue University.
Herman Rubin, Department of Statistics, Purdue University
hru...@stat.purdue.edu Phone: (765)494-6054 FAX: (765)494-0558