Statistical testing
Tim Frink
I've a question about on statistical hypothesis testing. Let's say I've
two approaches A and B and I would like to see if one approach is better
than the other.
These are my results (p-values) for the statistical testing:
A B
A - 0.002
B 0.998 -
using the alternative hypothesis that the approaches in the first column
are better than the approaches in the 2. and 3. column. The significance
level was 5%.
Is my interpretation correct that statically significant differences are
observed in the second row, i.e. it can be inferred that A outperforms B
(since 0.002 < 5%)?
If on the other hand, the table would be a result of significance testing
using the null hypothesis that approaches in the first column perform as
well as approaches in the 2. and 3. column. Could it be then inferred
that
a) the null hypothesis can be rejected for 0.002 (since smaller than
significance level), i.e. A and B do not perform equally
b) the null hypothesis is valid for 0.998, i.e. B and A perform equally?
So, is the interpretation of the p-value w.r.t. to a given significance
level oppositional for the null and alternative hypothesis? Can it be
said in general that if the null hypothesis is rejected then the
alternative hypothesis can be accepted?
Regards,
Tim
Ray Koopman
Your presentation is so non-standard as to be incomprehensible.
How you got those p-values?
Rich Ulrich
I agree with Ray that this is a non-standard presentation, and it
is possible that I don't understand it. What I take it for is this --
It is a very wordy presentation of one-tailed testing, with
some questions about "what does it mean."
>So, is the interpretation of the p-value w.r.t. to a given
> significance level oppositional for the null and alternative
> hypothesis? [snip, perhaps a restatement...]
"oppositional"? is that a word? Does that mean, "Are there
only two choices?" - Accept the null (because there is not
sufficient evidence to reject it); or accept the alternative,
for example "B is greater than A" according to evidence
using a test at the X% level.
In a designed experiment, we typically can accept the null
or accept the alternative. In observational studies, that is
a bit too sloppy -- we have to be clear that the alternative
is stated something like, "B is greater than A, or there are
influences that we have not yet accounted for that makes it
look that way."
- Designed studies are not always perfect and sometimes
need some of the same caution in interpretation. For instance,
subjects in clinical trials cannot always be kept blind to treatment
group, and double-blind (keeping all raters 'in the dark') can be
almost as difficult.
Does that cover it?
--
Rich Ulrich
Art Kendall
Rich said:
In a designed experiment, we typically can accept the null or accept the alternative. In observational studies, that is a bit too sloppy -- we have to be clear that the alternative is stated something like, "B is greater than A, or there are influences that we have not yet accounted for that makes it look that way."
At a given amount of statistical differentiation here are two decision outcomes (verdicts) not proven (not guilty) and proven (guilty). One can stick with the null, or go with the alternative. If the evidence does not make the Hs sufficiently statistically distinguishable, the the null H remains in place.
Art Kendall
Social Research Consultants
Rich Ulrich
wrote:
>Just to pick a nit. This is a very rare situation for me to disagree
>with him.
Art,
I was afraid that I would not say it very well, but I thought
I would try. I wanted to say more like what you say here, but
I wrote up null vs. alternative as appearing to be a symmetrical
choice, which it is not. As you point out, null is default.
I hope that I will start from that basis, next time.
>
>Rich said:
>
>> In a designed experiment, we typically can accept the null
>> or accept the alternative. In observational studies, that is
>> a bit too sloppy -- we have to be clear that the alternative
>> is stated something like, "B is greater than A, or there are
>> influences that we have not yet accounted for that makes it
>> look that way."
>In designed experiments, and even in quasi-experimental studies the null
>hypotheses is accepted a priori. The null H represents the currently
>accepted (status quo ante, presumed) theory/policy/practice/verdict. It
>is presumed unless the evidence in support of the alternative H about
>theory/policy/practice/verdict is shown to be acceptable at a given
>level of certainty.
>
>At a given amount of statistical differentiation here are two decision
>outcomes (verdicts) *not proven* (not guilty) and *proven* (guilty). One
>can stick with the null, or go with the alternative. If the evidence
>does not make the Hs sufficiently statistically distinguishable, the the
>null H remains in place.
>
>
>Art Kendall
>Social Research Consultants
[snip, previous posts]
--
Rich Ulrich
Tim Frink
> got those p-values?
The p-values are computed using the Mann-Whitney rank sum test.
Please see Table 3 from
http://www.tik.ethz.ch/~sop/pisa/publications/KTZ2005a.pdf
for more details.
I think it is best when you take a look into the paper before I try to
explain the details. If you still have any questions, please let me know.
Tim
Ray Koopman
Here is Table 3 and its description from the pdf:
| Table 3: Dominance ranking results using Mann-Whitney rank sum test.
| The tables contain for each pair of optimizers O_R (row) and O_C
| (column) the p-values with respect to the alternative hypothesis
| H_A that the dominance ranks for O_R are significantly better than
| those for O_C. On ZDT6 and DTLZ2 no approximation set dominates any
| other one and so all have the same rank, 1, and hence no differences
| between the algorithms are found. For the knapsack problem,
| differences between the algorithms are discovered but none of them
| is statistically significant (alpha = .05). In case of the QV
| problem, there are statistically significant differences (IBEA
| outperforms SPEA2 and NSGA2).
|
| Knapsack
| IBEA NSGA-II SPEA2
| IBEA -- .28 .19
| NSGA-II .72 -- .36
| SPEA2 .81 .64 --
|
| QV
| IBEA NSGA-II SPEA2
| IBEA -- .002 .0000211
| NSGA-II .998 -- .24
| SPEA2 1 .76 --
That presentation is misleading, for two reasons. First, the entries
in such tables should be AUC values. AUC = "Area Under the Curve" = P
(R > C). AUC is a descriptive statistic. It estimates the parameter
that the MW test tests. Second, since the authors are considering both
tails, two-tailed p-values should have been reported. P-values are not
measures of the size of the difference -- that's what AUC tells you --
but of how unusual the sample AUC is if the true AUC = 1/2.