On Tue, 27 Apr 2021 17:28:16 -0700 (PDT), Cosine <ase...@gmail.com
> Sometimes we need to perform many pair-comparisons.
When we do, we need to ask ourselves, WHY? What to
we intend to show with our presentation?
> When we need to do so, we would reduce the
>value of a significant level to reduce the type I error.
No, not too good, that is a sloppy statement. I prefer to think of
what we "preserve" rather than "reduce".
Or, using "reduce" - What we sometimes do is reduce the /nominal/
value of the alpha for each test in order to /preserve/ some overall
(global, as Duncan says) Type I error.
In my experience, the easiest way to "preserve" my power of
testing is to reduce the number of /important/ tests. Set up
hypotheses as "main" and subordinate (use hierarchies).
Test one variable, with one d.f. for hypothesis, for one hypothesis
that is most important to the study. Or two, or three at the most.
That worked for clinical research. Similar "indicators" could be
tested simultaneously by combining into a composite score.
If there are a dozen separate "hypotheses", then I wonder if they
aren't all equally exploratory ... or else, perhaps, deserve equal
attention as "clear hypotheses" that each deserve the same
respect and their own 5% level. - Huge surveys are an example
of studies that may have a dozen hyptheses. By virtue of huge
N, it might be possible to "adjust" for multiple testing, but journals
(wisely) say that you should use your large sample to estimate
effect sizes, and only report the results that are /interesting/, not
the ones that are "significant" by some standard.
> But when do we need to do so, i.e., reduce the
> alpha-value? Or how many is too many that we ought to reduce
> the alpha value?
Do you know about avoiding Type II error?
You might read about the Benjamini-Hochberg procedure
that allows /extra/ Type I error while reducing Type II error.
That is, your nominal p-value can increase to 0.1 or 0.2 or
even 0.5 (I've noted 0.5, while convincing a PI not to use
Benjamini-Hochberg in her research report).
IIRC, that B-H procedure came from electronic communication
theory, rather than clinical research. Its application in clinical
research, as of 15 year ago, was most often a mistake (IMO),
but I can't speak to modern developments.