Re: Retrospective significance testing

[David Jones]

Francogrex wrote:
> A little confused about significance testing: I have two senarios:
>
> 1) Retrospective: say I have an experiment with 1 sample group
> (injected with active ingredient) and 1 control group. It's just a
> collection of a fixed x number individuals in each group. Then I
> observe a specific reaction in the sample group compared to the
> control. Is it acceptable to test whether the number of a chosen
> reaction observed is significant in one group compared to the other
> (using fisher test for example)? Or
> 2) Should one before the onset of the experiment, determine the
> difference needed to be observed in the specific reaction between the
> 2 groups, and using power analysis determine a good sample size and
> then start the experiment and data collection and then at the end of
> that do statistical testing?
>
> Which is the more sane approach to determine whether there is an
> significant difference of a specific reaction bewteen the 2 groups?
> And suppose that in the 1st scenario, one starts picking and testing
> significance of any reaction? Is it acceptable?

Well you can do (1) if you like BUT, in order to make a reasonable inference from the sequence of tests, you should know the probabilities associated with the various possible outcomes (under the null hypothesis); of course these are not the same as the nominal signifance levels of the separate tests. You could try looking up something like "correction for multiple testing", or do a simulation study of the sequence of tests.

David Jones