paired samples t-test with bonferroni correction

[mmm...@yahoo.co.uk]

I have a question about performing a bonferroni correction with a
paired samples t-test.
In my experiment, I have measured reaction times to a sound at 6
different points under two conditions. I have run a paired samples
t-test to compare the means of these reaction times under the two
conditions. However, my supervisor has told me that I need to perform a
bonferoni correction. As far as I am aware, I would divide .05 by 6 to
do this.
Do I then adjust the confidence interval accordingly and check the
significance levels of the t-tests, or do I just check the significance
level of the t-tests without adjusting the confidence interval and see
wether they are more/less than my bonferroni adjusted alpha level?

[Richard Ulrich]

On 27 Aug 2006 05:46:21 -0700, mmm...@yahoo.co.uk wrote:

> I have a question about performing a bonferroni correction with a
> paired samples t-test.
> In my experiment, I have measured reaction times to a sound at 6
> different points under two conditions. I have run a paired samples
> t-test to compare the means of these reaction times under the two
> conditions. However, my supervisor has told me that I need to perform a
> bonferoni correction. As far as I am aware, I would divide .05 by 6 to
> do this.

Why do you have 6 "different points"?
Do you really have 6 hypotheses? - They are closely
connected, are they not?

Six tests, with 6 d.f., must have less power than doing
one test on a designed contrast, or doing one test on
the "most likely point". If you design a linear trend, or
an average score, or whatever gives the 1 d.f. test of
hypothesis, then you eliminate the need for correction.

> Do I then adjust the confidence interval accordingly and check the
> significance levels of the t-tests, or do I just check the significance
> level of the t-tests without adjusting the confidence interval and see
> wether they are more/less than my bonferroni adjusted alpha level?

Personally, I'm unhappy to see "adjusted CIs", for the most
part. The way that you describe the problem might serve to
justify that, but a different use of the "correction" goes like this:

The "overall test of differences" uses Bonferroni correction, or
it could be done by MANOVA. Once the overall test is rejected,
the *conclusion*, for a rather tightly-bound set of hypotheses,
is that "There is a real difference." After that, the estimation
problem is one of describing the differences, on each of the
variables. That can be conveniently done by giving the mean
differences with the ordinary, nominal 95% CIs. Carefully describe
them that way.

It could be 'fair' to mention to the reader how much wider
the 99% CI would be, as a fraction.

--
Rich Ulrich, wpi...@pitt.edu
http://www.pitt.edu/~wpilib/index.html

[the.lae...@gmail.com]

Can't give you a definite answers, because these multiple mean
comparisons adjustments should be hypothesis driven. But there are a
few things that may be helpful:

1. If there are x means, then a complete set of comparisons should be:

[x (x-1)] / 2

So, there are already 6 tests if you have only 4 means. If you have 6
means, it should be totally 15 pair-wise tests.

2. Point 1 leads to another fact that p < 0.003 (0.05 /15) may be too
stringent; and that is the characteristic of Bonferroni; easy to
understand, but tends to get too strict if number of means goes up.
You may consider consulting a statistician for better guidance.

[Bruce Weaver]

It sounds like you have a 2x6 design with repeated measures on both
factors. So why not do a 2x6 repeated measures ANOVA? Then the
question becomes whether or not there is an interaction between the two
factors. If the interaction is resoundingly non-significant, you can
stop with the F-test for the main effect of condition, I should think.
If the interaction is significant (or close), then you may wish to
explore further.

By the way, RT distributions are notoriously (positively) skewed. Do
you have several RTs per condition x time cell, or only one? In areas
of cognitive psychology that use RT as a dependent measure, it is
customary to have several raw RTs per cell, and then use medians (or
trimmed means) in the ANOVA.

--
Bruce Weaver
bwe...@lakeheadu.ca
www.angelfire.com/wv/bwhomedir

[mmm...@yahoo.co.uk]

Thanks for your replies they were very helpful,
Yes, the data was positively skewed and so I'm using median reaction
times. Yes, I did perform an ANOVA on the data. However, I've spent
hours and hours trying to work out how to perform the post hoc
contrasts and I've just about given up!
Ideally, I would like to be able contrast each of the 6 reaction times
in one condition to its "partner" in the other condition (i.e. RT1 in
the first condition with RT1 in the second condition and so on). The
other thing I need to with the data is compare RTs within each
condition (i.e. RT1 in condition 1 with RT2, RT3, RT4, RT5 and RT6 from
condition 1). SPSS will perform the tests, but won't perform the
contrasts automatically!
I've fiddled about with syntax for, as I say, hours on end and I can't
seem to work out how to do it!
The data for this experiment is in SPSS 12.0 in one row for each
participant, so I'm wondering if this is one of the problems, but to be
honest, I'm new to syntax anyway, so I think it's me.
I'm wondering if I should just give up and go back to the paired
samples t-tests with bonferroni correction................:((

[mmm...@yahoo.co.uk]

Thanks for your replies they were very helpful,
Yes, the data was positively skewed and so I'm using median reaction
times. Yes, I did perform an ANOVA on the data. However, I've spent
hours and hours trying to work out how to perform the post hoc
contrasts and I've just about given up!
Ideally, I would like to be able contrast each of the 6 reaction times
in one condition to its "partner" in the other condition (i.e. RT1 in
the first condition with RT1 in the second condition and so on). The
other thing I need to with the data is compare RTs within each
condition (i.e. RT1 in condition 1 with RT2, RT3, RT4, RT5 and RT6 from
condition 1). SPSS will perform the tests, but won't perform the
contrasts automatically!
I've fiddled about with syntax for, as I say, hours on end and I can't
seem to work out how to do it!
The data for this experiment is in SPSS 12.0 in one row for each
participant, so I'm wondering if this is one of the problems, but to be
honest, I'm new to syntax anyway, so I think it's me.
I'm wondering if I should just give up and go back to the paired
samples t-tests with bonferroni correction................:((

Anybody got any advice?
Danny