Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss
How to calculate Hedges g effect sizes when running an ANCOVA
135 views
Skip to first unread message
Gary
May 18, 2012, 5:57:01 AM5/18/12
to
I am trying to understand how to calculate effect sizes (standardized
effect sizes?) for more complicated designs. My impression from
textbooks is that these calculations have to be adjusted when the
design changes. So I have been trying various examples taken from
texts that I have found.
Brian Everitt reported data on a study of three treatments for
anorexia in young girls. One treatment was cognitive behaviour
therapy, a second was a control condition with no therapy, and a third
was a family therapy condition. There was pre-treatment and post-
treatment scores (body weights, I think) for all three conditions.
Suppose one wanted to calculate Hedges g to report the effect size of
the difference between the family therapy treatment and the control
condition after one had run an ANCOVA in which the dv = posttest
scores and the covariate was pretest scores. My question is: How
should the Hedges g value be calculated?
Looking at the formula given in Huitema (2011) “The analysis of
covariance and alternatives” (2nd edition), John Wiley and Sons, the
formula is often written with the difference between the two means
divided by the root of the error term in the ANCOVA, and sometimes as
the difference between the two means divided by the root of the sum of
the sum of squares of the dv (“y” in the formula) scores for each of
the two groups where this sum is divided by the sum of the n of the
two groups minus 2.
But I get quite different answers depending on which interpretation of
the formula I use. If I use the error term from the ANCOVA I get
0.116. If I use the residuals rather than the y scores (on the grounds
that the ANCOVA has adjusted the y scores for the effect of the
covariate) I get a very different answer, 0.874. Anyway I’m obviously
confused. Can anyone tell me how Hedges g should be calculated when
running an ANCOVA?
Lance.
The ANCOVA output is:
Analysis of Variance for Posttest, using Adjusted SS for Tests
Source DF Seq SS Adj SS Adj MS F P
Pretest 1 506.51 353.79 353.79 7.27 0.009
Group 2 766.27 766.27 383.14 7.87 0.001
Error 68 3311.26 3311.26 48.70
Total 71 4584.04
S = 6.97818 R-Sq = 27.77% R-Sq(adj) = 24.58%
And the data Everitt reported are:
Group Pretest Posttest Gain
1 80.5 82.2 1.7
1 84.9 85.6 0.7
1 81.5 81.4 -0.1
1 82.6 81.9 -0.7
1 79.9 76.4 -3.5
1 88.7 103.6 14.9
1 94.9 98.4 3.5
1 76.3 93.4 17.1
1 81 73.4 -7.6
1 80.5 82.1 1.6
1 85 96.7 11.7
1 89.2 95.3 6.1
1 81.3 82.4 1.1
1 76.5 72.5 -4
1 70 90.9 20.9
1 80.4 71.3 -9.1
1 83.3 85.4 2.1
1 83 81.6 -1.4
1 87.7 89.1 1.4
1 84.2 83.9 -0.3
1 86.4 82.7 -3.7
1 76.5 75.7 -0.8
1 80.2 82.6 2.4
1 87.8 100.4 12.6
1 83.3 85.2 1.9
1 79.7 83.6 3.9
1 84.5 84.6 0.1
1 80.8 96.2 15.4
1 87.4 86.7 -0.7
2 80.7 80.2 -0.5
2 89.4 80.1 -9.3
2 91.8 86.4 -5.4
2 74 86.3 12.3
2 78.1 76.1 -2
2 88.3 78.1 -10.2
2 87.3 75.1 -12.2
2 75.1 86.7 11.6
2 80.6 73.5 -7.1
2 78.4 84.6 6.2
2 77.6 77.4 -0.2
2 88.7 79.5 -9.2
2 81.3 89.6 8.3
2 78.1 81.4 3.3
2 70.5 81.8 11.3
2 77.3 77.3 0
2 85.2 84.2 -1
2 86 75.4 -10.6
2 84.1 79.5 -4.6
2 79.7 73 -6.7
2 85.5 88.3 2.8
2 84.4 84.7 0.3
2 79.6 81.4 1.8
2 77.5 81.2 3.7
2 72.3 88.2 15.9
2 89 78.8 -10.2
3 83.8 95.2 11.4
3 83.3 94.3 11
3 86 91.5 5.5
3 82.5 91.9 9.4
3 86.7 100.3 13.6
3 79.6 76.7 -2.9
3 76.9 76.8 -0.1
3 94.2 101.6 7.4
3 73.4 94.9 21.5
3 80.5 75.2 -5.3
3 81.6 77.8 -3.8
3 82.1 95.5 13.4
3 77.6 90.7 13.1
3 83.5 92.5 9
3 89.9 93.8 3.9
3 86 91.7 5.7
3 87.3 98 10.7
effect sizes?) for more complicated designs. My impression from
textbooks is that these calculations have to be adjusted when the
design changes. So I have been trying various examples taken from
texts that I have found.
Brian Everitt reported data on a study of three treatments for
anorexia in young girls. One treatment was cognitive behaviour
therapy, a second was a control condition with no therapy, and a third
was a family therapy condition. There was pre-treatment and post-
treatment scores (body weights, I think) for all three conditions.
Suppose one wanted to calculate Hedges g to report the effect size of
the difference between the family therapy treatment and the control
condition after one had run an ANCOVA in which the dv = posttest
scores and the covariate was pretest scores. My question is: How
should the Hedges g value be calculated?
Looking at the formula given in Huitema (2011) “The analysis of
covariance and alternatives” (2nd edition), John Wiley and Sons, the
formula is often written with the difference between the two means
divided by the root of the error term in the ANCOVA, and sometimes as
the difference between the two means divided by the root of the sum of
the sum of squares of the dv (“y” in the formula) scores for each of
the two groups where this sum is divided by the sum of the n of the
two groups minus 2.
But I get quite different answers depending on which interpretation of
the formula I use. If I use the error term from the ANCOVA I get
0.116. If I use the residuals rather than the y scores (on the grounds
that the ANCOVA has adjusted the y scores for the effect of the
covariate) I get a very different answer, 0.874. Anyway I’m obviously
confused. Can anyone tell me how Hedges g should be calculated when
running an ANCOVA?
Lance.
The ANCOVA output is:
Analysis of Variance for Posttest, using Adjusted SS for Tests
Source DF Seq SS Adj SS Adj MS F P
Pretest 1 506.51 353.79 353.79 7.27 0.009
Group 2 766.27 766.27 383.14 7.87 0.001
Error 68 3311.26 3311.26 48.70
Total 71 4584.04
S = 6.97818 R-Sq = 27.77% R-Sq(adj) = 24.58%
And the data Everitt reported are:
Group Pretest Posttest Gain
1 80.5 82.2 1.7
1 84.9 85.6 0.7
1 81.5 81.4 -0.1
1 82.6 81.9 -0.7
1 79.9 76.4 -3.5
1 88.7 103.6 14.9
1 94.9 98.4 3.5
1 76.3 93.4 17.1
1 81 73.4 -7.6
1 80.5 82.1 1.6
1 85 96.7 11.7
1 89.2 95.3 6.1
1 81.3 82.4 1.1
1 76.5 72.5 -4
1 70 90.9 20.9
1 80.4 71.3 -9.1
1 83.3 85.4 2.1
1 83 81.6 -1.4
1 87.7 89.1 1.4
1 84.2 83.9 -0.3
1 86.4 82.7 -3.7
1 76.5 75.7 -0.8
1 80.2 82.6 2.4
1 87.8 100.4 12.6
1 83.3 85.2 1.9
1 79.7 83.6 3.9
1 84.5 84.6 0.1
1 80.8 96.2 15.4
1 87.4 86.7 -0.7
2 80.7 80.2 -0.5
2 89.4 80.1 -9.3
2 91.8 86.4 -5.4
2 74 86.3 12.3
2 78.1 76.1 -2
2 88.3 78.1 -10.2
2 87.3 75.1 -12.2
2 75.1 86.7 11.6
2 80.6 73.5 -7.1
2 78.4 84.6 6.2
2 77.6 77.4 -0.2
2 88.7 79.5 -9.2
2 81.3 89.6 8.3
2 78.1 81.4 3.3
2 70.5 81.8 11.3
2 77.3 77.3 0
2 85.2 84.2 -1
2 86 75.4 -10.6
2 84.1 79.5 -4.6
2 79.7 73 -6.7
2 85.5 88.3 2.8
2 84.4 84.7 0.3
2 79.6 81.4 1.8
2 77.5 81.2 3.7
2 72.3 88.2 15.9
2 89 78.8 -10.2
3 83.8 95.2 11.4
3 83.3 94.3 11
3 86 91.5 5.5
3 82.5 91.9 9.4
3 86.7 100.3 13.6
3 79.6 76.7 -2.9
3 76.9 76.8 -0.1
3 94.2 101.6 7.4
3 73.4 94.9 21.5
3 80.5 75.2 -5.3
3 81.6 77.8 -3.8
3 82.1 95.5 13.4
3 77.6 90.7 13.1
3 83.5 92.5 9
3 89.9 93.8 3.9
3 86 91.7 5.7
3 87.3 98 10.7
Rich Ulrich
May 18, 2012, 8:59:19 PM5/18/12
to
On Fri, 18 May 2012 02:57:01 -0700 (PDT), Gary <lanc...@gmail.com>
wrote:
>I am trying to understand how to calculate effect sizes (standardized
>effect sizes?) for more complicated designs. My impression from
>textbooks is that these calculations have to be adjusted when the
>design changes. So I have been trying various examples taken from
>texts that I have found.
problem arises. Also, the way to deal with it is to be very clear
in your exposition when you write up what you have done.
Reviewers rightfully complain when you mis-describe whatever
you are doing. And you want to be clear when you are doing
something unusual, which this probalby is.
An "effect size" is a difference divided by some appropriate
standard deviation. The problem exists -- and is impossible
to dismiss -- because, when you have a change across time,
there are two estimates of the SD which are both relevant.
Or even more than two.
The effect size that you should use in a power analysis is
the change (or regressed change) divided by its error, if you
are just looking at change. It is the error of the differences
in change scores, if you are comparing change scores. That
is what describes the necessary N, and the inter-relations of
any power analysis. For an ANCOVA, that is from the error
term of the ANCOVA. That is the magnitude of your effect,
given that you have a within-subject design for effects.
On the other hand, the formula from Hedge that I see in Wikip
is the formula for a difference between uncorrelated samples.
And if you are reporting various tests, including the differences
that exist at baseline, it is entirely appropriate to report the
differences in terms of (say) the baseline SD, either from a
control group or pooled across experimental samples.
So - an X point difference on a particular scale (even if it
is well-behaved and the SD never varies with the mean)
is a different "effect size" depending on the comparison --
(1) Baseline differences; (2) change in one group; (3) difference
in changes between two groups. If I was ever looking at
(3), I would try to skip the version in (2), just to keep things
simpler.
>Brian Everitt reported data on a study of three treatments for
>anorexia in young girls. One treatment was cognitive behaviour
>therapy, a second was a control condition with no therapy, and a third
>was a family therapy condition. There was pre-treatment and post-
>treatment scores (body weights, I think) for all three conditions.
>Suppose one wanted to calculate Hedges g to report the effect size of
>the difference between the family therapy treatment and the control
>condition after one had run an ANCOVA in which the dv = posttest
>scores and the covariate was pretest scores. My question is: How
>should the Hedges g value be calculated?
>Looking at the formula given in Huitema (2011) �The analysis of
>covariance and alternatives� (2nd edition), John Wiley and Sons, the
>formula is often written with the difference between the two means
>divided by the root of the error term in the ANCOVA, and sometimes as
difference between the regressed-change scores.
You might check Huitema again and see if he ever mentions
the difference in circumstances. It looks like he should.
>the difference between the two means divided by the root of the sum of
>the sum of squares of the dv (�y� in the formula) scores for each of
>the two groups where this sum is divided by the sum of the n of the
>two groups minus 2.
>But I get quite different answers depending on which interpretation of
>the formula I use. If I use the error term from the ANCOVA I get
>0.116. If I use the residuals rather than the y scores (on the grounds
>that the ANCOVA has adjusted the y scores for the effect of the
>covariate) I get a very different answer, 0.874. Anyway I�m obviously
>confused. Can anyone tell me how Hedges g should be calculated when
>running an ANCOVA?
--
Rich Ulrich
Gary
May 19, 2012, 4:51:49 PM5/19/12
to
On May 19, 2:59 am, Rich Ulrich <rich.ulr...@comcast.net> wrote:
> On Fri, 18 May 2012 02:57:01 -0700 (PDT), Gary <lanceg...@gmail.com>
> >Looking at the formula given in Huitema (2011) �The analysis of
> >covariance and alternatives� (2nd edition), John Wiley and Sons, the
Lance
> On Fri, 18 May 2012 02:57:01 -0700 (PDT), Gary <lanceg...@gmail.com>
> >covariance and alternatives� (2nd edition), John Wiley and Sons, the
> >formula is often written with the difference between the two means
> >divided by the root of the error term in the ANCOVA, and sometimes as
>
> Using the error term from the ANCOVA is looking at the
> difference between the regressed-change scores.
>
> You might check Huitema again and see if he ever mentions
> the difference in circumstances. It looks like he should.
>
> >the difference between the two means divided by the root of the sum of
> >the sum of squares of the dv (�y� in the formula) scores for each of
> >divided by the root of the error term in the ANCOVA, and sometimes as
>
> Using the error term from the ANCOVA is looking at the
> difference between the regressed-change scores.
>
> You might check Huitema again and see if he ever mentions
> the difference in circumstances. It looks like he should.
>
> >the difference between the two means divided by the root of the sum of
> >the two groups where this sum is divided by the sum of the n of the
> >two groups minus 2.
> >But I get quite different answers depending on which interpretation of
> >the formula I use. If I use the error term from the ANCOVA I get
> >0.116. If I use the residuals rather than the y scores (on the grounds
> >that the ANCOVA has adjusted the y scores for the effect of the
> >covariate) I get a very different answer, 0.874. Anyway I�m obviously
> >two groups minus 2.
> >But I get quite different answers depending on which interpretation of
> >the formula I use. If I use the error term from the ANCOVA I get
> >0.116. If I use the residuals rather than the y scores (on the grounds
> >that the ANCOVA has adjusted the y scores for the effect of the
> >confused. Can anyone tell me how Hedges g should be calculated when
> >running an ANCOVA?
>
> [snip, example]
>
> --
> Rich Ulrich
Thanks for your reply. It is very informative.
> >running an ANCOVA?
>
> [snip, example]
>
> --
> Rich Ulrich
Lance
0 new messages