Calculating effect size when there is heterogeneity of variance
Liesl
I have a question regarding effect size calculation when there is
heterogeneity of variance, which I'm hoping that someone can please
help me with. I'm also afraid that this is a hypothetical question -
otherwise I would provide you with actual data/test results.
I'm interested in what happens if you run one-way between-subjects
ANOVA in SPSS and discover that Levene's test indicates that the
homogeneity of variance assumption is violated. I understand that you
can simply use the Welch F-ratio in place of the regular F-ratio.
However, my question is, is it okay to calculate eta-squared based on
the Welch F-ratio and adjusted df?
And assuming that you've then conducted Games-Howell post hoc tests,
is there an appropriate effect size estimate which can be readily
calculated for each pairwise comparison? I would normally calculate
an estimate of Cohen's d for each pairwise comparison, however, since
Cohen's d relies on the pooling of variance, I wouldn't think that it
is appropriate to use this effect size estimate when there is
heterogeneity of variance. So I'm wondering whether I could instead
calculate the proportion of variance explained, and if so, how I would
go about this?
Many thanks,
Liesl
Ray Koopman
There is no conventional "right" answer.
I haven't read either of the following papers,
but they look promising.
Measuring effect size: a robust heteroscedastic approach for two or
more groups
Rand Wilcox and Tian Tian
Journal of Applied Statistics, 2011, vol. 38, issue 7, pages 1359-1368
Abstract: Motivated by involvement in an intervention study, the paper
proposes a robust, heteroscedastic generalization of what is popularly
known as Cohen's d. The approach has the additional advantage of being
readily extended to situations where the goal is to compare more than
two groups. The method arises quite naturally from a regression
perspective in conjunction with a robust version of explanatory power.
Moreover, it provides a single numeric summary of how the groups
compare in contrast to other strategies aimed at dealing with
heteroscedasticity. Kulinskaya and Staudte [16] studied a
heteroscedastic measure of effect size similar to the one proposed
here, but their measure of effect size depends on the sample sizes
making it difficult for applied researchers to interpret the results.
The approach used here is based on a generalization of Cohen's d that
obviates the issue of unequal sample sizes. Simulations and
illustrations demonstrate that the new measure of effect size can make
a practical difference regarding the conclusions reached.
Interval estimates of weighted effect sizes in the one-way
heteroscedastic ANOVA
Kulinskaya E, Staudte RG.
Br J Math Stat Psychol. 2006 May;59(Pt 1):97-111.
Abstract: A framework for comparing normal population means in the
presence of heteroscedasticity and outliers is provided. A single
number called the weighted effect size summarizes the differences in
population means after weighting each according to the difficulty of
estimating their respective means, whether the difficulty is due to
unknown population variances, unequal sample sizes or the presence of
outliers. For an ANOVA weighted for unequal variances, we find
interval estimates for the weighted effect size. In addition, the
weighted effect size is shown to be a monotone function of a suitably
defined weighted coefficient of determination, which means that
interval estimates of the former are readily transformed into interval
estimates of the latter. Extensive simulations demonstrate the
accuracy of the nominal 95% coverage of these intervals for a wide
range of parameters.
Rich Ulrich
wrote:
correct answer. What we can have are "passable
alternatives" or work-arounds. (I don't think eta is very
workable choice, for an ANOVA with strong heterogeneity.)
And sometimes it may be helpful to present one set of results in
*two* competing ways -- I've done that when faced with showing
differences between groups in outcome, for longitudinal data.
Group differences can yield Cohen's d in the usual fashion. However,
within-subject changes, measured by a personal-change and SD
for within-subject change, comes up with entirely
different numbers for d.
The only great solution for the Groups problem is to manage to
establish a better metric (for instance, taking logs because it is
appropriate) which happens to remove the heterogeneity.
Otherwise, I've used the grouped Pre scores to establish
the SD size by which 'd' is assessed -- Typically, I standandize
a composite criterion score to have a Pre-mean of 50 and SD
of 10; and that leaves other group scores easy to read off.
However, that is misleading when the SD at Pre is smaller than
the SD at post, because changes then look too large.
- You do want to state clearly what you are presenting, so
it is helpful to Keep It Simple; but you don't want to over-state
the conclusion.
By extension of my own logic, I would prefer to use a Control
group for the mean and SD for effect size; but that, similarly,
is problematic when the Control has the smaller SD (d then looks
too big). In that case, I might standardize my 'd' by the larger
SD, whereever it happens to be; and use whatever is convenient
for the baseline-"50" if I am using that convention.
--
Rich Ulrich