Presenting ANOVA Results
Graham Ashe
Anova: Single Factor
SUMMARY
Groups Count Sum Average Variance
Column 1 1000 251.682 0.251682 0.000460858
Column 2 1000 295.13 0.29513 0.001289178
Column 3 1000 356.584 0.356584 0.00325257
ANOVA
Source of Variation SS df MS F P-value F crit
Between Groups 5.556250808 2 2.778125404 1666.007078 0 2.998731929
Within Groups 4.99760292 2997 0.001667535
Total 10.55385373 2999
What is the best way to present these results in a science paper? Usually for a two-sample t-test, I use the following.
t (150) = 14.6, p < 0.001
Can ANOVA results be presented in this form too? If so, how? Thanks.
Doug Morse
THE SHORT ANSWER
----------------
Here's what I'd write (fill in the ALL CAPS):
MY MEASURE. MY MEASURE was analyzed with a one-way analysis of variance
(ANOVA) with MY FACTOR as a between-subjects factor (i.e., FACTOR LEVEL 1
NAME, FACTOR LEVEL 2 NAME, and FACTOR LEVEL 3 NAME). There was a reliable
main effect of MY FACTOR, with means of 0.2517 UNITS, 0.2951 UNITS, and 3566
UNITS for the FACTOR LEVEL 1 NAME, FACTOR LEVEL 2 NAME, and FACTOR LEVEL 3
NAME, respectively, F(2, 2997) = 1666.01, p < .001, MSE = 0.001668.
(Note: F, p, and MSE should be italicized.)
THE LONG ANSWER
---------------
It depends on your discipline and, if applicable, the journal to which you
plan to submit your paper/report.
That said, the APA style is pretty widely used -- both in psychology and
several other disciplines. As such, my recommendation is based on the APA
conventions. These indicate that, at a minimum, the following information
should be provided:
A brief statement that makes clear the design being tested, including:
(a) what the dependent variable(s) is/are
(b) what the independent variable(s) is/are, noting all factor levels
(c) the type of analysis being conducted (e.g., ANOVA, MANOVA, etc.)
(c) which factors are within-subject and which are between-subject
and then the COMPLETE results of the statistical analysis:
(a) relevant central tendency estimates (e.g., means)
(b) relevant variability estimates (e.g., MSE's or SE's or SD's)
(c) relevant test statistics (e.g., F's, p's)
for:
(a) EVERY factor in the analysis
(b) EVERY interaction in the analysis
If a several factors and/or interactions are statistically non-significant,
it's considered OK to note that in a single statement and report a single
maximum F value.
Here's a real example that perhaps can serve as a useful template for you:
Distance preferences. Distances were analyzed with a 2x3x2x9
repeated-measures analysis of variance (ANOVA) with self-motion block order
(i.e., forward first or upward first) as a between-subjects factor and trial
(i.e., near, middle, or far position of the visual-field limiter), direction
of self-motion (i.e., forward, upward) and visual-field limiter type (i.e.,
4 occluders, 4 annuli, 1 fixation point) as within-subject factors. There
was a main effect of direction of self-motion, with means of 27.32 cm and
29.42 cm for the forward and upward conditions, respectively, F(1, 318) =
5.87, p < .02, MSE=0.4954. There was also a main effect of visual-field
limiter type, with means (in cm) of 24.64, 25.78, 31.02, 38.23, 25.06,
30.91, 23.81, 27.31, 28.56 for the four occluders (10 cm - 40 cm), the four
annuli (10 cm - 40 cm), and the fixation point, respectively, F(8, 318) =
12.21, p < .001, MSE=0.3432. No other main effects or interactions were
observed, all F's < 0.13.
Lastly, for analyses with statistically-significant 2-way interactions, it's
often more concise to report mean differences instead of means (at least at
the point where one is reporting the test statistics; the actual means should
be reported elsewhere or at least directly inferable somehow):
The interaction was also reliable, F(2, 57) = 3.85, p < .03, MSE = 0.0527.
The mean difference between the step-by-step and final diagram conditions
was .09 for the few-step objects but .37 for the many-step objects. In
contrast, the mean difference between the text only and final diagram
conditions showed the opposite pattern: .16 versus .05 for the few-step and
many-step objects, respectively.
Hope this helps.
Cheers,
Doug
Graham Ashe
"The differences between the mean scores were statistically significant; F (2, 2997) = 1666, p < 0.001, MS = 2.778, ANOVA single factor test."
What's the difference between F=1666 and F critical=2.999? Which one should I use here? Also, Excel only shows me "MS", not MSE or MSB. Are they the same?
Doug Morse
No problem -- glad to be of assistance. IMHO what you have written seems
fine, although be sure to report the actually means somewhere, too (and
indicate the proper units of measure).
Fcrit -- crit for "critical" value -- is the "cut-off" value; observing an
F(2, 2997) >= Fcrit(2, 2997) indicates that is appropriate to reject the null
hypothesis of no difference between the means. One does not need to report
the Fcrit, as anyone can look it up in a table (or do an equivalent look-up
using a software program).
MSE is "mean squared error"; basically the denominator term in your F ratio
test. For the ANOVA results you provided, the Within-Subjects MS is being
used as the error term. You can verify this in that MS(between) / MS(within)
equals your F value.
So, in a nutshell, you have reported the correct values (F, not Fcrit; MSE as
MS(within)).
Cheers!
Doug
Graham Ashe
I was looking over my results and I think it would be better if I just presented the information like this:
"The differences between the mean scores were statistically significant; F (2) = 1666, p < 0.001, MS = 2.778 using an analysis of variance, single factor test."
Since the df=2997 refers to "within groups" and my analysis is "between groups", it doesn't seem necessary to include it. What do you think? Also, there is no unit to my measurements. It's just an arbitrary point system generated by a computer's evaluation function. Is that a problem?
Come to think of it, I'm not even sure what these results say. I just want to show that the difference between the means of the three groups is significant. Does the ANOVA test I'm using actually analyze the 1,000 values in each group (in relation to each other) or just play with the three means?
Doug Morse
The wording changes are fine but unfortunately, no, the statistical changes
you propose won't work. First, the F distribution takes two parameters, a
numerator df and a denominator df (technically, there's a third parameter -- a
non-centrality parameter -- but for normal analyses of variance it's not used
and hence omitted). As such, it doesn't make sense to report F with just one
df (e.g, F(2)). Second, the MSE is to reported to provide a sense of variance
around the means. Your MSE is very low (0.001668), which is a "good thing".
The BS MS is not an "error" MS, so it's inappropriate and misleading to report
it as the MSE.
Regarding units, they're just what you said: points (or score) on a
computerized assessment. So, wherever you report your means, just say so
(e.g., "...with Group 1 having a mean score of 251.68, Group 2 having...").
Is this important? Well, for "correct scientific" reporting, yes -- units
should always be included.
Lastly, yes, the ANOVA does use all the data you provide. It computes mean
estimates and variance estimates both within and between groups. Does the
analysis show a statistically-reliable difference between the group means?
Yes, very much so, actually (the F(2,2997) value of 1666 is quite high and
similarly the estimated variablity around the means is quite small, 0.0017).
Hope this helps.
Cheers,
Doug
Doug Morse
I just noticed that I didn't catch the erroneous MSE in your previous post.
So, went I wrote earlier that what you had was fine, I misspoke; reporting "MS
= 2.778" was incorrect.
Doug
Bruce Weaver
> Hi Graham,
>
> THE SHORT ANSWER
> ----------------
>
> Here's what I'd write (fill in the ALL CAPS):
>
--- snip ---
I am in general agreement with the way Doug has laid things out.
However, I would not use the word "reliable" to describe any of the
main effects or interactions. I know that "reliable" has been used in
this way in the past in psychology journals, possibly in an attempt to
not say "statistically significant" repeatedly. But the correct term
is "statistically significant". Reliability of an effect is something
different.
--
Bruce Weaver
bwe...@lakeheadu.ca
www.angelfire.com/wv/bwhomedir
"When all else fails, RTFM."
Ray Koopman
> Hi Graham,
>
> THE SHORT ANSWER
> ----------------
>
> Here's what I'd write (fill in the ALL CAPS):
>
> MY MEASURE. MY MEASURE was analyzed with a one-way analysis of variance
> (ANOVA) with MY FACTOR as a between-subjects factor (i.e., FACTOR LEVEL 1
> NAME, FACTOR LEVEL 2 NAME, and FACTOR LEVEL 3 NAME). There was a reliable
> main effect of MY FACTOR, with means of 0.2517 UNITS, 0.2951 UNITS, and 3566
> UNITS for the FACTOR LEVEL 1 NAME, FACTOR LEVEL 2 NAME, and FACTOR LEVEL 3
> NAME, respectively, F(2, 2997) = 1666.01, p < .001, MSE = 0.001668.
>
> (Note: F, p, and MSE should be italicized.)
>
It's almost always better to report standard deviations rather than
variances, because standard deviations are in the same units as the
data. This information is more useful if given along with the means
rather than with the significance test results. A rough estimate of
the effect size can then be eyeballed from the numbers, by comparing
the size of the differences between the means to the within-group
s.d. (the square root of the within-group mean square). In your data,
the differences between adjacent means are a little bigger s.d.
(which is .04), so the effect size will be "large", no matter which
measure of effect size you use.
Graham Ashe
Okay, so the corrected presentation of my results looks like this.
"The differences between the mean scores were statistically significant; F (2, 2997) = 1666, p < 0.001, MSE = 0.0017 using an analysis of variance, single factor test."
Did I leave anything out? Doesn't the MSB (2.778) also need to be reported? Also, according to Excel, my P value is "0". Should I be reporting it as <0.001?
Ray,
Thanks for the tip on reporting SD with my means. I calculated the SD independenty for each group and put it something like this.
Mean = 0.252 (SD 0.021)
Doug Morse
Looks good. No need to report MS(Between) -- it can be derived if one knows
MS(Within/Error) and the F score. Indeed, this highlights why the ANOVA
results are reported they way they are: Not everything has to be reported,
because many things can be derived (e.g., effect sizes) as long as the right
"base" information is available (e.g., means and variances or SDs).
Regarding the p value, it technically isn't 0, but some very, very small value
-- beyond the number of significant digits Excel is using. So, p < .001 is
appropriate, or you could even say p < .0001 or p < .00001 to highlight just
how small of a p value was observed.
Bruce makes a good point re: using the word "reliable" -- technically
speaking, the reliability of an estimate has a different, specific meaning. I
agree with Ray's point as well re: reporting SDs is more immediately
interpretable and meaningful. However, I wouldn't recommend NOT reporting
MSE, for the very reason discussed above: a reader can derive the entire ANOVA
table if the "right" base information is provided. Rather, if desired, I'd
say report both (i.e., SD with your means, MSE with the ANOVA results).
Doug