ezANOVA and aov.car give radically different p values

[Danielle Smith]

I have a ANOVA, with View and Texture as within-subjects factors, TNOGroup as a between subjects factor and Age as a covariate. 

I specified the model in ezANOVA like so:

ezANOVA(

+     data=dataset

+     , dv=.(x)

+     , wid=.(ID)

+     , within=.(Texture,View)

+     , between=.(TNOGroup)

+     , between_covariate=.(Age)

+     , type=3

+     , return_aov=TRUE

+ )

I thought I specified it exactly the same way in aov.car:

aov.car(x~View*Texture*TNOGroup+Age+Error(ID/(View*Texture)), data=dataset)

However, I get radically different p-values in each case. Additionally, adding in the covariate via the ezANOVA method yields almost no difference in results (just slight differences in terms of effect size), whereas it has a huge impact for the model specified using aov.car.

Can anyone shed any light here? The main reason I tried aov.car is because I wanted to be able to see the F and p-values for the main effect of the covariate and its interactions. The dataset can be accessed here.

[Mike Lawrence]

I haven't used aov.car before, but upon installing it and running your code, I note that it prints a note "Converting to factor: Age", whereas ezANOVA has the warning "Warning: Covariate"Age" is numeric and will therefore be fit to a linear effect." So, the two analyses are treating age differently. Since you have a relatively continuous sample of ages, you likely don't want to treat Age as a factor as is default in aov.car. You can fix this by setting "factorize=FALSE", as in:

    

    aov.car(

        formula = x~View*Texture*TNOGroup+Age+Error(ID/(View*Texture))

        , data=dataset

        , factorize = FALSE

    )

But then you'll get another note saying that Age isn't centered on zero. To recenter, do:

    dataset$Age0 = dataset$Age - mean(dataset$Age)

    aov.car(

        formula = x~View*Texture*TNOGroup+Age0+Error(ID/(View*Texture))

        , data=dataset

        , factorize = FALSE

    )

In which case you'll get output that looks more similar to that of ezANOVA. However, there's still a few differences:

- aov.car reports the effect of Age as well as its interaction with any within-Ss variable; ezANOVA doesn't

- aov.car's MSE/F's are slightly different than those of ezANOVA

- aov.car reports error DFs that are 1 less than those of ezANOVA

I'm not familiar enough with aov.car to know from what these differences arise. Here's what ezANOVA does: when provided with a between covariate like Age, ez first collapses the dv to a mean across any within-Ss variables, performs a linear model predicting the collapsed dv ~ Age (automatically centering Age on 0 if it's a numeric variable), obtains the fitted values from this model, and then goes back to the uncollapsed data and, for each subject, replaces their dv values with those values minus the fitted value for that subject from the linear model of age. Finally, it is these residuals that are used as the dv for the ANOVA.

aov.car appears to be doing everything at once, rather than the two-step process that ezANOVA does with covariates. I can't really comment any further than that though, sorry!

Mike

--
Mike Lawrence
Graduate Student
Department of Psychology & Neuroscience
Dalhousie University

~ Certainty is (possibly) folly ~

--
You received this message because you are subscribed to the Google Groups "ez4r" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ez4r+uns...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Danielle Smith]

Thanks for your detailed reply. Not five minuted after I posted the question, I figured out how to set "factorize=FALSE" and also realised I hadn't centred Age. The other issues are confusing though - apparently Henrik is writing a reply as I type.

Danielle

[Henrik Singmann]

Hello,

What aov.car does is relatively simple as aov.car is just a wrapper for car::Anova. Hence, it only fits a linear model with lm (or multivariate linear model in the case of repated-measures) and hands this model over to car::Anova. In the case of covariates this model simply includes the covariates as is, so in your case it fits something like the following with lm (this can be obtained by setting return = "lm" in the call to aov.car:

cbind(a_a, a_b, b_a, b_b) ~ TNOGroup + Age0

Here the left hand sides of the formula represent your within-subject variables (which have been remained because in R prior to 3.0.2 there was a bug with long variables) and the right hand side represent your between-subject variables. Which in essence means something very similar as described by Mike, the effect of the between subject variable is calculated on the residuals from a model with only the covariate Age0 (differences can come from differences in using the multivariate models residuals as afex does or the mean of the within.subject variables as ez does).

And in effect, both approaches are very similar when only using a between subject variable:

ez <- ezANOVA(
  data=dataset
  , dv=.(x)
  , wid=.(ID)
  , between=.(TNOGroup)
  , between_covariate=.(Age)
  , type=3
)
ez$ANOVA$F
[1] 0.2157

afex <- aov.car(
  formula = x~TNOGroup+Age0+Error(ID)

  , data=dataset
  , factorize = FALSE

  , return = "Anova"
)
afex$`F value`[2]
[1] 0.2132

That here are still small differences is probably due to the fact that ez extracts the residuals and fits the factors on them whereas afex fits both variables directly (probably only differences due to numerical precision).

However, the issue of the different df remain. Here again, afex just uses what is handed over from car::Anova which seem to be the correct df.
In contrast, I have the feeling that ez uses the incorrect df, as every (numeric) covariate reduces the df by one. I double checked this with the classics on this issue. Table 10.2-4 of Winer (1971, p. 770 attached) shows this clearly for the case of one covariate. A similar reduction due to one covariate can also be found if comparing tables 4.3-2 (p. 140) and 14.3-2 (p. 730) from Kirk (1982, also attached).

So overall, both results should be roughly identical with respect to the estimates but the df seem to be off in ez.

Best,
Henrik

References:
Kirk, R. E. (1982). Experimental Design: Procedures for the Behavioral Sciences. Brooks/Cole.
Winer, B. J. (1971). Statistical principles in experimental design. New York u.a.: MacGraw-Hill.

2013/10/2 Mike Lawrence <mike....@gmail.com>

[Mike Lawrence]

I'll try to make time to run a simulation to evaluate to what degree ez is in error when using the +1 error DFs when doing ancova :O)

--
Mike Lawrence
Graduate Student
Department of Psychology & Neuroscience
Dalhousie University

~ Certainty is (possibly) folly ~