one-way repeated measures ANOVA with more levels than subjects

[Tim]

Hi everyone,

I have a quick question about a one-way repeated measures ANOVA in cases where the number of levels in a factor is larger than the number of subjects measured. Let me illustrate my question with a quick example: Say we presented 20 different colors in an experiment and recorded reaction times for every condition in 15 subjects (a total of 20*15=300 measurements). To my understanding, as we have 15 within-subject measurements for every cell, this should work perfectly fine.

Unfortunately, two of the statistics functions we tried using throw an error stating that the number of conditions is too large given the number of subjects (one seems to directly compare the two numbers, the other one computes negative df_error). However, the same kind of setting works just fine in SPSS (df1 = 19; df2 = 266).

Is there any principled reason why above setting should be problematic in a repeated-measures ANOVA? My search for an explanation has so far not brought up any reason why the number of levels in a factor should not exceed the number of subjects if all observations have been measured (n_obs = n_sub * n_cond).


Thanks a lot in advance, looking forward to your answers
Tim

[Bruce Weaver]

If you ran this with SPSS, you will have seen that it does not produce
any results in the "Multivariate Tests" table, and a footnote gives this
explanation:

"Cannot produce multivariate test statistics because of insufficient
residual degrees of freedom."

This makes me wonder if the two "statistics functions" you refer to are
attempting to carry out MANOVA rather than repeated measures ANOVA.
Perhaps you could provide more info about those "functions".

HTH.

--
Bruce Weaver
bwe...@lakeheadu.ca
http://sites.google.com/a/lakeheadu.ca/bweaver/Home
"When all else fails, RTFM."

[Art Kendall]

I do not know what the software other than SPSS was, but you may want to
check with a discussion list for that software. My guess is that you
did not use specifications that said you had repeated measures.

I can think of no reason that you cannot have more levels of repeat than
number of cases.

Times series is a particular kind of repeated measure and economists
routinely use a single case.

BTW color can be a nominal level variable but it can also be thought of
an array along a spectrum ROYGBIV, so you depending on exactly what you
have for color stimuli you may be able to put structure on your repeats.

What syntax did you use in SPSS?

Art Kendall
Social Research Consultants

[tkie...@gmail.com]

Hi Bruce,

Thanks a lot for your answer. As you said, SPSS did not produce output for the Multivariate Tests.

Sorry for originally not being more explicit about the function we are using, when I posted I was generally wondering whether there was a principled reason why the repeated measures ANOVA should not work properly in case of more levels than subjects.

We are in fact using the statfun_depsamplesF.m function of the fieldtrip package in matlab (we are looking at amplitudes of electroencephalography data). I will dive into the details tomorrow and try to track down whether it runs a MANOVA internally. Thanks for the hint, I will keep you updated.

Tim

[tkie...@gmail.com]

Hi Art,

also to you of course thanks a lot for the fast response. I put the color example above to explain the general kind of problem we are facing. We are in fact looking at different angles in a visual display, and recorded EEG data. Sorry for the confusion, I guess my attempt at simplifying things by coming up with an illustrative example was misleading.

Cheers
Tim

[Bruce Weaver]

On 23/10/2013 4:56 PM, tkie...@gmail.com wrote:
> Hi Bruce,
>
> Thanks a lot for your answer. As you said, SPSS did not produce output for the Multivariate Tests.
>
> Sorry for originally not being more explicit about the function we are using, when I posted I was generally wondering whether there was a principled reason why the repeated measures ANOVA should not work properly in case of more levels than subjects.
>
> We are in fact using the statfun_depsamplesF.m function of the fieldtrip package in matlab (we are looking at amplitudes of electroencephalography data). I will dive into the details tomorrow and try to track down whether it runs a MANOVA internally. Thanks for the hint, I will keep you updated.
>
> Tim

For those who are interested, here's the code:

http://code.google.com/p/fieldtrip/source/browse/trunk/statfun/statfun_depsamplesF.m?r=935

[Jeff Miller]

On Thursday, October 24, 2013 10:37:52 AM UTC+13, Bruce Weaver wrote:
> For those who are interested, here's the code:
> http://code.google.com/p/fieldtrip/source/browse/trunk/statfun/statfun_depsamplesF.m?r=935

This part of the code looks suspect to me:

nunits = max(design(cfg.uvar,:));
dfdenom = nunits - ncontrasts;

I'm not confident, but it looks to me like:

nunits = the number of subjects.
ncontrasts = (at least by default) the number of df's
for the repeated measures factor factor.

In that case, dfdenom for the factor effect in a repeated measures design should be (nunits-1)*ncontrasts.

[tkie...@gmail.com]

Hi Jeff,

Thanks for looking into this. The line you are referring to is the suspicious one (in fact the one that made me wonder whether I had a conceptual misunderstanding here).

Below the line, they compute the actual F statistic with

for smplindx=1:nsmpls
datonesmpl=reshape(dat(smplindx,poslabelsperunit),nunits,nconds);
contrasts=datonesmpl*cfg.contrastcoefs';
contrastavg=mean(contrasts,1);
dev=contrasts-repmat(contrastavg,nunits,1);
covmat=(dev'*dev)/(nunits-1);
s.stat(smplindx)=nunits*contrastavg*inv(covmat)*contrastavg';
end;

Apparently, they are comparing the factor levels to their average, which makes a MANOVA unlikely. I will go and see whether there is something on this in the fieldtrip mailing list. I will keep you updated.

If someone has a suggestion in the meantime, of course please let me know about it.

Thanks again for your help and advice
Tim

[tkie...@gmail.com]

OK, I have now heard back from the original author of the function. Though this is not stated in the code, the implementation is a MANOVA and not a repeated measures ANOVA.

I will now simply implement a repeated measures ANOVA for fieldtrip.

Thanks again for your help, your suggestions towards the MANOVA helped solving this riddle.
Tim