Within-subjects proportional data

[Ely]

I may be missing something very obvious, but I am stuck on the following
problem and hope someone can advise:

I want to do a psychology experiment where participants will be tested on
their ability to detect targets under two conditions. One each trial they
see one target paired with one distractor. They can either pick the target
(correct), pick the distractor (false alarm) or pick nothing (miss).

I am predicting that one condition will produce more correct hits, but am
also interested in whether condition affects the proportion of each type of
error (false alarm or miss). There are 12 trails per participant. I can't
use signal detection because there are 3 choices on each trail, and I can't
use chi-square because there are multiple trails per participant.

Testing the number of correct hits in each condition is easy enough by
calculating number of correct identifications out of 12 for each
participant. I could also separately compare the number of false alarms and
the number of misses between conditions. But obviously these measures are
not independent, and apart from the problem of multiple testing, what I
really want to know is whether the proportion of each type of error differs
between conditions.

I could take the errors for each participant and calculate the proportion of
false hits to misses in each condition and use these proportions for
analysis. However, the overall number of errors will be different for each
participant, and may be very small, meaning that proportions will tend to
take extreme values and the range of variation in proportions is restricted.
Also one condition is predicted to produce a smaller number of errors than
the other, meaning it will have more extreme proportions. I presume I should
do an arcsin transformation on the proportions, but I am concerned that the
test would still be unsound.

Does this sound like a serious violation of t-test assumptions, and should I
abandon this design and go back to the drawing board? Or is there some other
way of treating the data that I have missed? I would rather make sure that
the data will be analysable before collecting it.

Any thoughts appreciated...

Janet

[Ray Koopman]

See J.P.Shaffer, The analysis of variance mixed model with allocated
observations: Application to repeated measurement designs, Journal
of the American Statistical Association, 76 (1981), 607-611.

[Bill Howells]

Non-linear mixed model with logit link! As implemented in SAS NLMIXED.
(don't you just love it when people throw out terms like that!) :)

[Thom]

Bill Howells wrote:
> Non-linear mixed model with logit link! As implemented in SAS NLMIXED.
> (don't you just love it when people throw out terms like that!) :)

These suggestions are technically superior, but the analysis of
proportions is a lot simpler and would be a useful starting point
before doing the more complex analyses.

I'd also consider a simple repeated measures ANOVA looking at the
proportion of hits, misses and false positives. These data would be
ipsative, but there are papers arguing that this ANOVA still produces
sensible results in these cases.

Thom

[Ray Koopman]

Thom wrote:
> [...]

>
> I'd also consider a simple repeated measures ANOVA looking at the
> proportion of hits, misses and false positives. These data would be
> ipsative, but there are papers arguing that this ANOVA still produces
> sensible results in these cases.

That's the whole point of the Shaffer paper that I cited.

[Janet]

"Ray Koopman" <koo...@sfu.ca> wrote in message
news:1127430688....@g47g2000cwa.googlegroups.com...

Thanks for the responses. Not being a statistician and not having
encountered ipsative data before, I didn't conceptualise this as an ANOVA
factor, as the level (hit, false alarm and miss) is determined by
participant response rather than being manipulated by the experimenter.

So the idea is that we could use a fully repeated measures ANOVA with two
factors; condition and reponse type. It would give a zero for the main
effect of condition, but the interaction between condition and response type
should still be meaningful.

Although we would predict an interaction, it wouldn't actually tell us where
the shift in responses is occurring. What we really want to test the more
specific hypotheses is a comparison of hits between conditions, and probably
of the relative frequency of each type of error within each condition. I
assume that if the omnibus F test is valid, then post hoc or planned
comparisons for specific contrasts can also be performed with ipsative data?

I hit the wrong button before and didn't send the reply to your email
adress, sorry.

Janet

[Ray Koopman]

Janet wrote:
> [...]

> I assume that if the omnibus F test is valid, then post hoc or planned
> comparisons for specific contrasts can also be performed with ipsative
> data?

Yes. In fact, if the Bonferroni problem is handled properly then the
contrasts do not require the overall F to be done, let alone be
significant.

[Thom]

Thanks - I'll look it up.

I'd guessed it was a SAS PROC MIXED or NLMIXED type paper. I was
thinking of the Greer & Dunlop (1997) Psychological Methods paper.

Thom