D-Score Calculations Inquiry

[Adebimpe Adewoye]

Hello,

I used the Weight and Flower/Insect IATs and applied the Greenwald improved scoring algorithm to calculate the D-scores. The blocks were presented in randomized orders (e.g., Thin/Bad, Fat/Good vs. Fat/Bad, Thin/Good). To account for this randomization, I multiplied the final D-score by -1 if the first block was "Fat/Bad, Thin/Good" to correct for the reversed order. Could you please confirm that this approach is correct? Additionally, I would appreciate guidance on how to interpret the D-scores. Is there a universal range or standard for what different D-scores indicate?  Thank you for your assistance.

Thank you

Adebimpe Adewoye

[Yoav Bar-Anan]

Hi Adebimpe, 

This group is for questions regarding using Minno.js for programming web studies, rather than about the IAT in general. 

But, here are my answers:

On Wed, Aug 21, 2024 at 7:58 PM Adebimpe Adewoye <made...@udel.edu> wrote:

Hello,

I used the Weight and Flower/Insect IATs and applied the Greenwald improved scoring algorithm to calculate the D-scores. The blocks were presented in randomized orders (e.g., Thin/Bad, Fat/Good vs. Fat/Bad, Thin/Good). To account for this randomization, I multiplied the final D-score by -1 if the first block was "Fat/Bad, Thin/Good" to correct for the reversed order. Could you please confirm that this approach is correct?

YBA: If you computed the D score such that positive scores reflect better performance in Blocks 3 and 4 than in Blocks 6 and 7, then multiplying the score by -1 in one of the block order conditions (usually, when the "incompatible" pairing was the pairing in Blocks 3 and 4) would provide you a score that reflects a specific associations (usually, the compatibility effect). Another possibility is to recode the blocks with the compatible pairing (e.g., "Fat/Good, Thin/Bad") as Blocks 3 and 4, and blocks with the incompatible pairing (e.g., "Fat/Bad, Thin/Good") as Blocks 6 and 7, and only then compute the D score. That's what I usually do, and then I do not have to multiply by -1 based on the block order condition. 

 

Additionally, I would appreciate guidance on how to interpret the D-scores. Is there a universal range or standard for what different D-scores indicate?  Thank you for your assistance.

YBA: It is common to treat the D score 0 as indicating no preference. There have been some arguments against that, which would mean that, at best, the D score can only guide inference about differences between groups and individuals, but not for what specific preference they showed. However, I think that Greenwald et al.'s (2022, p. 1163) evidence and arguments in favor of treating a 0 D score as no-preference are compelling. Therefore, I believe that it is reasonable to interpret 0 as no preference, but only on the group level, and only if the block order condition is counterbalanced. For each individual, I do not believe that one IAT score is easily interpretable due to the block order effects. 

As for making inference regarding the size of the IAT preference, Nosek et al., 2007 once wrote "For exposition purposes, IAT scores

D > .15 or D < -.15 were counted as showing an evaluation in one direction or the other. This does not imply that there is something psychologically distinct about this break point in the continuous distribution of scores." Despite that warning Rubinstein et al. (2018) still used the .15 as a cutoff for inferring a preference or no preference. I think that it is a mistake. In my view, the interpretation of the D score should be like most other scores in social psychological research -- based on the effect size computed from those scores (e.g., Cohen's d effect size). So, I would not refer to a D of 0.6 (for example) as a strong preference. But if a group shows a mean of a D score of 0.18 and SD of 0.35, I would consider that a moderate effect because Cohen's d of that effect is d = 0.51.

All the best, 

Yoav

[Adebimpe Adewoye]

Thank you so much for responding. I did use the Minnos.JS and programmed IAT with Qualtrics, however I just noticed that the D scores are interpreted wrongly. For instance, I got an output that interpreted a -0.51 D score as a preference for thin over fat instead of a preference for fat over thin. What do you think could have been the issue and can this be fixed despite the fact that data have been collected? Thank you for your usual assistance.

[Yoav Bar-Anan]

Hi, 

I am not sure what you mean by an output of a D score "that was interperted". Perhaps you refer to the D score and the feedback message saved by the IAT script? Or perhaps to the D score computed with the R script and the feedback message saved by the IAT script?

As a reminder, it is strongly recommended to compute the D score on your own, from the raw data (e.g., in the blog post about using the IAT script in Qualtrics). It is reasonable to verify that you have done that correctly by comparing with the D score and the feedback message saved by the Minno.js IAT script with the data. Is that what you have done, and you are concerned about the mismatch between the D scores you computed and the feedback message recorded by Minno.js?

Yoav

 

--
You received this message because you are subscribed to the Google Groups "Minno.js" group.
To unsubscribe from this group and stop receiving emails from it, send an email to minnojs+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/minnojs/CAAEzqfXDiUoi94G%2BWDvq1G%2BV02XuZBmxnT0GxCYnEDWLTgRP%3Dw%40mail.gmail.com.

[Yoav Bar-Anan]

Hi Adebimpe, 

If you computed your own D-scores from the raw data, that's good. 

I do recommend to correlate it with the D score recorded by the IAT script itself, just as a sanity check. I typically get a correlation above 0.9 between the score I compute from the raw data and the score recorded by Minno.js. If you don't get a correlation around ~0.9, there might be some mistake in your computation. A highly negative correlation around -0.9 is also fine, and you just need to find out whether the 

Regarding the interpretation of the D score, it depends on which pairing condition you defined as Blocks 3 and 4, and which pairing condition you defined as Blocks 6 and 7, before computing the D score. If you defined Thin/Good, Fat/Bad as Blocks 3 and 4, then positive scores will reflect a pro-Thin preference. 

When I am not sure, I always search for some kind of sanity check. For example, I would look only at one specific session and compute the mean latency in one pairing condition (e.g., Thin/Good, Fat/Bad) and the mean latency in the other pairing condition (e.g., Thin/Bad, Fat/Good). From the difference between these mean latencies, I know whether I expect the D score of that session to be positive or negative. For example, let's say I find a mean response latency of 777 for a specific session in the block with the Thin/Good pairing and a mean latency of 877 for that session in the blocks with the Thin/Bad pairing. I would expect that session to have a positive D score. If that is not the case, I continue to test the other sessions, until I understand what went wrong.

I hope that helps,

Yoav

On Sun, Aug 25, 2024 at 11:49 AM Adebimpe Adewoye <made...@udel.edu> wrote:

Hi Yoav,

Thank you for responding. Yes, I am concerned about the mismatch between the D score computed by the IAT itself and the feedback message recorded.  For instance, a D score of -0.51 was interpreted as preferring thin over fat. After calculating the D score from the raw data the D-scores are different and I have read that that's fine. Additionally, I'd like to confirm how to interpret positive vs. negative D scores. Does a negative D score in the context of Fat/Bad words and Thin/Good words indicate a preference for fat over thin, while a positive D score indicates a preference for thin over fat? Thank you so much for your assistance.

Adebimpe