R Script for getting Qualtrics IAT results

[franck sarrazit]

Hi All

Has anyone tried the R Script to read IAT results (generated on Qualtrics)?

I imported the code into Rstudio, and changed the directory/path to my CSV file (generated in qualtrics), but i am running into multiple Warnings (see below). As a result, there is no graph being plotted.

mutate_()` is deprecated as of dplyr 0.7.0. Please use `mutate()` instead.

filter_()` is deprecated as of dplyr 0.7.0.

summarise_()` is deprecated as of dplyr 0.7.0. Please use `summarise()` instead

and many others like this . 

Would anyone know if this is due to Rstudio and the Dplyr package mismatch? 

Many thanks in advance

F

[Michael Pinus]

Hey Franck,

The author of the IAT R package hasn't updated it since 2016. dplyr has undergone major changes during this period, and the warnings you're seeing reflect that. 

Specifically, the underscore variants of dplyr's functions (`filter_()` instead of `filter()`) used another system for code evaluation, so that they can be used inside other functions.

dplyr now provides another mechanism for using its verbs inside other functions, using quosures and the nifty but somewhat confusing !! and !!! operators, so I wouldn't dig too deep into the underscored dplyr verbs.

In my experience, you can ignore these warnings for now, and IAT's functions should still produce the same output as before. 

The fact that a plot is not showing might be unrelated to these warnings. 

Which plot are you trying to produce? Could you attach the code you're using, and a minimal example of the data? (make sure to de-identify it beforehand, if that's relevant).

Going forward, you could either use the renv R package to keep a copy of the IAT and dplyr packages that works, or use another package to score the IAT. I tried the implicitMeasures package a few weeks ago, but the syntax is very different compared to Dan Martin's IAT package, and I didn't spend enough time looking into it. It is probably worth your time, if you'll use the IAT or another implicit measure again in the future.

On a side note, I've been meaning to fork and update the IAT package for ages, but haven't gotten around to it yet. It might happen in the next couple of months, but that's probably not relevant for the issue your facing right now. 

[franck sarrazit]

Thanks Michael 

I was referring to Elad zoltnick’s script for MinnoJs

I think you’re describing another R package?

Sent from my iPhone

On Sep 10, 2020, at 5:48 PM, Michael Pinus <michae...@gmail.com> wrote:

Hey Franck,

--
You received this message because you are subscribed to a topic in the Google Groups "Minno.js" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/minnojs/CXvFbScM0WI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to minnojs+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/minnojs/24c7a609-bcea-4fd3-a171-5088e1693f09n%40googlegroups.com.

[Elad Zlotnick]

Hi Franck,

Which script is this?

The only script I can think of is the qIAT specific script that I wrote and uses the latest version of tidyverse.

best,

Elad

[Yoav Bar-Anan]

Franck referred to "the R Script to read IAT results (generated on Qualtrics)" So, this is probably my script, based on Elad's script. 

I re-run my script now, did not see those warnings, and did see the graph. 

Franck, If you send me a link to your data (perhaps privately and not to the group), I can see whether my script runs over your data without issues. 

All the best,

Yoav

You received this message because you are subscribed to the Google Groups "Minno.js" group.
To unsubscribe from this group and stop receiving emails from it, send an email to minnojs+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/minnojs/b2bcc3d3-9e14-47ce-abef-38c56941dfc5n%40googlegroups.com.

[Elena Sammaciccio]

Dear all, 

I have been trying to program my IAT on Qualtrics using prof. Bar-Anan's code available here https://github.com/baranan/minno-tasks/blob/master/IAT/qualtrics/quiat10.js

and then using his R codes here https://github.com/baranan/minno-tasks/blob/master/IAT/qualtrics/minno.qualtrics.iat.process.rmd

I have three questions:

1) Is there a reason why the cleanIAT function of the IAT package calculates a d-score which has an opposite sign with respect to the one I get directly from Qualtrics using the MinnoJS code? 

2) Moreover, I wanted to ask whether the IAT d-score has a different interpretation if category1 is on the left first or on the right first (and hence if i should take into account the initial position of category1 when calculating the d-score with the cleanIAT function, for example by dividing the sample into two groups and get d-scores for the two groups). I assume not, but I am not sure.

3) I am using a gender-caree IAT, with Category1=Career, Category2=Family, Attribute1=Male, Attribute2=Female.

Am I understanding correctly that, on the basis of prof. Bar-Anan's code, if we display as feedback

'Your responses suggested a strong automatic preference for Family over Career.' 

we are meaning that the respondent is implicitly associating the couples Career/Male and Family/Female?

Thanks a lot!

Elena

[Mayan Navon]

Hi Elena,

1. Blocks 3,6 that are used by cleanIAT should present the stereotype-congruent trials. If they currently present the incongruent pairing then it is likely that your score would be the opposite.

2. Yes, some IATs show block-order effects and this should be taken into account in the analysis.

3. I'm not sure I understand, this is the exact feedback you got? The text should include both categories and attributes...

To view this discussion on the web visit https://groups.google.com/d/msgid/minnojs/d9b56096-ebb8-4974-b8d7-17081bf1e0f4n%40googlegroups.com.

--

Mayan

[Yoav Bar-Anan]

Hi Elena, 

In addition to Mayan's answers, here are a few more clarifications, below:

On Tue, Mar 30, 2021 at 10:44 PM Elena Sammaciccio <elena.sa...@gmail.com> wrote:

Dear all, 

I have been trying to program my IAT on Qualtrics using prof. Bar-Anan's code available here https://github.com/baranan/minno-tasks/blob/master/IAT/qualtrics/quiat10.js

and then using his R codes here https://github.com/baranan/minno-tasks/blob/master/IAT/qualtrics/minno.qualtrics.iat.process.rmd

I have three questions:

1) Is there a reason why the cleanIAT function of the IAT package calculates a d-score which has an opposite sign with respect to the one I get directly from Qualtrics using the MinnoJS code? 

YBA: In the extension, the compatible blocks are those in which category1 and attribute1 (and category2 and attribute2) share a key. A positive IAT D score would reflect faster response in the compatible blocks than in the incompatible blocks. The R script I shared uses the same rules. To make sure you know what the D score reflects, look at the data and see what 'cond' is when 'comp' is "compatible". You can even open the Qualtrics data file in notepad, and I think it will be easy to see. Like in this photo:

 

2) Moreover, I wanted to ask whether the IAT d-score has a different interpretation if category1 is on the left first or on the right first (and hence if i should take into account the initial position of category1 when calculating the d-score with the cleanIAT function, for example by dividing the sample into two groups and get d-scores for the two groups). I assume not, but I am not sure.

YBA: The interpretation is the same. But, note that as Mayan wrote, many IATs have a block-order effect that would influence the score. What to do about that is a bit beyond the scope of this group, but, briefly -- if you care about the means, then counterbalance between participants (50% perform each block-order condition). If you care mostly about correlations or comparison between groups (which is usually what really matters in research), then the block-order effect has no serious effect, and you can ignore it (or add the block-order as a factor when testing the effect of a manipulation).

3) I am using a gender-caree IAT, with Category1=Career, Category2=Family, Attribute1=Male, Attribute2=Female.

Am I understanding correctly that, on the basis of prof. Bar-Anan's code, if we display as feedback

'Your responses suggested a strong automatic preference for Family over Career.' 

we are meaning that the respondent is implicitly associating the couples Career/Male and Family/Female?

YBA: First, if you want to provide meaningful feedback to participants (and I recommend against it), then you should probably replace those messages (e.g., the parameter fb_strong_Att1WithCatA_Att2WithCatB).  I see that we forgot to do that in our example for the gender-career IAT. Here is what we're currently doing on Project Implicit's demonstration website:

   fb_strong_Att1WithCatA_Att2WithCatB : 'Your responses suggested a strong automatic association for attribute1 with categoryA and attribute2 with categoryB.',
fb_moderate_Att1WithCatA_Att2WithCatB : 'Your responses suggested a moderate automatic association for attribute1 with categoryA and attribute2 with categoryB.',
fb_slight_Att1WithCatA_Att2WithCatB : 'Your responses suggested a slight automatic association for attribute1 with categoryA and attribute2 with categoryB.',
fb_equal_CatAvsCatB : 'Your responses suggested little or no automatic association between attribute2 and attribute1 with categoryA and categoryB.'

If you are asking only for debugging purposes, then preference here is a stronger association with attribute2. So, indeed, I share your interpretation of the feedback.

All the best,

Yoav 

[Elena Sammaciccio]

Dear all,

thanks for your quick and extensive answers. It is everything clearer now.

Prof. Bar-Anan, can I just ask you a quick clarification? Sorry to bother.

I have tried many times the test, also "cheating" in order to get strong association between (Category1/Attribute1) Male/Career and (Category2/Attribute2) Female/Family. The compatible are Male/Career and Female/Family. I do not allow for randomization of the position of both categories and attributes (so Male always on the left, and Female on the right. Career on the left first)

You say that with your code a "positive IAT D score would reflect faster response in the compatible blocks than in the incompatible blocks."

Is there a reason why I get a negative d-score (calculated through your minnojs code, not through R) as I show here below?

(I get the same value, but positive (+1.5), using the R code you propose).

[Yoav Bar-Anan]

Hi Elena,

Indeed, the D score computed by the extension itself is smaller, the stronger the relation between cateogry1 and attribute1 and category2 and attribute2 (in the present case, Career+Male and Family+Female.

This D score is used by Project Implicit's website, internally, for uninteresting reasons, but it is does not use exactly the recommended scoring algorithm, so it is recommended to compute your own D score from your data.

All the best,

Yoav

--

You received this message because you are subscribed to the Google Groups "Minno.js" group.
To unsubscribe from this group and stop receiving emails from it, send an email to minnojs+u...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/minnojs/8d52bec8-4a34-45f6-a3a1-40e7fe0a70fen%40googlegroups.com.

[Elena Sammaciccio]

Thank you so much for the clarification! It was so helpful.

I wish you a good day and forgive me again for bothering with all my questions!

All the best,

Elena

[Julie Beadle]

Hi Yoav et al., 

I have used your R script to calculate D scores for an IAT that I ran with Qualtrics using the MinnoJS extension. The R script is working for me, but I have a few questions.

https://github.com/baranan/minno-tasks/blob/master/IAT/qualtrics/minno.qualtrics.iat.process.rmd

My questions are:

1. Using your R script, do resulting positive D scores always indicate a stronger association between the category-attribute pairs in the compatible block than the incompatible block?
2. Will the script automatically exclude people with problematic rt or accuracy performance, or does something need to be edited in order for an exclusion to happen? (No one is currently excluded from my data, n=30).
3. What is the b0rd variable and how is it involved in calculating D scores?
4. Is it correct that I can include data from participants who have completed the incompatible block first and participants who have completed the compatible block first in the same csv file that I use with the R Script, and that positive D scores would indicate the same thing for both participant groups? That is, in terms of calculating D scores, are differences in block order between participants corrected by using  "blockName"?

Please let me know if anything is not clear.

Thank you,

Julie 

[Mayan Navon]

Hi Julie,

My questions are:

1. Using your R script, do resulting positive D scores always indicate a stronger association between the category-attribute pairs in the compatible block than the incompatible block?

A: It depends on the way your IAT task defines the compatible and incompatible blocks. See these lines:

#Make sure the 'compatible' trials are indeed what you expected them to be.
table(iat.raw$comp, iat.raw$cond, exclude=NULL)

If you see the compatible/incompatible are opposite, then change that before you move on in the script. If they are correct, then yes, higher D scores indicate stronger stereotype-consistent responding. 

1. Will the script automatically exclude people with problematic rt or accuracy performance, or does something need to be edited in order for an exclusion to happen? (No one is currently excluded from my data, n=30).

A: The script uses the cleanIAT function of the IAT R package. That function uses Greenwald et al.'s recommendations regarding *trials*. It also marks participants with too many fast trials, with the SUBEXCL variable in the output. 'Good' participants are marked with 0, so you should only keep them. The script does not do this exclusion for you (only shows a frequency table -- table(iatscore$SUBEXCL, exclude=NULL)). If you want to use some other exclusion rules - then you'll need to add those to your script.

1. What is the b0rd variable and how is it involved in calculating D scores?

A: That variable codes the block-order (whether a participant started with the compatible pairing, e.g., White+good, Black+bad). If I recall correctly, the value of that variable is the pairing the participant started with. You can us that variable to test block-order effects in your study. 

1. Is it correct that I can include data from participants who have completed the incompatible block first and participants who have completed the compatible block first in the same csv file that I use with the R Script, and that positive D scores would indicate the same thing for both participant groups? That is, in terms of calculating D scores, are differences in block order between participants corrected by using  "blockName"?

A:  Yes, you can, because you are relying on the 'comp' variable, that codes the block's compatibility regardless of the block order. As you suggested, differences in block order between participants corrected by using  "blockName".

Best,

Mayan

To view this discussion on the web visit https://groups.google.com/d/msgid/minnojs/17fb286f-07b0-45b7-8413-e42a644c8034n%40googlegroups.com.

--

Mayan

[Julie Beadle]

Thank you, Mayan! Your responses are very clear and helpful!!

Best, 

Julie 

[Assaf Suberry]

Hello Maayan and Yoav 

1)

Your code for analyzing the D score directs to an "iat" type of file. I don't have such file (onle csv)

  dir = 'C:\\Users\\yoav\\Documents\\bigfiles\\qualtrics.iat'   

How can the file be found? 

2) 

My CSV file didn't create  a "bOrd"collum

Is it a problem? 

Thank you Assaf

ב-יום חמישי, 25 בנובמבר 2021 בשעה 20:34:22 UTC+2, jmar...@gmail.com כתב/ה:

[Mayan Navon]

Hi Assaf,

1) That is the directory where Yoav has saved his csv. 'qualtrics.iat' is probably the name of a subfolder. You can set your directory according to the location of your csv file.

2) Weird. Do you have a 'comp' column?

To view this discussion on the web visit https://groups.google.com/d/msgid/minnojs/0c6b2d5a-4f4c-41bb-a640-e9fc466a8e86n%40googlegroups.com.