Reaction Time in Online Experiments

[Catherine Arnett]

I want to run a lexical decision task, but I am worried about collecting reliable reaction time data. I'm mostly worried about distracted participants and unreliable internet connection. Can I rely on reaction time data collected online? 

In the demo on Tuesday, there was mention of some research on this. Does anyone have references? Recommendations on instructions, platform to use, etc. would also be helpful. 

[jkhart...@gmail.com]

The most up-to-date information here from this recent study. What you can see is that precision if generally quite good for most platforms. That is, the trial-to-trial error tends to be on the order of a couple ms. Which is more than good enough. 

However, there is an issue of lag. Basically, RTs are often recorded as 30-50 ms longer than they actually were. The exact amount depends on what software you are using, what browser and operating system the user has. For a within-subjects study, this won’t matter, because the lag will cancel out. 

But it’s potentially a problem for between-subjects studies if browser/OS aren’t randomized across conditions. An example would be a cross-cultural study, since people in different cultures may be more or less likely to use a particular operating system or browser, and the differences in lag would become a confound. Now, these confounds shouldn’t produce effects larger than 20-30 ms, so if you are looking for much larger effects, I wouldn’t worry too much about it. 

Nonetheless, it would be nice to reduce this source of noise. It’s something that the jsPsych folks are actively thinking about. We’d have done it years ago, but none of the grants we’ve written for that specific work have ever been funded!

[catherin...@gmail.com]

Thanks so much!