Response times reliability (I read the papers)

[Henryk Bukowski]

Dear Joshua,

As I was trying to convince other researchers to use JsPsych and they asked me about RT reliability.

I knew about the papers showing no significant difference in variance of psychological effects but I could not explain the specifics, which is:

WHY JSPYCH CAN BE SLOWER THAN SAY THAN PSYCHTOOL BOX?

- I read it is because of the queuie mechanism of java script but I am unclear about influence the lag?

- hosting server processing capacity?

- user's computer processing capacity?

- refresh rate of monitor?

- number of tabs opened on the web-browser?

Most importantly, is the gap constant within subject? i.e. identical across all subject's trials?

If gap is constant, the gap should disappear when calculated say a congruency effect ((incongruent trial: 450 ms +34ms gap) - (congruent trial: 340 ms + 34 ms gap) = 110 ms congruency effect)?

Can we state that's what happens with JsPsych?

Also, I read it is reliable for within-subject comparisons (which suggest the previous question's answer is yes) but what about between-subject comparisons or individual difference correlations?

If not, does that mean that thee gap is not constant within a subject's trials?

Finally, can we do something to measure the gap and account for it?

Can we reduce the gap by somehow managing to put stimulii display in sync with user's monitor refresh rate? This would be very useful subliminal presentations, which you cannot afford for now with a 10-40 ms gap in JsPsych.

That's a lot of questions but your answers will help me (and other readers) trust JsPsych with actual arguments.

[Josh de Leeuw]

Henryk,

These are great questions. I'm going to try and explicitly organize my response around the various themes.

1) Why does jsPsych measure longer RTs than Psychtoolbox?

There are several pieces to this, but the simple version is that Psychtoolbox (and any other well-built desktop experiment software) has greater control over the timing of stimuli than browser-based software like jsPsych. 

Any web-based system relies on web browsers to execute the code. PTB can do two things that JavaScript executing in a web browser cannot: 1) Take priority on the CPU and block external processes competing for computation, and 2) synchronize timing measurements with the display refresh of the monitor. The delay in RT measurements for jsPsych relative to PTB is most likely due to PTB having a better measurement of when the stimulus appeared on the screen. In jsPsych, there is some unknowable delay between the command to draw on the screen and the actual redraw. There may also be some (smaller) delay between when the user presses a key and the registration of the key press. Although desktop software like PTB has problems with this source of delay as well, it's possible that the delay is worse for JavaScript in a browser.

Importantly, there are a few things we can rule out as causes of the longer RT measurements: specifically anything related to network latency or hosting isn't relevant. All of the RT measurement in jsPsych happens locally on the user's computer. In fact, the entire experiment executes locally, so the only thing that the network can affect is how long the experiment takes to load at the beginning (and how long stimuli take to load during the experiment if you don't preload them at the beginning). 

2) Is the delay in RT consistent within subject?

Yes. It's not so consistent that we could do a subtraction of RT on every trial and get the absolute RT for every measurement. But it is consistent enough that any within-subjects comparison of means should be accurately estimated.

3) Is the delay in RT consistent between subjects?

This depends on what your error thresholds are. The variations in browsers (Edge vs. Chrome vs. Safari vs. Firefox) and platforms (Linux vs. OSX vs. Windows) means that code runs differently on different machines. As a result, there are RT measurement errors that are caused by subjects using different devices to run the experiment. Reimers & Stewart have a nice paper with a simulation showing that the errors from this can be overcome with a slightly larger sample size (https://link.springer.com/article/10.3758/s13428-014-0471-1). The risk here is if you are running a quasi-experimental design where subjects may be assigned non-randomly and therefore the device/OS/browser that subjects are using is possibly correlated with an experimental condition or demographic measure.

4) Delay in RT vs. Delay in Presentation Time

I want to clarify what might be a confusion point. The 2016 paper that estimated a 10-40ms average RT delay for jsPsych relative to PTB is only about RT, not display duration. It does not mean that stimuli are displayed 10-40ms longer than intended. A recent paper (https://psyarxiv.com/d6nu5/) does a fairly exhaustive comparison of different experiment software and finds that jsPsych performs pretty well on display duration, with the exception of Firefox where durations are more likely to be one refresh cycle (16ms) too long.   jsPsych currently uses what I would call a "naive" method for controlling display duration. There're definitely opportunities for doing better. There's a team of folks interested in web-based research who are currently trying to create a better method for controlling display duration, and once we find one I'll certainly be updating jsPsych to improve the accuracy of the timing.  So subliminal presentations are possible, though if you really need them you should get in touch with me because I have alpha versions of some plugins that do a better job at controlling display duration than the default set. (We've successfully replicated attentional blink studies with these, where small deviations in display timing would be a major problem).

Cheers,

Josh

--
You received this message because you are subscribed to the Google Groups "jsPsych" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jspsych+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jspsych/c549ef60-8072-4bb7-a54c-e869edc0d9ae%40googlegroups.com.

[Henryk Bukowski]

I could not hope a better reply ! Thanks a lot!

Regarding between-subject variability, one option could therefore be to encode each participant's web browser & OS to use this as control variables.

Looks pretty easy to collect with navigator.userAgent    https://www.w3schools.com/jsref/tryit.asp?filename=tryjsref_nav_all

[Josh de Leeuw]

It's possible that recording some information about browser and OS and correcting for it would reduce noise. There are other sources of between-computer variability (specific hardware) that are not captured by the userAgent string, so it wouldn't fully eliminate the problem.

--

You received this message because you are subscribed to the Google Groups "jsPsych" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jspsych+u...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jspsych/8ebc9be9-9100-4e57-a195-b90f2b39fb33%40googlegroups.com.