Evidence on e-prime timing accuracy compared to other similar software?

[CB]

I realise I risk annoying people here, as I imagine this has been
addressed here and elsewhere. But with an admittedly quite cursory
look so far (here, journal databases, pst's web site), I haven't found
any objective tests of e-prime against other similar packages. I'll go
on looking, but if anyone knows of anything that could save me a major
hunt, I'd appreciate it.

I'm well aware of the complexities involved (and the vagueness of the
question), and that hardware and OS issues are at least as significant
as the software. I've read the user and reference guides, thinned
Windows down, run the Refresh Clock tests, etc.

(background: I have an e-prime experiment that's nearly ready to run,
but there's some suggestion that I should redevelop it in
Presentation, because of what seem to me like some folklorish
impressions at large about e-prime's relative timing unreliability).

[CB]

One addition: I did come across one article comparing simple RT
measurements made with eprime using various mice, to those made more
directly (with an oscilloscope). It found tiny differences
attributable to the PC + Windows + Hardware contribution (of the order
of 1ms). But there were no comparisons there with other presentation
software.

[David McFarlane]

Two thoughts:

1) For an absolute measure of timing accuracy (as opposed to
comparisons with alternatives), PST addressed this in Chapter 3 of
the User's Guide that came with E-Prime, which I *insist* that anyone
who cares about critical timing read.

2) Awhile back I found a published review comparing E-Prime with some
alternatives, unfortunately I cannot find it in all the stacks on my
desk at the moment. But the upshot was that all the best task
software for Windows (including E-Prime) rely on Microsoft's DirectX
technology, and all provide similar timing accuracy.

But please don't take my word for this, keep looking, and maybe I
will post a link to the review when I find it. In the meantime, you
might try Google searches with terms including E-Prime, DirectRT,
Inquisit (by Millisecond), DirectX, etc. And then please let us know
what you find.

-- David McFarlane, Professional Faultfinder

[CB]

On Apr 22, 12:11 am, David McFarlane <mcfar...@msu.edu> wrote:
>
> 1) For an absolute measure of timing accuracy (as opposed to
> comparisons with alternatives), PST addressed this in Chapter 3 of
> the User's Guide that came with E-Prime, which I *insist* that anyone
> who cares about critical timing read.

Thanks David. I have read Ch3. and Appendix A of the UG, and agree
that they're essential reading.

> But the upshot was that all the best task
> software for Windows (including E-Prime) rely on Microsoft's DirectX
> technology, and all provide similar timing accuracy.

That goes along with my impression that differences between the
serious packages would be minimal, and swamped in practice by other
software and hardware differences between PCs.

> But please don't take my word for this, keep looking, and maybe I
> will post a link to the review when I find it.  In the meantime, you
> might try Google searches with terms including E-Prime, DirectRT,
> Inquisit (by Millisecond), DirectX, etc.  And then please let us know
> what you find.

If you do come across the reference I'd be interested. The only useful
published ref I've come across so far is http://www.ncbi.nlm.nih.gov/pubmed/12834085.
It's really comparing response devices, but is tangentially
interesting about e-prime (though in terms of absolute numbers; no
comparisons).

[CB]

Here's one comparative review: http://www.ncbi.nlm.nih.gov/pubmed/16955731

It's a feature comparison, not addressing timing at all, but might be
of interest to some here.

[CB]

There's also a fair bit of info, using external instrumentation, here:
http://www.psychology.heacademy.ac.uk/etsl/bench_results.html.
Packages examined are limited to e-prime, ERTS (running on DOS) and
SuperLab.

[David McFarlane]

Thanks, that's the one I referred to earlier, you
saved me a lot of trouble! This 2006 paper by
Christoph Stahl compares DirectRT, E-Prime,
Inquisit, and SuperLab (2004). Here is the
relevant excerpt from that paper -- note especially the final clause:

"All four packages rely on DirectX technology2 to
support stimulus presentation and response
registration with high temporal resolution, and
to interact with the output (e.g., video and
sound) and input interfaces (e.g., keyboard,
mouse, and joystick) available for the Windows
operating system. The reviewed packages claim
millisecond accuracy in stimulus presentation and
response registration; supporting evidence has
been reported for Inquisit (De Clercq, Crombez,
Buysse, & Roeyers, 2003). It is beyond the scope
of this review to provide a test of that claim
for the other packages. In general, some caution
is necessary regarding timing accuracy on a
Windows operating system: Because it supports
multithreading, (i.e., multiple processes running
at the same time, sharing one central processing
unit), perfect timing accuracy cannot be
warranted (see also Myors, 1999). If an
experiment program—be it one of the above
mentioned packages or self-programmed—is to
present a stimulus at time t, it can do so
accurately if no other processes are running. If,
however, another program is running in addition
to the experiment program, it might occupy the
central processing unit at time t and thus delay
stimulus presentation. Yet, within the DirectX
framework, several measures can be taken to
minimize this and other sources of timing error
(e.g., Forster & Forster, 2003), and it can thus
be assumed that timing accuracy is potentially
high for the reviewed packages (MacInnes &
Taylor, 2001; Plant, Hammond, & Whitehouse, 2002)."

Note that the E-Prime documentation also provides
supporting evidence for PST's claim of millisecond accuracy.

[CB]

On Apr 24, 3:53 am, David McFarlane <mcfar...@msu.edu> wrote:
> At 4/21/2009 04:06 PM Tuesday, CB wrote:
>
> >Here's one comparative review:http://www.ncbi.nlm.nih.gov/pubmed/16955731
>
>

> Thanks, that's the one I referred to earlier, you
> saved me a lot of trouble!  This 2006 paper by
> Christoph Stahl compares DirectRT, E-Prime,
> Inquisit, and SuperLab (2004).  Here is the
> relevant excerpt from that paper -- note especially the final clause:

It's still largely a matter of assumptions, however (albeit reasonable
ones).

There is some more empirical stuff in the heacademy page I linked to,
but I haven't yet had time to read and digest properly.