Tests results on psychopy keyboard event timing

[Jonas Lindeløv]

I just did some timing of response time iohub vs. event.getKeys() here at University of Amsterdam where they have some equipment to measure it. Long story short, PsychoPy has longer latency than Presentation and some strange jitter in addition. I'll report the results and hopefully some of you guys have some ideas on how to improve the performance or whether it's possible at all :-)

Setup: We timed to visual onset because we could not get the parallel port to work. A photoelectrode on the top-left of an 120 Hz LCD monitor triggered a motor to tap the keyboard. The actual movement of the motor was monitored using another photoelectrode and the total round-trip of this system is reliably 332 ms. So a recorded reaction time in PsychoPy of, say, 340 ms would indicate that keyboard+software imposed a delay of 340 - 332 = 8 ms. We tested this on three different keyboards (two external and one laptop) and two different pcs (powerful windows 7 desktop and a normal linux laptop). On both OSes, most background processes were disabled. Both ran PsychoPy 1.80.01.

Here's a picture of the setup: keyboard with motor on "x", oscilloscope and monitor with photoelectrode :-)

Presentation: The aivia keyboard has superior latency (~ 10 ms) to the Dell keyboard (>30 ms) so all results henceforth will report the aivia keyboard results. This is a nice reminder that external hardware matters! There's a jitter of ~ 3 ms and a slight drift from 10 to 12 ms latency. This system is known to deliver timing below 3 milliseconds with better keyboards so 7-10 ms this latency is most likely caused by the keyboard.

PsychoPy on Windows desktop: the testing script can be downloaded from http://lindeloev.net/misc/pp_keyboard_timing.py. It requires http://lindeloev.net/ppc/ppc.py to be in the same directory. Here's some observations:

• The performance of the event module is ~10 ms which is comparable to Presentation, indicating that the software-side of is very low-latency. However, event.getKeys() is "rounded to nearest frame" in the flip-loop, causing a big jitter of a 120th of a second (8.3 ms) in this case. For waitKeys the jitter is the same as Presentation ~ 3 ms, indicating that that's a lower level problem.
• Iohub shows a remarkable buffer-like behavior with a slow drift and sudden jumps. The jumps are suspiciously close to a frame in duration (8.3 ms) although visual stuff shouldn't have an impact. The event module was imported but not used for the iohub tests. I don't know if that has an impact in any way?
• Iohub had slightly longer latency at 13-18 ms.

PsychoPy on linux laptop: The top row shows results for the same monitor and keyboard as the Windows tests. The two bottom rows show results from the integrated keyboard and monitor. Some observations:

• The external devices had much lower latency and jitter than the integrated devices. The jitter is comparable to Presentation (1-3 ms). Remember that the low jitter on event.getKeys is illusory as it is frame-locked rather than locked to the actual onset of the keystroke.
• The latency is ~ 30 ms and thus longer than on the PC. Iohub is for the most part superior to the event module here. Event.getKeys() is ~ 65 ms delayed on this laptop. Once again it is clear that laptops are inappropriate for psychophysics!

All data is attached as a LibreOffice Calc document. There's also results from the Dell keyboard. The graphs are in separate tabs.

Best,
Jonas

[Jonas Lindeløv]

Correction! PsychoPy does not necessarily have longer latency than presentation. This is obvious from the results! I wrote the post as I was doing the testing and initially made the mistake of initiating iohub when testing the event module. That caused ~10 ms extra latency.

Best,

Jonas

[Michael MacAskill]

Hi Jonas,

What a great effort. That setup will be very useful. You should charge for this as a travelling service to other labs to validate their systems! A good way to finance a working holiday around Europe :-)

This is great information for the people who can absorb it. Perhaps a key finding was simply that the differences in the underpinning software approaches could be dwarfed simply by the variation between keyboards.

Sadly though, I despair that many of our users will benefit from this knowledge. Some obsess about "millisecond-level timing", without realising:

(1) Trial-to-trial *physiological* variability is usually far larger then the systematic or stochastic hardware/software timing errors. Sometimes, rough enough is good enough when the actual behaviour is so variable, particularly if you gather sufficient data per subject. One needs to know when to stop "polishing the cannonball".

(2) We are usually interested in differences between conditions rather than absolute reaction times per se, which cancels out any systematic lags.

(3) Some don't even know what a "frame" is, so it is hard to explain how if they use a faster monitor, they could get a better resolution of key press/device responses.

(4) So it is therefore even harder to explain how to break out of the simple and convenient but drawing-locked response checking method, to do faster continuous polling, or to use ioHub (which ideally will eventually just happen automatically without special action required).

A comprehensive guide to all of these issues would be useful: I find it a bit tiresome sometimes having to start from first principles, explaining basic hardware concepts in order to give what should be a simple answer to some user questions.

Anyway, rant over. I guess everyone on this list at least will understand the precision/convenience trade-off of flip-synchronised checking, and when to use alternative techniques. So most useful were your observations on the ioHub behaviour. Will be interested to hear Sol's thoughts on that. It seems very good apart from those periodic spikes. It wouldn't worry me if it has a slightly longer constant lag: the difference in magnitude is tiny, and being constant, can easily be corrected for. More important is that ioHub allows us to more easily break response checking out of the drawing loop with its associated errors, while still displaying dynamic stimuli. Having a hardware-based check of its performance will be great for giving us the necessary confidence in its performance.

Any way, this is a great advance on the recent PLoS One attempt to compare systems on their visual display timing (and it subsequent revisions…) i.e. "closing the loop" by looking at the full hardware response cycle is great.

Regards,

Mike

[Flip Phillips]

It's funny, my favorite things to complain about in modern RT experiments I review are exactly these sorts of issues. I keep meaning to write a whole paper on the subject of hardware + various timing issues w/ various busses + software layers. But I have other things to do and am glad you are doing this...

-f

[Yaroslav Halchenko]

On Thu, 17 Apr 2014, Michael MacAskill wrote:

> What a great effort.

I would like to wholeheartedly agree!!! ;)

> That setup will be very useful. You should charge for this as a travelling service to other labs to validate their systems! A good way to finance a working holiday around Europe :-)

But if traveling serviceman is not appealing to you Michael, I
wondered, could "blueprints" of such setup being detailed somewhere?

Detailed instructions could allow other labs to establish similar setups
and test their hardware (including keyboards in other laptops, since I
guess that is where you would also see large variability). I would be
really keen on trying it on our keyboards and response boxes ;)

--
Yaroslav O. Halchenko, Ph.D.
http://neuro.debian.net http://www.pymvpa.org http://www.fail2ban.org
Senior Research Associate, Psychological and Brain Sciences Dept.
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834 Fax: +1 (603) 646-1419
WWW: http://www.linkedin.com/in/yarik

[Sol Simpson]

Hi Jonas,

Great job on doing all this testing, comparing the different sw layers and how they interact with different levels of hardware quality. Testing at this level is the type of thing that I find fun, so I would be happy to help out if this is something you want to spend more time on. It would be hard for me to reproduce the physical testing set-up you used, so I am working on an alternative approach that will hopefully work out. 

I agree with Michael that many of the differences seen here would likely have no real influence on the statistical results in the majority of studies. However, if events can be timed with more accuracy and precision, then that can only be a good thing IMO. 

I have a few questions about the test code used and some of the observations you have found. I think I'll email you directly with them. A post back here with the results of our discussions would make a lot of sense.

Again, nice work, and talk soon.

[Jonathan Peirce]

Agreed too; nice one Jonas.

Yes, the timing of these things is definitely the next step. I'm pretty certain that timing of stimulus presentation is good in most cases (although i recently played with an LCD panel that was adding a variable lag at the actual display!). I've largely ignored the keyboard so far, but it is time to get this stuff better. I've just taken delivery of a blackbox toolkit for this, but it sounds like you're already on it. :-)

Let's chat about it at the code sprint.

Jon

--
You received this message because you are subscribed to the Google Groups "psychopy-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to psychopy-dev...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

-- 
Jonathan Peirce
Nottingham Visual Neuroscience

http://www.peirce.org.uk/

[Jonas Lindeløv]

Thanks guys! :-) I agree that there's no real-world show-stopper here, as Mike is giving various good reasons for. But it's just nice (and fun) to know that things work reliably. That's the way I feel about the visual module. Generally, if timesByFrames.py yields something sensible, it won't fail a hardware test.

The guys at Amsterdam are keen to help out and run new tests even after I've left. So we do have some collaborators there now. I'm discussing the results with Sol on mail and we'll post intermediate conclusions here. 

Discussing this stuff at the code sprint seems like a good idea. The audio timing would definitely be nice to take a look at too. I haven't seen good audio timing from PP yet and now even my winsound-solution failed on the Amsterdam-system. Once in a while I search for low-latency-cross-platform-open-source audio backends and I've seen many references to jack (http://jackaudio.org/). But I've got no clue whether it's feasible for PsychoPy. I'm just a poor psychologist :-)

Best,

Jonas

[Sol Simpson]

Given the great job done by Jonas in terms of looking at the timing differences between different event detection modules and hardware, over the last few days I 've been motivated to look into this in detail as well. Here are the results I've found so far. The test scripts, results files, and result plotting scripts, are in the attached zip file

Test Goals

The goal of the tests presented here are similar to those of Jonas I believe. My specific intent was to:

• Determine the absolute timestamp error between the actual time of a keyboard key press and the timestamp assigned by the event monitoring software to that keypress event.  
  
• Simultaneously record the keypress event timestamps from both psychopy.event and psychopy.iohub so the same external events are used by each event collection software module.
• When possible, Allow keypress events to occur at anytime during the monitor retrace period.
• Use as direct a method as possible in measuring the actual external keyboard event time.

Event Collection Software 

Only psychopy.event and psychopy.iohub key press data was compared. I do not have access to other commercial software, otherwise that could be tested as well I think.

Computer Specs Used

All tests have been performed using a Windows 7 desktop with a 3.4 Ghz i7 CPU, 8 GB RAM, standard 7200 RPM HD, and a SyncMaster P2770 60 Hz LCD. I did not make any effort to disable software services or programs that normally run. I have put the output from sysInfo.py in the attazhed zip as well.

Tests Conditions

Two test conditions were run, and can be distinguish by what psychopy functions were called and when in each test loop keypress events actually occurred and when these events were requested from psychopy.events and/or psychopy.iohub:

1. flip_keypress_getKeys:  This test matches situations where the psychopy program is:
• updating the display every retrace
  
• checking for new keyboard events each retrace using event.getKeys() or iohub keyboard.getEvents().
• calling win.flip() again to schedule the next display update.
• Go to next trial / iteration when a keypress event is found.
2. waitkeys_keypress: This test matches situations where the psychopy program is:
   
• Drawing graphics to the display (.draw() + win.flip()) once.
• Waiting until a key press event if received from event.waitKeys(), or the iohub keyboard.getEvents() method, in which case it was called in a tight loop. The display is not updated during the keyboard event detection phase.
• When a key press event is found, go to the next trial.

In both test conditions, keypress timestamps were collected simultaneously from psychopy.events and psychopy.iohub keyboard.

In each test, 150 key press events were collected as the sample data.

 

Keyboard Devices and External Key Press Time Calculation

Two keyboard 'devices' were used to run each of the two tests. Both devices were able to provide msec accurate information about the actual keyboard key press time.

• Asus PK100U USB keyboard: A generic low cost keyboard that came with a computer I have. It was used because I figured it would perform in a similar way to other cheap, non-specialty, keyboards. I also did not care if it became damaged. Actual keypress event time was calculated by modifying the keyboard so that when one of the keyboard's key pressure sensors was pushed, cause a key press event to be generated from the keyboard, the state of a digital input line on a microcontroller was also changed from high to low. When the object pressing the key sensor was released, the digital line state went back to high, and the keyboard reported a key release to the OS. The state of this digital input was monitored at 2000 Hz by a microcontroller, and any state changes were sent to the same computer that was running the psychopy test via serial -> usb interface. The round trip delay of this interface is less than 1 msec. Tests confirming the digital event notification delay can be provided on request, as can the details on exactly how the keyboard was modified.
• A Teensy 3 Keyboard Device: The Teensy 3 microcontroller can be programmed to appear as a generic USB keyboard to the operating system it is connected to via USB. The OS installs a generic hid usb keyboard driver when the device is detected. The Teensy 3 can be programmed to generate keyboard events based on any logic you want really, as long as you can program it.  
  It is important to note that with this setup, an external device (the Teensy 3) is connected to the computer via USB just like a 'real' USB keyboard is. The keyboard events created by the Teensy 3 are created externally to the OS and are sent to the OS in the same way that a real USB keyboard would register an event with the OS it is connected to. These are not'fake' keyboard events created by a software only keyboard event generator that injects the events into the processing loop at a application software level. 
  Actual keypress event time was known because the same psychopy program that was timestamping the key press events in psychopy.events or psychopy.iohub was also the program that actually controlled when theTeensy 3 sent the key press events to the OS. This was achieved using the ioSync python api and the ioSync Teensy 3 microcontroller program (source for both ioSync programs is available in psychopy.iohub.devices.mcu.iosync source module). As soon as a genkey command is issued to the ioSync programmed device, a key press event is issued by the device back to the PsychoPy computer. The genkey command also indicates how long after the press that the key release event should be issued. Key release events where done asynchronously to the genkey key press command. The round trip time it takes a python program to send the genkey command to the ioSync Teensy 3 and receive a reply back indicating that the key event has been issued is < 1 msec. 

The error between each keypress timestamp and the time the external key event actually occurred was calculated and used as the y-axis data. Timestamp errors for psychopy.event and psychopy.iohub events are plotted as separate lines, with the mean and stdev error reported for each in the figures title. The time that the external key press event occurred is used for the x axis. 

When the Asus keyboard was used, key press and release events were manually generated by pressing and releasing the modified keyboard key pressure sensor in relatively quick succession. Some small breaks were be taken to allow my wrist to rest as can be seen in the result plots. ;) Therefore when using the Asus keyboard, the time difference between an external keyboard press event and the time of the next monitor retrace will likely be uniformly distributed between 0.0 sec and 1/monitor_refresh_hz sec. The average difference would be expected to be 1/2 the monitor retrace rate. 

When the Teensy 3 Keyboard was used , the genkey commands specified a duration  between 100 and 900 msec that controlled when the key release event was issued. The duration of each key press was randomly selected from 9 equidistant 100 msec points. After the key release occurred, the test software slept for 100 msec before starting the next trial. 

Test Results

flip_keypress_getKeys Test

Remember that in this test, the psychopy.event module is able to timestamp new events at two discrete points for each retrace until the key press is received:

1. Within win.flip(), just prior to the actual call to the underlying pyglet flip method and the psychopy blocking logic that synchronizes the functions return time with start of the retrace it was waiting on.
2. Each time event.getKeys() is called.

In contrast, iohub timestamps keyboard and mouse events as they are received, independently of the graphics subsystem. On Windows, these events are timestamped at an interval equal to when the Windows PumpWaitingMessages function is called in the iohub process. This was previously set to be every 3.75 msec. As of April 23, 2014, I have modified the psychopy.iohub.default_config.yaml file to include a new setting, windows_msgpump_interval, which can be used to change this call interval. The new default has also been changed to 1 msec. The data collected in these tests used the modified iohub software with the 1 msec interval setting.

Asus Keyboard

• Displays a large timestamp error of 30 msec for psychopy.event and 22 msec for psychopy.iohub (largely due to the keyboards debounce time??)
• Large variability in the timestamp error as well, with a 4.7 msec stdev for psychopy.event vs. 2.5 msec for psychopy.iohub. For psychopy.event, some of this variability is due to key presses actually occurring at any pouint in time relative to display retrace onset. Based on results from the following tests, I would suggest that the the difference in stdev between .event and . iohub is because of this decoupling of retrace onset times and key press creation times. 
• psychopy.iohub keyboard events always have a lower timestamp error compared to psychopy.event. 
• Given this test called win.flip each time event.getKeys() was called, the range in error for psychopy.event timestamps is approx. equal to the monitor retrace rate as would be expepcted. 

Teensy 3 ioSync Keyboard

• With this keyboard, key press events are always created just after the onset of a retrace interval and a bimodal, almost binary, distribution of errors for events timestamped with the psychopy.event module is seen.This is obviously a result of when the actual keyboard events were being generated in this task and when keyboard events are time stamped by psychopy.events. The event is either timestamped just after it actually occurs, resulting in an error very close to that of psychopy.iohub, or it is not timestamped until the next retrace, in which case the timestamp error is equal to the monitor retrace interval (60 Hz, or 16.66 msec here). 
• psychopy.iohub has a 1 msec average time stamp error; compared to a 9.6 msec average error when using psychopy.event getKeys() in this task. 
• The variability of the iohub error is 0.5 msec, compared to 7.14 msec with psychopy.events.
• I would suggest that the ioSync Teensy 3 Test Keyboard is an ideal tool for assessing the keyboard event error caused by the different software layers used by the experiment (OS, libraries, application experiment software), independently of the average delay and variability caused by actual keyboard hardware. Therefore by using the ioSync Teensy 3 keyboard, a baseline can be observed for the given test condition, and any difference from the pattern of results found when a real keyboard device is used can be attributed to the keyboard hardware itself. 

waitkeys_keypress Test

In this test, a window is displayed and updated once in sync with the monitor vblank, by using win.flip(). Then psychopy.event.waitKeys() is called and returns the key press event with timestamp. Note that internally, waitKeys() constantly loops, checking for new keyboard events. This means that event timestamp error should be much better for psychopy.events in this test condition. 

Since waitKeys blocks until the key press event is found and returned, psychopy.iohub keyboard .getEvents() starts to be called in a tight loop after waitKeys returns().The looping is stopped when keyboard.getEvents() receives the keypress event, which is generally only after the first iteration of the loop in this task.  

Asus Keyboard

• As would be expected, psychopy.iohub timestamp errors have the same mean and stdev as in the flip_keypress_getKeys test. 
• psychopy.events.waitKeys() shows a very different pattern of timestamping error. In this case timestamp error follows that of psychopy.iohub almost exactly.
• psychopy.events timestamps are always slightly longer than psychopy.iohub. On Windows this is exactly what would be expected given iohub receives keyboard and mouse event notifications at an earlier point in the Windows event processing pipeline than psychopy.events. For example, psychopy.iohub keyboard event handlers could actually block keyboard events from even being issued to the Windows event handling layer used by psychopy.events. In practice It does not do this of course  
• The average difference in time stamps between the two event detection modules is 0.5 msec.
• Both event detection modules have a range of timestamp errors. The maximum error in psychopy.event  timestamps is about 2/3 of that found in the flip_keypress_getKeys test

Teensy 3 ioSync Keyboard

• Compared to the Asus keyboard in this condition, the degree and variability in timestamp error for both psychopy.event and .iohub is very low.
• Both event collection modules have a stdev of about 0.3 msec; psychopy.event displays an average error of 1.6 msec compared to 1.0 msec in psychopy.iohub 
• As with the Asus keyboard in this test condition psychopy.iohub timestamps are always slightly earlier than those from psychopy.event, with an average difference of approx 0.5 msec. 

Conclusions

The results of these tests confirm the general pattern found by Jonas tests in regard to the high delay and variability in timestamping key press events from a typical desktop keyboard. 

The performance of the ioSync Teensy 3 Test Keyboard demonstrates it's usefulness in determining how much of the timestamp error found in a task is due to the keyboard hardware being used compared the software used to timestamp the keyboard events.

 

Several differences in the performance of psychopy.iohub compared to psychopy.event were found compared to the test results obtained using the test procedures and equipment used in Jonas report. These result differences included the relative delay in keyboard event processing, the lack of occasional spikes in iohub keyboard event timestamps, and the increase in psychopy.event keyboard event processing when iohub was not started.

More testing should be done to determine what factors result in these differences in performance. I would suggest that these result differences may be due to the different approaches in measuring the psychopy time of the actual keypress event, and the tests built in assumption that measurement of screen retrace onset always == the actual key press event time. On the other hand, I could just have been lucky using the PC I do my dev on. More tests are needed for sure.

Thanks very much.

[Sol Simpson]

The images in the last post no longer seem to work, so here they are again.

flip_keypress_getKeys_error : Asus Keyboard

flip_keypress_getKeys_error : ioSync Keyboard

waitkeys_keypress_error : Asus Keyboard

waitkeys_keypress_error : ioSync Keyboard

[Jonathan Peirce]

Thanks for those Sol.

Did you see the thread on the users forum about USB polling and the BBTK button box by the way?
It seems related to this; the BBTK button box essentially fires rapid USB key events but doesn't do any internal time-stamping which makes it seem similar to your ioSync device if I remember correctly.

They were finding good timing with event.waitKeys() (suggesting that the box and OS were all working) but a lag when using iohub for the async polling (unless it was a coding error). I wasn't sure what to suggest to them

Jon

On 15/02/16 20:55, Sol Simpson wrote:

The images in the last post no longer seem to work, so here they are again.

flip_keypress_getKeys_error : Asus Keyboard

flip_keypress_getKeys_error : ioSync Keyboard

waitkeys_keypress_error : Asus Keyboard

waitkeys_keypress_error : ioSync Keyboard

--
You received this message because you are subscribed to the Google Groups "psychopy-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to psychopy-dev...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

-- 
Jonathan Peirce
University of Nottingham

http://www.peirce.org.uk

This message and any attachment are intended solely for the addressee
and may contain confidential information. If you have received this
message in error, please send it back to me, and immediately delete it. 

Please do not use, copy or disclose the information contained in this
message or in any attachment.  Any views or opinions expressed by the
author of this email do not necessarily reflect the views of the
University of Nottingham.

This message has been checked for viruses but the contents of an
attachment may still contain software viruses which could damage your
computer system, you are advised to perform your own checks. Email
communications with the University of Nottingham may be monitored as
permitted by UK legislation.

[jonathan.peirce]

Ah, I now see that you have replied there too Sol. Sorry for the confusion ;-)

On 16/02/16 09:53, Jonathan Peirce wrote:

Thanks for those Sol.

Did you see the thread on the users forum about USB polling and the BBTK button box by the way?
It seems related to this; the BBTK button box essentially fires rapid USB key events but doesn't do any internal time-stamping which makes it seem similar to your ioSync device if I remember correctly.

They were finding good timing with event.waitKeys() (suggesting that the box and OS were all working) but a lag when using iohub for the async polling (unless it was a coding error). I wasn't sure what to suggest to them

Jon

On 15/02/16 20:55, Sol Simpson wrote:

The images in the last post no longer seem to work, so here they are again.

flip_keypress_getKeys_error : Asus Keyboard

flip_keypress_getKeys_error : ioSync Keyboard

waitkeys_keypress_error : Asus Keyboard

waitkeys_keypress_error : ioSync Keyboard

--
You received this message because you are subscribed to the Google Groups "psychopy-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to psychopy-dev...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

-- 
Jonathan Peirce
University of Nottingham

http://www.peirce.org.uk

-- 
Jonathan Peirce
Nottingham Visual Neuroscience
School of Psychology
University of Nottingham

http://www.peirce.org.uk