EEG trigger timings for audiovisual stimuli
365 views
Skip to first unread message
Ashley
Aug 25, 2015, 8:33:56 AM8/25/15
to psychopy-users
Hi everyone,
Best,
Ashley
Michael MacAskill
Aug 25, 2015, 12:19:55 PM8/25/15
to psychop...@googlegroups.com
Hi Ashley,
[Sending the .py file isn't too helpful here. The .psyexp Builder file is easier to quickly make sense of, and if we have it, we can generate the .py file ourselves if necessary. So I'm going to guess rather than give a definite answer.]
At a guess, I would say this might be due to you loading a new .mpg move file on every trial. It can take quite a while to load a new movie file from disk (possible up to 150 ms, depending on size, disk speed, and so on). So when a movie object is created, you shouldn't assume that it instantly becomes available to play. If you send a trigger at the same time as applying a new file to a movie stimulus, the trigger will be sent immediately but the movie onset may have quite a lag.
i.e. it isn't that the trigger is early: it is very likely on time, but the movie is late.
A way to get around this in Builder is to load the movie in a "Static" or ISI period (the pink bit that appears by default in the background). This means the movie will be loaded in that period, PRIOR to it being needed. Then the movie will be able to start playing immediately, at some later point.
e.g. a common way to do this is to load the movie during a fixation point stimulus. Do you have such a period available to you during each trial, prior to the movie playing?
Regards,
Michael
--
Michael R. MacAskill, PhD 66 Stewart St
Research Director, Christchurch 8011
New Zealand Brain Research Institute NEW ZEALAND
Senior Research Fellow, michael....@nzbri.org
Te Whare Wānanga o Otāgo, Otautahi Ph: +64 3 3786 072
University of Otago, Christchurch http://www.nzbri.org/macaskill
[Sending the .py file isn't too helpful here. The .psyexp Builder file is easier to quickly make sense of, and if we have it, we can generate the .py file ourselves if necessary. So I'm going to guess rather than give a definite answer.]
At a guess, I would say this might be due to you loading a new .mpg move file on every trial. It can take quite a while to load a new movie file from disk (possible up to 150 ms, depending on size, disk speed, and so on). So when a movie object is created, you shouldn't assume that it instantly becomes available to play. If you send a trigger at the same time as applying a new file to a movie stimulus, the trigger will be sent immediately but the movie onset may have quite a lag.
i.e. it isn't that the trigger is early: it is very likely on time, but the movie is late.
A way to get around this in Builder is to load the movie in a "Static" or ISI period (the pink bit that appears by default in the background). This means the movie will be loaded in that period, PRIOR to it being needed. Then the movie will be able to start playing immediately, at some later point.
e.g. a common way to do this is to load the movie during a fixation point stimulus. Do you have such a period available to you during each trial, prior to the movie playing?
Regards,
Michael
Michael R. MacAskill, PhD 66 Stewart St
Research Director, Christchurch 8011
New Zealand Brain Research Institute NEW ZEALAND
Senior Research Fellow, michael....@nzbri.org
Te Whare Wānanga o Otāgo, Otautahi Ph: +64 3 3786 072
University of Otago, Christchurch http://www.nzbri.org/macaskill
Ashley
Aug 26, 2015, 5:37:34 AM8/26/15
to psychopy-users
Hi Michael,
Thank yo so much for your response. This morning I tried loading the movie during an ISI period but it doesn't seem to have made a difference. I'm sure that I am just doing something stupid but for the life of me I cannot figure out what it is. I've attached the .psyexp file which will hopefully be more helpful. Thank you again for your input into this problem.
Best,
Ashley
Michael MacAskill
Aug 26, 2015, 5:55:02 AM8/26/15
to psychop...@googlegroups.com
Hi Ashley,
For this to work, that static period needs to occur before the video file starts playing, so that it is loading the movie required for the current trial. And the ITI needs to have a duration which is longer than the time it takes to load a movie file.
At the moment, you
— haven't connected the video file loading to a static period. The idea is that in the movie component, next to the "Movie file" field, instead of "Set every repeat", you select "Set during" and choose the name of a static ISI period.
— have the static period after the video routine. You should shift it to occur beforehand in the sequence of events.
I can't tell how long the ITI period is supposed to be: it is specified as the variable "ITI", but that doesn't appear to exist in your conditions file.
NB avoid giving things the same names. At the moment, you have a routine called ITI, as well as a variable called ITI. PsychoPy can't know which you mean to refer to in any given instance. This will almost certainly lead to errors. Most likely, they will be very obvious and PsychoPy will complain, but in the worst case, you might not notice them.
So I would suggest shifting your ITI routine so that it occurs before your trial routine. From the subject's point of view, everything will seem the same except on the first trial, which will commence with a pause period instead of starting immediately (although of course, at the moment the video isn't starting immediately).
What you do in that ITI period depends on your requirements. Does it make sense just to have a blank screen, a fixation target, written instructions etc, wait for a keyboard response, etc? That is up to you, as long as the period will always have a minimum duration that is safely longer than the time it takes to load a movie file.
Make sense?
Regards,
Michael
> On 26/08/2015, at 11:37, Ashley <aeliza...@gmail.com> wrote:
>
> Hi Michael,
>
> Thank yo so much for your response. This morning I tried loading the movie during an ISI period but it doesn't seem to have made a difference. I'm sure that I am just doing something stupid but for the life of me I cannot figure out what it is. I've attached the .psyexp file which will hopefully be more helpful. Thank you again for your input into this problem.
>
> Best,
> Ashley
For this to work, that static period needs to occur before the video file starts playing, so that it is loading the movie required for the current trial. And the ITI needs to have a duration which is longer than the time it takes to load a movie file.
At the moment, you
— haven't connected the video file loading to a static period. The idea is that in the movie component, next to the "Movie file" field, instead of "Set every repeat", you select "Set during" and choose the name of a static ISI period.
— have the static period after the video routine. You should shift it to occur beforehand in the sequence of events.
I can't tell how long the ITI period is supposed to be: it is specified as the variable "ITI", but that doesn't appear to exist in your conditions file.
NB avoid giving things the same names. At the moment, you have a routine called ITI, as well as a variable called ITI. PsychoPy can't know which you mean to refer to in any given instance. This will almost certainly lead to errors. Most likely, they will be very obvious and PsychoPy will complain, but in the worst case, you might not notice them.
So I would suggest shifting your ITI routine so that it occurs before your trial routine. From the subject's point of view, everything will seem the same except on the first trial, which will commence with a pause period instead of starting immediately (although of course, at the moment the video isn't starting immediately).
What you do in that ITI period depends on your requirements. Does it make sense just to have a blank screen, a fixation target, written instructions etc, wait for a keyboard response, etc? That is up to you, as long as the period will always have a minimum duration that is safely longer than the time it takes to load a movie file.
Make sense?
Regards,
Michael
> On 26/08/2015, at 11:37, Ashley <aeliza...@gmail.com> wrote:
>
> Hi Michael,
>
> Thank yo so much for your response. This morning I tried loading the movie during an ISI period but it doesn't seem to have made a difference. I'm sure that I am just doing something stupid but for the life of me I cannot figure out what it is. I've attached the .psyexp file which will hopefully be more helpful. Thank you again for your input into this problem.
>
> Best,
> Ashley
Michael MacAskill
Aug 26, 2015, 6:08:29 AM8/26/15
to psychop...@googlegroups.com
Ooh, Ashley, I just scrolled a bit further and see that you have multiple repetitions of your trial loops.
It is a bit of a dangerous practice to have so much duplication. e.g. to make the change to the routine sequence as suggested, you would have to change it 8 times, and there is always the possibility of making an error. A more efficient structure is to have two loops: an inner one (which you currently have) and an outer one which envelops it. That way, your trial routines are specified only once, instead of 8 times, and you have just two loops instead of 8.
The outer loop has its own conditions file, which will just look like this, with only one variable:
conditionsFile
Run1np.xlsx
Run2np.xlsx
Run3np.xlsx
Run4np.xlsx
etc
Then in the inner loop dialog box, instead of having a fixed file name like "Run1np.xlsx", just put the variable name $conditionsFile in the Conditions field.
ie on each iteration of the outer loop, a new conditions file is selected for the inner loop. As well as hugely simplifying your experiment layout, that also enables you the flexibility to do some randomisation of condition orders across subjects, if that is important to you. Otherwise, just cycle through the list in order to replicate your current arrangement.
Cheers,
It is a bit of a dangerous practice to have so much duplication. e.g. to make the change to the routine sequence as suggested, you would have to change it 8 times, and there is always the possibility of making an error. A more efficient structure is to have two loops: an inner one (which you currently have) and an outer one which envelops it. That way, your trial routines are specified only once, instead of 8 times, and you have just two loops instead of 8.
The outer loop has its own conditions file, which will just look like this, with only one variable:
conditionsFile
Run1np.xlsx
Run2np.xlsx
Run3np.xlsx
Run4np.xlsx
etc
Then in the inner loop dialog box, instead of having a fixed file name like "Run1np.xlsx", just put the variable name $conditionsFile in the Conditions field.
ie on each iteration of the outer loop, a new conditions file is selected for the inner loop. As well as hugely simplifying your experiment layout, that also enables you the flexibility to do some randomisation of condition orders across subjects, if that is important to you. Otherwise, just cycle through the list in order to replicate your current arrangement.
Cheers,
> I've attached the .psyexp file which will hopefully be more helpful. Thank you again for your input into this problem.
Ashley
Sep 8, 2015, 5:07:56 AM9/8/15
to psychopy-users
Hi Michael,
Thanks again for your help. I created an entirely new version of the experiment with a 1 second ITI during which the videos were loaded but I had the same problem with the timings. Strangely enough, the timings looked OK when I reverted to a previous version of the experiment with separate .avi and .wav files for the video and audio respectively. I merged the files because I am manipulating the synchrony between auditory and visual streams and wanted to minimize the potential for timing errors but it looks like I just managed to create more...
Jonathan Peirce
Sep 8, 2015, 5:16:50 AM9/8/15
to psychop...@googlegroups.com
Hi Ashley,
I don't think that combining the streams into a single file will make a difference to the timings. When the movie is loaded by the computer the two streams will be extracted from the file and played one on the sound card and one on the graphics card, which is what would have happened anyway. The main timing issues here are in the lags sending the sound to the audio card, which will occur whether or not the audio was initially combined into a single movie file.
Basically, I don't think audio timing is very good in most software (including psychopy) using standard hardware so I'd be quite wary of the whole approach unless you can get some dedicated hardware for precise sound onsets (like the CRS AudioFile).
best wishes
Jon
I don't think that combining the streams into a single file will make a difference to the timings. When the movie is loaded by the computer the two streams will be extracted from the file and played one on the sound card and one on the graphics card, which is what would have happened anyway. The main timing issues here are in the lags sending the sound to the audio card, which will occur whether or not the audio was initially combined into a single movie file.
Basically, I don't think audio timing is very good in most software (including psychopy) using standard hardware so I'd be quite wary of the whole approach unless you can get some dedicated hardware for precise sound onsets (like the CRS AudioFile).
best wishes
Jon
--
You received this message because you are subscribed to the Google Groups "psychopy-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to psychopy-user...@googlegroups.com.
To post to this group, send email to psychop...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/psychopy-users/b2b955b0-a555-4eaf-a149-ae43874af25b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
-- Jonathan Peirce University of Nottingham http://www.peirce.org.uk
This message and any attachment are intended solely for the addressee and may contain confidential information. If you have received this message in error, please send it back to me, and immediately delete it. Please do not use, copy or disclose the information contained in this message or in any attachment. Any views or opinions expressed by the author of this email do not necessarily reflect the views of the University of Nottingham. This message has been checked for viruses but the contents of an attachment may still contain software viruses which could damage your computer system, you are advised to perform your own checks. Email communications with the University of Nottingham may be monitored as permitted by UK legislation.
Richard Höchenberger
Sep 8, 2015, 8:24:04 AM9/8/15
to psychop...@googlegroups.com
Hello, I would like to add to that.
On Tue, Sep 8, 2015 at 11:16 AM, Jonathan Peirce <jon.p...@gmail.com> wrote:
> Basically, I don't think audio timing is very good in most software
> (including psychopy) using standard hardware so I'd be quite wary of the
> whole approach unless you can get some dedicated hardware for precise sound
> onsets (like the CRS AudioFile).
Several years back, I implemented an experiment using audio-visual
speech stimuli (McGurk stimuli, that is). Back then it was impossible
for me to perfectly align the A and V onsets or to maintain a fully
synchronized AV playback using PsychoPy when using ordinary movie
files with embedded audio. I have to add that probably much in this
area has changed in the meantime (for example, there is MovieStim2
now...). At any rate, it seemed that the "video player" would
sometimes even intentionally drop frames to keep the A and V streams
in sync. It was horrible, each trial would present the visual stimuli
a bit differently than the others.
After a few intermediate steps I finally found a solution, which was
decoding the video to a number of bitmap images (one image per frame)
and presenting them using a computer dedicated for video stimulus
delivery only (running ActiveSTIM, http://www.activestim.com/); and
using a separate machine for audio playback via a National Instruments
data-acquisition board.
I would do it differently today (using PsychoPy and one computer
only), but I would always, always separate both streams (A and V)
manually; triple-check their duration; pre-load every single frame of
the video stimulus when possible (which means one will need many
gigabytes of RAM for longer streams); pre-load the whole audio stream
onto a specifically designed piece of hardware (in my case, it would
be an NI board); and trigger the audio playback using a photo-diode
placed on the monitor (it's very easy to trigger analog output via an
external signal on NI boards).
How the exact implementation would look like in PsychoPy I don't know;
I just wanted to share my experiences.
All the best,
Richard
On Tue, Sep 8, 2015 at 11:16 AM, Jonathan Peirce <jon.p...@gmail.com> wrote:
> Basically, I don't think audio timing is very good in most software
> (including psychopy) using standard hardware so I'd be quite wary of the
> whole approach unless you can get some dedicated hardware for precise sound
> onsets (like the CRS AudioFile).
speech stimuli (McGurk stimuli, that is). Back then it was impossible
for me to perfectly align the A and V onsets or to maintain a fully
synchronized AV playback using PsychoPy when using ordinary movie
files with embedded audio. I have to add that probably much in this
area has changed in the meantime (for example, there is MovieStim2
now...). At any rate, it seemed that the "video player" would
sometimes even intentionally drop frames to keep the A and V streams
in sync. It was horrible, each trial would present the visual stimuli
a bit differently than the others.
After a few intermediate steps I finally found a solution, which was
decoding the video to a number of bitmap images (one image per frame)
and presenting them using a computer dedicated for video stimulus
delivery only (running ActiveSTIM, http://www.activestim.com/); and
using a separate machine for audio playback via a National Instruments
data-acquisition board.
I would do it differently today (using PsychoPy and one computer
only), but I would always, always separate both streams (A and V)
manually; triple-check their duration; pre-load every single frame of
the video stimulus when possible (which means one will need many
gigabytes of RAM for longer streams); pre-load the whole audio stream
onto a specifically designed piece of hardware (in my case, it would
be an NI board); and trigger the audio playback using a photo-diode
placed on the monitor (it's very easy to trigger analog output via an
external signal on NI boards).
How the exact implementation would look like in PsychoPy I don't know;
I just wanted to share my experiences.
All the best,
Richard
Ashley
Oct 1, 2015, 3:34:09 AM10/1/15
to psychopy-users
Thank you all for sharing your experiences and advice. What still puzzles me is that when I look at the data in SPM, the trigger timings are too early. For instance, there is supposed to be a 520ms delay between visual and motion onset but when I subtract the differences between the actual triggers (as displayed in SPM), the difference between visual and motion onset ranges from ~330 to 430ms (see attached).
Since the trigger timings are called from an Excel sheet (my conditions file), are they still dependent on the actual presentation of the video? Im not sure if this makes sense but if there was an issue with the loading of the stimulus, I'd expect the trigger timings to be correct but the ERPs to be incorrect. However, the trigger timings appear to be what is incorrect...sorry if this is a stupid question (my excuse is that I'm a 1st year PhD student and have never worked with such complicated stimuli) but I'd really appreciate any input and advice on this matter
Best,
Ashley
Reply all
Reply to author
Forward
0 new messages