Saving data to file and performance

[or duek]

Hi,
I ran an experiment with Psychtoolbox which I now wrote again in
Psychopy (it was just too much fun to stop).
But I have one question regarding the way the data is saved.
In psychotoolbox I used changing vector that on each iteration (i.e.
each trial) added the value I wanted to write (for example if I take
RT, after each after first trial the vector wiil bee [0.34], after the
second trial [,0.34,0.86] etc.)
At the end of the experiment everything was written to a .mat file
(all the vectors toghether).

As I understand, in Psychopy I open a .csv file at the beginning of
the experiment and then in each trial I write down the data and move
another row down (\n). Isn't it affect performance? writing to the
file in each iteration? I feel like I'm missing something here and
would love to get some explanation.
Thank you very much,
Or.

[Jeremy Gray]

Hi Or,

you are exactly right, file access can take significant time, potentially messing with your experiment timing. (however, file-writes are typically buffered, which will make things less disruptive.)

so instead of writing to a file every iteration, you can save your values in a vector like you did in PTB (array in python), and write that out at the end, or save to a pickle file, etc. I tend to append formatted text to a string variable, including trial-type identifiers, timing information, etc, and "\n". when the timing-critical part of the code is finished, I then save that big multi-line string to a file. the key thing is to avoid doing file-access during timing-critical bits of your experiment.

--Jeremy

--
You received this message because you are subscribed to the Google Groups "psychopy-users" group.
To post to this group, send email to psychop...@googlegroups.com.
To unsubscribe from this group, send email to psychopy-user...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/psychopy-users?hl=en.

[or duek]

I'm a bit new so I'll try to understand.
I used your example here: http://www.psychopy.org/coder/tutorial2.html
- is it what you referring to?

> > psychopy-user...@googlegroups.com<psychopy-users%2Bunsu...@googlegroups.com>
> > .

[Jeremy Gray]

I'm a bit new so I'll try to understand.
I used your example here: http://www.psychopy.org/coder/tutorial2.html
- is it what you referring to?

ah, ok. in that tutorial, the lines

    staircase.addData(thisResp)

    dataFile.write('%i,%.3f,%i\n' %(targetSide, thisIncrement, thisResp))

come in between timing-critical sections, that is, between trials. so if it takes 80ms to save to the file it does not matter (probably much much faster for a line of text), and the subject won't notice that tiny pause (and often it would be buffered, so taking hardly any time at all).

so: yes, saving takes time. you do need to avoid doing so during timing-critical bits of your experiment. you can either defer everything until the end of the experiment, or save as you go, saving in between trails.
 

To unsubscribe from this group, send email to psychopy-user...@googlegroups.com.

[Jonathan Peirce]

If you're using the StairHandler or TrialHandler, this data will be stored for you anyway, probably in a more-usable format and certainly in a way that doesn't impact timing (no disk access).
The tutorial shows how to write files manually as well as using the TrialHandler for demonstration purposes, but this shouldn't be necessary. Just use trials.saveAsExcel(filename) or saveAsText(filename) and you should be done. (ACTUALLY my PhD student found a buglet in the stairs.saveAsExcel() code - the fixed code is in the git repository).

As a 3rd option, you *could* do exactly what you did in matlab; keep your own vector of data and add your new data each trial::
responses=[]
for thisTrial in trials:
    #do stuff and measure resp
    responses.append(resp)
#after trials save your list of vals to a text file or something else.

But the advantage of the Handler objects is that they keep track of which trial type went with which response after randomisation and also provide an automatic way to save excel files without you having to write file-saving code. They're a big help for most basic studies.

Jon

-- 
Dr. Jonathan Peirce
Nottingham Visual Neuroscience

http://www.peirce.org.uk/