Neo: Plexon IO

[Thomas Akam]

Hi,

I have a couple of questions about using Neo to import plexon files into python.

I am using the following commands to import the data:

from neo import io
r=io.PlexonIO(filename='myfile.plx')
seg=r.read_segment()

Questions:

1).  When I load .plx files containing spike train data, the spike times are loaded but the waveforms are not.  Is it possible to load the waveforms as well?

Example:

s=seg.spiketrains[1]
print s.times
[  2.17825000e-01   2.19475000e-01   2.20900000e-01 ...,   6.28836575e+02   6.28845150e+02   6.28916125e+02] s
print s.waveforms
None

2.  PlexonIO appears not to load any information about which channel data was recorded on, i.e it does not create any Recording Channel objects.  This is problematic if any online sorting has been done on the data, in which case the number of spiketrains imported by neo is different from the number of electrodes.  Is there any way of loading this information?

3) Loading data takes a very long time compared to loading the .plx files in Plexon Offline Sorter.  For example a 200 MB file that takes < 2 seconds to load in offline sorter takes ~ 4 minutes to load using r.read_segment().  Is this unavoidable or can the process be accelerated in any way?   I am running neo 0.2.0 under 32 bit python 2.7.2 on a 64 bit Windows 7 PC with 8 GB ram and an i7-2600 CPU.

Thanks for you help.

best,

Thomas Akam

--
Thomas Akam
Henry Wellcome Postdoctoral Research Fellow
Neurobiology of Action Lab
Champalimaud Neuroscience Program
Lisboa, Portugal
 

[Thomas Akam]

Some futher information about the speed of loading .plx files:

Reading a 1 minute 16 channel continuous recording (40 kHz sample rate) is taking approximately 140 times longer than writing the same data to a .mat file!  I'm no expert but this suggests to me that PlexonIO read segment is not operating as efficiently as it might?

In [1]: import neo

In [2]: r = neo.io.PlexonIO(filename='12-10-m1wb.plx')

In [3]: %timeit seg = r.read_segment()

1 loops, best of 3: 46.5 s per loop

In [4]: seg = r.read_segment()

In [5]: bl = neo.Block(name = 'my block')

In [6]: bl.segments.append(seg)

In [7]: w = neo.io.NeoMatlabIO(filename = 'm1wb.mat')

1 loops, best of 3: 335 ms per loop

I have not been able to compare the read speed for the matlab format data as I get the following error when I try and load it back into python:

In [8]: bl2 = w.read_block()
------------------------------------------------------------
Traceback (most recent call last):
  File "<ipython console>", line 1, in <module>
  File "C:\Python27\lib\site-packages\neo-0.2.0-py2.7.egg\neo\io\neomatlabio.py", line 196, in read_block
    bl =  self.create_ob_from_struct(bl_struct, 'Block', cascade = cascade, lazy = lazy)
  File "C:\Python27\lib\site-packages\neo-0.2.0-py2.7.egg\neo\io\neomatlabio.py", line 307, in create_ob_from_struct
    for c in range(len(getattr(struct,attrname))):
TypeError: object of type 'mat_struct' has no len()

[Samuel Garcia]

Hi,

Questions:

1).  When I load .plx files containing spike train data, the spike times are loaded but the waveforms are not.  Is it possible to load the waveforms as well?

From what I see in the code waveform should be loaded.
Could you send me the file on an http sendfile service ?

2.  PlexonIO appears not to load any information about which channel data was recorded on, i.e it does not create any Recording Channel objects.  This is problematic if any online sorting has been done on the data, in which case the number of spiketrains imported by neo is different from the number of electrodes.  Is there any way of loading this information?

Few IO support directly the RecordignChannel and RecordingChannelGroup object because it is more work.

You can use:
  * AnalogSignal.annotations['channel_index'] or AnalogSignal.annotations['channel_name']. Many IO (PlexonIO) use this
  * The function neo.io.tools.populate_RecordingChannel help to cerate theses objects based on AnalogSignal.annotations['channel_index'].

Note that RecordingChannel and RecordingChannelGroup have sens in a Block so you should do

r=io.PlexonIO(filename='myfile.plx')

bl=r.read() ro bl = r.read_block()

3) Loading data takes a very long time compared to loading the .plx files in Plexon Offline Sorter.  For example a 200 MB file that takes < 2 seconds to load in offline sorter takes ~ 4 minutes to load using r.read_segment().  Is this unavoidable or can the process be accelerated in any way?   I am running neo 0.2.0 under 32 bit python 2.7.2 on a 64 bit Windows 7 PC with 8 GB ram and an i7-2600 CPU.

In python32 (even in win64) can only allocate 2Go.

Here some win dostribu for win64
https://store.continuum.io/cshop/anaconda
http://code.google.com/p/winpython/
http://www.lfd.uci.edu/~gohlke/pythonlibs/

Thanks for you help.

Samuel

-- 
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Samuel Garcia
Lyon Neuroscience
CNRS - UMR5292 -  INSERM U1028 -  Universite Claude Bernard LYON 1
Equipe R et D
50, avenue Tony Garnier
69366 LYON Cedex 07
FRANCE
Tél : 04 37 28 74 24
Fax : 04 37 28 76 01
http://olfac.univ-lyon1.fr/unite/equipe-07/
http://neuralensemble.org/trac/OpenElectrophy
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

[Samuel Garcia]

Reading a 1 minute 16 channel continuous recording (40 kHz sample rate) is taking approximately 140 times longer than writing the same data to a .mat file!  I'm no expert but this suggests to me that PlexonIO read segment is not operating as efficiently as it might?

In [1]: import neo

In [2]: r = neo.io.PlexonIO(filename='12-10-m1wb.plx')

In [3]: %timeit seg = r.read_segment()

1 loops, best of 3: 46.5 s per loop

Bad news. I need to spent some time in the code. The main reason is certainly that in actual implementation the file is read 2 times.
One for counting and allocating, the other one for extracting.

In [4]: seg = r.read_segment()

In [5]: bl = neo.Block(name = 'my block')

In [6]: bl.segments.append(seg)

In [7]: w = neo.io.NeoMatlabIO(filename = 'm1wb.mat')

1 loops, best of 3: 335 ms per loop

I do not understand is "w" for reading or writing ?

I would except something like that:

r = neo.io.PlexonIO(filename='12-10-m1wb.plx')

%timeit bl = r.read()

w = neo.io.NeoMatlabIO(filename = 'm1wb.mat')

w.write_block(bl)
r2 = neo.io.NeoMatlabIO(filename = 'm1wb.mat')
%timeit bl = r2.read()

[Samuel Garcia]

I have spent more that 30 seconds in the code and there is a option for waveform that is set to False

load_spike_waveform = False


seg = r.read(load_spike_waveform = True)
should load waveforms

Samuel

--
You received this message because you are subscribed to the Google Groups "Neural Ensemble" group.
To post to this group, send an email to neurale...@googlegroups.com.
To unsubscribe from this group, send email to neuralensembl...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/neuralensemble?hl=en-GB.

[Thomas Akam]

Samuel,

Thanks for getting back to me about those questions, unfortunately I am still having some problems importing plexon data with Neo.

I have spent more that 30 seconds in the code and there is a option for waveform that is set to False

load_spike_waveform = False


seg = r.read(load_spike_waveform = True)
should load waveforms

When I try either

seg = r.read(load_spike_waveform = True)

or

seg = r.read_segment(load_spike_waveform = True)

I get the following errors:

------------------------------------------------------------
Traceback (most recent call last):
  File "<ipython console>", line 1, in <module>

  File "C:\Python27\lib\site-packages\neo-0.2.0-py2.7.egg\neo\io\plexonio.py", line 264, in read_segment
    waveform = fromstring( fid.read(n1*n2*2) , dtype = 'i2').reshape(n1,n2).astype('f')
NameError: global name 'fromstring' is not defined

Do I need to import an additional dependency to make this work?

 Bad news. I need to spent some time in the code. The main reason is certainly that in actual implementation the file is read 2 times.

In my hands the neo.io.PlexonIO is about 100x slower at loading files than Plexon Offline sorter, and about 20x slower than importing files into matlab using the functions supplied by plexon which use C++ mex files.  This suggests that even after removing the double file read it will still be an order of magnitude slower.  To get round this I thought I would try using the Neo NeuroshareIO to see if import using the Plexon C DLLs was faster.  Unfortunately I have not been able to get the NeuroshareIO to work:

>>> from neo import io
>>> r = io.NeuroshareIO(filename='12-10-m1wb.plx', dllname='nsPlxLibrary.dll')
>>> seg = r.read_segment()

Traceback (most recent call last):

  File "<stdin>", line 1, in <module>
  File "C:\Python27\lib\site-packages\neo-0.2.0-py2.7.egg\neo\io\neuroshareio.py", line 110, in read_segment
    seg = Segment( file_origin = os.path.basename(self.filename), )
  File "C:\Python27\lib\ntpath.py", line 198, in basename
    return split(p)[1]
  File "C:\Python27\lib\ntpath.py", line 170, in split
    d, p = splitdrive(p)
  File "C:\Python27\lib\ntpath.py", line 125, in splitdrive
    if p[1:2] == ':':
TypeError: 'NoneType' object is not subscriptable

I am not sure why this error is occuring as the command seg.file_origin = os.path.basename(self.filename) works fine in the Neo PlexonIO, and the following code work fine in the console:

>>> filename='12-10-m1wb.plx'
>>> print os.path.basename(filename)
12-10-m1wb.plx

Given the above, I tried replacing the line causing problems in NeuroshareIO (line 110):   seg = Segment( file_origin = os.path.basename(self.filename), )                 

with:  seg = Segment( file_origin = self.filename, )

But then I get the following error:

>>> seg = r.read_segment()

Traceback (most recent call last):

  File "<stdin>", line 1, in <module>
  File "C:\Python27\lib\site-packages\neo-0.2.0-py2.7.egg\neo\io\neuroshareio.py", line 120, in read_segment
    if not cascade:
NameError: global name 'cascade' is not defined

So I replaced line 103 of NeuroshareIO  def read_segment(self, import_neuroshare_segment = True):

with:  def read_segment(self, cascade = True, import_neuroshare_segment = True):

This returned a different error.

>>> seg = r.read_segment()

Traceback (most recent call last):

  File "<stdin>", line 1, in <module>
  File "C:\Python27\lib\site-packages\neo-0.2.0-py2.7.egg\neo\io\neuroshareio.py", line 126, in read_segment
    neuroshare.ns_OpenFile(c_char_p(self.filename) ,byref(hFile))
WindowsError: exception: access violation reading 0x00000000

Any idea what I need to do to get the Neuroshare IO to work?

many thanks,

Thomas

 

[Samuel Garcia]

Hi Thomas,
I push on github a fast patch for your problem.
You should be able to open waveform of spikes now with PlexonIO.
load_waveform are now True by default.
You need to re install with github version.

For the problem of performences (wich are bad!) If you send me your file I will able to play with that IO.


When more time I will also focused on NeuroshareIO wrapper.


Samuel

--
You received this message because you are subscribed to the Google Groups "Neural Ensemble" group.
To post to this group, send an email to neurale...@googlegroups.com.
To unsubscribe from this group, send email to neuralensembl...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/neuralensemble?hl=en-GB.

[Thomas Akam]

Samuel,

The PlexonIO is now loading spike wavforms fine, many thanks for fixing that.

I will email you a link to some plexon data files.

best,

Thomas

[Samuel Garcia]

Hi Thomas,
I modify PlexonIO, it should faster.
You can reinstall with github dev version.

I am not in a good mood enough to open a window box and  touch NeuroshareIO now.
So I will do it later.

I hope it help.

Best,

Samuel