Converting .continuous into .dat and opening it with neuroscope : what voltage range ?

[Laetitia LALLA]

Dear Open Ephys community,

thank you all for this amazing help on the Open Ephys system. I am using it for my recordings (32-channel silicon probe in freely moving rat) and I find it amazing.

I just have one small question : I'm not using the .kwik format yet. I'm converting the 32 continuous files into a dat file with a Matlab function because I'd like to open it with Neuroscope. However, when opening with Neuroscope, I have to know the “Voltage Range” value (by default 20V). I tried to find this value into the Intan documentation but I found nothing...

Would any of you tried the same process than me (converting into dat file and open with neuroscope) and would know what voltage range is used by Intan for the digitalization ?

Thank you all for your help!

Laetitia Lalla

PhD student in Robbe's lab (INMED, Marseille, FRANCE)

Dynamics and functions of the basal ganglia

[Chris Rodgers]

I've been wondering about this too.

If you look at this datasheet: http://www.intantech.com/files/Intan_RHD2000_series_datasheet.pdf

You'll see that they recommend multiplying the digital signals (ranging from -2^15 to +2^15) by .195 in order to convert them to microvolts. So, the total voltage range is 2^16 * .195 = 12.78mV.

I've used this conversion factor in my own data and it looks correct to my eye, but it would be nice if someone who knows more could confirm that it's correct. It's a little confusing because the datasheet also says the input voltage range is +/- 5mV, which would imply a conversion factor of .152. Perhaps this is just the linear portion of the input voltage range.

[Josh Siegle]

That’s correct…the conversion simply entails multiplying by the “bitVolts” value in the header, which is 0.195 for the RHD2000-series chips. 

Josh

-- 
You received this message because you are subscribed to the Google Groups "Open Ephys" group.
To unsubscribe from this group and stop receiving emails from it, send an email to open-ephys+...@googlegroups.com.
To post to this group, send email to open-...@googlegroups.com.
Visit this group at http://groups.google.com/group/open-ephys.
To view this discussion on the web visit https://groups.google.com/d/msgid/open-ephys/2bc34fb7-5726-4e04-a5b3-65abb439029d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Flame Tang]

Hello, Laetitia:

How to convert continuous files into .dat file?

thanks

flame

在 2015年6月24日星期三 UTC+2上午11:00:05，Laetitia LALLA写道：

[Laetitia LALLA]

Hello, 

based on the information on the wiki and Git Hub about the data format (https://open-ephys.atlassian.net/wiki/display/OEW/Data+format), you can do a custom script that "reorganizes" your data of all your continuous files into one dat file. 

Laetitia

[Cleiton Aguiar]

Dear all,

I'm using OpenEphys.py from https://github.com/open-ephys/analysis-tools to convert .continuous in .dat. This is working great for short-term recordings (around 10 min) but I'm get the following error when try to convert (> 2 hours recordings).

>>> import OpenEphys

>>> OpenEphys.loadFolder('/home/cleiton/Documents/dataconv')

Loading continuous data from /home/cleiton/Documents/dataconv/100_CH8_2.continuous

Traceback (most recent call last):

  File "<stdin>", line 1, in <module>

  File "OpenEphys.py", line 67, in loadFolder

    data[f.replace('.continuous','')] = loadContinuous(os.path.join(folderpath, f))

  File "OpenEphys.py", line 164, in loadContinuous

    print type(index)

NameError: global name 'index' is not defined

I was wondering if some Open Ephys user is getting this error as well and if there is something we could change in the code to fix it. 

Thank you,

Cleiton Lopes Aguiar

[Chris Rodgers]

I wonder if this is related to the fact that this is the second recording from the same session. You see the filename is 100_CH8_2, with the _2 indicating the 2nd recording.

I don't think OpenEphys.py supports this loading of multiple recordings from the same session. I submitted a pull request to add this functionality.

https://github.com/open-ephys/analysis-tools/pull/11

[Cleiton Aguiar]

Hi Chris,

Thank you for your quick reply!

I renamed the file by removing the _2 extension but I keep getting exactly the same error.

Any other suggestion for this error?

All the best,

Cleiton

--
You received this message because you are subscribed to a topic in the Google Groups "Open Ephys" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/open-ephys/uT81gVVoVpQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to open-ephys+...@googlegroups.com.

To post to this group, send email to open-...@googlegroups.com.

Visit this group at https://groups.google.com/group/open-ephys.
To view this discussion on the web visit https://groups.google.com/d/msgid/open-ephys/94a4408b-2bfc-45eb-9978-fe4a5f82211b%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Dr. Cleiton Lopes Aguiar

Postdoctoral Scholar - São Paulo Research Foundation (FAPESP)

Dr. João Pereira Leites's lab - Ribeirão Preto School of Medicine (FMRP)

University of São Paulo 

Ribeirão Preto - SP - Brazil

Postal code: 14049-900

Phone: +55 16 33154535

Mobile: +55 19 993194438

Skype: cleitonbiousp

[Chris Rodgers]

The actual error you are getting is due to a clear typo in OpenEphys.py (it refers to a variable "index" when it should be "indices"). But more importantly, it is in error-handling code because another line has failed:

    samples[indices[recordNumber]:indices[recordNumber+1]] = data

You can change this error-handling code to print out the values of recordNumber and the shapes of "indices" and "samples". This is how I would debug it. If you can upload the file somewhre, I can try to load it and see if I can figure it out.

Eyeballing the code, most likely the problem is these constants at the top:

MAX_NUMBER_OF_SPIKES = 1e6

MAX_NUMBER_OF_RECORDS = 1e6

MAX_NUMBER_OF_CONTINUOUS_SAMPLES = 1e8

MAX_NUMBER_OF_EVENTS = 1e6

Your data is probably larger than this (perhaps you have more than MAX_NUMBER_OF_CONTINUOUS_SAMPLES samples). Increasing these constants should fix it. Be aware that these constants were probably chosen to avoid out-of-memory errors, so you will need to have enough memory on your system to load all of your data at once.

How big are your files exactly (how many bytes, how many channels, and approximately how many seconds of recording time)?

[Chris Rodgers]

I got your email containing sample data. You have 13295 seconds of data, which is about 400e6 samples. In OpenEphys.py, you need to change this line:

  MAX_NUMBER_OF_CONTINUOUS_SAMPLES = 1e8

to this:

  MAX_NUMBER_OF_CONTINUOUS_SAMPLES = 1e9

It works for me now.

Note that it will require 800MB of memory per channel, so you probably won't be able to load all your channels at once (unless you have a ton of RAM).

In general the OpenEphys.py file is in need of some work, I think (no offense intended to the authors). Instead of pre-allocating the arrays to a fixed size, I think they could be allocated to the correct size, and optionally issue a warning if this will exceed some threshold of memory. I will work on it and probably submit some pull requests. It's good to know that there are other Python users out there so we can work together on identifying problems and solutions.

[Cleiton Aguiar]

Hi Chris!!!

Thank you very much!!!

Indeed, it worked here as well. 

Just a couple of useful comments:

(1) When I tried to change MAX_NUMBER_OF_CONTINUOUS_SAMPLES to 1e10 I got memory error. 

So, I guess some user that have gathered data of more than 3 hours will probably have problems to load and convert it using OpenEphys.py.

(2) OpenEphys.py support files with _2 extension (e.g. 100_CH8_2).  

(3) Using a PC with 16 GB of RAM, i7 (OS: Ubuntu 14.04.4 LTS) I could load 14 channels of 13295 seconds of data. Then I got memory error. 

Loading only part of the channels generated another problem. Now I need to figure out a way to import and combine multiple .dat files to further open it in neuroscope... If anyone has suggestions please let me know.

Thank you again,

Cleiton. 

To view this discussion on the web visit https://groups.google.com/d/msgid/open-ephys/7e68af98-fb1f-4d9d-bd06-5954e15a6b31%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Chris Rodgers]

Yeah, you can't allocate enough memory to load the entire set of channels. The best way to combine them into a single Neuroscope-like file would be to write your own code to read the files:

1) Read 1000 seconds of data from each channel and write each to the outputfile

2) Clear memory

3) Repeat 1 and 2 until you have read all of the file

This would take a little work to write.

There is a simpler way to do it where you only have to use the existing OpenEphy.loadContinuous, but it is very inefficient.

1) Read entire channel using loadContinuous, but keep only the first 1000 seconds of data

2) Continue for rest of channels

3) When you have all the channels, write to output file

4) Repeat 1-3 for the next epoch of time until you have used all of the data

It is inefficient because you read every file multiple times, but it should be simpler to get working.

[Cleiton Aguiar]

Hi Chris,

I very much appreciate your help!

I'll work on this and back to the community as soon as it's solved.

Cheers,

Cleiton. 

To view this discussion on the web visit https://groups.google.com/d/msgid/open-ephys/47f96bd6-2844-4ff7-a6d9-d00013c0c9d9%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Dr. Cleiton Lopes Aguiar

Postdoctoral Scholar - São Paulo Research Foundation (FAPESP)

Dr. João Pereira Leites's lab - Ribeirão Preto School of Medicine (FMRP)

University of São Paulo 

Ribeirão Preto - SP - Brazil

Phone: +55 16 33154634

[Ainhoa]

Dear all,

 

I'm using Update load_open_ephys_data_faster.m from https://github.com/open-ephys/analysis-tools to convert .continuous in .dat. This is working great, it reads the .continous really fast, but when I want to write the data of all the .continuous  in one .dat  file using a fwrite matlab  function   it goes very slowly  (900s each .continuous  of 2 hours recording at 30kHz). Here is a piece of my code:

 

fid = fopen(‘RawData’,W);

for k = 1:length(ContinuousOrder) % loop over .continuous files

    data = load_open_ephys_data_faster(DataPath{k,1});    

    fseek(fid,2*(k-1),'bof');

    fwrite(fid, data(1), 'int16');

    fwrite(fid, data(2:end), 'int16', (length(DataPath)-1)*2);  

    clear data

    disp([num2str((c/length(DataPath))*100) '%']);

end 

fclose(fid)

 

As you can see I write the data skiping some bytes in other to write in them the next .continuous samples during the next loop step.

Is there any other way to do it faster?  I think that the problems is that I am writing the data skiping some positions, but I am not really sure...

Thank you very much for your help!!!

Ainhoa

[Aarón Cuevas]

Hi Ainhoa,

Writing with skips is definitely the issue there. The faster way, as long as you have enough memory in your system, is to load all the continuous files first into a single MATLAB matrix and then write it all at once in one single frwrite call. That sould be several times faster.

Aarón.

[Ainhoa Hermoso Mendizabal]

Hi Aarón

Thank you very much for your fast response. I tryed to do like that but as the recordings were huge (395MB each .continuous file), I had out-of-memory error. Any other suggestion?

Thank you again,

Ainhoa

To view this discussion on the web visit https://groups.google.com/d/msgid/open-ephys/3f818234-b499-4d91-8523-5a99599e3d21%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Ainhoa Hermoso Mendizabal
Cortical Circuit Dynamics Group.
Institut d'investigacions Biomèdiques August Pi i Sunyer (IDIBAPS)
C/ Roselló 149 - 153
Barcelona 08036
Spain
Tel. +34 932275400 (4304)