example data clip using Ephys lib to share?

[Korey Kam]

Hi all,

In the future I'd like to use Bonsai to record data from a Rhythm FPGA (Open-Ephys and Intan). Currently, I am unable to begin my own recordings. However, I was wondering if anyone had a short data clip they'd be willing to share that was acquired using Bonsai's Ephys Library. Something with LFP and triggers would be awesome to begin playing with.

Cheers,

Korey

[goncaloclopes]

Hi Korey,

You can find example multi-electrode silicon probe datasets at the Kampff-Lab website:

http://www.kampff-lab.org/validating-electrodes/

You can open them in Bonsai using the attached workflow.

Let me know how it works!

Bass: thanks for the PSTH and LFP workflow :-)

At some point the flow will be simplified into a couple of built-in nodes.

G

[Korey Kam]

Hi Goncalo,

-After downloading the adc/amp .bin files and bonsai workflow, I was able to read the spike data in Bonsai. Really cool!

-Now, I'm trying to import the bin files into matlab. I must be doing something wrong, because all I get is a vector of this size (expected a matrix):

 Name              Size                 Bytes  Class     Attributes

  A         297830400x1             2382643200  double   

-The plot looks like this:

-Any recommendations on how to read in this raw file type into matlab?

--
You received this message because you are subscribed to a topic in the Google Groups "Bonsai Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/bonsai-users/rzXpXU1Z3ZM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to bonsai-users...@googlegroups.com.
Visit this group at http://groups.google.com/group/bonsai-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/bonsai-users/15c52764-367a-47d3-b9a6-df35b3dec696%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[goncaloclopes]

Hi Korey,

What matlab code are you using to open the files? The raw binary data file is saved as unsigned 16-bit integers. You need to use matlab's fread to make sure it interprets the data as such and then reshape the vector of numbers into a matrix. Basically you need to reshape it into a matrix of size (numchannels, length / numchannels) where "length" is the length of the vector. I use python myself so unfortunately I don't have matlab code handy, but I know other users have used fread successfully.

Does anyone have a small snippet of matlab code they could share to do this? It would be nice to have it somewhere posted in the forums.

Thanks everyone,

G

[Korey Kam]

Now we're talking:

The key was knowing apriori the number of channels (ie. no header info).

mcode:

%read it

fileName = 'amplifier2014-11-25T22_09_28.bin'; %amplifierData, numChannels = 32;

data = fopen(fileName);

numChannels = 32;

A = fread(data,[numChannels,Inf],'uint16'); %read in m x n matrix, m = numChannels

%sanity

if size(A,1) ~= numChannels

    error('numChanels does not match desired raw data import')

else

    warning('sanity check input passed')

end

whos A

%plot it

figure,

kk_stackedLinePlot(A(1:numChannels/4,1:1e4)), title(sprintf('plot of %s',fileName)); %plot first 8 channels, first 1e4 samps

xlabel('samples'), ylabel('Chan#')

--
You received this message because you are subscribed to a topic in the Google Groups "Bonsai Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/bonsai-users/rzXpXU1Z3ZM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to bonsai-users...@googlegroups.com.
Visit this group at http://groups.google.com/group/bonsai-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/bonsai-users/0809d343-a1bf-4b95-94f6-34fb5eec9a64%40googlegroups.com.

[goncaloclopes]

Cool, glad to hear you got it working :-)

We may add support for more structured formats in the future. Our current practice is to include the Bonsai file that generated the data with the files, which has this info. I tend to prefer metadata that is stored externally and human-readable so that the raw data remains accessible. Alternatively, you can add metadata in the file name.

Cheers,

G

[Sarah P]

Hi Goncalo,

Based on your reply to my question about MatrixWriter provided here, and using the MatLab code provide in this post, I've encountered a confusion.

Attached is my Bonsai code that I'm working with. AnalogInput is receiving data through 2 channels. (This is an example code, so I setup channel ai0 twice to simulate reading in 2 channels.) MatrixWriter is set to ColumnMajor, which I understand to mean the rows are time points or samples (N = samples unknown a priori) and the columns are channels (in my example code, 2 channels).

To my understanding, MatrixWriter set to ColumnMajor will save the data of both channels in one column of data, and I will have to reshape this data in MatLab into 2 columns of N samples (N x 2). Is this correct?

When I read in the binary data in MatLab using the code provided in this post, I get an array of 2 x N (row x col). Is this the appropriate way to reshape the data saved with MatrixWriter from my example Bonsai code? Or, should I reshape the MatrixWriter data in MatLab to be N x 2 (row x col)? I'm wondering if the code should be as follows:

%% ____start code____

data = fopen('filename.bin');
A = fread(data,'int16'); % read in entire data set to know how many total samples of data; must indicate bit depth to get the correct # of samples

numChannels = 2;

samplesperchannel = size(A,1) / numChannels; % samples per channel

data = fopen('filename.bin');
B = fread(data, [samplesperchannel, numChannels], 'int16'); % read in m x n matrix, m = # samples, n = numChannels

fclose(data);

figure;

subplot(2,1,1); plot(B(:,1));

subplot(2,1,2); plot(B(:,2));

%%_____end code_____

Just to see whether the output is the same regardless of how the binary data is read in MatLab, I plotted the data reshaped both ways (N x 2, 2 x N). The plots look different. I'm confused. How should the MatrixWriter binary data be read in the example I've provided? I just want to make sure I'm reading the binary data correctly in MatLab.

Thank you!

Sarah

To unsubscribe from this group and all its topics, send an email to bonsai...@googlegroups.com.

Visit this group at http://groups.google.com/group/bonsai-users.

[Gonçalo Lopes]

Hi Sarah,

The ColumnMajor property specifies how any 2D multi-channel data matrix is serialized (sequenced) into a 1D raw binary file. Whichever property you choose, subsequent buffers will always be appended one after the other to the file. Once the file data is laid out, it's up to analysis scripts to pull it out and reconstruct in whichever way is most convenient.

Let me go through a quick example that illustrates all the steps in this process:

1. By default in Bonsai all time-series data is represented using rows as channels, and columns as time points. This has advantages for signal processing performance in OpenCV has byte alignment padding is performed on a row-by-row basis, including for cropping / subsampling. This means that a typical buffer will be arranged like so:

C1S1 , C1S2 , C1S3

C2S1 , C2S2 , C2S3

C3S1 , C3S2 , C3S3

where C is channel and S is sample.

2. When you serialize matrix data into a 1D binary file, there are two possible natural representations you might use: column-major (scan columns first) and row-major (scan rows first). By default Bonsai uses column-major, which scans columns first, so the matrix data will be appended to the file like so:

C1S1 , C2S1 , C3S1 , C1S2 , C2S2 , C3S2 , C1S3 , C2S3 , C3S3

If we used row-major, on the other hand, the data would look like this:

C1S1 , C1S2 , C1S3 , C2S1 , C2S2 , C2S3 , C3S1 , C3S2 , C3S3

3. The main reason why column-major is the default mode becomes apparent if you see what happens when you try to serialize two buffers into the file (one after the other). As I mentioned above, subsequent buffers will simply be appended to the end of the file, to maximize streaming performance, so for instance if you have the following two buffers:

C1S1 , C1S2 , C1S3    -    C1S4 , C1S5 , C1S6

C2S1 , C2S2 , C2S3    -    C2S4 , C2S5 , C2S6

C3S1 , C3S2 , C3S3    -    C3S4 , C3S5 , C3S6

In this case column-major serialization will give you:

C1S1 , C2S1 , C3S1 , C1S2 , C2S2 , C3S2 , C1S3 , C2S3 , C3S3, C1S4 , C2S4 , C3S4 , C1S5 , C2S5 , C3S5 , C1S6 , C2S6 , C3S6

while row-major serialization will give you:

C1S1 , C1S2 , C1S3 , C2S1 , C2S2 , C2S3 , C3S1 , C3S2 , C3S3, C1S4 , C1S5 , C1S6 , C2S4 , C2S5 , C2S6 , C3S4 , C3S5 , C3S6

4. As you may realize, the column-major representation is inherently "streamable": as long as you know how many channels you recorded from, it doesn't matter how many samples are in the file, you will always be able to interpret and reshape the data. In the row-major representation it is much more tricky, as data is inherently "chunked" by buffer size, since for each channel you have to write out all the samples in the buffer before you move to the next channel. This means that to decode this file format, you would need to know not just the number of channels, but also the size of each buffer.

This is the main reason why we prefer the column-major representation. This does mean that representation of data on file is by default transposed from how Bonsai holds data in memory during processing. In MATLAB however, arrays are actually stored in column-major representation by default, so you can parse it directly using fread as in your example B above where you pass the number of channels and Inf directly.

The best way however, is probably to try it out with a file with known data and make sure you can reconstruct the expected output, e.g. using FunctionGenerator. When pulling the data out in MATLAB, one way will look obviously scrambled, and the other will be correct.

Hope this helps.

You received this message because you are subscribed to the Google Groups "Bonsai Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bonsai-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bonsai-users/7b1f5b86-30f7-4057-877c-12dde7ab06bb%40googlegroups.com.

[Sarah P]

Thank you for this detailed explanation, Goncalo!

This helps me understand how Bonsai is holding the data.

I did try fread-ing the binary data using Korey's code and my code and found that Korey's code reconstructs the expected output correctly. So, the issue is resolved!

Thank you!

Sarah

[KPerks]

I will be trying this in Python tomorrow so needed to look up equivalent code for what was provided here in Matlab.

Hope this is helpful to others (and that it works tomorrow). If not I will delete this message. But wanted to put it here for my reference as well when I go back to it tomorrow! ;)

Matlab Code for a 2-Dimensional Array

Matlab's fread has the ability to read the data into a matrix of form [m, n] instead of just reading it into a column vector. For instance, to read data into a matrix with 2 rows use:

fid = fopen(inputfilename, 'r'); 

data_array = fread(fid, [2, inf], 'int16'); 

fclose(fid); 

Equivalent Python Code for a 2-Dimensional Array

You can handle this scenario in Python using Numpy's shape and transpose.

import numpy as np 

with open(inputfilename, 'rb') as fid: 

     data_array = np.fromfile(fid, np.int16).reshape((-1, 2)).T

• The -1 tells numpy.reshape to infer the length of the array for that dimension based on the other dimension—the equivalent of Matlab's inf infinity representation.
• The .T transposes the array so that it is a 2-dimensional array with the first dimension—the axis—having a length of 2

*np.fromfile note: if preferred, a filename can be passed to the function instead of an open fid.