kwik format - mean_waveform_raw

[Kevin Bolding]

Hello,

I'm interested in measuring waveform attributes from my sorted units. It seems from the documentation that average waveforms should be stored in the KWIK file and retrievable by something like one of these commands (in MATLAB):

waveform = h5readatt(spikekwikfile, '/channel_groups/0/clusters/main/26','mean_waveform_raw')
waveform = hdf5read(spikekwikfile, '/channel_groups/0/clusters/main/26/mean_waveform_raw')

The first returns: 
waveform = 'N.'

The second returns:
hdf5.h5string:

Name: ' '
Length: 2
Padding: 'nullterm'
Data: 'N.'

Can you advise whether we are using the h5read commands incorrectly or if this data is simply not in the KWIK file?

Best,
Kevin Bolding
Franks Lab
Duke Univ., Dept. of Neurobiology

[Cyrille Rossant]

You are right, this has not been implemented yet. For the time being, you'll have to compute the average manually, by selecting the spikes in a given cluster, then selecting the waveforms, and finally performing an average along the first axis (or maybe the last one if you use MATLAB).

--
To subscribe, send email to klustaviewa...@googlegroups.com
---
You received this message because you are subscribed to the Google Groups "KlustaViewas" group.
To unsubscribe from this group and stop receiving emails from it, send an email to klustaviewas...@googlegroups.com.
To post to this group, send email to klusta...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[vesm...@gmail.com]

Dear Cyrille,

Related to this question, I could not still find a way to get the spike waveforms into Matlab. Could you please give information about where the spike waveforms are saved and how you could access them? I was also wondering if the channel information is available for each cluster in the kwik file? and if so where?

Best regards,

Vahid

[yany...@gmail.com]

By using h5disp function in matlab, you can see the whole structure of the kwik file. Then, you can easily read different kinds of stacks you want.

Yili

Tsinghua Univ

[Vahid Esmaeili]

Thanks for your response. I have been going through the structure of the kwik file, but the only information I find about the waveform in that is the one Kevin mentioned as the mean_waveform_raw which as Cyrille replied it is not implemented yet.

​ I could not understand what "

selecting the spikes in a given cluster, then selecting the waveforms

​" means.​ It could be very trivial to you but I am a beginner with Klustasuite!

I

​have

 explor

​ed

 the kwx file and I

​have extracted the

 (N x NWAVESAMPLES x NCHANNELS)

​​

but

​this I guess only gives me the spikes detected on each channel and not related to each cluster.

Another question is regarding the each cluster recording depth. In Klustakwik is each cluster assigned to one channel and is this information available in the kwik file? I could find this either by going through different stacks of the kwik file.

​Bests,

Vahid​

[Kevin Bolding]

Vahid,
This page is useful:
https://github.com/klusta-team/kwiklib/wiki/Kwik-format

It's also necessary to check out the klusta-suite article:
http://biorxiv.org/content/biorxiv/early/2015/02/16/015198.full.pdf

This will make it more clear to you that spikes occur across multiple channels and wouldn't be likely to be assigned to just one.

Here are commands I use to pull waveforms and clusternumbers in MATLAB:
probe = '0';
clusternumbers = double(hdf5read(KWIKfile, ['/channel_groups/',probe,'/spikes/clusters/main']));
allwaveforms = hdf5read(KWXfile, ['/channel_groups/',probe,'/waveforms_filtered']);

If you want all the waveforms for cluster 36, you can do this:
unit36_waveforms = allwaveforms(:,:, clusternumbers==36);

Hope that helps,
KB

[Max Hunter]

Hi,

Yes - if you want to do this in MATLAB, you'll have to look at the spikes/clusters/main array in the .kwik file, and select only the indices of the spikes array which correspond to the clusters you're looking for (i.e. manually filter it)...

Max

[Brendon Watson]

Should we set up a git repo for matlab reading of kwik files?

[Cyrille Rossant]

phy (phy.cortexlab.net), in addition to being the replacement of
KlustaViewa, contains Python functions to easily retrieve the data
from Kwik files, including mean waveforms, mean features, etc. Later
on we'll add further analysis functions for ephys data.

We could also provide some Python/MATLAB conversion routines to make
it easier to integrate phy with your MATLAB code. (see
http://arokem.github.io/python-matlab-bridge/ for example)

[Vahid Esmaeili]

Thank you everyone for your fast responses. Thanks Kevin for the codes. It was very helpful. Right, I could indeed get the spike info from the kwx file looking only at the indices that belong to a specific cluster. 

Of course any improvement in making it easier for the Matlab users to work with phy would be a great help. 

Bests,

Vahid

[Vahid Esmaeili]

Dear all,

How can one extract waveforms in phy? Waveform information is not stored in .Kwx file anymore?

Bests,

Vahid

[Max Hunter]

Hi,

You need to run my phy2kv script (see previous thread) which will extract them for you, if you have any old analysis code.

Alternatively, it's very easy to access with the KwikModel, using model.waveforms[...] and using slice notation to access the waveforms you want. In phy, the waveforms are automatically generated from the .dat file on-the-fly, so if you only need a subset of them, it's easier to do this on-the-fly rather than write them all to disk.

Max

[JT]

hi Kevin, 

I just found your post, just wondering if you have managed this also with phy yet?

In phy, as I've just found out after quite some struggling, the waveforms are no longer written into the kwx file, which means your solution here doesn't work. 

Have you by chance found a matlab solution that works for phy to get the waveforms out? The KWIK format docs give some hint (http://phy.readthedocs.org/en/latest/kwik-format/), but I'm just not getting it to work since I don't understand their docs, particularly this bit under 'spikes' which appears be what I need:

waveforms_filtered

        hdf5_path* [='{kwx}/channel_groups/X/waveforms_filtered']

I don't get the [] notation here.. nor the meaning of ={kwx}, and not getting anything to work by trial-and-error so far. Would be great to hear from someone who got this to work!

cheers,

John

[Nick Steinmetz]

Hi all, 

There is a phy-users group, here: https://groups.google.com/forum/#!forum/phy-users

Copying from a thread there, Max wrote: 

Hi Nivaldo, all,

As I mentioned on Gitter, I did build a quick script to allow you to view your phy .kwik files in KlustaViewa: at the moment, it has two major limitations:

1) it doesn't properly support data dtypes other than int16 (the old Klusta-Suite didn't, either)

2) you need at least as much RAM as you have waveforms since it converts all in one go rather than in chunks. (This is about half the size of your final KWX file, and depends entirely on how many spikes / samples / channels you have, but generally it's the same order of magnitude as your raw .dat file).

I plan to fix these in the coming days but if anybody wants to beta-test this and give me some feedback, please feel free!

1) Download the Python script here: https://github.com/nippoo/kwiktools/blob/master/phy2kv.py

2) Activate the phy virtual environment, if you set one when you installed the program (skip this step if you didn't)

3) Make a backup of your existing .kwik and .kwx files for the moment (this shouldn't make any irreversible changes but it's very much untested!)

4) Run the script: python phy2kv.py myfile.kwik

Max

[Kevin Bolding]

Hi John,

If you haven't already got a solution in place, the solution Nick
posted is a good choice given you have enough RAM.
My method is very idiosyncratic to my data pipeline and involves
pulling in the spiketimes, then seeking to each location in the DAT
file and grabbing the waveforms. I don't have any analysis that really
cares what the waveforms look like on a spike-by-spike basis after
sorting, so I usually pick a random subset (50-100) for averaging and
computing waveform shape statistics.

Here's a little snippet that you could repeat for every unit. There
are some things that might need explaining in there, but it gives the
gist if you know what you're looking for:

% Nchannels x Nsamples x Nwaveforms
RawChannelCount = double(h5readatt(FilesKK.KWIK,
'/application_data/spikedetekt','n_channels'));
fdata=fopen(FilesKK.DAT);
subspks =
TSECS{unit+1}(randperm(length(TSECS{unit+1}),min(wfsubsample,length(TSECS{unit+1}))));
WFx = zeros(RawChannelCount,48,length(subspks));
for spk = 1:length(subspks)

fseek(fdata,round((subspks(spk)*30000)-24)*RawChannelCount*2,'bof');
WFdata = fread(fdata,48*RawChannelCount,'*int16');
WFdata = reshape(WFdata,RawChannelCount,[]);
fWF = filtfilt(B,A,double(WFdata'));
WFx(:,:,spk) = fWF';
end
fclose(fdata);
% To make extracted phy waveforms be like klustakwik waveforms
% let's eliminate the ignored channels.
WFx = WFx(realchannellist+1,:,:);
avgwaveform{unit+1} = mean(WFx,3);

hope that helps,
kb