Optimizing settings for spike detection/fitting

[Stephan Boltzmann]

Hello everybody out there using Spyking-Circus,

A few single units are already isolated from my data. 

Using phy's traceview (see attached screenshots), I realized that some spikes are not detected. Since there are much smaller spikes, which get detected, this is probably not due to the threshold settings for spike detection (for which I already tried different values). 

I carefully read the manual and related posts and think this could be due to 2 reasons:

- There is no template matching these spikes:

I increased the parameters max_elts and nb_elts to ensure there is a sufficient number of templates.

- The allowed variation of the amplitude size is to small:

I increased amp_limits (and turned amp_auto off) to ensure all spikes are captured.

Are there any further diagnostics I could run to pin down the problem?

Kind regards

Stephan

Full list of settings:

### Note that depending on the file format selected, the parameters in the data section can vary
### You should refer to the documentation to know what are the needed parameters for a given
### file format. Otherwise, launch the code and a message will tell you what is needed

[data]
file_format    =  raw_binary          # Can be raw_binary, openephys, hdf5, ... See >> spyking-circus help -i for more info
stream_mode    = None       # None by default. Can be multi-files, or anything depending to the file format
mapping        =  probes.prb  # Mapping of the electrode (see http://spyking-circus.rtfd.org)
suffix         =            # Suffix to add to generated files, if needed
overwrite      = True       # Filter or remove artefacts on site (if write access is possible). Data are duplicated otherwise
parallel_hdf5  = True       # Use the parallel HDF5 feature (if available)
output_dir     =            # By default, generated data are in the same folder as the data.
data_dtype     = int16
sampling_rate  = 20000
shared_memory  = True


[detection]
radius         = auto       # Radius [in um] (if auto, read from the prb file)
N_t            = 6          # Width of the templates [in ms]
spike_thresh   = 5          # Threshold [in MAD] for spike detection
peaks          = both       # Can be negative (default), positive or both
dead_channels  =            # If not empty or specified in the probe, a dictionary {channel_group : [list_of_valid_ids]}
weird_thresh   =            # If not empty, threshold [in MAD] for artefact detection

[filtering]
cut_off        = 400, auto  # Min and Max (auto=nyquist) cut off frequencies for the band pass butterworth filter [Hz]
filter         = True       # If True, then a low-pass filtering is performed
remove_median  = True       # If True, medians over all channels within shanks are substracted (movement artifacts)
common_ground  =            # If you want to use channels as ground within shanks {channel_group : ground_channel}
sat_value      =            # Values higher than sat_value are set to 0 during filtering

[triggers]
trig_file      =            # External stimuli to be considered as putative artefacts [in trig units] (see documentation)
trig_windows   =            # The time windows of those external stimuli [in trig units]
trig_unit      = ms         # The unit in which times are expressed: can be ms or timestep
clean_artefact = False      # If True, external artefacts induced by triggers will be suppressed from data
dead_file      =            # Portion of the signals that should be excluded from the analysis [in dead units]
dead_unit      = ms         # The unit in which times for dead regions are expressed: can be ms or timestep
ignore_times   = False      # If True, any spike in the dead regions will be ignored by the analysis
make_plots     =            # Generate sanity plots of the averaged artefacts [Nothing or None if no plots]

[whitening]
spatial        = True       # Perform spatial whitening
max_elts       = 2000       # Max number of events per electrode (should be compatible with nb_elts)
nb_elts        = 0.9        # Fraction of max_elts that should be obtained per electrode [0,1]
output_dim     = 9          # Can be in percent of variance explain, or num of dimensions for PCA on waveforms

[clustering]
extraction     = median-raw # Can be either median-raw (default) or mean-raw
sub_dim        = 16         # Number of dimensions to keep for local PCA per electrode
max_elts       = 20000      # Max number of events per electrode (should be compatible with nb_elts)
nb_elts        = 0.75       # Fraction of max_elts that should be obtained per electrode [0,1]
nb_repeats     = 5          # Number of passes used for the clustering
smart_search   = Fals       # Activate the smart search mode
merging_method = dip  # Method to perform local merges (distance, dip, folding, nd-folding, bhatta, nd-bhatta)
merging_param  = default    # Merging parameter (see docs) (3 if distance, 0.5 if dip, 1e-9 if folding, 2 if bhatta)
sensitivity    = 3          # Single parameter for clustering sensitivity. The lower the more sensitive
cc_merge       = 0.80       # If CC between two templates is higher, they are merged
dispersion     = (5, 5)     # Min and Max dispersion allowed for amplitudes [in MAD]
fine_amplitude = True       # Optimize the amplitudes and compute a purity index for each template
make_plots     =            # Generate sanity plots of the clustering [Nothing or None if no plots]

[fitting]
amp_limits     = (0.8, 2)   # Amplitudes for the templates during spike detection [if not auto]
amp_auto       = auto      # True if amplitudes are adjusted automatically for every templates
collect_all    = False      # If True, one garbage template per electrode is created, to store unfitted spikes
ratio_thresh   = 0.8        # Ratio of the spike_threshold used while fitting [0-1]. The lower the slower
mse_error      = False      # If True, RMS is collected over time, to assess quality of reconstruction

[merging]
erase_all      = True       # If False, a prompt will ask you to remerge if merged has already been done
cc_overlap     = 0.55       # Only templates with CC higher than cc_overlap may be merged
cc_bin         = 1          # Bin size for computing CC [in ms]
default_lag    = 4          # Default length of the period to compute dip in the CC [ms]
auto_mode      = 0.95       # Between 0 (aggressive) and 1 (no merging). If empty, GUI is launched
remove_noise   = True       # If True, meta merging will remove obvious noise templates (weak amplitudes)
noise_limit    = 0.65       # Amplitude at which templates are classified as noise
sparsity_limit = 0          # Sparsity level (in percentage) for selecting templates as putative noise (in [0,1])
time_rpv       = 2          # Time [in ms] to consider for Refraction Period Violations (RPV) (0 to disable)
rpv_threshold  = 0.02       # Percentage of RPV allowed while merging
merge_drifts   = True       # Try to automatically merge drifts, i.e. non overlapping spiking neurons
drift_limit    = 1          # Distance for drifts. The higher, the more non-overlapping the activities should be
clean_merging  = False      # When templates are merged, automatically remove spike duplicated (less than 0.5ms appart)

[converting]
erase_all      = True       # If False, a prompt will ask you to export if export has already been done
export_pcs     = prompt     # Can be prompt [default] or in none, all, some
export_all     = False      # If True, unfitted spikes will be exported as the last Ne templates
sparse_export  = True       # For recent versions of phy, and large number of templates/channels
prelabelling   = False      # If True, putative labels (good, noise, best, mua) are pre-assigned to neurons
rpv_threshold  = 0.05       # Percentage of RPV allowed while labelling neurons as good neurons

[Pierre Yger]

Dear Stephan

By looking at the .params file, I am not sure that your parameters are optimal, so here are some comments below. I don't think you should change them the way you did. I guess your issue comes from the fact that clustering is not perfect, and to appreciate it, you can set make_plots = png in the [clustering] section. It will generates plots in the results folder, heping to understand the behavior wrt your parameters

[whitening]
output_dim     = 9          # Why did you increase that? If this 5 is a standard value...

[clustering]

sub_dim        = 16         # Why did you increase that? 
max_elts       = 20000      # Why not, but clustering is slower
nb_repeats     = 5          # 3 is enough but why not
smart_search   = False      # Better if True
merging_method = dip        # Why did you change the default?
cc_merge       = 0.80       # Maybe this is too low. Depends if you have lots of "local merges" in the .log. If yes, then set it back to default value

make_plots     =            # Generate sanity plots of the clustering [Nothing or None if no plots]

[fitting]

amp_limits     = (0.8, 2)   # This is not used because amplitude are automated...

Hope this helps

Best

Pierre

--
You received this message because you are subscribed to the Google Groups "SpyKING CIRCUS" group.
To unsubscribe from this group and stop receiving emails from it, send an email to spyking-circus-u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/spyking-circus-users/9a9f3ffe-bfcf-4dd7-a9c2-7b82ad9a158cn%40googlegroups.com.

[Stephan Boltzmann]

Hi Pierre,

Thanks a lot for your support.

I kept changing and adapting the parameters starting from default (including your suggestions), but didn't manage to get good results so far.

I uploaded a recording with a tetrode for 1 minute (as well as the corresponding parameter and probe file) on 

https://drive.google.com/drive/folders/1BIrl4fq2co2IsJZ1cvrTil5P4RLbP7Tf?usp=sharing

(4 channels with 20000 Hz sampling rate in raw_binary, i.e. 16bit integer values).

The real spikes get detected, but there are also many false positives. These do unfortunately get grouped into one cluster with the real spikes.

My idea was to raise the threshold for spike detection and also increase the minimal amplitude for template matching, but this didn't really help.

To my surprise, phy is indicating a purity of 0.92 although I would say that none of my results showed a clear refractory period.

Kind regards,

Stephan