The problem in analysis of 400 electrodes' recording
79 views
Skip to first unread message
Fukuaki Kinoshita
Oct 9, 2021, 8:36:07 AM10/9/21
to SpyKING CIRCUS
Dear Pierre,
I have a problem when I was analyzing 400 electrodes recording. The number of clusters should be more than 500, but I got only 100 clusters and 2 templates.
Why did I get much fewer clusters than I had expected ?
I use mouse cortical neurons and, the interval of electrode is about 70um.
This is the content of the parameter file:
[data]
file_format =numpy # Can be raw_binary, openephys, hdf5, ... See >> spyking-circus help -i for more info
sampling_rate = 10000
#data_dtype = float32
#nb_channels = 400
stream_mode = multi-files # None by default. Can be multi-files, or anything depending to the file format
mapping = probes/map_400s.prb # Mapping of the electrode (see http://spyking-circus.rtfd.org)
suffix = # Suffix to add to generated files, if needed
overwrite = True # Filter or remove artefacts on site (if write access is possible). Data are duplicated otherwise
parallel_hdf5 = True # Use the parallel HDF5 feature (if available)
output_dir = # By default, generated data are in the same folder as the data.
[detection]
radius = auto # Radius [in um] (if auto, read from the prb file)
N_t = 3 # Width of the templates [in ms]
spike_thresh = 4.5 # Threshold [in MAD] for spike detection
peaks = both # Can be negative (default), positive or both
dead_channels = # If not empty or specified in the probe, a dictionary {channel_group : [list_of_valid_ids]}
weird_thresh = # If not empty, threshold [in MAD] for artefact detection
[filtering]
cut_off = 300, auto # Min and Max (auto=nyquist) cut off frequencies for the band pass butterworth filter [Hz]
filter = True # If True, then a low-pass filtering is performed
remove_median = False # If True, medians over all channels within shanks are substracted (movement artifacts)
common_ground = # If you want to use channels as ground within shanks {channel_group : ground_channel}
sat_value = # Values higher than sat_value are set to 0 during filtering (in % of max dtype) [0,1]
[whitening]
spatial = True # Perform spatial whitening
max_elts = 10000 # Max number of events per electrode (should be compatible with nb_elts)
nb_elts = 0.8 # Fraction of max_elts that should be obtained per electrode [0,1]
output_dim = 5 # Can be in percent of variance explain, or num of dimensions for PCA on waveforms
[clustering]
extraction = median-raw # Can be either median-raw (default) or mean-raw
sub_dim = 5 # Number of dimensions to keep for local PCA per electrode
max_elts = 10000 # Max number of events per electrode (should be compatible with nb_elts)
nb_elts = 0.8 # Fraction of max_elts that should be obtained per electrode [0,1]
nb_repeats = 5 # Number of passes used for the clustering
smart_search = True # Activate the smart search mode
merging_method = nd-bhatta # Method to perform local merges (distance, dip, folding, nd-folding, bhatta, nd-bhatta)
merging_param = default # Merging parameter (see docs) (3 if distance, 0.5 if dip, 1e-9 if folding, 2 if bhatta)
sensitivity = 3 # Single parameter for clustering sensitivity. The lower the more sensitive
cc_merge = 1 # If CC between two templates is higher, they are merged
dispersion = (5, 5) # Min and Max dispersion allowed for amplitudes [in MAD]
fine_amplitude = True # Optimize the amplitudes and compute a purity index for each template
make_plots = png # Generate sanity plots of the clustering [Nothing or None if no plots]
[fitting]
amp_limits = (0.3, 5) # Amplitudes for the templates during spike detection [if not auto]
amp_auto = True # True if amplitudes are adjusted automatically for every templates
collect_all = False # If True, one garbage template per electrode is created, to store unfitted spikes
ratio_thresh = 0.75 # Ratio of the spike_threshold used while fitting [0-1]. The lower the slower
mse_error = False # If True, RMS is collected over time, to assess quality of reconstruction
[merging]
erase_all = True # If False, a prompt will ask you to remerge if merged has already been done
cc_overlap = 0.99 # Only templates with CC higher than cc_overlap may be merged
cc_bin = 2 # Bin size for computing CC [in ms]
default_lag = 5 # Default length of the period to compute dip in the CC [ms]
auto_mode = 0.95 # Between 0 (aggressive) and 1 (no merging). If empty, GUI is launched
remove_noise = True # If True, meta merging will remove obvious noise templates (weak amplitudes)
noise_limit = 0.75 # Amplitude at which templates are classified as noise
sparsity_limit = 0 # Sparsity level (in percentage) for selecting templates as putative noise (in [0,1])
time_rpv = 5 # Time [in ms] to consider for Refraction Period Violations (RPV) (0 to disable)
rpv_threshold = 0.02 # Percentage of RPV allowed while merging
merge_drifts = True # Try to automatically merge drifts, i.e. non overlapping spiking neurons
drift_limit = 1 # Distance for drifts. The higher, the more non-overlapping the activities should be
clean_merging = True # When templates are merged, automatically remove spike duplicated (less than 0.5ms appart)
and the result of the analysis above is belows:
---------------------- Informations -------------------------
| Number of recorded channels : 400
| Number of analyzed channels : 400
| File format : NUMPY
| Data type : float32
| Sampling rate : 10000 Hz
| Duration of the recording : 60 min 0 s 0 ms
| Width of the templates : 3 ms
| Spatial radius considered : 150 um
| Threshold crossing : positive and negative
| Streams : multi-files (6 found)
------------------------------------------------------------------
------------------------- Informations -------------------------
| Using only 8 out of 16 local CPUs available (-c to change)
------------------------------------------------------------------
[33m------------------------- Informations -------------------------
[33m| Filtering has already been done
[37m------------------------------------------------------------------
[37mAnalyzing data to get whitening matrices and thresholds...
[37mFound 20.2181s to compute the whitening matrix...
[37mBecause of whitening, need to recompute the thresholds...
[37mSearching spikes to construct the PCA basis...
100%|####################################|[16:16<00:00, 37.55s/it]
[37mFound 5117294 waveforms over 6400000 requested
[37mSearching isolated random spikes to sample amplitudes...
100%|###################################|[43:41<00:00, 113.98s/it]
[37mFound 1511531 spikes over 6400000 requested
[37mEstimating amplitudes distributions...
[37mSmart Search of good isolated spikes for the clustering (1/5)...
100%|#################################|[1:34:48<00:00, 247.31s/it]
[37mFound 1511531 isolated spikes over 6400000 requested (65299924 rejected)
[37mComputing density estimations...
[37mSearching random spikes to refine the clustering (2/5)...
100%|###################################|[44:50<00:00, 116.96s/it]
[37mFound 1302831 spikes over 6400000 requested
[37mRefining density estimations...
[37mSearching random spikes to refine the clustering (3/5)...
100%|###################################|[45:20<00:00, 118.29s/it]
[37mFound 1196350 spikes over 6400000 requested
[37mRefining density estimations...
[37mSearching random spikes to refine the clustering (4/5)...
100%|###################################|[45:02<00:00, 117.50s/it]
[37mFound 1057709 spikes over 6400000 requested
[37mRefining density estimations...
[37mSearching random spikes to refine the clustering (5/5)...
100%|###################################|[44:30<00:00, 116.12s/it]
[37mFound 950568 spikes over 6400000 requested
[37mRefining density estimations...
[37mRunning density-based clustering...
100%|####################################|[01:19<00:00, 1.59s/it]
[33m------------------------- Informations -------------------------
[33m| Number of clusters found : 106
[33m| Number of local merges : 93 (method nd-bhatta, param 2)
[37m------------------------------------------------------------------
[37mEstimating the templates with the median-raw procedure ...
100%|####################################|[00:30<00:00, 1.66it/s]
[37mRemoving 104 strongly shifted or noisy/mixture templates...
[33m------------------------- Informations -------------------------
[33m| Templates on few channels only (1), cc_merge set to 1 automatically
[37m------------------------------------------------------------------
[37mRemoving mixtures of templates...
100%|##################################|[00:00<00:00, 3199.41it/s]
100%|############################################|[00:00<?, ?it/s]
[33m------------------------- Informations -------------------------
[33m| Number of global merges : 0
[33m| Number of mixtures removed : 0
[37m------------------------------------------------------------------
[37mComputing optimal amplitudes for the templates...
100%|####################################|[00:01<00:00, 1.73s/it]
100%|####################################|[01:03<00:00, 1.28s/it]
[37mPre-computing the overlaps of templates...
100%|##################################|[00:00<00:00, 3198.78it/s]
[37mHere comes the SpyKING CIRCUS using 8 CPUs and 2 templates...
100%|####################################|[12:05<00:00, 6.21it/s]
[37mGathering spikes from 8 nodes...
100%|###################################|[00:00<00:00, 511.86it/s]
[33m------------------------- Informations -------------------------
[33m| Number of spikes fitted : 27942
[37m------------------------------------------------------------------
[33m------------------------- Informations -------------------------
[33m| Automatic merging with a threshold of 0.95
[37m------------------------------------------------------------------
[37mUpdating the data...
100%|############################################|[00:00<?, ?it/s]
[33m------------------------- Informations -------------------------
[33m| No pairs were found given the parameters
[37m------------------------------------------------------------------
[33m------------------------- Informations -------------------------
[33m| We kept 2 templates out of 2 after merging
[33m| As meta merging is still experimental, you need
[33m| to use the extension option to see its results
I wolud appreciate it if you could reply
Best regards,
Fukuaki
Pierre Yger
Oct 12, 2021, 3:09:51 AM10/12/21
to Fukuaki Kinoshita, SpyKING CIRCUS
Dear Fukuaki
Indeed, something is weird here. First, did you check the data were properly loaded (using the preview GUI, spyking-circus mydata.npy -p). If yes, then why such a low threshold, and not the default value at 6? If this is still not solving the problem, then have a look at the plots generated during the clustering step (in the results/plots folder). Can you share some with me? Maybe decreasing the sensitivity (from 3 to 2) in the [clustering] section can help
Best
Pierre
--
You received this message because you are subscribed to the Google Groups "SpyKING CIRCUS" group.
To unsubscribe from this group and stop receiving emails from it, send an email to spyking-circus-u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/spyking-circus-users/4438832b-25f5-468c-b79b-bb2be37fda78n%40googlegroups.com.
Fukuaki Kinoshita
Oct 17, 2021, 12:01:03 PM10/17/21
to SpyKING CIRCUS
Dear Pierre
Thank you for your concern.
I tried threshold 6, clustering sensitivity 2. I could obtain 893 clusters.
However, I had "Memory Error" when I tried to analyze data recorded in a hour. So I analyzed in 30 min recording.
Could you tell me the way to deal with memory error?
Best regards,
Fukuaki
2021年10月12日火曜日 16:09:51 UTC+9 pierr...@gmail.com:
Reply all
Reply to author
Forward
0 new messages