[Neo][NeuralynxIO] How to treat different card starting times?

[Peter N. Steinmetz]

So I am making some changes to NeuralynxRawIO to handle older and different style ncs files. (Despite having the same block structure, the headers and timing conventions can be different for different versions of the software creating these .ncs files.)

Some of the older systems had multiple ADCs cards which could be started and stopped at different times during one experiment. Sometimes this would simply be due to the time it takes to execute the startup code and quite a small difference. Other times it might be due to a particular card's driver failing and create a larger difference. Myriad reasons can cause the set of .ncs files for a single experiment to start and stop at slightly different times. 

The question is how best to map this onto the Neo core object model? 

Alternative 1. Since it is a single experiment, this should be one block. One could define separate segments for each set of slightly different timebases arising for different groups of channels. This seems a little odd as segments normally refer to discontiguous periods of time, not overlapping periods. 

Alternative 2. Again a single experiment in a single block. One could define a single timebase for all the channels which is the minimal common set of time periods. This would then exclude some samples from cards which started earlier and those which started later, but provides a single fairly intuitive view of the experiment. The cost is losing some of the samples at the start and end for some channels. It also creates the question of how much data loss to tolerate to construct this type of model. Sometimes 3 of 4 cards might continue running after one fails. It seems a pity to lose a large time segment of samples for 3/4 of the data in such a case.

Alternative 3. Map each group of channels with a common timebase to a separate block. 

How has this been dealt with generally within Neo, perhaps for other IO types?

I suppose one could provide an option when constructing the IO, or alternately just be sure to provide directory version with alternative 2 and also a single file construction of the IO for users that care about the missing data. 

[Chadwick Boulay]

I'm sorry I can't help you with your question. I just wanted to mention that this problem also exists for Blackrock multi-NSP setups, and maybe a consistent solution is desired. I don't know how Neo handles it but there are some Blackrock NSx file experts who work on Neo and probably know the answer.

> When there are multiple NSPs, each one gets its own file. The file that is started first starts its sample index at 0 and counts up (30,000 /s) until it detects the sync mechanism at which point it resets its sample index to 0 and both files are on the same clock. As far as I know, Blackrock's analysis software will discard the data in the first file up until the clock sync - this is consistent with your Alternative 2. I suppose it's also possible that one of the files will crash while the other continues, so maybe a less strict solution could be useful here too.

[Samuel Garcia]

Hi Peter,
In my mind I would say solution 2 but without loosing some sample.

A Segment by definition just handle bunch of objects that share the same clock but not necessary the same (sampling_rate/t_start/t_stop).
Segment.t_start is the minimum t_start of all object (and reverse for Segment.t_stop)
The way to go, I think is to split channel into groups that share the same (sampling_rate/t_start/t_stop) at neo.rawio level.
Then at neo.io level each channels groups will give a different AnalogSignal (multi channel) object having the same size.
After this, I think you will need to re-align all channels sharing the same sampling rate : so you will need some function to "crop" each AnalogSignal to align t_start and t_stop.
This would lead to your solution 2. but with some utility function afterward.


The complicated question is : what to do if one channel group get lost buffer in a middle of a Segment (and other don't have the drop) ?
And for this I don't know!! It is complicated.

Solution 3 is not really in the spirit I think.

Julia, Andrew ?


Cheers,

Samuel

--
You received this message because you are subscribed to the Google Groups "Neural Ensemble" group.
To unsubscribe from this group and stop receiving emails from it, send an email to neuralensembl...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/neuralensemble/8267c3fa-0b87-4719-afd5-0eb29b914acfn%40googlegroups.com.

[Andrew Davison]

Hi Peter,

As Samuel says, the defining feature of a Segment is that time has the same meaning for all objects within it, so that if you have two different AnalogSignals, t=324.5 ms in both signals refers to the same physical time during the recording. It is _not necessary_ that all objects within the segment exist for the full duration of the segment (this means, for example, that the t_start of a Segment is the smallest t_start of all the objects within it).

So I think the solution is: a single Block with a single Segment. Where you have channels that drop out, I would suggest using a separate AnalogSignal for each, and then use the new Group object which is coming in Neo 0.9 (due for release within the next couple of weeks) to group the signals that come from the same ADC card.

Cheers,

Andrew

[Peter N. Steinmetz]

Thanks, that does make this simpler. There may be a future question regarding how different the timebases on two cards have to be to necessitate a different Segment, but that is for the future. 

[Peter N. Steinmetz]

In looking to implement this now at the rawio level I have another question. For NeuralynxRawIO in a more general sense, if a single directory is going to have one block and one segment (all on the same time base), I think there is going to be an issue when there are gaps in the recordings. One can map a given channel of recording with the gaps to multiple AnalogSignals in the only segment. 

But then these AnalogSignals are no longer unique to channels and the channel_uid as presently constructed. It seems like that needs to be the case for the interface to get_analog_signal_chunk. 

I guess one could expand the notion of the channel_uid from having the present 2 elements (number and name) to include a contiguous section  number (would 'section' be a good name for this so as not to overload 'block' or 'segment'?).

Does that seem right to people? Or is a multiple segments approach likely better? The latter seems more consistent with the diagram of the Neo object model, though that diagram doesn't really show overlapping segments. 

[Peter N. Steinmetz]

Also, there is this statement in the description of the rawio interface "AnalogSignals must have the same characteristcs across all Blocks and Segments: sampling_rate, shape[1], dtype". 

I am not sure, but does that imply that all AnalogSignals should have the same number of samples in them in the rawio interface? If so, then it would seem to require either chopping the data to the minimal common set with no gaps or alternately filling in the missing data with either a null value like 0. The null value could be user selectable and visually on a graph this could make sense as recording channels do occasionally just drop out to a rail value or 0. 

[Samuel Garcia]

Hi Peter,
yes,, with gaps there are 2 possibilties:
  * one segment and several analogsignal represent the same channel
  * several segment and one analogsignal per segment.

With neo.io there is not constrain you can choose what ever you want.
You the neo.rawio layer (which was construct for simple reader in mind) we can not implement the first solution.

This is why for neurlaynx we chosse to split into several segment when there are gaps.

If I understand correctly what you want (one channel have several object in the same segment), I think it is not not possible in neo.rawio.
Unless a very strong change.

I think multi segment is better because it is more intuitive for a beginner user.
Inside a Bock the neo object model do not impose clock coherency but you can have time compatibility across segment (universal no reseted clock).
Many IO do it by ajusting t_start/t_stop across segment.


I am sure to be clear.



Samuel

--
You received this message because you are subscribed to the Google Groups "Neural Ensemble" group.
To unsubscribe from this group and stop receiving emails from it, send an email to neuralensembl...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/neuralensemble/e768375d-18e1-4e61-8bb4-61011e9f018dn%40googlegroups.com.

[Samuel Garcia]

Hi,
yes it imply that all channel inside a group need to have the same number of sample because the basefromrawio will construct one object per group so we need to have the same number of sample.

The neuralynx case with theses gaps is complicated....
If gaps are same across channel then is more easy we create a new segment.
If there is a gap only for one channel then ... i don't known. In the user perspective we should fill with 0 I think.
But it is more complicated to implement.


Samuel

--

You received this message because you are subscribed to the Google Groups "Neural Ensemble" group.
To unsubscribe from this group and stop receiving emails from it, send an email to neuralensembl...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/neuralensemble/9767a784-8b11-40b1-a2ca-e2ae34a156d0n%40googlegroups.com.