Data Loss in v0.5.4

[Danny Siu]

Hello,

My lab has been performing 24/7 recordings with up to 96 channels for over a year now. We've noticed two issues recently in 10% of our recordings, where it occurred more often as we upgraded from v0.4 to v0.5.4:

1) Byte difference: Some channels have more blocks than the others. We are noticing exactly byte differences in multiples of 2070, consistent with how OEP records blocks. By the end of a 24hr recording, we've seen differences of up to 5 blocks between CH1 and 64 (Fig 1).

2) Timestamp difference: This is related to #1, where we see timestamps diverge from CH1 and CH64 by a constant amount (Fig 2)

3) Data Loss: Upon investigating #1 and #2, we noticed significant data loss, up to 12 minutes in a 22hr recording. We did not see any error from the machine, or have any logs that indicated this. However, we saved a screenshot of the GUI after we stopped the recording, indicating we recorded for 1341min 49s, or 80509s (Fig 3a). The amount of datapoints outputted only reflected 79767s of data. 

We looked at the timestamp variable and noticed it jumped within the first 15min of recording, indicating acquisition problems of up to 10s before resuming (Fig 3b).

We looked at the cross correlation across our 64 channels towards the end of our recording and noticed significant time lag (Fig 3c). To us, this indicates we would need to adjust each channel to its own timestamp.

Our questions are:

1) Have these de-sync issues across channels been seen before?

2) Are there supposed to be error messages associated with this data loss? 

Thanks.

Danny

[Josh Siegle]

Hi Danny,

Thanks for documenting this in detail. I’m not sure we’ve seen this exact issue come up before, but data corruption has been reported with the Open Ephys data format, e.g. see this issue.

In your case, since the data can be loaded successfully, it means that some data blocks were not written to disk, but the data files are otherwise intact. This can happen in cases where the GUI fails to keep up with the rate of incoming data (in which case the CPU meter would spike to 100%), or the disk writing is not happening fast enough (in which case the buffer monitor in the Record Node would fill up). Unfortunately there are no warnings that appear when these events occur, although this is something we would like to add before the next release.

For the data you’ve already collected, you will need to load the timestamps separately for each channel and align them individually. Hopefully once you’ve accounted for the timing of individual data blocks, the cross correlation plots look as expected.

For future experiments, I’d strongly recommend upgrading to the latest version of the GUI (0.5.5.3) and switching to the default Binary format. This will require some changes to your offline analysis code, but it should help prevent data loss, as writing to a single file is much more efficient than writing to 64. Check out the open-ephys-python-tools and open-ephys-matlab-tools packages if you need help reading in the data.

Josh

--
You received this message because you are subscribed to the Google Groups "Open Ephys" group.
To unsubscribe from this group and stop receiving emails from it, send an email to open-ephys+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/open-ephys/31656edc-2d31-47f2-9a2c-5d7da521a26fn%40googlegroups.com.
<Fig3_DataLoss.png><Fig1_ByteDifference.png><Fig2_TimestampDifference.png>

[Danny Siu]

Hi Josh,

Thank you for your reply. We have seen the issue you linked once or twice but haven't gotten to recovering that data yet. 

Yeah we're hoping that the timestamp alignment will fix our issue as well, glad we came to the same conclusion. Have you seen our kind of issue when recording in the binary format? Would we be able to retrieve any data if the recording failed mid-way or if there was data loss like we see in the .continuous OpenEphys format?

Thanks!

Danny

[Josh Siegle]

Hi Danny,

In a similar situation (CPU overload or failure to write fast enough), the Binary format would have a gap in the timestamps and zeros for a subset of samples. But because writing to one file is more efficient (especially for 64 simultaneous channels), this issue is less likely to occur.

Can you send the settings.xml file for one of these recordings to sup...@open-ephys.org? We can take a look and make sure there’s nothing in your signal chain that could be causing problems.

Thanks,

Josh

To view this discussion on the web visit https://groups.google.com/d/msgid/open-ephys/4ece62f0-653b-4755-84f2-2230e003dec0n%40googlegroups.com.