NPX TIMESTAMP JUMP

[Anan moran]

Hi,

The command line of the Open-Ephys GUI reported the following information/error messages:

Received message.
[Neuropix-PXI] NPX TIMESTAMP JUMP: 34, expected 3 or 4...Possible data loss on slot 4, probe 2 at sample number 111819997
[Neuropix-PXI] NPX TIMESTAMP JUMP: 32, expected 3 or 4...Possible data loss on slot 4, probe 2 at sample number 112480350
[Neuropix-PXI] NPX TIMESTAMP JUMP: 35, expected 3 or 4...Possible data loss on slot 4, probe 2 at sample number 149290550
[Neuropix-PXI] NPX TIMESTAMP JUMP: 3221225475, expected 3 or 4...Possible data loss on slot 4, probe 2 at sample number 322118436
[Neuropix-PXI] NPX TIMESTAMP JUMP: 3221225476, expected 3 or 4...Possible data loss on slot 4, probe 2 at sample number 644237159
[Neuropix-PXI] NPX TIMESTAMP JUMP: 3221225476, expected 3 or 4...Possible data loss on slot 4, probe 2 at sample number 966355904
[Neuropix-PXI] NPX TIMESTAMP JUMP: 3221225476, expected 3 or 4...Possible data loss on slot 4, probe 2 at sample number 1288474797
[Neuropix-PXI] NPX TIMESTAMP JUMP: 3221225475, expected 3 or 4...Possible data loss on slot 4, probe 2 at sample number 1610593765
[Neuropix-PXI] NPX TIMESTAMP JUMP: 3221225475, expected 3 or 4...Possible data loss on slot 4, probe 2 at sample number 1932712760
[Neuropix-PXI] NPX TIMESTAMP JUMP: 3221225475, expected 3 or 4...Possible data loss on slot 4, probe 2 at sample number 2254831757
[Neuropix-PXI] NPX TIMESTAMP JUMP: 3221225476, expected 3 or 4...Possible data loss on slot 4, probe 2 at sample number 2576950738
[Neuropix-PXI] NPX TIMESTAMP JUMP: 3221225475, expected 3 or 4...Possible data loss on slot 4, probe 2 at sample number 2899069758

Can you please tell me what does it mean? What type of data was lost? How can I assess the severity of the data loss? If it is messing with synchronization, can I overcome this sync problem in some way?

Thanks

Anan

[Josh Siegle]

Hi Anan,

This indicates that your PC wasn’t able to keep up with the incoming data stream, and some blocks of samples will be missing from the saved data. In the past, we’ve only seen this happen when some other process is using the PCIe bus, such as file transfers or spike sorting. Were you running any other software when these messages appeared?

As long as you have sync pulses before and after each of the segments of dropped samples, you’ll be able to align the data offline by chunking it into contiguous blocks.

How long was this recording? Sample number 2899069758 would normally only be reached after ~27 hours.

Josh

--
You received this message because you are subscribed to the Google Groups "Open Ephys" group.
To unsubscribe from this group and stop receiving emails from it, send an email to open-ephys+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/open-ephys/25cf2e9d-9d85-496f-a393-24f0057b8cd3n%40googlegroups.com.

[Anan moran]

Thanks Josh for your reply,

The only thing that I am running on the computer apart from the OE Gui is a python program that captures usb camera images. I am not sure, however, that this process is the culprit as it runs continuously and I encounter this phenomenon only for few times over the past 49 hours. I am not sure even if USB3 and PCIe share the same bus, but I would assume they share some common bottleneck in the data pipeline that may cause data loss if some other internal process of the computer starts. Trying to eliminate interactions between video capture and Ephys data collection I am saving them on different physical disks. 

To solve the issue I will try to:

1) Find internal processes of the computer and disable them.

2) Move the video capture to a different computer/RPi

Other suggestions?

As to your question regarding the time of recording, I indeed record for many hours (actually days). My aim is to record for 3 days. As it takes about 2.4 TB/day, I am using a 16 TB 7000 rpm disk for the ephys data. I know you generally recommend SSDs for the task, but HDDs worked for me flawlessly for long recordings (~10 hours).

Thanks again

Anan

[Josh Siegle]

Hi Anan,

This is an interesting use case – we have tested 12+ hour Neuropixels recordings, but not much longer than that due to SSD size constraints.

It looks like most of the jumps occurred at 3-hour intervals, which suggests there is either some recurring process (e.g. anti-virus) that is interfering with data acquisition, or there is something (e.g. disk defragging) that is triggered each time the file grows by ~250 GB.

For this reason, I would recommend splitting up your recordings into smaller chunks. To automate this, you can use the Network Events plugin and the open_ephys.control package to start/stop recording and acquisition at regular intervals.

I am also curious about how Kilosort performs on 10+ hour recordings. Is it able to track cells reliably for the entire session? If not, recording smaller snippets and matching cells based on their waveform shape and firing patterns may be a better approach that trying to save everything into one giant file.

Josh

To view this discussion on the web visit https://groups.google.com/d/msgid/open-ephys/25bc9002-d2b7-44f7-9df9-2a5594457012n%40googlegroups.com.

[Anan moran]

Hi Josh,

I did the following tests:

1) Disabled defragmentation on the disk

2) Disabled antivirus

3) Change to SSD instead of HDD

4) Revert to OEphys 0.5.3.1

5) Stopped the video capture

All have failed to stop the data loss reported. Your observation regarding the timing of the data loss is accurate: it happens exactly every 3 hours starting from the initial time of the recording. This tells us that the cause is probably related to incremental size of the file, in chunks if ~300-350GB. Looking at the process manager when the data loss was reported, I could not find a new process with high CPU that can be the source for the interference, nor a new disk process.

Sorting with KiloSort was long, but eventually finished. It was about 1-2 hour sorting for each hour of recording. The sorting was not the greatest, but this is probably due to unstable recording conditions.

Can you tell from the log how much data was lost? From what I see the disk does not stop writing while the log is printed. What does the statement " 3221225475, expected 3 or 4... " means?

I try to avoid the technique of sorting of a small file and matching templates and firing rates.  I found that neurons, especially under differnt states and learning conditions, tend to change their firing rate more than expected. Spike shapes may also change a little over a long recording, due to drift effect, but KS partitions can be corrected with Phy; which is easier for me when using a single file. I can think of appending several recordings into a single recorded file. Do you know if that have been done before?

Thanks

Anan

[Josh Siegle]

Hi Anan,

I figured out what’s going on – there is not any data loss occurring, but instead the 100 kHz clock onboard the Neuropixels headstage is rolling back to 0. 3,221,225,476 / 100,000 = 10,737 seconds, which corresponds to 2.98 hours. This is nearly identical to the intervals seen in the 30 kHz AP band timestamps. So the warning is being triggered by the clock resetting, and is just a false alarm.

That said, the jumps of ~32 timestamps earlier in the session likely indicate that some data was dropped. How often do warnings like that appear? And are they accompanied by the PXI buffer monitor filling up (visible in the Neuropix-PXI editor)?

What type of signal are you using for synchronization? I would recommend using something like this, which sends randomized barcodes at regular intervals (30 s). That way if there are any dropped samples, they are super easy to spot, and at most you will lose a 30 second chunk of data.

Josh

To view this discussion on the web visit https://groups.google.com/d/msgid/open-ephys/83e4b08d-0de1-4f43-bfc1-0dbbbdf01454n%40googlegroups.com.

[Anan moran]

Wow, that is sooooo good to hear. You've made my day! It fits with all my current observations (including implementing your previous suggestion of splitting the recording file to smaller chunks, which did not solve the problem).

The shorter, ~32 sample drops are super rare. I actually think I did not see them apart from the one I posted here, so I don't think it is a huge concern.

I use the imec 1Hz signal coming from the SMA for syncing.  I will look into your suggestion for an external barcode signal as a measure of extra care.

On the bright side, it seems that OEphys GUI seems to handle long hours of recording quite reliably with large HDDs, at least for now. I will update how it goes after some more experiments.

Thanks again for solving this issue

Anan

[Anan moran]

Hi Josh,

I have some updates on the data loss issue. I have been recording NP data for several days and suffered from occasional data loss incidents. Some were minor (<1000 samples) and some larger than that. In some cases, the OEphys GUI continue to work, but many channels were corrupted. In one case the recording was stopped (no disk activity). I tried both HDD and SSD, with the latter probably showing less of that data loss, but it was still present. The recordings were in a behaving rat, and we tried to secure the headstage connector to the cable with a tape to reduce optional jittering. I will try to check different probes/cable/disks and see if data loss is reduced. Oddly, when I tried to test the same configuration with SpikeGLX it gave me NO_LOCK 24 error and refused to run, while Open Ephys GUI has no issues.

 

I will be happy to hear your comments/ideas

Thanks

Anan

[Josh Siegle]

Hi Anan,

Can you try a long recording with the preview version of the next major GUI release?

https://openephysgui.jfrog.io/artifactory/Test/windows/open-ephys-v0.6.0-rc2-windows-beta.zip

This version includes a number of under-the-hood changes that should reduce overall CPU load. It’s not quite ready for widespread release, but we’ve been using it for our Neuropixels recordings at the Allen Institute and it’s working great.

What does the channel data look like when it becomes corrupted? We can add a check that warns the user when this is happening.

Josh

To view this discussion on the web visit https://groups.google.com/d/msgid/open-ephys/85a140a3-9cf3-4f26-860e-2fab3ff19c2fn%40googlegroups.com.