continuous channel buffer filling on a different HD

67 views
Skip to first unread message

Tobias Niebur

unread,
Oct 22, 2024, 6:33:09 PM10/22/24
to Open Ephys
Hello,

I am trying to use OE to write to a different HD than OE is installed on. I am able to record onto the C: drive (OE install) with no problems, but when I try to write to a second HD on the same computer, the buffer fills within a couple seconds.

Is there any solution to this besides either running a different install of OE or switching to an SSD?

Thanks

Josh Siegle

unread,
Oct 24, 2024, 5:58:20 PM10/24/24
to Tobias Niebur, Open Ephys
Hi Tobias,

How many channels are you trying to write simultaneously? You could try using multiple record nodes to distribute data writing across more than one thread. But the HD will likely still be the bottleneck. Switching to an SSD or recording to the C: drive is your best bet for solving this issue.

Josh


--
You received this message because you are subscribed to the Google Groups "Open Ephys" group.
To unsubscribe from this group and stop receiving emails from it, send an email to open-ephys+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/open-ephys/d61aafaa-276e-4b71-b004-b4580d0dfc9an%40googlegroups.com.

Tobias Niebur

unread,
Oct 25, 2024, 3:13:19 PM10/25/24
to Open Ephys
Thank you Josh,

We are currently trying to write 64 channels, but plan to write 400 in the near future.
Would you anticipate a bottleneck to exist with an external SSD? Installing an internal SSD is not a large operation but is more difficult than simply using an external SSD.

Thanks


On Thursday, October 24, 2024 at 5:58:20 PM UTC-4 Josh Siegle wrote:
Hi Tobias,

How many channels are you trying to write simultaneously? You could try using multiple record nodes to distribute data writing across more than one thread. But the HD will likely still be the bottleneck. Switching to an SSD or recording to the C: drive is your best bet for solving this issue.

Josh


On Tue, Oct 22, 2024 at 3:33 PM, Tobias Niebur <teni...@gmail.com> wrote:
Hello,

I am trying to use OE to write to a different HD than OE is installed on. I am able to record onto the C: drive (OE install) with no problems, but when I try to write to a second HD on the same computer, the buffer fills within a couple seconds.

Is there any solution to this besides either running a different install of OE or switching to an SSD?

Thanks

--
You received this message because you are subscribed to the Google Groups "Open Ephys" group.
To unsubscribe from this group and stop receiving emails from it, send an email to open-ephys+...@googlegroups.com.

Josh Siegle

unread,
Oct 27, 2024, 2:32:35 PM10/27/24
to Tobias Niebur, Open Ephys
Hi Tobias,

If you're planning to scale to 100+ channels, I would strongly recommend installing an internal SSD. An external SSD may work, but in the scheme of things an internal SSD is not much of a price to pay to guarantee that recording will be reliable.

Josh


On Fri, Oct 25, 2024 at 12:13 PM, Tobias Niebur <teni...@gmail.com> wrote:
Thank you Josh,

We are currently trying to write 64 channels, but plan to write 400 in the near future.
Would you anticipate a bottleneck to exist with an external SSD? Installing an internal SSD is not a large operation but is more difficult than simply using an external SSD.

Thanks


On Thursday, October 24, 2024 at 5:58:20 PM UTC-4 Josh Siegle wrote:
Hi Tobias,

How many channels are you trying to write simultaneously? You could try using multiple record nodes to distribute data writing across more than one thread. But the HD will likely still be the bottleneck. Switching to an SSD or recording to the C: drive is your best bet for solving this issue.

Josh


On Tue, Oct 22, 2024 at 3:33 PM, Tobias Niebur <teni...@gmail.com> wrote:
Hello,

I am trying to use OE to write to a different HD than OE is installed on. I am able to record onto the C: drive (OE install) with no problems, but when I try to write to a second HD on the same computer, the buffer fills within a couple seconds.

Is there any solution to this besides either running a different install of OE or switching to an SSD?

Thanks

--
You received this message because you are subscribed to the Google Groups "Open Ephys" group.
To unsubscribe from this group and stop receiving emails from it, send an email to open-ephys+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Open Ephys" group.
To unsubscribe from this group and stop receiving emails from it, send an email to open-ephys+unsubscribe@googlegroups.com.

Tobias Niebur

unread,
Jan 15, 2025, 5:06:20 PMJan 15
to Open Ephys
Hello,

Sorry to revive an old thread. I did as suggested and installed a new internal SSD. The buffer problem disappeared for about a month, the SSD is now half full and I am once again running into this problem, it's not consistently every recording but it is common enough to pose a real problem with our experiments. I'm currently waiting on extra storage for more backups, but requiring 2tb of storage to be unused in order to record seems like a waste. Is there anything else I could to do avoid this?

Thanks

On Sunday, October 27, 2024 at 2:32:35 PM UTC-4 Josh Siegle wrote:
Hi Tobias,

If you're planning to scale to 100+ channels, I would strongly recommend installing an internal SSD. An external SSD may work, but in the scheme of things an internal SSD is not much of a price to pay to guarantee that recording will be reliable.

Josh


On Fri, Oct 25, 2024 at 12:13 PM, Tobias Niebur <teni...@gmail.com> wrote:
Thank you Josh,

We are currently trying to write 64 channels, but plan to write 400 in the near future.
Would you anticipate a bottleneck to exist with an external SSD? Installing an internal SSD is not a large operation but is more difficult than simply using an external SSD.

Thanks


On Thursday, October 24, 2024 at 5:58:20 PM UTC-4 Josh Siegle wrote:
Hi Tobias,

How many channels are you trying to write simultaneously? You could try using multiple record nodes to distribute data writing across more than one thread. But the HD will likely still be the bottleneck. Switching to an SSD or recording to the C: drive is your best bet for solving this issue.

Josh


On Tue, Oct 22, 2024 at 3:33 PM, Tobias Niebur <teni...@gmail.com> wrote:
Hello,

I am trying to use OE to write to a different HD than OE is installed on. I am able to record onto the C: drive (OE install) with no problems, but when I try to write to a second HD on the same computer, the buffer fills within a couple seconds.

Is there any solution to this besides either running a different install of OE or switching to an SSD?

Thanks

--
You received this message because you are subscribed to the Google Groups "Open Ephys" group.
To unsubscribe from this group and stop receiving emails from it, send an email to open-ephys+...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Open Ephys" group.
To unsubscribe from this group and stop receiving emails from it, send an email to open-ephys+...@googlegroups.com.

Josh Siegle

unread,
Jan 15, 2025, 7:29:27 PMJan 15
to Tobias Niebur, Open Ephys
I don't think I've ever seen a case where SSD write speed depends on the amount of space remaining, so this is new. What's the model of the SSD, and what are the specs of the machine you're using for acquisition?

Josh


On Wed, Jan 15, 2025 at 2:06 PM, Tobias Niebur <teni...@gmail.com> wrote:
Hello,

Sorry to revive an old thread. I did as suggested and installed a new internal SSD. The buffer problem disappeared for about a month, the SSD is now half full and I am once again running into this problem, it's not consistently every recording but it is common enough to pose a real problem with our experiments. I'm currently waiting on extra storage for more backups, but requiring 2tb of storage to be unused in order to record seems like a waste. Is there anything else I could to do avoid this?

Thanks

On Sunday, October 27, 2024 at 2:32:35 PM UTC-4 Josh Siegle wrote:
Hi Tobias,

If you're planning to scale to 100+ channels, I would strongly recommend installing an internal SSD. An external SSD may work, but in the scheme of things an internal SSD is not much of a price to pay to guarantee that recording will be reliable.

Josh


On Fri, Oct 25, 2024 at 12:13 PM, Tobias Niebur <teni...@gmail.com> wrote:
Thank you Josh,

We are currently trying to write 64 channels, but plan to write 400 in the near future.
Would you anticipate a bottleneck to exist with an external SSD? Installing an internal SSD is not a large operation but is more difficult than simply using an external SSD.

Thanks


On Thursday, October 24, 2024 at 5:58:20 PM UTC-4 Josh Siegle wrote:
Hi Tobias,

How many channels are you trying to write simultaneously? You could try using multiple record nodes to distribute data writing across more than one thread. But the HD will likely still be the bottleneck. Switching to an SSD or recording to the C: drive is your best bet for solving this issue.

Josh


On Tue, Oct 22, 2024 at 3:33 PM, Tobias Niebur <teni...@gmail.com> wrote:
Hello,

I am trying to use OE to write to a different HD than OE is installed on. I am able to record onto the C: drive (OE install) with no problems, but when I try to write to a second HD on the same computer, the buffer fills within a couple seconds.

Is there any solution to this besides either running a different install of OE or switching to an SSD?

Thanks

--
You received this message because you are subscribed to the Google Groups "Open Ephys" group.
To unsubscribe from this group and stop receiving emails from it, send an email to open-ephys+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Open Ephys" group.
To unsubscribe from this group and stop receiving emails from it, send an email to open-ephys+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Open Ephys" group.
To unsubscribe from this group and stop receiving emails from it, send an email to open-ephys+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/open-ephys/56dd454e-e81d-48f2-8ec8-d27b38162033n%40googlegroups.com.

Tobias Niebur

unread,
Apr 9, 2025, 3:32:46 PMApr 9
to Open Ephys
Sorry for the delay, I did indeed move most of the data off of the SSD and it solved the issue but now it is back (1.3TB remaining). I am able to record with SpikeGLX without an issue.

This is a WD Blue 4TB SATA drive (SA510).

Intel Xeon CPU E5-2630 v4
32GB Ram
NVS310 
To unsubscribe from this group and stop receiving emails from it, send an email to open-ephys+...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Open Ephys" group.
To unsubscribe from this group and stop receiving emails from it, send an email to open-ephys+...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Open Ephys" group.
To unsubscribe from this group and stop receiving emails from it, send an email to open-ephys+...@googlegroups.com.

Tobias Niebur

unread,
Apr 9, 2025, 3:40:40 PMApr 9
to Open Ephys
As an addition, I've tested this by reducing the number of recorded channels on the record node from 384 to 200, and the buffer problem persists.

Josh Siegle

unread,
Apr 9, 2025, 11:43:53 PMApr 9
to Tobias Niebur, Open Ephys
Hi Tobias,

The WD Blue SATA is actually a hard drive, not an SSD (solid state drive). It uses a spinning magnetic platter, and it can definitely slow down as it fills up, due to the need to move the write head between physical locations where data is stored. So that explains the difficulty of writing many channels of data, and also the drop in performance as the space is used up.

The reason for the recording performance difference between the Open Ephys GUI and SpikeGLX is that the GUI uses 32-bit floating point numbers to pass data between plugins, which must be converted to 16-bit integers before saving to disk, creating additional overhead when writing data. This allows electrode signals to be represented in physical units (microvolts), which makes it much easier for plugin developers to know how to handle them. SpikeGLX uses integers internally, which makes disk writing more efficient, but comes at the cost of greater complexity when writing code for real-time processing or visualization.

Josh




To unsubscribe from this group and stop receiving emails from it, send an email to open-ephys+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/open-ephys/cfdabd22-154f-4ec8-9f6e-f9b161ac5d2an%40googlegroups.com.

Tobias Niebur

unread,
Apr 10, 2025, 12:42:58 PMApr 10
to Open Ephys
Hello Josh,

The SA510 is a SSD, not HDD. 

Thanks for the clarification on the differences between SPGLX and OE data writing.

Tobias

Josh Siegle

unread,
Apr 10, 2025, 5:42:57 PMApr 10
to Tobias Niebur, Open Ephys
Sorry about that! I just searched for "WD Blue 4TB SATA" and didn't include the product number.

Given that it is an SSD, my guess would be that the processor (2.2 GHz) is having trouble keeping up with the conversions that happen prior to disk writing. We recommend a minimum of 3.5 GHz processor speed for Neuropixels acquisition (see hardware requirements page for more details).

Josh



To unsubscribe from this group and stop receiving emails from it, send an email to open-ephys+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Open Ephys" group.
To unsubscribe from this group and stop receiving emails from it, send an email to open-ephys+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/open-ephys/1994b353-1e5f-4a1d-896c-e0206733d6bcn%40googlegroups.com.

Tobias Niebur

unread,
Apr 15, 2025, 1:24:54 PMApr 15
to Open Ephys
Ok thank you!

--
You received this message because you are subscribed to the Google Groups "Open Ephys" group.
To unsubscribe from this group and stop receiving emails from it, send an email to open-ephys+...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/open-ephys/1994b353-1e5f-4a1d-896c-e0206733d6bcn%40googlegroups.com.

Reply all
Reply to author
Forward
0 new messages