Pushing Terrabytes of files across a WAN

[Jay Askren]

We need to push 40 TB of images per day from our scanning department in Utah to our storage servers in Virginia and then we download about 4 TB of processed images per day back to Utah.  In our previous process we had no problem getting the throughput we needed by using Robocopy which comes with Windows, but our old storage servers were here in Utah.  We can get Robocopy to work across the WAN but we have to run 3 or 4 Robocopy processes under different Windows users which is somewhat fragile and feels like a bad hack.  The files here in Utah are on a Windows server because of the proprietary software needed to run the scanner.  All of our servers in Virginia run Centos.

Any thoughts on how to transfer files over long distance and still get high throughput?  I believe the issue we are running into is high latency.

[Todd Montgomery]

Hi Jay.

Going to assume Robocopy uses TCP....

As you had no real issues with things without a WAN, I would assume the TCP window sizes, etc. are all good for the rates you need.

Latency will play a role, but more likely loss is a more impactful factor as congestion control will be more of a throttle than flow control. With TCP (low loss rate), RTT scales linearly with throughput. Well, as RTT goes up, throughput goes down, but it is linear. With loss, even low loss, throughput scales with sqrt(loss rate). After about 5%, TCP-Reno goes into stop-and-wait. 1 MSS per RTT. This scale is non-linear and in the < 5% loss rate area is really painful on throughput.

In short, WANs will slow down with loss quite a lot. Latency will also have an impact, though. Just not as much potential.

Running multiple TCP connections over the same path will mean that they will fight with one another via congestion control trying to find a fairness point that jumps around and can end up underutilizing the bandwidth at times. This is where things like TCP BBR can be helpful. But still, loss will cause quite a slow down.

What can you do? Well, it depends on what your links between the areas actually are..... terrestrial vs. satellite, etc. Lots of options.

On Thu, Sep 13, 2018 at 9:41 AM Jay Askren <jay.a...@gmail.com> wrote:

We need to push 40 TB of images per day from our scanning department in Utah to our storage servers in Virginia and then we download about 4 TB of processed images per day back to Utah.  In our previous process we had no problem getting the throughput we needed by using Robocopy which comes with Windows, but our old storage servers were here in Utah.  We can get Robocopy to work across the WAN but we have to run 3 or 4 Robocopy processes under different Windows users which is somewhat fragile and feels like a bad hack.  The files here in Utah are on a Windows server because of the proprietary software needed to run the scanner.  All of our servers in Virginia run Centos.

Any thoughts on how to transfer files over long distance and still get high throughput?  I believe the issue we are running into is high latency.

--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-symp...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Thomas Lindgren]

So you will be transferring 320 Tb per day in one direction. That means you need to transfer 3.7 Gbps just to be done in 24 hours. (Or 11.1 Gbps to be done in 8 hours.)

Let's take Verizon as an example. Round-trip latency in North America appears to be 35-42ms in practice with an SLA of 45ms. You might have different numbers in practice. 

http://www.verizonenterprise.com/about/network/latency/

Your TCP window size (B) will have to be Latency(s) * Connection (Bps). Assuming a single Verizon connection, that looks like a (11.1/8*1000*0.045) 62 MB TCP window. Assuming nothing goes wrong and utilization is 100%, of course. 

http://bradhedlund.com/2008/12/19/how-to-calculate-tcp-throughput-for-long-distance-links/

Also see that post for a couple of tricks you could perhaps use. 

Best,

/Thomas

--

Thomas Lindgren

[Jay Askren]

Todd Montgomvery,

Correct, robocopy uses TCP.  We have a 10 Gbps terrestrial line and are working on getting a second line for redundancy.

Thomas, 

Thanks for the link.  I will look at that page.

Jay

To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-sympathy+unsub...@googlegroups.com.

[Kevin Bowling]

Most the performance will be dictated by the sender side.  As others have said make sure the send buffer and reccive buffer are large enough for the BDP.

You can try experimenting with CUBIC:

https://blogs.technet.microsoft.com/networking/2017/07/13/core-network-stack-features-in-the-creators-update-for-windows-10/

Multiplatform makes this a bit more challenging so If you really need it to work optimally you might want to get a copy of something like Aspera.

On Thu, Sep 13, 2018 at 1:15 PM Jay Askren <jay.a...@gmail.com> wrote:

Correct, robocopy uses TCP.  We have a 10 Gbps terrestrial line and are working on getting a second line for redundancy.

On Thursday, September 13, 2018 at 11:04:54 AM UTC-6, Todd L. Montgomery wrote:

Hi Jay.

Going to assume Robocopy uses TCP....

As you had no real issues with things without a WAN, I would assume the TCP window sizes, etc. are all good for the rates you need.

Latency will play a role, but more likely loss is a more impactful factor as congestion control will be more of a throttle than flow control. With TCP (low loss rate), RTT scales linearly with throughput. Well, as RTT goes up, throughput goes down, but it is linear. With loss, even low loss, throughput scales with sqrt(loss rate). After about 5%, TCP-Reno goes into stop-and-wait. 1 MSS per RTT. This scale is non-linear and in the < 5% loss rate area is really painful on throughput.

In short, WANs will slow down with loss quite a lot. Latency will also have an impact, though. Just not as much potential.

Running multiple TCP connections over the same path will mean that they will fight with one another via congestion control trying to find a fairness point that jumps around and can end up underutilizing the bandwidth at times. This is where things like TCP BBR can be helpful. But still, loss will cause quite a slow down.

What can you do? Well, it depends on what your links between the areas actually are..... terrestrial vs. satellite, etc. Lots of options.

On Thu, Sep 13, 2018 at 9:41 AM Jay Askren <jay.a...@gmail.com> wrote:

We need to push 40 TB of images per day from our scanning department in Utah to our storage servers in Virginia and then we download about 4 TB of processed images per day back to Utah.  In our previous process we had no problem getting the throughput we needed by using Robocopy which comes with Windows, but our old storage servers were here in Utah.  We can get Robocopy to work across the WAN but we have to run 3 or 4 Robocopy processes under different Windows users which is somewhat fragile and feels like a bad hack.  The files here in Utah are on a Windows server because of the proprietary software needed to run the scanner.  All of our servers in Virginia run Centos.

Any thoughts on how to transfer files over long distance and still get high throughput?  I believe the issue we are running into is high latency.

--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-symp...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-symp...@googlegroups.com.

[yawkat]

UDT ( https://en.wikipedia.org/wiki/UDP-based_Data_Transfer_Protocol ) was made exactly for this purpose, but I'm not familiar with how well it works as a standalone tool.

[high-voltage]

Have considered using BitTorrent for this purpose ? It scales out quite well. You may need to add some peers to the make the transfer faster.

[Gil Tene]

Assuming your 40TB is spread across many files, the main thing I'd play with is the number of threads used in the copy (Robocopy has a /MT:n option which defaults to 8 but can be set as high as 128 per https://docs.microsoft.com/en-us/windows-server/administration/windows-commands/robocopy). On the assumption that each thread uses blocking APIs on a TCP connection, using the highest number of threads (128) will allow you to keep the throughput and TCP window sizes of each TCP connection closer to the range that mere mortals work at. and still be able to saturate this cool 10Gbps high latency link.

To saturate the 10Gbps link, your 128 connections will average ~80Mbps per connection. With an e.g. ~50msec actual RTT, this means connections will need at least a ~512KB send buffer (and probably 2-3x that since you'll see variance across them, and some will need to be faster than the average). On older OSs this could be a challenge (and require tweaking default send and receive buffer limits), but it should not be a problem if RFC 1323 TCP auto-scaling is used an supported on both sides. I would like to assume that any Windows and CentOS setup that has a 10Gbps NIC also has TCP window scaling support on by default and that all network elements on the path (including e.g. firewalls) don't block TCP Window Scale Option in RFC 1323, but assumptions like that could be a bit presumptuous. Just in case, there is some useful discussion here .

One other obvious question to ask, since (as noted by Thomas) your average transfer speed will need to be at least 3.7Gbps in order to complete 40TB in each 24hr period, is whether anyone else is using that nice fat 10Gbps link... It the link dedicated to you? You'll be using well over 30% of it's capacity,, and it's enough for two or more other users like you to want to do the same for the math to never work out...

[ has window scaling enabled (which I'd hope to see on on any machine with a 10Gbps network card), this shouldn't be a problem. But just in case, I'd 

[Carl Mastrangelo]

+1 to BBR.  Reports I have seen shows it maintains a much higher throughput in lossy environments.   It should easy to turn on in a recent kernel.   

https://reproducingnetworkresearch.wordpress.com/2017/06/05/rebbr/

To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-sympathy+unsub...@googlegroups.com.

[Kevin Bowling]

There is no current BBR stack for Windows that I am aware of without
getting into userland transports. Windows' default congestion control
is either a hybrid called Compound TCP that should do better than
traditional loss-based CC in environments where they fall down or
CUBIC depending on version, however this sounds like a nearly perfect
situation of server-class hardware on each end (realistically no need
to bound send/receive windows) on a terrestrial line (close to zero
channel loss). The sender side TCP stack and congestion control will
dictate the performance in this scenario for any reasonably modern
receiver stack of the past 20 years.

Make sure you have datacenter-class switches on each side so you
aren't dropping packets on the floor on the easy part of the journey,
crank the send and receive buffers way up to match the BDP, and try
switching between Windows CTCP and CUBIC for a comparison, enabling
more aggressive window scaling
(https://blogs.technet.microsoft.com/netgeeks/2018/05/21/a-word-about-autotuninglevel-tcp-receive-auto-tuning-level-explained/)

Regards,
Kevin

>>> To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-symp...@googlegroups.com.

>>> For more options, visit https://groups.google.com/d/optout.
>

> --
> You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-symp...@googlegroups.com.

[Richard Warburton]

Hi Jay,

There's a lot of good advice in this thread around networking, but have you tried compressing the files before sending and decompressing on the other side? I know its not always practical, but its often a really simple win. Especially if you use some of the more modern compression algorithms - I've had a dataset I've been using recently that compresses about 2x better with zstd than gzip for example.

On Thu, Sep 13, 2018 at 5:41 PM Jay Askren <jay.a...@gmail.com> wrote:

We need to push 40 TB of images per day from our scanning department in Utah to our storage servers in Virginia and then we download about 4 TB of processed images per day back to Utah.  In our previous process we had no problem getting the throughput we needed by using Robocopy which comes with Windows, but our old storage servers were here in Utah.  We can get Robocopy to work across the WAN but we have to run 3 or 4 Robocopy processes under different Windows users which is somewhat fragile and feels like a bad hack.  The files here in Utah are on a Windows server because of the proprietary software needed to run the scanner.  All of our servers in Virginia run Centos.

Any thoughts on how to transfer files over long distance and still get high throughput?  I believe the issue we are running into is high latency.

--

You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-symp...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

regards,

  Richard Warburton

  http://insightfullogic.com

  @RichardWarburto

[Chris Miller]

I'm not sure if https://bvckup2.com/ will be suitable or not but might be worth benchmarking for your use case. [disclaimer: I'm not affiliated to the product in any way, just a satisfied user]

[Jay Askren]

Thanks everyone for the great ideas.  To answer several questions and offer some clarification.  We have 7 Windows servers on site that do the pushing to our storage server in Virginia.  If we can get around 80% saturation of each of the 1Gbps NICs on these machines that is more than enough throughput for our purposes now.  The 10 Gbps line has very minimal traffic other than us uploading these files.  I believe the storage server has a 10 Gbps NIC.

As far as compression, we looked at that early on but decided against it because the compression plus the file transfer took more time than the file transfer and we were trying to move the images off the Windows server as quickly as possible.  But, I think it would be worth revisiting as I only tried gzip before.

I am looking into the other technologies you suggested.  Thank you!

[Edward Ribeiro]

One last tip that maybe could help you: Globus' GridFTP. It is used in academic research for transferring large datasets. See more here: http://toolkit.globus.org/toolkit/docs/latest-stable/gridftp/ and https://www.globus.org/data-transfer It has been many years since I dabbled on Globus and GridFTP and it was sort of hard to configure and use back then, but it should be easier today (looks like they even sell a SaaS for this nowadays).

[Alex Kashchenko]

Hi,

On 09/19/2018 09:14 PM, Jay Askren wrote:
> Thanks everyone for the great ideas. To answer several questions and offer
> some clarification. We have 7 Windows servers on site that do the pushing
> to our storage server in Virginia. If we can get around 80% saturation of
> each of the 1Gbps NICs on these machines that is more than enough
> throughput for our purposes now. The 10 Gbps line has very minimal traffic
> other than us uploading these files. I believe the storage server has a 10
> Gbps NIC.
>
> As far as compression, we looked at that early on but decided against it
> because the compression plus the file transfer took more time than the file
> transfer and we were trying to move the images off the Windows server as
> quickly as possible. But, I think it would be worth revisiting as I only
> tried gzip before.

On compression, it obviously depends on data, but in the past for
"receive-filter-save-read-send" operations under load I've had a
positive experience with Snappy compression. All IO operations wrapped
with Snappy were consistently faster than no-compression mode, when gzip
was 2x+ slower due to high CPU usage.

> I am looking into the other technologies you suggested. Thank you!
>
>
> On Thursday, September 13, 2018 at 10:41:50 AM UTC-6, Jay Askren wrote:
>>
>> We need to push 40 TB of images per day from our scanning department in
>> Utah to our storage servers in Virginia and then we download about 4 TB of
>> processed images per day back to Utah. In our previous process we had no
>> problem getting the throughput we needed by using Robocopy which comes with
>> Windows, but our old storage servers were here in Utah. We can get
>> Robocopy to work across the WAN but we have to run 3 or 4 Robocopy
>> processes under different Windows users which is somewhat fragile and feels
>> like a bad hack. The files here in Utah are on a Windows server because of
>> the proprietary software needed to run the scanner. All of our servers in
>> Virginia run Centos.
>>
>> Any thoughts on how to transfer files over long distance and still get
>> high throughput? I believe the issue we are running into is high latency.
>>
>

--
-Alex