Optimal configuration for sending large files using Netty

[Ravikant Dindokar]

I want to transfer files of sizes ranging from 100MB to 800MB using netty. I am running my distributed application on cluster which has bandwidth of ~112 MB/s between nodes. With the current configuration, it takes ~6sec to transfer a file of size 180MB while ~20sec for a file of size 360MB. I have specified following config:

nettyClientThreads=4
nettyServerThreads=16
useNettyPooledAllocator=true
nettyRequestEncoderBufferSize=1048576
receiveBufferSize=1048576
sendBufferSize=1048576







With the bandwidth available, I suppose that the transfer time can be reduced. I am not able to figure out what should be the optimal values for my requirements.


Please help, how to fine tune the netty to improve the file transfer performance.

Thanks

Ravikant

[Tim Boudreau]

First you might want to post the code you're using to do the file transfer.

Then you probably want to figure out the theoretical minimum time it could possibly take to do that transfer on your network (if it were carrying absolutely no other traffic), and compare your numbers to that.

And find out what else is happening on your network.

And find out what all the computers (on both ends!) are actually doing.  For that, I'd suggest *not* starting with, say, a Java profiler.  Instead, start with system counters - say, poll the results of vmstat while the code is running and graph them.  If you're wondering where your code is spending time, a non-invasive approach is to take periodic thread dumps over a period of hours - say one every 30 seconds - and then sort them by how frequently specific ones come up.  The more frequent ones are where your code is spending a lot of time (ignore "waiting for something to do" states).

After that, there are a lot of knobs you can turn, and they are not all the ones you listed.

The point is, in this sort of thing, there are a huge number of variables that affect performance.  The first thing is to figure out where you're spending time - which sometimes is easy to find, sometimes a process of eliminating what things it definitely isn't, one by one.  There is never a silver-bullet, one-size-fits-all answer - for example, one obvious thing to tune is buffer sizes, and the right buffer size can change based on what settings your kernel was compiled with, what kind of network card you're talking to, the MTU of the network you're on, etc.  Depending on where the file is (network drive or SAN?  Congratulations, you're doing twice the network I/O you thought you were!) you might be bottlenecking on that.

Or, for that matter, it could be completely unrelated traffic on your network that's the problem.  Or a network configuration issue - I once spent a day and a half diagnosing random periods of 70-80% packet loss on a staging server on an excellent network - and finally the network admin realized he had given me an IP address that was already in use by a video-conferencing system, so things only went crazy when a meeting was happening (talk about literally a server that works great except when you're out of the room!).

I'd suggest resisting the temptation to randomly turn knobs until something looks like it works, especially without a good understanding of what the knobs do.  For example, back when I worked on NetBeans, every now and then a user would post to the mailing list that NetBeans would get really slow after working for a while.  Inevitably I'd find out that they'd set their Java heap size so big that the Java heap was swapped to disk, and every JVM garbage collection was thrashing massive amounts of virtual memory into and out of physical memory.  It was a thing that looked like the magic "run real fast" knob, so people would turn it, and it would look like things ran real fast - until they ran straight into a brick wall.  So random adjustments can look like they work when they're actually pathological.

Performance work is fun, and you can learn a lot about how things under the hood really work that will make you a better engineer for the rest of your life.  Here are some resources from one of the best in the performance business:

http://www.brendangregg.com/usemethod.html

http://www.brendangregg.com/USEmethod/use-linux.html

The bottom line is, there is no magic answer to your question that isn't going to be dependent on things that are really specific to your set-up.

-Tim