Why is Netty Client only pushing small amount of data through NIO Socket Channel When Network resources permit larger amounts of data to be pushed through

[L_DisplayName]

Greetings All,

  I have a Netty client  that sends two 5GB files to a Netty server. The Netty client sends one 5GB file directly to the Netty Server through a 10Gb link.   The Netty Client sends the other 5GB file through two Netty Proxy Servers to the Netty Server using 1Gb links.   The client, proxy servers and servers are directly connected to each other and do not have background traffic. I would like to know why the Netty Client only pushes KiloBytes of data through the socket instead of the specified 100MB? The network is more than capable of pushing MB of data. In the code I pass in a  NioChunkedInput to the ChunkedWriteHandler that fetches 100MB chunks of data to send through the socket, but only KiloBytes and a max of 1 or 2MB of data is actually pushed through the NioSocketChannel at a time. Each of the Channels were bootstrapped with a socket send buffer size of 100MB, socket receive buffer size of 100MB and TCP_NODELAY was set to true to disable Nagle's algorithm. I even set the NioEventLoopGroup's I/O ratio to 100 to spend 100% of the time processing I/O events.  Each channel shares the same NioEventLoopGroup. in addition the Netty proxy servers and the Netty Server channels were configured with the same configuration.  Please see the below log file that shows amount of data being pushed through the NioSocket. Note, I collected metrics from the io.netty.channel.socket.nio.NioSocketChannel doWrite method, please see a portion of the log file below containing these metrics In addition, when the client sends the two files through both paths as specified below the throughput of the transfer for both Paths decreases by approximately 50% as opposed to a Single One 5GB file transfer through only one path. For Example the throughput rendered for each path when sending one 5GB down path 1 and one 5GB down path 2 is: 

 Path 1: Netty Client (WS5), Netty Server (WS7)  -- 345 Mb/s

 Path 2: Netty Client (WS5), Netty Proxy Server 1 (WS11), Netty Proxy Server 2 (WS12), Netty Server (WS7) -- 318 Mb/s

Throughput rendered from the client just sending one 5GB file down path 1 and nothing down path 2 is:

Path 1: Netty Client (WS5), Netty Server (WS7)  -- 842 Mb/s

Throughput rendered from the client just sending one 5GB file down path 2 and nothing down path 1 is:  

Path 2: Netty Client (WS5), Netty Proxy Server 1 (WS11), Netty Proxy Server 2 (WS12), Netty Server (WS7) -- 724 Mb/s

My question is why does the throughput decrease for each path when sending data down 2 disjoint paths that share the same source and destination as opposed to sending data down one single path? Am I missing something that Netty is doing behind the scenes?

                           

Netty Client = Machine WS5

Netty Server = Machine WS7

Netty Proxy Servers = Machines WS11 & WS12

Path 1: WS5,WS7

 ----Channel: Thread ID: 15

Path 2 WS5,WS11,WS12,WS7

 ----Channel: Thread ID: 12

Portion of the log file showing the number of bytes being transferred through the socket at a given time

nioEventLoopGroup-3-2, Thread ID: 15 NioSocketChannel:doWrite(), (Call #: 17, Attempting to Write 104857600 Bytes to the Socket, Remaining bytes in Buffer = 104857600, Buffer Capacity = 104857600, Number of Flushed Msgs in the ChannelOutboundBuffer = 1, current Time: 1494460673317

---- Wrote: 101360 Bytes to the Socket, remaining Bytes to Write = 104756240

---- Wrote: 0 Bytes to the Socket, remaining Bytes to Write = 104756240

**Total Bytes Wrote to Socket = 101360 Bytes, Start Time 1494460673317, The endTime = 1494460673317, Elapsed Time = 0, Total Elapsed Time Since First Write: 509 ms

nioEventLoopGroup-2-2, Thread ID: 12 NioSocketChannel:doWrite(), (Call #: 17, Attempting to Write 104857600 Bytes to the Socket, Remaining bytes in Buffer = 104857600, Buffer Capacity = 104857600, Number of Flushed Msgs in the ChannelOutboundBuffer = 1, current Time: 1494460673334

---- Wrote: 101360 Bytes to the Socket, remaining Bytes to Write = 104756240

---- Wrote: 0 Bytes to the Socket, remaining Bytes to Write = 104756240

**Total Bytes Wrote to Socket = 101360 Bytes, Start Time 1494460673334, The endTime = 1494460673334, Elapsed Time = 0, Total Elapsed Time Since First Write: 529 ms

In addition, each machine running the client, proxy servers and servers have the following TCP Setting: 

net.ipv4.tcp_rmem = 4096 87380 67108864 

net.ipv4.tcp_wmem = 4096 87389 67108864

#
net.core.rmem_max = 67108864
net.core.wmem_max = 67108864

FileSenderHandler.java - Sends the file header info - File Name, offset, length and then the Actual File Contents

@Override
public void channelActive(ChannelHandlerContext ctx) throws Exception {
 try {
 String fileRequest = "ftp WS5/home/5GB_File1.dat WS7/tmp/5GB_File1_Copy.dat";

 //Source File to send / transfer to the Destination Node

 String theSrcFilePath =  "/home/5GB_File1.dat";

 //File Name to write on the destination node, once the file is received  
 String theDestFilePath = "/tmp/5GB_File1_Copy.dat";

//Get the source file to send
 File theFile = new File(theSrcFilePath);
 FileChannel theFileChannel = new RandomAccessFile(theFile, "r").getChannel();

//Get the length of the file
 long fileLength = theFileChannel.size();

 //Get the offset

 long offSet = 0;
 
 //Copy the offset to the ByteBuf
 ByteBuf offSetBuf = Unpooled.copyLong(offSet);
 //Copy the file length to the ByteBuf
 ByteBuf fileLengthBuf = Unpooled.copyLong(fileLength);

 //Get the Destination Filename (including the file path) in Bytes
 byte[] theDestFilePathInBytes = theDestFilePath.getBytes();
 //Get the length of theFilePath
 int theDestSize = theDestFilePathInBytes.length;
 //Copy the Dest File Path length to the ByteBuf
 ByteBuf theDestSizeBuf = Unpooled.copyInt(theDestSize);
 //Copy the theDestFilePathInBytes to the Byte Buf
 ByteBuf theDestFileBuf = Unpooled.copiedBuffer(theDestFilePathInBytes);
 

 //Send the file Headers: FileName Length, the FileName, the Offset and the file length
 ctx.write(theDestSizeBuf);
 ctx.write(theDestFileBuf);
 ctx.write(offSetBuf);
 ctx.write(fileLengthBuf);
 ctx.flush();

 //Send the 5GB File in 100MB chunks as specified by the following chunk size (1024*1024*100)
 ctx.write(new ChunkedNioFile(theFileChannel, offSet, fileLength, 1024 * 1024 * 100)); //Send File in 100MB Chunks
 ctx.flush();
 
 }catch(Exception e){
 System.err.printf("FileSenderHandler: Channel Active: Error: "+e.getMessage());
 e.printStackTrace();
 }
} //End channelActive