big object transfer performance

[Sara]

Hi Netty Team,
 
      I tried to send one big java object of size 64MB with ctx.writeAndFlush then i see performance drops to 200MB/sec whereas if i send 2MB * 32 times using ctx.writeAndFlush call then iam seeing around 800MB/sec.

     one thing i noticed during 64MB transfer most of the time EAGAIN happens and EPOLL OUT being added again and again and then write syscall being called. so it may be related to EAGAIN or memory allocation for big object..

     This i tried with EpollServerSocketChannel class.

     my requirement here is to send one big java object of size 64MB without performance penalty... can you please suggest how can i split and send the big java object object in to 2MB chunks such that i can avoid the EAGAIN or memory allocation issue.

     Thanks team for your great work...

Regards,
Sara
   

[Jörg Prante]

Hi Sara,

are you using Linux ? Did you check socket buffer default write memory size?

/proc/sys/net/core/wmem_default

where the default is 256k, and

/proc/sys/net/core/wmem_max

where the default is 1m, so socket options that set a socket write buffer size can use values up to 1m.

To change for IPv4, you can modify

/proc/sys/net/ipv4/tcp_wmem 

But be careful of huge values like 64m. I do not recommend such extreme values. They are not worth the price because each and every socket might be able to use such large buffer size for each write operation, which wastes a lot of memory, and increases the risk of running out of memory. And, because TCP frames can not exceed 64k (and typical MTU for ethernet is 1500 bytes), there is not much effect. If you care for performance on the Linux kernel and network interface level, you would have to use a Linux network performance tuning guide like /https://access.redhat.com/sites/default/files/attachments/20150325_network_performance_tuning.pdf

Best,

Jörg

[Sara]

Hi Prante,

         Thanks for your response...

         Iam using linux... i already increased wmem to 16MB max but still seeing low throughput with 64MB netty blob push..

         [sara@s0 ~]$ cat /proc/sys/net/core/wmem_default
         16777216
         [sara@s0 ~]$ cat /proc/sys/net/core/wmem_max
         16777216
         [sara@s0 ~]$ cat /proc/sys/net/ipv4/tcp_wmem
         4096    87380   16777216

 
Regards,
Saravanan