question on SO_BACKLOG of ChannelOption.

[Ronggen Liu]

Hello,

problem:

some connections were lost when there are too many clients were trying to connect to the server during a short time, for example, send out about 10k connections during about 3 seconds.

findings:

as i know, all the connections would be accepted in method doReadMessage of NioServerSocketChannel, and, in fact, we did find out that some connections were not accepted from there, but, actually, the TCP have been created successfully with completed 3-ways handsharke. 

meanwhile, we found, to some degree, we can avoid this issue by increasing the option SO_BACKLOG of ChannelOption, currently, we set it with 8, should be a little bit low.

questions:

how to handle this issue? as i know, the client should receive connection refused error when the number of the connections exceed the SO_BACKLOG, but the client didn't receive it.

Thanks,

Gary

[Norman Maurer]

Hey,

comments inside…

On 20. Jul 2017, at 08:50, Ronggen Liu <rongg...@gmail.com> wrote:

Hello,

problem:

some connections were lost when there are too many clients were trying to connect to the server during a short time, for example, send out about 10k connections during about 3 seconds.

findings:

as i know, all the connections would be accepted in method doReadMessage of NioServerSocketChannel, and, in fact, we did find out that some connections were not accepted from there, but, actually, the TCP have been created successfully with completed 3-ways handsharke. 

meanwhile, we found, to some degree, we can avoid this issue by increasing the option SO_BACKLOG of ChannelOption, currently, we set it with 8, should be a little bit low.

8 is most likely a way too low. I would not touch this setting at all if you not want to increase it to a high-number.

questions:

how to handle this issue? as i know, the client should receive connection refused error when the number of the connections exceed the SO_BACKLOG, but the client didn't receive it.

No it will not receive a connect refused error but the connection will just timeout. You should check if you block the EventLoop somehow and so not be able to accept fast enough.

Thanks,

Gary

--
You received this message because you are subscribed to the Google Groups "Netty discussions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to netty+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/netty/3f5aae56-1fd6-4e0b-8a6e-cfed66653c1e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Ronggen Liu]

Thanks, but I'm still confused why the server can't accept the connection when the TCP 3-way handshake is completed.

[Norman Maurer]

Like I said you may just block the EventLoop. Attach profiler and check if the EventLoop is blocked that is used by the SocketChannel.

To view this discussion on the web visit https://groups.google.com/d/msgid/netty/bbccbf19-aa67-4346-83dd-566257893296%40googlegroups.com.

[Ronggen Liu]

is it code related? the server is pretty simple like below.

public class EchoServer {
  private final int port;
  EventLoopGroup serverEventLoop = new NioEventLoopGroup(1);
  EventLoopGroup btsEventLoop = new NioEventLoopGroup();

  public EchoServer(int port) {
    this.port = port;
  }

  public static void main(String[] args)
      throws Exception {
    int port = Integer.parseInt(args[0]);
    int backlog = Integer.parseInt(args[1]);
    new EchoServer(port).start(backlog);
  }

  public void start(int backlog) throws Exception {
    final EchoServerHandler serverHandler = new EchoServerHandler();
    try {
      ServerBootstrap b = new ServerBootstrap();
      b.group(serverEventLoop, btsEventLoop)
          .channel(NioServerSocketChannel.class)
          .localAddress(new InetSocketAddress(port))
          .childHandler(new ChannelInitializer<SocketChannel>() {
            @Override
            public void initChannel(SocketChannel ch) throws Exception {
              ch.pipeline().addLast(serverHandler);
            }
          })
          .option(ChannelOption.SO_BACKLOG, backlog)
//          .option(ChannelOption.AUTO_READ, false)
          .childOption(ChannelOption.SO_KEEPALIVE, true);
      ChannelFuture f = b.bind().sync();
      System.out.println("backlog: " + b.config().options().get(ChannelOption.SO_BACKLOG));
//      System.out.println("autoRead: " + b.config().options().get(ChannelOption.AUTO_READ));
      System.out.println("Server started and listening for connections on " + f.channel().localAddress());
      f.channel().closeFuture().sync();
    } finally {
      serverEventLoop.shutdownGracefully().sync();
      btsEventLoop.shutdownGracefully().sync();
      System.out.println("finished...");
    }
  }
}

public class EchoServerHandler extends ChannelInboundHandlerAdapter {
    private static volatile int channelNum;
    @Override
    public void channelRead(ChannelHandlerContext ctx, Object msg) {
        ByteBuf in = (ByteBuf) msg;
        System.out.println(
                "Server received: " + in.toString(CharsetUtil.UTF_8));
        ctx.write(Unpooled.copiedBuffer("msg from server: " + ctx.channel().remoteAddress(), CharsetUtil.UTF_8));
    }

    @Override
    public void channelActive(ChannelHandlerContext ctx) throws Exception {
        channelNum++;
        System.out.println("Active Channel..." + channelNum);
    }

    @Override
    public void channelRegistered(ChannelHandlerContext ctx) throws Exception {
//        System.out.println("Register...");
    }

    @Override
    public void channelReadComplete(ChannelHandlerContext ctx)
            throws Exception {
        ctx.writeAndFlush(Unpooled.EMPTY_BUFFER)
                .addListener(ChannelFutureListener.CLOSE);
    }

    @Override
    public void exceptionCaught(ChannelHandlerContext ctx,
        Throwable cause) {
        cause.printStackTrace();
        ctx.close();
    }
}

[Ronggen Liu]

Hello,

according to the definition of SO_BACKLOG below, the connection would be refused in case of the queue is full, how it refuse the connection? would the server send a TCP message to the client, and then the client would send a re-connection when received the connection refuse/timeout error?

Description of SO_BACKLOG option:

The maximum queue length for incoming connection indications (a request to connect) is set to the backlog parameter. If a connection indication arrives when the queue is full, the connection is refused.

[Ronggen Liu]

Norman Maurer,

why the EventLoop maybe blocked when the SO_BACKLOG is low? you know, when i increase it to somewhat hight, the issue did is gone.

[Ronggen Liu]

Norman Maurer,

could you help to check the profiler attached? the server did was blocked when it ran in Linux OS, instead, the server would forcedly close the connection in Windows OS, any advice about how to resolve it?

BTW, i also attached the source code for my project. Thanks!