"Too many open files" errors and many "Connection reset by peer"

[poiuytrez]

Hello, 

I am running a vert.x 3.2 webserver. When I run the server I get many:

12:52:32.874 [vert.x-eventloop-thread-1] ERROR i.vertx.core.net.impl.ConnectionBase - java.io.IOException: Connection reset by peer

during the first 5 minutes and after 5 minutes I get:

java.io.IOException: Too many open files
 at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) ~[na:1.8.0_66-internal]
 at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422) ~[na:1.8.0_66-internal]
 at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250) ~[na:1.8.0_66-internal]
 at io.vertx.core.http.impl.VertxNioServerSocketChannel.doReadMessages(VertxNioServerSocketChannel.java:35) ~[blueberry2-fat.jar:na]
 at io.netty.channel.nio.AbstractNioMessageChannel$NioMessageUnsafe.read(AbstractNioMessageChannel.java:69) ~[blueberry2-fat.jar:na]
 at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) [blueberry2-fat.jar:na]
 at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) [blueberry2-fat.jar:na]
 at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) [blueberry2-fat.jar:na]
 at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:349) [blueberry2-fat.jar:na]
 at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:112) [blueberry2-fat.jar:na]
 at java.lang.Thread.run(Thread.java:745) [na:1.8.0_66-internal]

The server is running in linux debian 8. The max open files seems to be configured correctly:

# ulimit -n
65536

When I check the number of file descriptors open, the number is around 65000 after 5 minutes:

# ls -la /proc/1740/fd | grep socket | wc -l
65412

What would you recommend to debug the issue?

Best Regards,

poiuytrez

[Alexander Lehmann]

you may be able to figure out where file descriptors are leaking by running lsof when you get the error.

Probably there is a close operation missing at the end of a handler

[Hui Zhang]

65535 is too low for server side, please exam your client implementation, it seems your clients close connection improperly.

Hui