Got 'Too many open file' exception

[Buddha Shrestha]

Hi, I've been using vertx for my application which connects to other apis and makes HTTP post,get requests and also run some database operations. 

Everything is fine, but after certain amount of time (few days), the server start to throw exception as in:

2016-08-08 16:37:20,576 [WARN ][vert.x-acceptor-thread-0][y.channel.DefaultChannelPipeline] - An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.

java.io.IOException: Too many open files

at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)

at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)

at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)

at io.netty.channel.socket.nio.NioServerSocketChannel.doReadMessages(NioServerSocketChannel.java:135)

at io.netty.channel.nio.AbstractNioMessageChannel$NioMessageUnsafe.read(AbstractNioMessageChannel.java:75)

at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:563)

at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:504)

at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:418)

at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:385)

at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:742)

Initially I thought that I could have not properly closed the http connection, so I set setKeepAlive option to false. But still this problem persists.

Heres my reproducer (This runs fine for a while... but after certain time... the exception starts to pop up and the server doesnot accept any further requests.):

https://github.com/buddhashrestha/vertx-jdbc-service

Please do suggest. Thanks!

[Julien Viet]

Hi,

can you give OS details ?

--
You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.
Visit this group at https://groups.google.com/group/vertx.
To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/f3a65e19-4463-4b02-bcec-f310f8fef0a0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Jez P]

Also is it possible to share a simple reproducer? Are you using a lot of Vertx.vertx() calls? If so that would lead to this kind of problem (as you'll be creating multiple Vertx instances).

[Alexander Lehmann]

You could do a lsof when you are seeing the issue, this should show which files or sockets are open (probably http or the db connection), which would narrow it down a bit.

In Java, there are some small errors possible where the connection is not closed until the garbage collection happens, which might cause your issue.

[Buddha Shrestha]

Hi Julien, 

Here's my CPU info: I'm using CentOS Linux release 7.1.1503 (Core) (AWS service)

processor : 0

vendor_id : GenuineIntel

cpu family : 6

model : 63

model name : Intel(R) Xeon(R) CPU E5-2676 v3 @ 2.40GHz

stepping : 2

microcode : 0x25

cpu MHz : 2394.518

cache size : 30720 KB

physical id : 0

siblings : 2

core id : 0

cpu cores : 1

apicid : 0

initial apicid : 0

fpu : yes

fpu_exception : yes

cpuid level : 13

wp : yes

bogomips : 4789.03

clflush size : 64

cache_alignment : 64

address sizes : 46 bits physical, 48 bits virtual

power management:

processor : 1

vendor_id : GenuineIntel

cpu family : 6

model : 63

model name : Intel(R) Xeon(R) CPU E5-2676 v3 @ 2.40GHz

stepping : 2

microcode : 0x25

cpu MHz : 2394.518

cache size : 30720 KB

physical id : 0

siblings : 2

core id : 0

cpu cores : 1

apicid : 1

initial apicid : 1

fpu : yes

fpu_exception : yes

cpuid level : 13

wp : yes

bogomips : 4789.03

clflush size : 64

cache_alignment : 64

address sizes : 46 bits physical, 48 bits virtual

power management:

[Buddha Shrestha]

Hi Jez,

We took reference from : https://github.com/cescoffier/vertx-workshop

And we call Vertx.vertx() only once. After all the verticles are deployed, we then use event bus to communicate with them. Please let me know if you have any suggestion.

Thanks,

[Buddha Shrestha]

Hi Alexander,

Doing lsof, I saw that my database clients were open in cumulative order. It turned out that I had not closed the JDBC client. What I'm wondering is that the document says that we should close the client, but I haven't seen any example that specifically closes the client.

Well, for now I've closed the client. Hope it works out.

Thanks for the suggestion.

[Alexander Lehmann]

I guess the jdbc client keeps a connection pool, so if you create a client for each connection, you are leaking files if you don't close it.

[Jez P]

And this suggests that a jdbc client is being created for each connection: https://github.com/buddhashrestha/vertx-jdbc-service/blob/master/service-db/src/main/java/com/test/db/Utils/NonSharedDatabase.java

[Buddha Shrestha]

Yup thanks Alexander.

Jez, yup I didn't close the client previously. Now I've closed it and pushed it. 

Thanks guys for the help.