Too many files open on hiveserver

[Sonny Heer]

After kylo / nifi is running for a period of time.  the server we are connecting to via thrift in application.properites goes down and fills up the hiveserver2.log with the following exceptions:

2017-11-16 20:59:32,802 WARN  [Thread-13]: server.TThreadPoolServer (TThreadPoolServer.java:serve(206)) - Transport error occurred during acceptance of message.

org.apache.thrift.transport.TTransportException: java.net.SocketException: Too many open files (Accept failed)

        at org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:134)

        at org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:35)

        at org.apache.thrift.transport.TServerTransport.accept(TServerTransport.java:60)

        at org.apache.thrift.server.TThreadPoolServer.serve(TThreadPoolServer.java:161)

        at org.apache.hive.service.cli.thrift.ThriftBinaryCLIService.run(ThriftBinaryCLIService.java:99)

        at java.lang.Thread.run(Thread.java:748)

Caused by: java.net.SocketException: Too many open files (Accept failed)

        at java.net.PlainSocketImpl.socketAccept(Native Method)

        at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)

        at java.net.ServerSocket.implAccept(ServerSocket.java:545)

        at java.net.ServerSocket.accept(ServerSocket.java:513)

        at org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:129)

        ... 5 more

The ulimit is unlimited and hive process is configured to accept 32k open files.  checked here: /proc/<prod_id>/limits

Limit                     Soft Limit           Hard Limit           Units

Max cpu time              unlimited            unlimited            seconds

Max file size             unlimited            unlimited            bytes

Max data size             unlimited            unlimited            bytes

Max stack size            8388608              unlimited            bytes

Max core file size        unlimited            unlimited            bytes

Max resident set          unlimited            unlimited            bytes

Max processes             16000                16000                processes

Max open files            32000                32000                files

Max locked memory         65536                65536                bytes

Max address space         unlimited            unlimited            bytes

Max file locks            unlimited            unlimited            locks

Max pending signals       128260               128260               signals

Max msgqueue size         819200               819200               bytes

Max nice priority         0                    0

Max realtime priority     0                    0

Max realtime timeout      unlimited            unlimited            us

Any ideas on how to track this down?  Maybe its not closing a connection somewhere?

[Greg Hart]

Hi Sonny,

There are command-line programs available for seeing what sockets are open and what program is using them. You would need to ask in the community forums for your Linux distro which command-line programs they support.

You can also ask in the Apache Hive mailing list to see if there are configuration options for avoiding this issue.

If you believe the issue is with Kylo then please try to document how to reproduce the issue and open a JIRA. Sometimes issues like this one are tricky to reproduce so please include as much information as you can.

https://kylo-io.atlassian.net/

[Sonny Heer]

Yeah the service has been restarted and recovered.  I will capture more info when it happens again.  I wanted to just mention here in case someone else knew of an outstanding issue with connections left open.  I'll paste back here some more info when it happens again.  It could be something else, but we have used a process of elimination and it appears to be when kylo/nifi is running.