Lot of IOException: "Too many files open"

[Mainak Ghosh]

Hello,

I am testing the groupby code in an older version of Druid 0.9.0. I have data using the following schema 

      'timestamp': long(time.time() * 1000),

      'name': 'query/time',

      'host': '192.168.' + str(random.randrange(1, 254)) + '.' + str(random.randrange(1, 254)),

      'page': str(int(max(1, random.gauss(5, 4)))) + '.html',

      'value': max(0, int(random.gauss(200, 80)))

I am running the following groupby query: (datasource, granularity="all", intervals=<variable>, dimensions="host", aggregations={"count": doublesum("count")}). I do not have filters and any other thing.

When I check the broker log, I see a lot of these:

broker-0.log:java.io.IOException: Too many open files

broker-0.log-   at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) ~[?:?]

broker-0.log-   at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)

~[?:?]

broker-0.log-   at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:377) ~[

jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]

broker-0.log-   at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.ja

va:500) [jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]

broker-0.log-   at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:

620) [jetty-util-9.2.5.v20141112.jar:9.2.5.v20141112]

broker-0.log-   at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:5

40) [jetty-util-9.2.5.v20141112.jar:9.2.5.v20141112]

broker-0.log-   at java.lang.Thread.run(Thread.java:745) [?:?]

broker-0.log-2017-05-05 02:51:49,349 WARN o.e.j.s.ServerConnector [qtp996440505-159-acceptor-1@

4d336fdc-ServerConnector@44e30523{HTTP/1.1}{0.0.0.0:8082}]

broker-0.log:java.io.IOException: Too many open files

broker-0.log-   at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) ~[?:?]

broker-0.log-   at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)

~[?:?]

broker-0.log-   at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:377) ~[

jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]

broker-0.log-   at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.ja

va:500) [jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]

broker-0.log-   at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:

620) [jetty-util-9.2.5.v20141112.jar:9.2.5.v20141112]

broker-0.log-   at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:5

40) [jetty-util-9.2.5.v20141112.jar:9.2.5.v20141112]

broker-0.log-   at java.lang.Thread.run(Thread.java:745) [?:?]

I saw a JIRA entry https://github.com/druid-io/druid/issues/3568. It does not speak of the reason why one faces this. Any help will be highly appreciated.

Thanks and Regards,

Mainak

[Gian Merlino]

I would start by doing two things.

- use lsof to see what the open files are, and if they look reasonable or not. If not it might be a misconfiguration or leak of some kind.

- increase the open files limit using ulimit -n or /etc/security/limits.conf

Gian

--
You received this message because you are subscribed to the Google Groups "Druid Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to druid-development+unsubscribe@googlegroups.com.
To post to this group, send email to druid-development@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/druid-development/309da126-b2e7-4e25-a863-191b9a8bf78c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.