Middle manager not able to create new Peons

[Vraj Parikh]

Hi,

For our druid cluster the middle manager is not able to handle more than 3 tasks. If any more task is supplied to the middle manager, it fails with following exception - 

java.lang.OutOfMemoryError: unable to create new native thread
    at java.lang.Thread.start0(Native Method) ~[?:1.7.0_79]
    at java.lang.Thread.start(Thread.java:714) ~[?:1.7.0_79]
    at org.eclipse.jetty.util.thread.QueuedThreadPool.startThreads(QueuedThreadPool.java:430) ~[jetty-util-9.2.5.v20141112.jar:9.2.5.v20141112]
    at org.eclipse.jetty.util.thread.QueuedThreadPool.doStart(QueuedThreadPool.java:104) ~[jetty-util-9.2.5.v20141112.jar:9.2.5.v20141112]
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) [jetty-util-9.2.5.v20141112.jar:9.2.5.v20141112]
    at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132) [jetty-util-9.2.5.v20141112.jar:9.2.5.v20141112]
    at org.eclipse.jetty.server.Server.start(Server.java:387) [jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]
    at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114) [jetty-util-9.2.5.v20141112.jar:9.2.5.v20141112]
    at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61) [jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]
    at org.eclipse.jetty.server.Server.doStart(Server.java:354) [jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) [jetty-util-9.2.5.v20141112.jar:9.2.5.v20141112]
    at io.druid.server.initialization.jetty.JettyServerModule$1.start(JettyServerModule.java:204) [druid-server-0.9.2.jar:0.9.2]
    at com.metamx.common.lifecycle.Lifecycle.start(Lifecycle.java:259) [java-util-0.27.10.jar:?]
    at io.druid.guice.LifecycleModule$2.start(LifecycleModule.java:155) [druid-api-0.9.2.jar:0.9.2]
    at io.druid.cli.GuiceRunnable.initLifecycle(GuiceRunnable.java:101) [druid-services-0.9.2.jar:0.9.2]
    at io.druid.cli.CliPeon.run(CliPeon.java:274) [druid-services-0.9.2.jar:0.9.2]
    at io.druid.cli.Main.main(Main.java:106) [druid-services-0.9.2.jar:0.9.2]

Our middle manager's runtime.properties is -

druid.service=druid/middleManager
druid.host=<Host>
druid.port=<Port>


# Number of tasks per middleManager
druid.worker.capacity=4


# Task launch parameters
druid.indexer.runner.javaOpts=-server -Xms2g -Xmx9g -Duser.timezone=UTC -Dfile.encoding=UTF-8 -XX:MaxPermSize=2g


# Task Logging


# Processing threads and buffers
druid.processing.buffer.sizeBytes=536870912
druid.processing.numThreads=2


# Hadoop indexing
druid.indexer.task.hadoopWorkingPath=hdfs://<path>
druid.indexer.task.defaultHadoopCoordinates=["org.apache.hadoop:hadoop-client:2.6.0"]
druid.indexer.fork.property.druid.indexer.task.hadoopWorkingPath=hdfs://<path>

and the jvm.config looks like - 

-server
-Xmx3g
-Xms1g
-XX:+PrintGCDetails
-Duser.timezone=UTC
-Dfile.encoding=UTF-8
-Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager
-Djava.io.tmpdir=/mnt/tmp
-Dlog4j.configurationFile=file:/opt/druid/druid/conf/druid/middleManager/log4j2.xml

Can someone please help us for this? 

Thanks!

[Gian Merlino]

Try checking what your process limit is with "ulimit -u" and increasing it using "ulimit -u <something higher>".

Gian

--
You received this message because you are subscribed to the Google Groups "Druid User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to druid-user+unsubscribe@googlegroups.com.
To post to this group, send email to druid...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/druid-user/1d1cb5d9-b544-40c3-aa50-c58f4e243881%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Vraj Parikh]

Thanks Gian for getting back. So we have the following ulimit  - 

soft   nofile         32768

hard   nofile         32768

How much ulimit should we ideally have?

Gian

To unsubscribe from this group and stop receiving emails from it, send an email to druid-user+...@googlegroups.com.

[Gian Merlino]

Those are open files limits; threads/processes are a different kind of limit.

Gian

To unsubscribe from this group and stop receiving emails from it, send an email to druid-user+unsubscribe@googlegroups.com.

To post to this group, send email to druid...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/druid-user/dc709db7-1907-47f7-ae5e-796e66260d82%40googlegroups.com.

[Vraj Parikh]

Oh sorry. Yeah we have that set to 1024.

Thanks!

Gian