tranquility goes out of memory

[Rishi Mishra]

Tranquility consuming from kafka goes out of memory for high volume topic. How do I increase the memory for tranquility?

2016-04-12 14:17:56,311 [ConsumerFetcherThread-tranquility-consumer_c14-og2a-cm-druid-overlord-1.srv.media.net-1460468796515-ec381872-0-2] ERROR k.network.BoundedByteBufferReceive - OOME with size 25166359

java.lang.OutOfMemoryError: Java heap space

        at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57) ~[na:1.8.0_77]

        at java.nio.ByteBuffer.allocate(ByteBuffer.java:335) ~[na:1.8.0_77]

        at kafka.network.BoundedByteBufferReceive.byteBufferAllocate(BoundedByteBufferReceive.scala:80) [org.apache.kafka.kafka_2.11-0.8.2.2.jar:na]

        at kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:63) [org.apache.kafka.kafka_2.11-0.8.2.2.jar:na]

        at kafka.network.Receive$class.readCompletely(Transmission.scala:56) [org.apache.kafka.kafka_2.11-0.8.2.2.jar:na]

        at kafka.network.BoundedByteBufferReceive.readCompletely(BoundedByteBufferReceive.scala:29) [org.apache.kafka.kafka_2.11-0.8.2.2.jar:na]

        at kafka.network.BlockingChannel.receive(BlockingChannel.scala:111) [org.apache.kafka.kafka_2.11-0.8.2.2.jar:na]

        at kafka.consumer.SimpleConsumer.liftedTree1$1(SimpleConsumer.scala:71) [org.apache.kafka.kafka_2.11-0.8.2.2.jar:na]

        at kafka.consumer.SimpleConsumer.kafka$consumer$SimpleConsumer$$sendRequest(SimpleConsumer.scala:68) [org.apache.kafka.kafka_2.11-0.8.2.2.jar:na]

        at kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(SimpleConsumer.scala:112) [org.apache.kafka.kafka_2.11-0.8.2.2.jar:na]

        at kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(SimpleConsumer.scala:112) [org.apache.kafka.kafka_2.11-0.8.2.2.jar:na]

        at kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(SimpleConsumer.scala:112) [org.apache.kafka.kafka_2.11-0.8.2.2.jar:na]

        at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33) [org.apache.kafka.kafka_2.11-0.8.2.2.jar:na]

        at kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply$mcV$sp(SimpleConsumer.scala:111) [org.apache.kafka.kafka_2.11-0.8.2.2.jar:na]

        at kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(SimpleConsumer.scala:111) [org.apache.kafka.kafka_2.11-0.8.2.2.jar:na]

        at kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(SimpleConsumer.scala:111) [org.apache.kafka.kafka_2.11-0.8.2.2.jar:na]

        at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33) [org.apache.kafka.kafka_2.11-0.8.2.2.jar:na]

        at kafka.consumer.SimpleConsumer.fetch(SimpleConsumer.scala:110) [org.apache.kafka.kafka_2.11-0.8.2.2.jar:na]

        at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:94) [org.apache.kafka.kafka_2.11-0.8.2.2.jar:na]

        at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:86) [org.apache.kafka.kafka_2.11-0.8.2.2.jar:na]

        at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:60) [org.apache.kafka.kafka_2.11-0.8.2.2.jar:na]

2016-04-12 14:17:56,311 [ConsumerFetcherThread-tranquility-consumer_c14-og2a-cm-druid-overlord-1.srv.media.net-1460468796515-ec381872-0-2] INFO  kafka.consumer.SimpleConsumer - Reconnect due to socket error: jav

a.lang.OutOfMemoryError: Java heap space

[Gian Merlino]

Hey Rishi,

You should be able to specify a heap size on the command line like this,

  bin/tranquility -J-Xmx1g kafka

(or any other Xmx you like)

Gian

--
You received this message because you are subscribed to the Google Groups "Druid User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to druid-user+...@googlegroups.com.
To post to this group, send email to druid...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/druid-user/d5d8f813-0397-404e-9fc4-c15cc1409920%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Rishi Mishra]

Thanks Gian

--
You received this message because you are subscribed to a topic in the Google Groups "Druid User" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/druid-user/5DNrJ2_1_Ew/unsubscribe.
To unsubscribe from this group and all its topics, send an email to druid-user+...@googlegroups.com.

To post to this group, send email to druid...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/druid-user/CACZNdYCYkfjkFMm%2B9x%2B5kKNJJYh21XbBKrL9cJn-30wqG91V4g%40mail.gmail.com.

[Kenji Noguchi]

I'm using the Imply suite.  Tranquility is started up by the supervise command.
The command line option for the JVM that I put in the conf/supervise/data.conf seems to be ignored.

The bin/tranquility loads strings from dist/tranquility/conf/application.ini 

Any ideas?

thanks,

Kenji