Tranquility to fail new realtime segment creation after every one other segmentgranularity period

Fatih Mermerkaya

unread,

Aug 27, 2016, 5:27:17 AM8/27/16

to Druid User

Hi,

I have setup single server for PoC to test druid, I am using default configurations of quicksetup of imply.io. Tranquility kafka setup is done. It consume from kafka topic and writes to druid. segmentgranularity is "hour". It works ever one hour, and fails to create realtime index job for the next hour then it success to create realtime index job again. As summary it work one hour then fails to create index task then stop, then works in next hour.

What i am missing in settings?

Below is the tranquility kafka log during creation of new realtime index task. ( # of record per hour is ~1.2m)

"exceptionMessage":"Tasks are all gone: index_realtime_igwkafka_2016-08-27T09:00:00.000Z_0_0"}}]

2016-08-27 09:02:36,193 [Hashed wheel timer #1] WARN c.m.tranquility.druid.TaskClient - Emitting alert: [anomaly] Loss of Druid redundancy: igwkafka

{

"dataSource" : "igwkafka",

"task" : "index_realtime_igwkafka_2016-08-27T09:00:00.000Z_0_0",

"status" : "TaskFailed"

}

2016-08-27 09:02:36,194 [Hashed wheel timer #1] INFO c.metamx.emitter.core.LoggingEmitter - Event [{"feed":"alerts","timestamp":"2016-08-27T09:02:36.194Z","service":"tranquility","host":"localhost","severity":"anomaly","description":"Loss of Druid redundancy: igwkafka","data":{"dataSource":"igwkafka","task":"index_realtime_igwkafka_2016-08-27T09:00:00.000Z_0_0","status":"TaskFailed"}}]

2016-08-27 09:02:36,196 [Hashed wheel timer #1] WARN c.m.tranquility.beam.ClusteredBeam - Emitting alert: [anomaly] Beam defunct: druid:overlord/igwkafka

{

"eventCount" : 1,

"timestamp" : "2016-08-27T09:00:00.000Z",

"beam" : "MergingPartitioningBeam(DruidBeam(interval = 2016-08-27T09:00:00.000Z/2016-08-27T10:00:00.000Z, partition = 0, tasks = [index_realtime_igwkafka_2016-08-27T09:00:00.000Z_0_0/igwkafka-009-0000-0000]))"

}

com.metamx.tranquility.beam.DefunctBeamException: Tasks are all gone: index_realtime_igwkafka_2016-08-27T09:00:00.000Z_0_0

at com.metamx.tranquility.druid.DruidBeam$$anonfun$sendAll$2$$anonfun$6$$anonfun$apply$6.apply(DruidBeam.scala:115) ~[io.druid.tranquility-core-0.8.2.jar:0.8.2]

at scala.Option.getOrElse(Option.scala:121) ~[org.scala-lang.scala-library-2.11.7.jar:na]

at com.metamx.tranquility.druid.DruidBeam$$anonfun$sendAll$2$$anonfun$6.apply(DruidBeam.scala:112) ~[io.druid.tranquility-core-0.8.2.jar:0.8.2]

at com.metamx.tranquility.druid.DruidBeam$$anonfun$sendAll$2$$anonfun$6.apply(DruidBeam.scala:109) ~[io.dru

Nishant Bangarwa

unread,

Aug 29, 2016, 3:51:26 AM8/29/16

to Druid User

First make sure you have enough capacity on your middlemanagers to run the tasks for consecutive hours parallely.

i.e number of middlemanagers * druid.worker.capacity > 2 * (partitions + replication factor )

also, when you see DefunctBeamExceptions, check overlord console if any of the tasks FAILED, task logs would have more info on the root cause behind the failure.

Cheers,

Nishant

--
You received this message because you are subscribed to the Google Groups "Druid User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to druid-user+...@googlegroups.com.
To post to this group, send email to druid...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/druid-user/7e34e21d-c492-48cc-b48c-cfc6a541e52f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Fatih Mermerkaya

unread,

Aug 29, 2016, 4:55:56 AM8/29/16

to Druid User

I saw that there is lack of memory error in middlemanager peons once creating realtime index task.

I increased the settings and reduce the segment to 15min but still it works for one it fails for others.

At the end, I give up on tranquility kafka

I used kafka indexing service that works without problem with default settings that means middlemanager settings was correct.

Reply all

Reply to author

Forward