"Failed to create merged beam" exception while using tranquility

[rohit kochar]

I am tranquility 0.8.1 for pushing events to druid from a java application.

This application has been working fine for few days and all of sudden all the events started getting dropped with exception "Failed to create merged beam".

On restart of the application everything returned to normal.

Is there a known cause for this exception?

Here is the full stack trace:

java.lang.IllegalStateException: Failed to create merged beam: druid:overlord/metricDS

at com.metamx.tranquility.beam.ClusteredBeam$$anonfun$sendAll$2$$anonfun$26.apply(ClusteredBeam.scala:383) ~[tranquility-core_2.11-0.8.1.jar:0.8.1]

at com.metamx.tranquility.beam.ClusteredBeam$$anonfun$sendAll$2$$anonfun$26.apply(ClusteredBeam.scala:379) ~[tranquility-core_2.11-0.8.1.jar:0.8.1]

at com.twitter.util.Promise$Transformer.liftedTree1$1(Promise.scala:112) ~[util-core_2.11-6.30.0.jar:6.30.0]

at com.twitter.util.Promise$Transformer.k(Promise.scala:112) ~[util-core_2.11-6.30.0.jar:6.30.0]

at com.twitter.util.Promise$Transformer.apply(Promise.scala:122) ~[util-core_2.11-6.30.0.jar:6.30.0]

at com.twitter.util.Promise$Transformer.apply(Promise.scala:103) ~[util-core_2.11-6.30.0.jar:6.30.0]

at com.twitter.util.Promise$$anon$1.run(Promise.scala:366) ~[util-core_2.11-6.30.0.jar:6.30.0]

at com.twitter.concurrent.LocalScheduler$Activation.run(Scheduler.scala:178) [util-core_2.11-6.30.0.jar:6.30.0]

at com.twitter.concurrent.LocalScheduler$Activation.submit(Scheduler.scala:136) [util-core_2.11-6.30.0.jar:6.30.0]

at com.twitter.concurrent.LocalScheduler.submit(Scheduler.scala:207) [util-core_2.11-6.30.0.jar:6.30.0]

at com.twitter.concurrent.Scheduler$.submit(Scheduler.scala:92) [util-core_2.11-6.30.0.jar:6.30.0]

at com.twitter.util.Promise.runq(Promise.scala:350) [util-core_2.11-6.30.0.jar:6.30.0]

at com.twitter.util.Promise.updateIfEmpty(Promise.scala:721) [util-core_2.11-6.30.0.jar:6.30.0]

at com.twitter.util.ExecutorServiceFuturePool$$anon$2.run(FuturePool.scala:107) [util-core_2.11-6.30.0.jar:6.30.0]

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_25]

at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_25]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_25]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_25]

at java.lang.Thread.run(Thread.java:745) [na:1.8.0_25]

Caused by: java.lang.IllegalStateException: Failed to save new beam for identifier[druid:overlord/metricDS] timestamp[2016-08-16T05:00:00.000Z]

at com.metamx.tranquility.beam.ClusteredBeam$$anonfun$2.applyOrElse(ClusteredBeam.scala:289) ~[tranquility-core_2.11-0.8.1.jar:0.8.1]

at com.metamx.tranquility.beam.ClusteredBeam$$anonfun$2.applyOrElse(ClusteredBeam.scala:286) ~[tranquility-core_2.11-0.8.1.jar:0.8.1]

at com.twitter.util.Future$$anonfun$rescue$1.apply(Future.scala:924) ~[util-core_2.11-6.30.0.jar:6.30.0]

at com.twitter.util.Future$$anonfun$rescue$1.apply(Future.scala:922) ~[util-core_2.11-6.30.0.jar:6.30.0]

... 17 common frames omitted

Caused by: com.fasterxml.jackson.databind.JsonMappingException: 128 (through reference chain: com.fasterxml.jackson.module.scala.deser.MapBuilderWrapper["beams"])

at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:210) ~[jackson-databind-2.6.1.jar:2.6.1]

at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:177) ~[jackson-databind-2.6.1.jar:2.6.1]

at com.fasterxml.jackson.databind.deser.std.ContainerDeserializerBase.wrapAndThrow(ContainerDeserializerBase.java:88) ~[jackson-databind-2.6.1.jar:2.6.1]

at com.fasterxml.jackson.databind.deser.std.MapDeserializer._readAndBindStringMap(MapDeserializer.java:507) ~[jackson-databind-2.6.1.jar:2.6.1]

at com.fasterxml.jackson.databind.deser.std.MapDeserializer.deserialize(MapDeserializer.java:341) ~[jackson-databind-2.6.1.jar:2.6.1]

at com.fasterxml.jackson.module.scala.deser.UnsortedMapDeserializer.deserialize(UnsortedMapDeserializerModule.scala:76) ~[jackson-module-scala_2.11-2.6.0.jar:na]

at com.fasterxml.jackson.module.scala.deser.UnsortedMapDeserializer.deserialize(UnsortedMapDeserializerModule.scala:39) ~[jackson-module-scala_2.11-2.6.0.jar:na]

at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3702) ~[jackson-databind-2.6.1.jar:2.6.1]

at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2807) ~[jackson-databind-2.6.1.jar:2.6.1]

at com.metamx.tranquility.beam.ClusteredBeamMeta$.fromBytes(ClusteredBeam.scala:497) ~[tranquility-core_2.11-0.8.1.jar:0.8.1]

at com.metamx.tranquility.beam.ClusteredBeam$$anon$1$$anonfun$modify$1.apply(ClusteredBeam.scala:156) ~[tranquility-core_2.11-0.8.1.jar:0.8.1]

at com.metamx.tranquility.beam.ClusteredBeam$$anon$1$$anonfun$modify$1.apply(ClusteredBeam.scala:152) ~[tranquility-core_2.11-0.8.1.jar:0.8.1]

at com.twitter.util.Try$.apply(Try.scala:13) ~[util-core_2.11-6.30.0.jar:6.30.0]

... 6 common frames omitted

Caused by: java.lang.ArrayIndexOutOfBoundsException: 128

at com.fasterxml.jackson.core.sym.ByteQuadsCanonicalizer.addName(ByteQuadsCanonicalizer.java:853) ~[jackson-core-2.6.1.jar:2.6.1]

at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.addName(UTF8StreamJsonParser.java:2340) ~[jackson-core-2.6.1.jar:2.6.1]

at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.findName(UTF8StreamJsonParser.java:2224) ~[jackson-core-2.6.1.jar:2.6.1]

at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.parseLongName(UTF8StreamJsonParser.java:1831) ~[jackson-core-2.6.1.jar:2.6.1]

at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.parseMediumName2(UTF8StreamJsonParser.java:1786) ~[jackson-core-2.6.1.jar:2.6.1]

at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.parseMediumName(UTF8StreamJsonParser.java:1743) ~[jackson-core-2.6.1.jar:2.6.1]

at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._parseName(UTF8StreamJsonParser.java:1678) ~[jackson-core-2.6.1.jar:2.6.1]

at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.nextFieldName(UTF8StreamJsonParser.java:1007) ~[jackson-core-2.6.1.jar:2.6.1]

at com.fasterxml.jackson.databind.deser.std.MapDeserializer._readAndBindStringMap(MapDeserializer.java:483) ~[jackson-databind-2.6.1.jar:2.6.1]

at com.fasterxml.jackson.databind.deser.std.MapDeserializer.deserialize(MapDeserializer.java:341) ~[jackson-databind-2.6.1.jar:2.6.1]

at com.fasterxml.jackson.module.scala.deser.UnsortedMapDeserializer.deserialize(UnsortedMapDeserializerModule.scala:76) ~[jackson-module-scala_2.11-2.6.0.jar:na]

at com.fasterxml.jackson.module.scala.deser.UnsortedMapDeserializer.deserialize(UnsortedMapDeserializerModule.scala:39) ~[jackson-module-scala_2.11-2.6.0.jar:na]

at com.fasterxml.jackson.module.scala.deser.UntypedObjectDeserializer.mapObject(UntypedObjectDeserializerModule.scala:44) ~[jackson-module-scala_2.11-2.6.0.jar:na]

at com.fasterxml.jackson.databind.deser.std.UntypedObjectDeserializer.deserialize(UntypedObjectDeserializer.java:222) ~[jackson-databind-2.6.1.jar:2.6.1]

at com.fasterxml.jackson.databind.deser.std.MapDeserializer._readAndBindStringMap(MapDeserializer.java:495) ~[jackson-databind-2.6.1.jar:2.6.1]

[pja...@yahoo-inc.com]

Hello,

Looks like you are hitting this issue -

https://groups.google.com/forum/#!topic/druid-user/3xRvOuXf3Ks
https://github.com/druid-io/tranquility/issues/185

-Parag

[Akshay Goyal]

I faced the same error but my stack trace was different. 

Caused by: java.util.concurrent.ExecutionException: java.lang.NoSuchMethodError: org.jboss.netty.handler.codec.http.HttpMessage.headers()Lorg/jboss/netty/handler/codec/http/HttpHeaders;

    ... 6 more

Caused by: java.lang.NoSuchMethodError: org.jboss.netty.handler.codec.http.HttpMessage.headers()Lorg/jboss/netty/handler/codec/http/HttpHeaders;

    at com.twitter.finagle.http.netty.HttpMessageProxy$class.headers(HttpMessageProxy.scala:20)

    at com.twitter.finagle.http.Message.headers(Message.scala:27)

    at com.twitter.finagle.http.MessageHeaderMap.$plus$eq(HeaderMap.scala:145)

    at com.twitter.finagle.http.MessageHeaderMap.$plus$eq(HeaderMap.scala:123)

    at scala.collection.mutable.MapLike$class.update(MapLike.scala:88)

    at com.twitter.finagle.http.HeaderMap.update(HeaderMap.scala:20)

It seems netty had conflicts between hadoop's and druid's transitive dependencies. Moving to latest hadoop version fixed the issue.