Tranquility and Curator exception

[Mahesh Padmanabhan]

Hello,

I am trying to run a simple Tranquility client to send data to a local mode Druid.

I am getting the following exception and it seems to only happen when a new firehose id is created. Once the id is created, things seems to work ok.

015-02-23 20:02:33,941 [ClusteredBeam-ZkFuturePool-1bf63f50-38ac-4af0-969b-b8215efe974c] INFO  com.metamx.common.scala.net.finagle.DiscoResolver - Updating instances for service[druid:local:firehose:foo-03-0000-0000] to Set()
2015-02-23 20:02:33,942 [ClusteredBeam-ZkFuturePool-1bf63f50-38ac-4af0-969b-b8215efe974c] WARN  finagle - Name resolution is pending
2015-02-23 20:02:33,945 [ClusteredBeam-ZkFuturePool-1bf63f50-38ac-4af0-969b-b8215efe974c] INFO  com.metamx.tranquility.finagle.FinagleRegistry - Created client for service: druid:local:firehose:foo-03-0000-0000
2015-02-23 20:02:39,386 [ServiceCache-0] ERROR org.apache.curator.framework.listen.ListenerContainer - Listener (org.apache.curator.x.discovery.details.ServiceCacheImpl@3a2ef3ba) threw an exception
java.lang.NoSuchMethodError: org.apache.curator.framework.listen.ListenerContainer.forEach(Lcom/google/common/base/Function;)V
        at org.apache.curator.x.discovery.details.ServiceCacheImpl.childEvent(ServiceCacheImpl.java:157)
        at org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:494)
        at org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:488)
        at org.apache.curator.framework.listen.ListenerContainer$1.run(ListenerContainer.java:92)
        at org.spark-project.guava.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:293)
        at org.apache.curator.framework.listen.ListenerContainer.forEach(ListenerContainer.java:83)
        at org.apache.curator.framework.recipes.cache.PathChildrenCache.callListeners(PathChildrenCache.java:485)
        at org.apache.curator.framework.recipes.cache.EventOperation.invoke(EventOperation.java:35)
        at org.apache.curator.framework.recipes.cache.PathChildrenCache$11.run(PathChildrenCache.java:755)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)

In this case, the next run with the same data results in no exception.

Any idea what could be wrong?

Thanks,
Mahesh

[Prajwal Tuladhar]

Seems like curator lib is missing from your class path.

--
You received this message because you are subscribed to the Google Groups "Druid Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to druid-developm...@googlegroups.com.
To post to this group, send email to druid-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/druid-development/fa0ac271-7bf4-44d6-aa08-b42c1000036e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Praj

[Mahesh Padmanabhan]

Thanks for responding. Is that the Curator jar?

That is confusing to me. This exception only appears when a new firehose id is being created. I think by default it is the start of every hour. Once that id is created, subsequent runs of the same simple client have no exceptions. If the Curator jar is missing, won't the exception appear with each run?

Thanks,
Mahesh

To unsubscribe from this group and stop receiving emails from it, send an email to druid-development+unsubscribe@googlegroups.com.
To post to this group, send email to druid-development@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/druid-development/fa0ac271-7bf4-44d6-aa08-b42c1000036e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Praj

[Gian Merlino]

That's really strange. Curator's ListenerContainer hasn't changed in a long time so it doesn't seem like a jar version mismatch. Also, org.apache.curator.framework.listen.ListenerContainer.forEach (the method that triggered the NoSuchMethodError) is actually present earlier in the stack trace, so it's strange that it couldn't be found later on.

Is this error reliably reproducible for you? Is it possible something is messing with classes or jars on your classpath while your program is running? Or maybe there's some bytecode manipulation going awry? The relocated "org.spark-project.guava.common.util.concurrent.MoreExecutors" is possibly an indicator of that.

[Mahesh Padmanabhan]

Thanks for responding Gian.

Yes, the error is reliable reproducible and it happens every time a new firehose id is created.

Good catch on the changed guava package path. I am using Tranquility within a Spark 1.2.1 app and it seems to be picking up the "shaded" Guava jar (14.0.1) file at runtime.

Any idea what I can try? I am using sbt and checked that it is indeed pulling in the right Curator Guava (version 16.0.1) during assembly creation.

Thanks,
Mahesh

[Gian Merlino]

It seems like something in the build process is relocating Guava and also messing with Curator in order to use the relocated Guava. But the job doesn't appear to be complete, since the NoSuchMethodError was for a version of a Curator method that expected to receive a non-relocated Guava Function. It's possible the relocations are not being done properly and something needs to change in the relocation config.

Some other general thoughts that may or may not make sense: you might be able to get everything on the same versions of Curator and Guava and skip the relocations, or maybe relocate tranquility's Curator and Guava so they don't conflict with Spark's, or maybe get Spark to load your classes with an isolated classloader.

[Prajwal Tuladhar]

You can also try to exclude Guava while including Spark dependency and use Gauva as local dependency. In maven, local dependency always gets used but don't know how sbt handles it.

To unsubscribe from this group and stop receiving emails from it, send an email to druid-developm...@googlegroups.com.
To post to this group, send email to druid-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/druid-development/d50ec283-eca0-41b8-b82d-2e3b940ab766%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

-- Praj

[Mahesh Padmanabhan]

Hi Gian,

This is what I tried and I am running into the same issue:

1) Since I use sbt, I added the following:

  "io.druid" % "druid-indexing-service" % "0.6.164" exclude("org.slf4j", "slf4j-log4j12") exclude("log4j", "log4j") exclude("org.hyperic", "sigar") exclude("net.java.dev.jets3t", "jets3t") exclude("org.apache.curator", "curator-x-discovery") exclude("org.apache.curator", "curator-framework") exclude("org.apache.curator", "curator-client") exclude("org.apache.curator", "curator-recipes"),
  "com.metamx" % "tranquility_2.10" % "0.3.2" exclude("net.java.dev.jets3t", "jets3t") exclude("org.apache.curator", "curator-x-discovery") exclude("org.apache.curator", "curator-framework") exclude("org.apache.curator", "curator-client") exclude("org.apache.curator", "curator-recipes"),

2) I then downloaded curator and shaded guava 16.0.1 within each of the jars that I excluded above

3) Created a lib directory and added those jar to it

I verified that Guava is present in all those curator jars that I created so I am puzzled as to why it is not working. I also verified that my app assembly contains my Curator jars.

Do you have any additional suggestions?

Thanks,
Mahesh

[David Westbrook]

Was there any resolution here?  I'm not getting the exact same issue, but I have been having trouble with NoClassDef errors from curator and am wondering if it is part of the build process.  Here is the stack trace I'm seeing:

SEVERE: Fatal exception propagated to the root monitor!
java.lang.NoSuchMethodError: org.apache.curator.framework.CuratorFramework.clearWatcherReferences(Lorg/apache/zookeeper/Watcher;)V
	at org.apache.curator.framework.recipes.cache.PathChildrenCache.close(PathChildrenCache.java:373)
	at com.google.common.io.Closeables.close(Closeables.java:80)
	at org.apache.curator.utils.CloseableUtils.closeQuietly(CloseableUtils.java:61)
	at org.apache.curator.x.discovery.details.ServiceCacheImpl.close(ServiceCacheImpl.java:107)
	at com.metamx.common.scala.net.curator.Disco$$anon$2.stop(Disco.scala:67)
	at com.metamx.common.lifecycle.Lifecycle.stop(Lifecycle.java:267)
	at com.metamx.common.scala.net.finagle.DiscoResolver$$anonfun$bind$1$$anon$2$$anonfun$close$1.apply$mcV$sp(DiscoResolver.scala:53)
	at com.metamx.common.scala.net.finagle.DiscoResolver$$anonfun$bind$1$$anon$2$$anonfun$close$1.apply(DiscoResolver.scala:51)
	at com.metamx.common.scala.net.finagle.DiscoResolver$$anonfun$bind$1$$anon$2$$anonfun$close$1.apply(DiscoResolver.scala:51)
	at com.twitter.util.Try$.apply(Try.scala:13)
	at com.twitter.util.Future$.apply(Future.scala:84)
	at com.metamx.common.scala.net.finagle.DiscoResolver$$anonfun$bind$1$$anon$2.close(DiscoResolver.scala:51)
	at com.twitter.util.Var$$anon$4$$anonfun$3.apply(Var.scala:314)
	at com.twitter.util.Var$$anon$4$$anonfun$3.apply(Var.scala:304)
	at com.twitter.util.Closable$$anon$5.close(Closable.scala:67)
	at com.twitter.util.Closable$$anon$3.com$twitter$util$Closable$$anon$$closeSeq(Closable.scala:54)
	at com.twitter.util.Closable$$anon$3$$anonfun$com$twitter$util$Closable$$anon$$closeSeq$1.apply(Closable.scala:54)
	at com.twitter.util.Closable$$anon$3$$anonfun$com$twitter$util$Closable$$anon$$closeSeq$1.apply(Closable.scala:54)
	at com.twitter.util.Future$$anonfun$flatMap$1.apply(Future.scala:791)
	at com.twitter.util.Future$$anonfun$flatMap$1.apply(Future.scala:790)
	at com.twitter.util.ConstFuture$$anonfun$transform$1.apply(Future.scala:1084)
	at com.twitter.util.ConstFuture$$anonfun$transform$1.apply(Future.scala:1083)
	at com.twitter.util.ConstFuture$$anon$5$$anonfun$run$1.apply$mcV$sp(Future.scala:1073)
	at com.twitter.util.Monitor$.apply(Monitor.scala:128)
	at com.twitter.util.ConstFuture$$anon$5.run(Future.scala:1073)
	at com.twitter.concurrent.LocalScheduler$Activation.run(Scheduler.scala:186)
	at com.twitter.concurrent.LocalScheduler$Activation.submit(Scheduler.scala:157)
	at com.twitter.concurrent.LocalScheduler.submit(Scheduler.scala:212)
	at com.twitter.concurrent.Scheduler$.submit(Scheduler.scala:86)
	at com.twitter.util.ConstFuture.respond(Future.scala:1069)
	at com.twitter.util.ConstFuture.transform(Future.scala:1083)
	at com.twitter.util.Future.flatMap(Future.scala:790)
	at com.twitter.util.Closable$$anon$3.com$twitter$util$Closable$$anon$$closeSeq(Closable.scala:54)
	at com.twitter.util.Closable$$anon$3.close(Closable.scala:57)
	at com.twitter.finagle.loadbalancer.LoadBalancerFactory$$anon$2$$anon$1.close(LoadBalancerFactory.scala:154)
	at com.twitter.finagle.ProxyServiceFactory$class.close(Service.scala:180)
	at com.twitter.finagle.ServiceFactoryProxy.close(Service.scala:189)
	at com.twitter.finagle.ProxyServiceFactory$class.close(Service.scala:180)
	at com.twitter.finagle.ServiceFactoryProxy.close(Service.scala:189)
	at com.twitter.finagle.ProxyServiceFactory$class.close(Service.scala:180)
	at com.twitter.finagle.ServiceFactoryProxy.close(Service.scala:189)
	at com.twitter.finagle.Filter$$anon$3.close(Filter.scala:85)
	at com.twitter.finagle.ProxyServiceFactory$class.close(Service.scala:180)
	at com.twitter.finagle.builder.ClientBuilder$$anon$2.close(ClientBuilder.scala:782)
	at com.twitter.finagle.FactoryToService.close(Service.scala:204)
	at com.twitter.finagle.Filter$$anon$2.close(Filter.scala:73)
	at com.twitter.finagle.ServiceProxy.close(Service.scala:104)
	at com.twitter.finagle.builder.ClientBuilder$$anon$3.close(ClientBuilder.scala:823)
	at com.twitter.finagle.ServiceProxy.close(Service.scala:104)
	at com.metamx.tranquility.finagle.FinagleRegistry$$anon$1.close(FinagleRegistry.scala:68)
	at com.metamx.tranquility.finagle.SharedService.close(SharedService.scala:54)
	at com.metamx.tranquility.finagle.FinagleRegistry$$anon$2.close(FinagleRegistry.scala:100)
	at com.twitter.util.Closable$class.close(Closable.scala:17)
	at com.twitter.finagle.Service.close(Service.scala:33)
	at com.metamx.tranquility.druid.DruidBeam$$anonfun$8.apply(DruidBeam.scala:197)
	at com.metamx.tranquility.druid.DruidBeam$$anonfun$8.apply(DruidBeam.scala:197)
	at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
	at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
	at scala.collection.immutable.List.foreach(List.scala:318)
	at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
	at scala.collection.AbstractTraversable.map(Traversable.scala:105)
	at com.metamx.tranquility.druid.DruidBeam.close(DruidBeam.scala:197)
	at com.metamx.tranquility.beam.ClusteredBeam$$anonfun$10$$anonfun$14$$anonfun$16$$anonfun$apply$3$$anonfun$apply$4.apply(ClusteredBeam.scala:243)
	at com.metamx.tranquility.beam.ClusteredBeam$$anonfun$10$$anonfun$14$$anonfun$16$$anonfun$apply$3$$anonfun$apply$4.apply(ClusteredBeam.scala:243)
	at com.metamx.common.scala.Predef$FinallyOps$$anonfun$withFinally$1.apply(Predef.scala:56)
	at com.metamx.common.scala.Predef$FinallyOps$$anonfun$withFinally$1.apply(Predef.scala:56)
	at com.metamx.tranquility.beam.ClusteredBeam$$anonfun$10$$anonfun$14$$anonfun$16$$anonfun$apply$3.apply(ClusteredBeam.scala:243)
	at com.metamx.tranquility.beam.ClusteredBeam$$anonfun$10$$anonfun$14$$anonfun$16$$anonfun$apply$3.apply(ClusteredBeam.scala:243)
	at scala.collection.mutable.MapLike$class.getOrElseUpdate(MapLike.scala:189)
	at scala.collection.mutable.AbstractMap.getOrElseUpdate(Map.scala:91)
	at com.metamx.tranquility.beam.ClusteredBeam$$anonfun$10$$anonfun$14$$anonfun$16.apply(ClusteredBeam.scala:240)
	at com.metamx.tranquility.beam.ClusteredBeam$$anonfun$10$$anonfun$14$$anonfun$16.apply(ClusteredBeam.scala:239)
	at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
	at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
	at scala.collection.immutable.Range.foreach(Range.scala:141)
	at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
	at scala.collection.AbstractTraversable.map(Traversable.scala:105)
	at com.metamx.tranquility.beam.ClusteredBeam$$anonfun$10$$anonfun$14.apply(ClusteredBeam.scala:238)
	at com.metamx.tranquility.beam.ClusteredBeam$$anonfun$10$$anonfun$14.apply(ClusteredBeam.scala:229)
	at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
	at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
	at scala.collection.immutable.List.foreach(List.scala:318)
	at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
	at scala.collection.AbstractTraversable.map(Traversable.scala:105)
	at com.metamx.tranquility.beam.ClusteredBeam$$anonfun$10.apply(ClusteredBeam.scala:229)
	at com.metamx.tranquility.beam.ClusteredBeam$$anonfun$10.apply(ClusteredBeam.scala:194)
	at com.metamx.tranquility.beam.ClusteredBeam$$anon$1$$anonfun$modify$1.apply(ClusteredBeam.scala:152)
	at com.metamx.tranquility.beam.ClusteredBeam$$anon$1$$anonfun$modify$1.apply(ClusteredBeam.scala:141)
	at com.twitter.util.Try$.apply(Try.scala:13)
	at com.twitter.util.ExecutorServiceFuturePool$$anon$2.run(FuturePool.scala:111)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
	at java.util.concurrent.FutureTask.run(FutureTask.java:166)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:722)

[Mahesh Padmanabhan]

I forgot to post the resolution to my original error. My bad.

Mine was more Apache Spark and Tranquility not playing nice with respect to Apache Curator so the workaround was to relocate the Curator package namespace within the Spark assembly POM. That fixed the issue for me.

[Gian Merlino]

Mahesh, that's great to know!

David, that error sounds like you have mismatched versions of curator-framework and curator-recipes. Can you check that your runtime versions of those two things match?

[David Westbrook]

Shading Curator did the trick.  I'm surprised it hasn't caused issues until now.  Updating to Tranquility 0.3.3 did seem to Trigger the issue, so anyone using Storm should shade that jar if they aren't already.