Tranquility and cardinality metrics

已查看 95 次
跳至第一个未读帖子

Darrin Thompson

未读,
2017年2月2日 08:52:072017/2/2
收件人 Druid User
I tried to add a cardinality metric to a tranquility server and got a NullPointerException at startup.

After some spelunking I found that tranquility (even on master) is built against older versions of druid which have slightly different ideas about how a metricSpec of type cardinality should look.

(Note: I'm finding this somewhat concerning as I would think that since cardinality is so prominently featured in the druid docs and tranquility is the most common way to do real-time ingest into druid, I shouldn't be discovering this.)

My goal at the moment is to generate indexing and tranquility ingest configurations from a set of common sources, and partition these sources into their separate ops and dev portions. I can work around tranquility's and druid's version mismatch by including both fieldNames, and fields in the metricSpec.

So a couple of questions:

Am I just the first to run across an annoying bug?

Am I using tranquility in a very different way from the rest of the community for no good reason?

Did I make a brown paper bag error? (I'm open to this.)

--
Darrin

Darrin Thompson

未读,
2017年2月2日 16:58:302017/2/2
收件人 Druid User
I found that I was running tranquility-0.8.0 and upgraded to 0.8.2, but I still get the NullPointerException. Here is more info:

The offending metricSpec:

        {
          "name" : "thing",
          "type" : "cardinality",
          "fields" : ["guild1", "guid2", "id3"],
          "byRow": false
        },

I can work around this by providing "fieldNames" with fields. But the docs don't indicate that; I think I had to read code to figure it out: http://druid.io/docs/0.9.2/querying/aggregations.html And I probably would have had problems later if I tried to use the same metricSpec in a batch task.

Tranquility 0.8.2 ships with druid-processing 0.9.1. If I'm reading the code right the change was introduced into 0.9.2 here:


Rolling a new release of tranquility with druid-0.9.2 would solve this in theory.

The exception:

2017-02-02 21:39:49,276 [main] INFO  io.druid.guice.JsonConfigurator - Loaded class[class io.druid.guice.ExtensionsConfig] from props[druid.extensions.] as [ExtensionsConfig{searchCurrentClassloader=true, directory='extensions', hadoopDependenciesDir='hadoop-dependencies', hadoopContainerDruidClasspath='null', loadList=null}]
java.lang.NullPointerException
        at java.util.AbstractCollection.addAll(AbstractCollection.java:341)
        at io.druid.segment.indexing.DataSchema.getParser(DataSchema.java:105)
        at com.metamx.tranquility.druid.DruidBeams$.fromConfigInternal(DruidBeams.scala:301)
        at com.metamx.tranquility.druid.DruidBeams$.fromConfig(DruidBeams.scala:204)
        at com.metamx.tranquility.server.http.ServerMain$$anonfun$2.apply(ServerMain.scala:118)
        at com.metamx.tranquility.server.http.ServerMain$$anonfun$2.apply(ServerMain.scala:98)
        at com.metamx.common.scala.collection.package$MapLikeOps$$anonfun$strictMapValues$1.apply(package.scala:143)
        at com.metamx.common.scala.collection.package$MapLikeOps$$anonfun$strictMapValues$1.apply(package.scala:143)
        at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
        at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
        at scala.collection.immutable.Map$Map1.foreach(Map.scala:116)
        at scala.collection.TraversableLike$class.map(TraversableLike.scala:245)
        at scala.collection.AbstractTraversable.map(Traversable.scala:104)
        at com.metamx.common.scala.collection.package$MapLikeOps.strictMapValues(package.scala:143)
        at com.metamx.tranquility.server.http.ServerMain$.createServlet(ServerMain.scala:98)
        at com.metamx.tranquility.server.http.ServerMain$.main(ServerMain.scala:77)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at com.twitter.app.App$$anonfun$nonExitingMain$3.apply(App.scala:168)
        at com.twitter.app.App$$anonfun$nonExitingMain$3.apply(App.scala:167)
        at scala.Option.foreach(Option.scala:257)
        at com.twitter.app.App$class.nonExitingMain(App.scala:167)
        at com.metamx.tranquility.server.http.ServerMain$.nonExitingMain(ServerMain.scala:49)
        at com.twitter.app.App$class.main(App.scala:133)
        at com.metamx.tranquility.server.http.ServerMain$.main(ServerMain.scala:49)
        at com.metamx.tranquility.distribution.DistributionMain$.main(DistributionMain.scala:34)
        at com.metamx.tranquility.distribution.DistributionMain.main(DistributionMain.scala)
Exception thrown in main on startup

Gian Merlino

未读,
2017年2月2日 17:39:452017/2/2
收件人 druid...@googlegroups.com
Hey Darrin,

You're right, "fields" is newer, introduced in Druid 0.9.2. Tranquility is built against the Druid 0.9.1 API so it expects "fieldNames" rather than "fields" for the cardinality aggregator. The 0.9.1 docs are available here: http://druid.io/docs/0.9.1/querying/aggregations.html and I'd suggest referring to those when writing Tranquility specs.

We'll probably skip Druid 0.9.2 for Tranquility, since Druid 0.10.0 is coming soon and we are focused on making sure Tranquility works with that API.

Gian

--
You received this message because you are subscribed to the Google Groups "Druid User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to druid-user+unsubscribe@googlegroups.com.
To post to this group, send email to druid...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/druid-user/dc6c926d-2755-4acb-95ee-b516dcd1d104%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Arpan Khagram

未读,
2017年2月27日 06:58:012017/2/27
收件人 Druid User
Hi Gian, Do you know when 0.10 Release is planned to be release ? Any tentative dates decided for the release ?

Regards,
Arpan Khagram

Gian

To unsubscribe from this group and stop receiving emails from it, send an email to druid-user+...@googlegroups.com.

To post to this group, send email to druid...@googlegroups.com.
回复全部
回复作者
转发
0 个新帖子