Tranquility and cardinality metrics

[Darrin Thompson]

I tried to add a cardinality metric to a tranquility server and got a NullPointerException at startup.

After some spelunking I found that tranquility (even on master) is built against older versions of druid which have slightly different ideas about how a metricSpec of type cardinality should look.

(Note: I'm finding this somewhat concerning as I would think that since cardinality is so prominently featured in the druid docs and tranquility is the most common way to do real-time ingest into druid, I shouldn't be discovering this.)

My goal at the moment is to generate indexing and tranquility ingest configurations from a set of common sources, and partition these sources into their separate ops and dev portions. I can work around tranquility's and druid's version mismatch by including both fieldNames, and fields in the metricSpec.

So a couple of questions:

Am I just the first to run across an annoying bug?

Am I using tranquility in a very different way from the rest of the community for no good reason?

Did I make a brown paper bag error? (I'm open to this.)

--

Darrin

[Darrin Thompson]

I found that I was running tranquility-0.8.0 and upgraded to 0.8.2, but I still get the NullPointerException. Here is more info:

The offending metricSpec:

        {

          "name" : "thing",

          "type" : "cardinality",

          "fields" : ["guild1", "guid2", "id3"],

          "byRow": false

        },

I can work around this by providing "fieldNames" with fields. But the docs don't indicate that; I think I had to read code to figure it out: http://druid.io/docs/0.9.2/querying/aggregations.html And I probably would have had problems later if I tried to use the same metricSpec in a batch task.

Tranquility 0.8.2 ships with druid-processing 0.9.1. If I'm reading the code right the change was introduced into 0.9.2 here:

https://github.com/druid-io/druid/commit/4e91330a179d6140d54bb58b922f90f907797619#diff-33d523f826a29726a0051e50f5eb25d6R118

Rolling a new release of tranquility with druid-0.9.2 would solve this in theory.

The exception:

2017-02-02 21:39:49,276 [main] INFO  io.druid.guice.JsonConfigurator - Loaded class[class io.druid.guice.ExtensionsConfig] from props[druid.extensions.] as [ExtensionsConfig{searchCurrentClassloader=true, directory='extensions', hadoopDependenciesDir='hadoop-dependencies', hadoopContainerDruidClasspath='null', loadList=null}]

java.lang.NullPointerException

        at java.util.AbstractCollection.addAll(AbstractCollection.java:341)

        at io.druid.segment.indexing.DataSchema.getParser(DataSchema.java:105)

        at com.metamx.tranquility.druid.DruidBeams$.fromConfigInternal(DruidBeams.scala:301)

        at com.metamx.tranquility.druid.DruidBeams$.fromConfig(DruidBeams.scala:204)

        at com.metamx.tranquility.server.http.ServerMain$$anonfun$2.apply(ServerMain.scala:118)

        at com.metamx.tranquility.server.http.ServerMain$$anonfun$2.apply(ServerMain.scala:98)

        at com.metamx.common.scala.collection.package$MapLikeOps$$anonfun$strictMapValues$1.apply(package.scala:143)

        at com.metamx.common.scala.collection.package$MapLikeOps$$anonfun$strictMapValues$1.apply(package.scala:143)

        at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)

        at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)

        at scala.collection.immutable.Map$Map1.foreach(Map.scala:116)

        at scala.collection.TraversableLike$class.map(TraversableLike.scala:245)

        at scala.collection.AbstractTraversable.map(Traversable.scala:104)

        at com.metamx.common.scala.collection.package$MapLikeOps.strictMapValues(package.scala:143)

        at com.metamx.tranquility.server.http.ServerMain$.createServlet(ServerMain.scala:98)

        at com.metamx.tranquility.server.http.ServerMain$.main(ServerMain.scala:77)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:606)

        at com.twitter.app.App$$anonfun$nonExitingMain$3.apply(App.scala:168)

        at com.twitter.app.App$$anonfun$nonExitingMain$3.apply(App.scala:167)

        at scala.Option.foreach(Option.scala:257)

        at com.twitter.app.App$class.nonExitingMain(App.scala:167)

        at com.metamx.tranquility.server.http.ServerMain$.nonExitingMain(ServerMain.scala:49)

        at com.twitter.app.App$class.main(App.scala:133)

        at com.metamx.tranquility.server.http.ServerMain$.main(ServerMain.scala:49)

        at com.metamx.tranquility.distribution.DistributionMain$.main(DistributionMain.scala:34)

        at com.metamx.tranquility.distribution.DistributionMain.main(DistributionMain.scala)

Exception thrown in main on startup

[Gian Merlino]

Hey Darrin,

You're right, "fields" is newer, introduced in Druid 0.9.2. Tranquility is built against the Druid 0.9.1 API so it expects "fieldNames" rather than "fields" for the cardinality aggregator. The 0.9.1 docs are available here: http://druid.io/docs/0.9.1/querying/aggregations.html and I'd suggest referring to those when writing Tranquility specs.

We'll probably skip Druid 0.9.2 for Tranquility, since Druid 0.10.0 is coming soon and we are focused on making sure Tranquility works with that API.

Gian

--
You received this message because you are subscribed to the Google Groups "Druid User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to druid-user+unsubscribe@googlegroups.com.
To post to this group, send email to druid...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/druid-user/dc6c926d-2755-4acb-95ee-b516dcd1d104%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Arpan Khagram]

Hi Gian, Do you know when 0.10 Release is planned to be release ? Any tentative dates decided for the release ?

Regards,
Arpan Khagram

+91 8308993200

Gian

To unsubscribe from this group and stop receiving emails from it, send an email to druid-user+...@googlegroups.com.

To post to this group, send email to druid...@googlegroups.com.