thetaSketch error

Manohar Rao

unread,

Jan 29, 2016, 2:05:23 AM1/29/16

to Druid User

Hi

I m trying thetaSketch and running into some error

{"version":"0.8.3","modules":[{"name":"io.druid.examples.ExamplesDruidModule","artifact":"druid-examples","version":"0.8.3"},{"name":"io.druid.firehose.kafka.KafkaEightDruidModule","artifact":"druid-kafka-eight","version":"0.8.3"}],"memory":{"maxMemory":5726797824,"totalMemory":442499072,"freeMemory":381018376,"usedMemory":61480696}}

My index spec is attached

Error :

<html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 500 </title>
</head>
<body>
<h2>HTTP ERROR: 500</h2>
<p>Problem accessing /druid/indexer/v1/task. Reason:
<pre>    javax.servlet.ServletException: com.fasterxml.jackson.databind.JsonMappingException: Could not resolve type id 'thetaSketch' into a subtype of [simple type, class io.druid.query.aggregation.AggregatorFactory]
 at [Source: HttpInputOverHTTP@b11c130; line: 1, column: 652] (through reference chain: Object[][3])</pre></p>
<hr /><i><small>Powered by Jetty://</small></i>
</body>
</html>

What could be wrong?

Thanks and Regards

Manohar

thetatest.json

Fangjin

unread,

Jan 29, 2016, 2:21:08 AM1/29/16

to Druid User

You need to include the theta sketches extension.

--
You received this message because you are subscribed to the Google Groups "Druid User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to druid-user+...@googlegroups.com.
To post to this group, send email to druid...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/druid-user/437e6c5e-f2f9-4d7c-9dec-3677b82f6255%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Manohar Rao

unread,

Jan 29, 2016, 2:33:29 AM1/29/16

to Druid User

Thanks ,

Sorry , but do you have any link on how to add the extension? .I am pretty new to druid.

Thanks and Regards

Manohar

mcap...@kochava.com

unread,

Jan 29, 2016, 2:57:34 AM1/29/16

to Druid User

It goes in the druid.extensions.coordinates of your common.runtime.properties config for all your druid nodes. Should look something like this (give or take the other extensions):

druid.extensions.coordinates=["io.druid.extensions:druid-datasketches","io.druid.extensions:druid-kafka-eight","io.druid.extensions:mysql-metadata-storage"]

Michael

Manohar Rao

unread,

Jan 29, 2016, 4:00:04 AM1/29/16

to Druid User

Great !

Thanks

Lovenish Goyal

unread,

Jun 30, 2016, 10:25:37 AM6/30/16

to Druid User

Hi Fangjin,

We are also facing same issue while using thetaSkech.

We have sketches extensions added in common property file.

druid.extensions.loadList=["druid-datasketches","druid-kafka-indexing-service"]

All druid services are loading this extension while getting up but when we start tranquility and ingest event it gets down with below error. Seems like tranquility is not getting this extension.

2016-06-30 12:09:42,767 [KafkaConsumer-0] INFO io.druid.guice.JsonConfigurator - Loaded class[class io.druid.guice.ExtensionsConfig] from props[druid.extensions.] as [ExtensionsConfig{searchCurrentClassloader=true, directory='extensions', hadoopDependenciesDir='hadoop-dependencies', loadList=null}]
2016-06-30 12:09:42,910 [KafkaConsumer-0] ERROR c.m.tranquility.kafka.KafkaConsumer - Exception:
java.lang.IllegalArgumentException: Could not resolve type id 'thetaSketch' into a subtype of [simple type, class io.druid.query.aggregation.AggregatorFactory]
at [Source: N/A; line: -1, column: -1] (through reference chain: Object[][1])
at com.fasterxml.jackson.databind.ObjectMapper._convert(ObjectMapper.java:2774) ~[com.fasterxml.jackson.core.jackson-databind-2.4.6.jar:2.4.6]
at com.fasterxml.jackson.databind.ObjectMapper.convertValue(ObjectMapper.java:2700) ~[com.fasterxml.jackson.core.jackson-databind-2.4.6.jar:2.4.6]
at com.metamx.tranquility.druid.DruidBeams$.makeFireDepartment(DruidBeams.scala:406) ~[io.druid.tranquility-core-0.8.0.jar:0.8.0]
at com.metamx.tranquility.druid.DruidBeams$.fromConfigInternal(DruidBeams.scala:291) ~[io.druid.tranquility-core-0.8.0.jar:0.8.0]
at com.metamx.tranquility.druid.DruidBeams$.fromConfig(DruidBeams.scala:199) ~[io.druid.tranquility-core-0.8.0.jar:0.8.0]
at com.metamx.tranquility.kafka.KafkaBeamUtils$.createTranquilizer(KafkaBeamUtils.scala:40) ~[io.druid.tranquility-kafka-0.8.0.jar:0.8.0]
at com.metamx.tranquility.kafka.KafkaBeamUtils.createTranquilizer(KafkaBeamUtils.scala) ~[io.druid.tranquility-kafka-0.8.0.jar:0.8.0]
at com.metamx.tranquility.kafka.writer.TranquilityEventWriter.<init>(TranquilityEventWriter.java:64) ~[io.druid.tranquility-kafka-0.8.0.jar:0.8.0]
at com.metamx.tranquility.kafka.writer.WriterController.createWriter(WriterController.java:171) ~[io.druid.tranquility-kafka-0.8.0.jar:0.8.0]
at com.metamx.tranquility.kafka.writer.WriterController.getWriter(WriterController.java:98) ~[io.druid.tranquility-kafka-0.8.0.jar:0.8.0]
at com.metamx.tranquility.kafka.KafkaConsumer$2.run(KafkaConsumer.java:231) ~[io.druid.tranquility-kafka-0.8.0.jar:0.8.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_65]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_65]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_65]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_65]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65]

Is there anything which we need to configure for tranquility as well.

Here is the historical cluster log which shows that extension is added in common config file.

2016-06-30T11:25:44,248 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.guice.ExtensionsConfig] from props[druid.extensions.] as [ExtensionsConfig{searchCurrentClassloader=true, directory='extensions', hadoopDependenciesDir='hadoop-dependencies', hadoopContainerDruidClasspath='null', loadList=[druid-datasketches, druid-kafka-indexing-service]}]

Waiting for your response.

Thanks,

mcap...@kochava.com

unread,

Jun 30, 2016, 10:46:41 AM6/30/16

to Druid User

Hi try adding -Ddruid.extensions.loadList='["druid-datasketches"]' -Ddruid.extensions.directory=/druid/path/extensions to your tranquility JVM properties. https://github.com/druid-io/tranquility/blob/master/docs/configuration.md

Lovenish Goyal

unread,

Jun 30, 2016, 11:00:14 AM6/30/16

to Druid User

Thanks Mcap, Its woking

Shantanu Deshmukh

unread,

Nov 9, 2016, 5:22:42 AM11/9/16

to Druid User

I tried the same. It's not working for me. Where exactly did you add these options? I added them in bin/tranquility shell file. Added those arguments like

addJava "-Dlogback.configurationFile=${app_home}/../conf/logback.xml -Ddruid.extensions.loadList='[\"druid-datasketches\"]' -Ddruid.extensions.directory=dist/druid/extensions"

I am still getting the same error

Lovenish Goyal

unread,

Nov 9, 2016, 5:33:40 AM11/9/16

to Druid User

Shantanu,

Add druid.extensions.loadList=["druid-datasketches","druid-kafka-indexing-service"] in your common property file and pass same argument while running tranquility.

Eg:

bin/tranquility kafka -configFile conf/kafka.json -Ddruid.extensions.loadList='["druid-datasketches"]' -Ddruid.extensions.directory=/opt/druid/extensions

druid-datasketches jar should be in your extension directory path.

Shantanu Deshmukh

unread,

Nov 9, 2016, 8:56:25 AM11/9/16

to Druid User

Hey,

Thanks a lot. That helped! :-)

ravi bhatta

unread,

Sep 8, 2017, 11:31:47 AM9/8/17

to Druid User

Hi , I am trying to include a community extension "druid-distinctcount". I have downoaded the extension by using pull-deps and included it in common.runtime.properties.

But i am getting error ;

{"error":"Could not resolve type id 'distinctCount' into a subtype of [simple type, class io.druid.query.aggregation.AggregatorFactory]\n at [Source: HttpInputOverHTTP@30582c42[c=5289,q=1,[0]=EOF,s=STREAM]; line: 1, column: 4882] (through reference chain: Object[][3])"}

I have included druid.extensions.loadList=["druid-distinctcount"] in common.runtime.properties. I have also restarted all druid nodes after this.

Plz help me solve this issue.

Thank You

Ravi Bhatta

Reply all

Reply to author

Forward