Problem in ingestion with Tranquility

291 views
Skip to first unread message

liuyanc...@gmail.com

unread,
Feb 24, 2016, 3:36:06 AM2/24/16
to Druid User
I try to ingest data with Tranquility from Kafka. But work can't start,below is my json  configuration  and log for Tranquility.

{

   "dataSources" : [

      {

         "spec" : {

            "dataSchema" : {

               "parser" : {

                  "type" : "string",

                  "parseSpec" : {

                     "timestampSpec" : {

                        "format" : "auto",

                        "column" : "timestamp"

                     },

                     "dimensionsSpec" : {

                        "spatialDimensions" : [

                           { }

                        ],

                        "dimensions" : [

                           "page","lag","lagtype"

                        ]

                     },

                     "format" : "json"

                  }

               },

               "dataSource" : "api-tranquility",

               "granularitySpec" : {

                  "segmentGranularity" : "hour",

                  "type" : "uniform",

                  "queryGranularity" : "none"

               },

               "metricsSpec" : [

                  {

                     "type" : "count",

                     "name" : "count"

                  }

               ]

            },

            "tuningConfig" : {

               "maxRowsInMemory" : "100000",

               "type" : "realtime",

               "windowPeriod" : "PT10M",

               "intermediatePersistPeriod" : "PT10M"

            }

         },

         "properties" : {

            "topicPattern" : "api-druid",

            "topicPattern.priority" : "1",

         }

      }

   ],

   "properties" : {

       "zookeeper.connect" : "10.10.10.10:2181",

       "zookeeper.timeout" : "PT15S",

       "druid.selectors.indexing.serviceName" : "overlord",

       "druid.discovery.curator.path" : "/data/druid/realtime/tranquility",

       "kafka.zookeeper.connect" : "10.10.10.11:2181",

       "kafka.group.id" : "tranquility-api",

       "consumer.numThreads" : "15",

       "commit.periodMillis" : "500",

       "reportDropsAsExceptions" : "false"

    }

}


below is log:

2016-02-24 08:28:20,625 [main] INFO  o.s.c.ConfigurationObjectFactory - Assigning value [2] for [consumer.numThreads] on [com.metamx.tranquility.kafka.model.PropertiesBasedKafkaConfig#getConsumerNumThreads()]

2016-02-24 08:28:20,636 [main] INFO  o.s.c.ConfigurationObjectFactory - Assigning default value [(?!)] for [topicPattern] on [com.metamx.tranquility.kafka.model.PropertiesBasedKafkaConfig#getTopicPattern()]

2016-02-24 08:28:20,637 [main] INFO  o.s.c.ConfigurationObjectFactory - Assigning value [tranquility-kafka] for [kafka.group.id] on [com.metamx.tranquility.kafka.model.PropertiesBasedKafkaConfig#getKafkaGroupId()]

2016-02-24 08:28:20,637 [main] INFO  o.s.c.ConfigurationObjectFactory - Assigning value [localhost:2181] for [kafka.zookeeper.connect] on [com.metamx.tranquility.kafka.model.PropertiesBasedKafkaConfig#getKafkaZookeeperConnect()]

2016-02-24 08:28:20,637 [main] INFO  o.s.c.ConfigurationObjectFactory - Assigning default value [false] for [useTopicAsDataSource] on [com.metamx.tranquility.kafka.model.PropertiesBasedKafkaConfig#useTopicAsDataSource()]

2016-02-24 08:28:20,638 [main] INFO  o.s.c.ConfigurationObjectFactory - Assigning default value [1] for [topicPattern.priority] on [com.metamx.tranquility.kafka.model.PropertiesBasedKafkaConfig#getTopicPatternPriority()]

2016-02-24 08:28:20,638 [main] INFO  o.s.c.ConfigurationObjectFactory - Assigning value [15000] for [commit.periodMillis] on [com.metamx.tranquility.kafka.model.PropertiesBasedKafkaConfig#getCommitPeriodMillis()]

2016-02-24 08:28:20,639 [main] INFO  o.s.c.ConfigurationObjectFactory - Assigning value [false] for [reportDropsAsExceptions] on [com.metamx.tranquility.kafka.model.PropertiesBasedKafkaConfig#reportDropsAsExceptions()]

2016-02-24 08:28:20,639 [main] INFO  o.s.c.ConfigurationObjectFactory - Assigning value [PT20S] for [zookeeper.timeout] on [com.metamx.tranquility.config.PropertiesBasedConfig#zookeeperTimeout()]

2016-02-24 08:28:20,726 [main] INFO  o.s.c.ConfigurationObjectFactory - Assigning value [druid/overlord] for [druid.selectors.indexing.serviceName] on [com.metamx.tranquility.config.PropertiesBasedConfig#druidIndexingServiceName()]

2016-02-24 08:28:20,726 [main] INFO  o.s.c.ConfigurationObjectFactory - Using method itself for [task.partitions] on [com.metamx.tranquility.config.PropertiesBasedConfig#taskPartitions()]

2016-02-24 08:28:20,727 [main] INFO  o.s.c.ConfigurationObjectFactory - Using method itself for [task.replicants] on [com.metamx.tranquility.config.PropertiesBasedConfig#taskReplicants()]

2016-02-24 08:28:20,728 [main] INFO  o.s.c.ConfigurationObjectFactory - Using method itself for [task.warmingPeriod] on [com.metamx.tranquility.config.PropertiesBasedConfig#taskWarmingPeriod()]

2016-02-24 08:28:20,728 [main] INFO  o.s.c.ConfigurationObjectFactory - Assigning value [localhost:2181] for [zookeeper.connect] on [com.metamx.tranquility.config.PropertiesBasedConfig#zookeeperConnect()]

2016-02-24 08:28:20,728 [main] INFO  o.s.c.ConfigurationObjectFactory - Using method itself for [tranquility.maxBatchSize] on [com.metamx.tranquility.config.PropertiesBasedConfig#tranquilityMaxBatchSize()]

2016-02-24 08:28:20,732 [main] INFO  o.s.c.ConfigurationObjectFactory - Using method itself for [tranquility.maxPendingBatches] on [com.metamx.tranquility.config.PropertiesBasedConfig#tranquilityMaxPendingBatches()]

2016-02-24 08:28:20,733 [main] INFO  o.s.c.ConfigurationObjectFactory - Using method itself for [tranquility.lingerMillis] on [com.metamx.tranquility.config.PropertiesBasedConfig#tranquilityLingerMillis()]

2016-02-24 08:28:20,733 [main] INFO  o.s.c.ConfigurationObjectFactory - Assigning value [/druid/discovery] for [druid.discovery.curator.path] on [com.metamx.tranquility.config.PropertiesBasedConfig#discoPath()]

2016-02-24 08:28:20,733 [main] INFO  o.s.c.ConfigurationObjectFactory - Using method itself for [druidBeam.firehoseGracePeriod] on [com.metamx.tranquility.config.PropertiesBasedConfig#firehoseGracePeriod()]

2016-02-24 08:28:20,733 [main] INFO  o.s.c.ConfigurationObjectFactory - Using method itself for [druidBeam.firehoseQuietPeriod] on [com.metamx.tranquility.config.PropertiesBasedConfig#firehoseQuietPeriod()]

2016-02-24 08:28:20,734 [main] INFO  o.s.c.ConfigurationObjectFactory - Using method itself for [druidBeam.firehoseRetryPeriod] on [com.metamx.tranquility.config.PropertiesBasedConfig#firehoseRetryPeriod()]

2016-02-24 08:28:20,734 [main] INFO  o.s.c.ConfigurationObjectFactory - Using method itself for [druidBeam.firehoseChunkSize] on [com.metamx.tranquility.config.PropertiesBasedConfig#firehoseChunkSize()]

2016-02-24 08:28:20,734 [main] INFO  o.s.c.ConfigurationObjectFactory - Using method itself for [druidBeam.randomizeTaskId] on [com.metamx.tranquility.config.PropertiesBasedConfig#randomizeTaskId()]

2016-02-24 08:28:20,735 [main] INFO  o.s.c.ConfigurationObjectFactory - Using method itself for [druidBeam.indexRetryPeriod] on [com.metamx.tranquility.config.PropertiesBasedConfig#indexRetryPeriod()]

2016-02-24 08:28:20,735 [main] INFO  o.s.c.ConfigurationObjectFactory - Using method itself for [druidBeam.firehoseBufferSize] on [com.metamx.tranquility.config.PropertiesBasedConfig#firehoseBufferSize()]

2016-02-24 08:28:20,925 [main] INFO  o.s.c.ConfigurationObjectFactory - Assigning value [2] for [consumer.numThreads] on [com.metamx.tranquility.kafka.model.PropertiesBasedKafkaConfig#getConsumerNumThreads()]

2016-02-24 08:28:20,926 [main] INFO  o.s.c.ConfigurationObjectFactory - Assigning value [twitter] for [topicPattern] on [com.metamx.tranquility.kafka.model.PropertiesBasedKafkaConfig#getTopicPattern()]

2016-02-24 08:28:20,926 [main] INFO  o.s.c.ConfigurationObjectFactory - Assigning value [tranquility-kafka] for [kafka.group.id] on [com.metamx.tranquility.kafka.model.PropertiesBasedKafkaConfig#getKafkaGroupId()]

2016-02-24 08:28:20,926 [main] INFO  o.s.c.ConfigurationObjectFactory - Assigning value [localhost:2181] for [kafka.zookeeper.connect] on [com.metamx.tranquility.kafka.model.PropertiesBasedKafkaConfig#getKafkaZookeeperConnect()]

2016-02-24 08:28:20,927 [main] INFO  o.s.c.ConfigurationObjectFactory - Assigning default value [false] for [useTopicAsDataSource] on [com.metamx.tranquility.kafka.model.PropertiesBasedKafkaConfig#useTopicAsDataSource()]

2016-02-24 08:28:20,927 [main] INFO  o.s.c.ConfigurationObjectFactory - Assigning value [1] for [topicPattern.priority] on [com.metamx.tranquility.kafka.model.PropertiesBasedKafkaConfig#getTopicPatternPriority()]

2016-02-24 08:28:20,928 [main] INFO  o.s.c.ConfigurationObjectFactory - Assigning value [15000] for [commit.periodMillis] on [com.metamx.tranquility.kafka.model.PropertiesBasedKafkaConfig#getCommitPeriodMillis()]

2016-02-24 08:28:20,928 [main] INFO  o.s.c.ConfigurationObjectFactory - Assigning value [false] for [reportDropsAsExceptions] on [com.metamx.tranquility.kafka.model.PropertiesBasedKafkaConfig#reportDropsAsExceptions()]

2016-02-24 08:28:20,928 [main] INFO  o.s.c.ConfigurationObjectFactory - Assigning value [PT20S] for [zookeeper.timeout] on [com.metamx.tranquility.config.PropertiesBasedConfig#zookeeperTimeout()]

2016-02-24 08:28:20,929 [main] INFO  o.s.c.ConfigurationObjectFactory - Assigning value [druid/overlord] for [druid.selectors.indexing.serviceName] on [com.metamx.tranquility.config.PropertiesBasedConfig#druidIndexingServiceName()]

2016-02-24 08:28:20,929 [main] INFO  o.s.c.ConfigurationObjectFactory - Using method itself for [task.partitions] on [com.metamx.tranquility.config.PropertiesBasedConfig#taskPartitions()]

2016-02-24 08:28:20,929 [main] INFO  o.s.c.ConfigurationObjectFactory - Using method itself for [task.replicants] on [com.metamx.tranquility.config.PropertiesBasedConfig#taskReplicants()]

2016-02-24 08:28:20,930 [main] INFO  o.s.c.ConfigurationObjectFactory - Using method itself for [task.warmingPeriod] on [com.metamx.tranquility.config.PropertiesBasedConfig#taskWarmingPeriod()]

2016-02-24 08:28:20,930 [main] INFO  o.s.c.ConfigurationObjectFactory - Assigning value [localhost:2181] for [zookeeper.connect] on [com.metamx.tranquility.config.PropertiesBasedConfig#zookeeperConnect()]

2016-02-24 08:28:20,930 [main] INFO  o.s.c.ConfigurationObjectFactory - Using method itself for [tranquility.maxBatchSize] on [com.metamx.tranquility.config.PropertiesBasedConfig#tranquilityMaxBatchSize()]

2016-02-24 08:28:20,930 [main] INFO  o.s.c.ConfigurationObjectFactory - Using method itself for [tranquility.maxPendingBatches] on [com.metamx.tranquility.config.PropertiesBasedConfig#tranquilityMaxPendingBatches()]

2016-02-24 08:28:20,931 [main] INFO  o.s.c.ConfigurationObjectFactory - Using method itself for [tranquility.lingerMillis] on [com.metamx.tranquility.config.PropertiesBasedConfig#tranquilityLingerMillis()]

2016-02-24 08:28:20,931 [main] INFO  o.s.c.ConfigurationObjectFactory - Assigning value [/druid/discovery] for [druid.discovery.curator.path] on [com.metamx.tranquility.config.PropertiesBasedConfig#discoPath()]

2016-02-24 08:28:20,931 [main] INFO  o.s.c.ConfigurationObjectFactory - Using method itself for [druidBeam.firehoseGracePeriod] on [com.metamx.tranquility.config.PropertiesBasedConfig#firehoseGracePeriod()]

2016-02-24 08:28:20,932 [main] INFO  o.s.c.ConfigurationObjectFactory - Using method itself for [druidBeam.firehoseQuietPeriod] on [com.metamx.tranquility.config.PropertiesBasedConfig#firehoseQuietPeriod()]

2016-02-24 08:28:20,932 [main] INFO  o.s.c.ConfigurationObjectFactory - Using method itself for [druidBeam.firehoseRetryPeriod] on [com.metamx.tranquility.config.PropertiesBasedConfig#firehoseRetryPeriod()]

2016-02-24 08:28:20,932 [main] INFO  o.s.c.ConfigurationObjectFactory - Using method itself for [druidBeam.firehoseChunkSize] on [com.metamx.tranquility.config.PropertiesBasedConfig#firehoseChunkSize()]

2016-02-24 08:28:20,932 [main] INFO  o.s.c.ConfigurationObjectFactory - Using method itself for [druidBeam.randomizeTaskId] on [com.metamx.tranquility.config.PropertiesBasedConfig#randomizeTaskId()]

2016-02-24 08:28:20,933 [main] INFO  o.s.c.ConfigurationObjectFactory - Using method itself for [druidBeam.indexRetryPeriod] on [com.metamx.tranquility.config.PropertiesBasedConfig#indexRetryPeriod()]

2016-02-24 08:28:20,933 [main] INFO  o.s.c.ConfigurationObjectFactory - Using method itself for [druidBeam.firehoseBufferSize] on [com.metamx.tranquility.config.PropertiesBasedConfig#firehoseBufferSize()]

2016-02-24 08:28:20,937 [main] INFO  o.s.c.ConfigurationObjectFactory - Assigning value [2] for [consumer.numThreads] on [com.metamx.tranquility.kafka.model.PropertiesBasedKafkaConfig#getConsumerNumThreads()]

2016-02-24 08:28:20,938 [main] INFO  o.s.c.ConfigurationObjectFactory - Assigning value [wikipedia.*] for [topicPattern] on [com.metamx.tranquility.kafka.model.PropertiesBasedKafkaConfig#getTopicPattern()]

2016-02-24 08:28:20,939 [main] INFO  o.s.c.ConfigurationObjectFactory - Assigning value [tranquility-kafka] for [kafka.group.id] on [com.metamx.tranquility.kafka.model.PropertiesBasedKafkaConfig#getKafkaGroupId()]

2016-02-24 08:28:20,939 [main] INFO  o.s.c.ConfigurationObjectFactory - Assigning value [localhost:2181] for [kafka.zookeeper.connect] on [com.metamx.tranquility.kafka.model.PropertiesBasedKafkaConfig#getKafkaZookeeperConnect()]

2016-02-24 08:28:20,939 [main] INFO  o.s.c.ConfigurationObjectFactory - Assigning default value [false] for [useTopicAsDataSource] on [com.metamx.tranquility.kafka.model.PropertiesBasedKafkaConfig#useTopicAsDataSource()]

2016-02-24 08:28:20,940 [main] INFO  o.s.c.ConfigurationObjectFactory - Assigning value [1] for [topicPattern.priority] on [com.metamx.tranquility.kafka.model.PropertiesBasedKafkaConfig#getTopicPatternPriority()]

2016-02-24 08:28:20,940 [main] INFO  o.s.c.ConfigurationObjectFactory - Assigning value [15000] for [commit.periodMillis] on [com.metamx.tranquility.kafka.model.PropertiesBasedKafkaConfig#getCommitPeriodMillis()]

2016-02-24 08:28:20,941 [main] INFO  o.s.c.ConfigurationObjectFactory - Assigning value [false] for [reportDropsAsExceptions] on [com.metamx.tranquility.kafka.model.PropertiesBasedKafkaConfig#reportDropsAsExceptions()]

2016-02-24 08:28:20,941 [main] INFO  o.s.c.ConfigurationObjectFactory - Assigning value [PT20S] for [zookeeper.timeout] on [com.metamx.tranquility.config.PropertiesBasedConfig#zookeeperTimeout()]

2016-02-24 08:28:20,941 [main] INFO  o.s.c.ConfigurationObjectFactory - Assigning value [druid/overlord] for [druid.selectors.indexing.serviceName] on [com.metamx.tranquility.config.PropertiesBasedConfig#druidIndexingServiceName()]

2016-02-24 08:28:20,942 [main] INFO  o.s.c.ConfigurationObjectFactory - Assigning value [2] for [task.partitions] on [com.metamx.tranquility.config.PropertiesBasedConfig#taskPartitions()]

2016-02-24 08:28:20,942 [main] INFO  o.s.c.ConfigurationObjectFactory - Assigning value [2] for [task.replicants] on [com.metamx.tranquility.config.PropertiesBasedConfig#taskReplicants()]

2016-02-24 08:28:20,942 [main] INFO  o.s.c.ConfigurationObjectFactory - Using method itself for [task.warmingPeriod] on [com.metamx.tranquility.config.PropertiesBasedConfig#taskWarmingPeriod()]

2016-02-24 08:28:20,942 [main] INFO  o.s.c.ConfigurationObjectFactory - Assigning value [localhost:2181] for [zookeeper.connect] on [com.metamx.tranquility.config.PropertiesBasedConfig#zookeeperConnect()]

2016-02-24 08:28:20,943 [main] INFO  o.s.c.ConfigurationObjectFactory - Using method itself for [tranquility.maxBatchSize] on [com.metamx.tranquility.config.PropertiesBasedConfig#tranquilityMaxBatchSize()]

2016-02-24 08:28:20,943 [main] INFO  o.s.c.ConfigurationObjectFactory - Using method itself for [tranquility.maxPendingBatches] on [com.metamx.tranquility.config.PropertiesBasedConfig#tranquilityMaxPendingBatches()]

2016-02-24 08:28:20,943 [main] INFO  o.s.c.ConfigurationObjectFactory - Using method itself for [tranquility.lingerMillis] on [com.metamx.tranquility.config.PropertiesBasedConfig#tranquilityLingerMillis()]

2016-02-24 08:28:20,943 [main] INFO  o.s.c.ConfigurationObjectFactory - Assigning value [/druid/discovery] for [druid.discovery.curator.path] on [com.metamx.tranquility.config.PropertiesBasedConfig#discoPath()]

2016-02-24 08:28:20,944 [main] INFO  o.s.c.ConfigurationObjectFactory - Using method itself for [druidBeam.firehoseGracePeriod] on [com.metamx.tranquility.config.PropertiesBasedConfig#firehoseGracePeriod()]

2016-02-24 08:28:20,944 [main] INFO  o.s.c.ConfigurationObjectFactory - Using method itself for [druidBeam.firehoseQuietPeriod] on [com.metamx.tranquility.config.PropertiesBasedConfig#firehoseQuietPeriod()]

2016-02-24 08:28:20,945 [main] INFO  o.s.c.ConfigurationObjectFactory - Using method itself for [druidBeam.firehoseRetryPeriod] on [com.metamx.tranquility.config.PropertiesBasedConfig#firehoseRetryPeriod()]

2016-02-24 08:28:20,945 [main] INFO  o.s.c.ConfigurationObjectFactory - Using method itself for [druidBeam.firehoseChunkSize] on [com.metamx.tranquility.config.PropertiesBasedConfig#firehoseChunkSize()]

2016-02-24 08:28:20,946 [main] INFO  o.s.c.ConfigurationObjectFactory - Using method itself for [druidBeam.randomizeTaskId] on [com.metamx.tranquility.config.PropertiesBasedConfig#randomizeTaskId()]

2016-02-24 08:28:20,946 [main] INFO  o.s.c.ConfigurationObjectFactory - Using method itself for [druidBeam.indexRetryPeriod] on [com.metamx.tranquility.config.PropertiesBasedConfig#indexRetryPeriod()]

2016-02-24 08:28:20,946 [main] INFO  o.s.c.ConfigurationObjectFactory - Using method itself for [druidBeam.firehoseBufferSize] on [com.metamx.tranquility.config.PropertiesBasedConfig#firehoseBufferSize()]

2016-02-24 08:28:20,967 [main] WARN  o.a.c.retry.ExponentialBackoffRetry - maxRetries too large (500). Pinning to 29

Exception in thread "main" java.lang.NoSuchMethodError: java.util.List.sort(Ljava/util/Comparator;)V

at com.metamx.tranquility.kafka.writer.WriterController.<init>(WriterController.java:58)

at com.metamx.tranquility.kafka.KafkaMain.run(KafkaMain.java:122)

at com.metamx.tranquility.kafka.KafkaMain.main(KafkaMain.java:71)

at com.metamx.tranquility.distribution.DistributionMain$.main(DistributionMain.scala:37)

at com.metamx.tranquility.distribution.DistributionMain.main(DistributionMain.scala)


It is the same problem with example test.

Venkat Reddy Amalla

unread,
Feb 24, 2016, 7:23:58 AM2/24/16
to Druid User
Hi,

I resolved the same exception by upgrading java7 to java-8. 
...

Gian Merlino

unread,
Feb 24, 2016, 8:52:15 PM2/24/16
to druid...@googlegroups.com
Sorry, yes, this was a mistake in the original release of tranquility-kafka. It was intended to support java 7 too. The next release will support both java 7 and 8 (master already does). Sorry for the inconvenience.

Gian

--
You received this message because you are subscribed to the Google Groups "Druid User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to druid-user+...@googlegroups.com.
To post to this group, send email to druid...@googlegroups.com.
 To view this discussion on the web visit https://groups.google.com/d/msgid/druid-user/d8cd7d33-b87f-48d5-a52f-9c025386b55c%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

luo...@conew.com

unread,
Feb 24, 2016, 9:29:35 PM2/24/16
to Druid User
Hi, Gian:

There are some exception when spark streaming ingest data with tranquility. 

在 2016年2月25日星期四 UTC+8上午9:52:15，Gian Merlino写道：
...

Gian Merlino

unread,
Feb 24, 2016, 9:32:45 PM2/24/16
to druid...@googlegroups.com
yep, was just commenting there.

Gian

--
You received this message because you are subscribed to the Google Groups "Druid User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to druid-user+...@googlegroups.com.
To post to this group, send email to druid...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/druid-user/8c4eda9f-2753-4395-be3d-2ce9971d3ca6%40googlegroups.com.
Reply all
Reply to author
Forward
0 new messages