Problem in ingestion with Tranquility

已查看 291 次
跳至第一个未读帖子

liuyanc...@gmail.com

未读，
2016年2月24日 03:36:062016/2/24
收件人 Druid User
I try to ingest data with Tranquility from Kafka. But work can't start,below is my json  configuration  and log for Tranquility.

{

   "dataSources" : [

      {

         "spec" : {

            "dataSchema" : {

               "parser" : {

                  "type" : "string",

                  "parseSpec" : {

                     "timestampSpec" : {

                        "format" : "auto",

                        "column" : "timestamp"

                     },

                     "dimensionsSpec" : {

                        "spatialDimensions" : [

                           { }

                        ],

                        "dimensions" : [

                           "page","lag","lagtype"

                        ]

                     },

                     "format" : "json"

                  }

               },

               "dataSource" : "api-tranquility",

               "granularitySpec" : {

                  "segmentGranularity" : "hour",

                  "type" : "uniform",

                  "queryGranularity" : "none"

               },

               "metricsSpec" : [

                  {

                     "type" : "count",

                     "name" : "count"

                  }

               ]

            },

            "tuningConfig" : {

               "maxRowsInMemory" : "100000",

               "type" : "realtime",

               "windowPeriod" : "PT10M",

               "intermediatePersistPeriod" : "PT10M"

            }

         },

         "properties" : {

            "topicPattern" : "api-druid",

            "topicPattern.priority" : "1",

         }

      }

   ],

   "properties" : {

       "zookeeper.connect" : "10.10.10.10:2181",

       "zookeeper.timeout" : "PT15S",

       "druid.selectors.indexing.serviceName" : "overlord",

       "druid.discovery.curator.path" : "/data/druid/realtime/tranquility",

       "kafka.zookeeper.connect" : "10.10.10.11:2181",

       "kafka.group.id" : "tranquility-api",

       "consumer.numThreads" : "15",

       "commit.periodMillis" : "500",

       "reportDropsAsExceptions" : "false"

    }

}


below is log:

2016-02-24 08:28:20,625 [main] INFO  o.s.c.ConfigurationObjectFactory - Assigning value [2] for [consumer.numThreads] on [com.metamx.tranquility.kafka.model.PropertiesBasedKafkaConfig#getConsumerNumThreads()]

2016-02-24 08:28:20,636 [main] INFO  o.s.c.ConfigurationObjectFactory - Assigning default value [(?!)] for [topicPattern] on [com.metamx.tranquility.kafka.model.PropertiesBasedKafkaConfig#getTopicPattern()]

2016-02-24 08:28:20,637 [main] INFO  o.s.c.ConfigurationObjectFactory - Assigning value [tranquility-kafka] for [kafka.group.id] on [com.metamx.tranquility.kafka.model.PropertiesBasedKafkaConfig#getKafkaGroupId()]

2016-02-24 08:28:20,637 [main] INFO  o.s.c.ConfigurationObjectFactory - Assigning value [localhost:2181] for [kafka.zookeeper.connect] on [com.metamx.tranquility.kafka.model.PropertiesBasedKafkaConfig#getKafkaZookeeperConnect()]

2016-02-24 08:28:20,637 [main] INFO  o.s.c.ConfigurationObjectFactory - Assigning default value [false] for [useTopicAsDataSource] on [com.metamx.tranquility.kafka.model.PropertiesBasedKafkaConfig#useTopicAsDataSource()]

2016-02-24 08:28:20,638 [main] INFO  o.s.c.ConfigurationObjectFactory - Assigning default value [1] for [topicPattern.priority] on [com.metamx.tranquility.kafka.model.PropertiesBasedKafkaConfig#getTopicPatternPriority()]

2016-02-24 08:28:20,638 [main] INFO  o.s.c.ConfigurationObjectFactory - Assigning value [15000] for [commit.periodMillis] on [com.metamx.tranquility.kafka.model.PropertiesBasedKafkaConfig#getCommitPeriodMillis()]

2016-02-24 08:28:20,639 [main] INFO  o.s.c.ConfigurationObjectFactory - Assigning value [false] for [reportDropsAsExceptions] on [com.metamx.tranquility.kafka.model.PropertiesBasedKafkaConfig#reportDropsAsExceptions()]

2016-02-24 08:28:20,639 [main] INFO  o.s.c.ConfigurationObjectFactory - Assigning value [PT20S] for [zookeeper.timeout] on [com.metamx.tranquility.config.PropertiesBasedConfig#zookeeperTimeout()]

2016-02-24 08:28:20,726 [main] INFO  o.s.c.ConfigurationObjectFactory - Assigning value [druid/overlord] for [druid.selectors.indexing.serviceName] on [com.metamx.tranquility.config.PropertiesBasedConfig#druidIndexingServiceName()]

2016-02-24 08:28:20,726 [main] INFO  o.s.c.ConfigurationObjectFactory - Using method itself for [task.partitions] on [com.metamx.tranquility.config.PropertiesBasedConfig#taskPartitions()]

2016-02-24 08:28:20,727 [main] INFO  o.s.c.ConfigurationObjectFactory - Using method itself for [task.replicants] on [com.metamx.tranquility.config.PropertiesBasedConfig#taskReplicants()]

2016-02-24 08:28:20,728 [main] INFO  o.s.c.ConfigurationObjectFactory - Using method itself for [task.warmingPeriod] on [com.metamx.tranquility.config.PropertiesBasedConfig#taskWarmingPeriod()]

2016-02-24 08:28:20,728 [main] INFO  o.s.c.ConfigurationObjectFactory - Assigning value [localhost:2181] for [zookeeper.connect] on [com.metamx.tranquility.config.PropertiesBasedConfig#zookeeperConnect()]

2016-02-24 08:28:20,728 [main] INFO  o.s.c.ConfigurationObjectFactory - Using method itself for [tranquility.maxBatchSize] on [com.metamx.tranquility.config.PropertiesBasedConfig#tranquilityMaxBatchSize()]

2016-02-24 08:28:20,732 [main] INFO  o.s.c.ConfigurationObjectFactory - Using method itself for [tranquility.maxPendingBatches] on [com.metamx.tranquility.config.PropertiesBasedConfig#tranquilityMaxPendingBatches()]

2016-02-24 08:28:20,733 [main] INFO  o.s.c.ConfigurationObjectFactory - Using method itself for [tranquility.lingerMillis] on [com.metamx.tranquility.config.PropertiesBasedConfig#tranquilityLingerMillis()]

2016-02-24 08:28:20,733 [main] INFO  o.s.c.ConfigurationObjectFactory - Assigning value [/druid/discovery] for [druid.discovery.curator.path] on [com.metamx.tranquility.config.PropertiesBasedConfig#discoPath()]

2016-02-24 08:28:20,733 [main] INFO  o.s.c.ConfigurationObjectFactory - Using method itself for [druidBeam.firehoseGracePeriod] on [com.metamx.tranquility.config.PropertiesBasedConfig#firehoseGracePeriod()]

2016-02-24 08:28:20,733 [main] INFO  o.s.c.ConfigurationObjectFactory - Using method itself for [druidBeam.firehoseQuietPeriod] on [com.metamx.tranquility.config.PropertiesBasedConfig#firehoseQuietPeriod()]

2016-02-24 08:28:20,734 [main] INFO  o.s.c.ConfigurationObjectFactory - Using method itself for [druidBeam.firehoseRetryPeriod] on [com.metamx.tranquility.config.PropertiesBasedConfig#firehoseRetryPeriod()]

2016-02-24 08:28:20,734 [main] INFO  o.s.c.ConfigurationObjectFactory - Using method itself for [druidBeam.firehoseChunkSize] on [com.metamx.tranquility.config.PropertiesBasedConfig#firehoseChunkSize()]

2016-02-24 08:28:20,734 [main] INFO  o.s.c.ConfigurationObjectFactory - Using method itself for [druidBeam.randomizeTaskId] on [com.metamx.tranquility.config.PropertiesBasedConfig#randomizeTaskId()]

2016-02-24 08:28:20,735 [main] INFO  o.s.c.ConfigurationObjectFactory - Using method itself for [druidBeam.indexRetryPeriod] on [com.metamx.tranquility.config.PropertiesBasedConfig#indexRetryPeriod()]

2016-02-24 08:28:20,735 [main] INFO  o.s.c.ConfigurationObjectFactory - Using method itself for [druidBeam.firehoseBufferSize] on [com.metamx.tranquility.config.PropertiesBasedConfig#firehoseBufferSize()]

2016-02-24 08:28:20,925 [main] INFO  o.s.c.ConfigurationObjectFactory - Assigning value [2] for [consumer.numThreads] on [com.metamx.tranquility.kafka.model.PropertiesBasedKafkaConfig#getConsumerNumThreads()]

2016-02-24 08:28:20,926 [main] INFO  o.s.c.ConfigurationObjectFactory - Assigning value [twitter] for [topicPattern] on [com.metamx.tranquility.kafka.model.PropertiesBasedKafkaConfig#getTopicPattern()]

2016-02-24 08:28:20,926 [main] INFO  o.s.c.ConfigurationObjectFactory - Assigning value [tranquility-kafka] for [kafka.group.id] on [com.metamx.tranquility.kafka.model.PropertiesBasedKafkaConfig#getKafkaGroupId()]

2016-02-24 08:28:20,926 [main] INFO  o.s.c.ConfigurationObjectFactory - Assigning value [localhost:2181] for [kafka.zookeeper.connect] on [com.metamx.tranquility.kafka.model.PropertiesBasedKafkaConfig#getKafkaZookeeperConnect()]

2016-02-24 08:28:20,927 [main] INFO  o.s.c.ConfigurationObjectFactory - Assigning default value [false] for [useTopicAsDataSource] on [com.metamx.tranquility.kafka.model.PropertiesBasedKafkaConfig#useTopicAsDataSource()]

2016-02-24 08:28:20,927 [main] INFO  o.s.c.ConfigurationObjectFactory - Assigning value [1] for [topicPattern.priority] on [com.metamx.tranquility.kafka.model.PropertiesBasedKafkaConfig#getTopicPatternPriority()]

2016-02-24 08:28:20,928 [main] INFO  o.s.c.ConfigurationObjectFactory - Assigning value [15000] for [commit.periodMillis] on [com.metamx.tranquility.kafka.model.PropertiesBasedKafkaConfig#getCommitPeriodMillis()]

2016-02-24 08:28:20,928 [main] INFO  o.s.c.ConfigurationObjectFactory - Assigning value [false] for [reportDropsAsExceptions] on [com.metamx.tranquility.kafka.model.PropertiesBasedKafkaConfig#reportDropsAsExceptions()]

2016-02-24 08:28:20,928 [main] INFO  o.s.c.ConfigurationObjectFactory - Assigning value [PT20S] for [zookeeper.timeout] on [com.metamx.tranquility.config.PropertiesBasedConfig#zookeeperTimeout()]

2016-02-24 08:28:20,929 [main] INFO  o.s.c.ConfigurationObjectFactory - Assigning value [druid/overlord] for [druid.selectors.indexing.serviceName] on [com.metamx.tranquility.config.PropertiesBasedConfig#druidIndexingServiceName()]

2016-02-24 08:28:20,929 [main] INFO  o.s.c.ConfigurationObjectFactory - Using method itself for [task.partitions] on [com.metamx.tranquility.config.PropertiesBasedConfig#taskPartitions()]

2016-02-24 08:28:20,929 [main] INFO  o.s.c.ConfigurationObjectFactory - Using method itself for [task.replicants] on [com.metamx.tranquility.config.PropertiesBasedConfig#taskReplicants()]

2016-02-24 08:28:20,930 [main] INFO  o.s.c.ConfigurationObjectFactory - Using method itself for [task.warmingPeriod] on [com.metamx.tranquility.config.PropertiesBasedConfig#taskWarmingPeriod()]

2016-02-24 08:28:20,930 [main] INFO  o.s.c.ConfigurationObjectFactory - Assigning value [localhost:2181] for [zookeeper.connect] on [com.metamx.tranquility.config.PropertiesBasedConfig#zookeeperConnect()]

2016-02-24 08:28:20,930 [main] INFO  o.s.c.ConfigurationObjectFactory - Using method itself for [tranquility.maxBatchSize] on [com.metamx.tranquility.config.PropertiesBasedConfig#tranquilityMaxBatchSize()]

2016-02-24 08:28:20,930 [main] INFO  o.s.c.ConfigurationObjectFactory - Using method itself for [tranquility.maxPendingBatches] on [com.metamx.tranquility.config.PropertiesBasedConfig#tranquilityMaxPendingBatches()]

2016-02-24 08:28:20,931 [main] INFO  o.s.c.ConfigurationObjectFactory - Using method itself for [tranquility.lingerMillis] on [com.metamx.tranquility.config.PropertiesBasedConfig#tranquilityLingerMillis()]

2016-02-24 08:28:20,931 [main] INFO  o.s.c.ConfigurationObjectFactory - Assigning value [/druid/discovery] for [druid.discovery.curator.path] on [com.metamx.tranquility.config.PropertiesBasedConfig#discoPath()]

2016-02-24 08:28:20,931 [main] INFO  o.s.c.ConfigurationObjectFactory - Using method itself for [druidBeam.firehoseGracePeriod] on [com.metamx.tranquility.config.PropertiesBasedConfig#firehoseGracePeriod()]

2016-02-24 08:28:20,932 [main] INFO  o.s.c.ConfigurationObjectFactory - Using method itself for [druidBeam.firehoseQuietPeriod] on [com.metamx.tranquility.config.PropertiesBasedConfig#firehoseQuietPeriod()]

2016-02-24 08:28:20,932 [main] INFO  o.s.c.ConfigurationObjectFactory - Using method itself for [druidBeam.firehoseRetryPeriod] on [com.metamx.tranquility.config.PropertiesBasedConfig#firehoseRetryPeriod()]

2016-02-24 08:28:20,932 [main] INFO  o.s.c.ConfigurationObjectFactory - Using method itself for [druidBeam.firehoseChunkSize] on [com.metamx.tranquility.config.PropertiesBasedConfig#firehoseChunkSize()]

2016-02-24 08:28:20,932 [main] INFO  o.s.c.ConfigurationObjectFactory - Using method itself for [druidBeam.randomizeTaskId] on [com.metamx.tranquility.config.PropertiesBasedConfig#randomizeTaskId()]

2016-02-24 08:28:20,933 [main] INFO  o.s.c.ConfigurationObjectFactory - Using method itself for [druidBeam.indexRetryPeriod] on [com.metamx.tranquility.config.PropertiesBasedConfig#indexRetryPeriod()]

2016-02-24 08:28:20,933 [main] INFO  o.s.c.ConfigurationObjectFactory - Using method itself for [druidBeam.firehoseBufferSize] on [com.metamx.tranquility.config.PropertiesBasedConfig#firehoseBufferSize()]

2016-02-24 08:28:20,937 [main] INFO  o.s.c.ConfigurationObjectFactory - Assigning value [2] for [consumer.numThreads] on [com.metamx.tranquility.kafka.model.PropertiesBasedKafkaConfig#getConsumerNumThreads()]

2016-02-24 08:28:20,938 [main] INFO  o.s.c.ConfigurationObjectFactory - Assigning value [wikipedia.*] for [topicPattern] on [com.metamx.tranquility.kafka.model.PropertiesBasedKafkaConfig#getTopicPattern()]

2016-02-24 08:28:20,939 [main] INFO  o.s.c.ConfigurationObjectFactory - Assigning value [tranquility-kafka] for [kafka.group.id] on [com.metamx.tranquility.kafka.model.PropertiesBasedKafkaConfig#getKafkaGroupId()]

2016-02-24 08:28:20,939 [main] INFO  o.s.c.ConfigurationObjectFactory - Assigning value [localhost:2181] for [kafka.zookeeper.connect] on [com.metamx.tranquility.kafka.model.PropertiesBasedKafkaConfig#getKafkaZookeeperConnect()]

2016-02-24 08:28:20,939 [main] INFO  o.s.c.ConfigurationObjectFactory - Assigning default value [false] for [useTopicAsDataSource] on [com.metamx.tranquility.kafka.model.PropertiesBasedKafkaConfig#useTopicAsDataSource()]

2016-02-24 08:28:20,940 [main] INFO  o.s.c.ConfigurationObjectFactory - Assigning value [1] for [topicPattern.priority] on [com.metamx.tranquility.kafka.model.PropertiesBasedKafkaConfig#getTopicPatternPriority()]

2016-02-24 08:28:20,940 [main] INFO  o.s.c.ConfigurationObjectFactory - Assigning value [15000] for [commit.periodMillis] on [com.metamx.tranquility.kafka.model.PropertiesBasedKafkaConfig#getCommitPeriodMillis()]

2016-02-24 08:28:20,941 [main] INFO  o.s.c.ConfigurationObjectFactory - Assigning value [false] for [reportDropsAsExceptions] on [com.metamx.tranquility.kafka.model.PropertiesBasedKafkaConfig#reportDropsAsExceptions()]

2016-02-24 08:28:20,941 [main] INFO  o.s.c.ConfigurationObjectFactory - Assigning value [PT20S] for [zookeeper.timeout] on [com.metamx.tranquility.config.PropertiesBasedConfig#zookeeperTimeout()]

2016-02-24 08:28:20,941 [main] INFO  o.s.c.ConfigurationObjectFactory - Assigning value [druid/overlord] for [druid.selectors.indexing.serviceName] on [com.metamx.tranquility.config.PropertiesBasedConfig#druidIndexingServiceName()]

2016-02-24 08:28:20,942 [main] INFO  o.s.c.ConfigurationObjectFactory - Assigning value [2] for [task.partitions] on [com.metamx.tranquility.config.PropertiesBasedConfig#taskPartitions()]

2016-02-24 08:28:20,942 [main] INFO  o.s.c.ConfigurationObjectFactory - Assigning value [2] for [task.replicants] on [com.metamx.tranquility.config.PropertiesBasedConfig#taskReplicants()]

2016-02-24 08:28:20,942 [main] INFO  o.s.c.ConfigurationObjectFactory - Using method itself for [task.warmingPeriod] on [com.metamx.tranquility.config.PropertiesBasedConfig#taskWarmingPeriod()]

2016-02-24 08:28:20,942 [main] INFO  o.s.c.ConfigurationObjectFactory - Assigning value [localhost:2181] for [zookeeper.connect] on [com.metamx.tranquility.config.PropertiesBasedConfig#zookeeperConnect()]

2016-02-24 08:28:20,943 [main] INFO  o.s.c.ConfigurationObjectFactory - Using method itself for [tranquility.maxBatchSize] on [com.metamx.tranquility.config.PropertiesBasedConfig#tranquilityMaxBatchSize()]

2016-02-24 08:28:20,943 [main] INFO  o.s.c.ConfigurationObjectFactory - Using method itself for [tranquility.maxPendingBatches] on [com.metamx.tranquility.config.PropertiesBasedConfig#tranquilityMaxPendingBatches()]

2016-02-24 08:28:20,943 [main] INFO  o.s.c.ConfigurationObjectFactory - Using method itself for [tranquility.lingerMillis] on [com.metamx.tranquility.config.PropertiesBasedConfig#tranquilityLingerMillis()]

2016-02-24 08:28:20,943 [main] INFO  o.s.c.ConfigurationObjectFactory - Assigning value [/druid/discovery] for [druid.discovery.curator.path] on [com.metamx.tranquility.config.PropertiesBasedConfig#discoPath()]

2016-02-24 08:28:20,944 [main] INFO  o.s.c.ConfigurationObjectFactory - Using method itself for [druidBeam.firehoseGracePeriod] on [com.metamx.tranquility.config.PropertiesBasedConfig#firehoseGracePeriod()]

2016-02-24 08:28:20,944 [main] INFO  o.s.c.ConfigurationObjectFactory - Using method itself for [druidBeam.firehoseQuietPeriod] on [com.metamx.tranquility.config.PropertiesBasedConfig#firehoseQuietPeriod()]

2016-02-24 08:28:20,945 [main] INFO  o.s.c.ConfigurationObjectFactory - Using method itself for [druidBeam.firehoseRetryPeriod] on [com.metamx.tranquility.config.PropertiesBasedConfig#firehoseRetryPeriod()]

2016-02-24 08:28:20,945 [main] INFO  o.s.c.ConfigurationObjectFactory - Using method itself for [druidBeam.firehoseChunkSize] on [com.metamx.tranquility.config.PropertiesBasedConfig#firehoseChunkSize()]

2016-02-24 08:28:20,946 [main] INFO  o.s.c.ConfigurationObjectFactory - Using method itself for [druidBeam.randomizeTaskId] on [com.metamx.tranquility.config.PropertiesBasedConfig#randomizeTaskId()]

2016-02-24 08:28:20,946 [main] INFO  o.s.c.ConfigurationObjectFactory - Using method itself for [druidBeam.indexRetryPeriod] on [com.metamx.tranquility.config.PropertiesBasedConfig#indexRetryPeriod()]

2016-02-24 08:28:20,946 [main] INFO  o.s.c.ConfigurationObjectFactory - Using method itself for [druidBeam.firehoseBufferSize] on [com.metamx.tranquility.config.PropertiesBasedConfig#firehoseBufferSize()]

2016-02-24 08:28:20,967 [main] WARN  o.a.c.retry.ExponentialBackoffRetry - maxRetries too large (500). Pinning to 29

Exception in thread "main" java.lang.NoSuchMethodError: java.util.List.sort(Ljava/util/Comparator;)V

at com.metamx.tranquility.kafka.writer.WriterController.<init>(WriterController.java:58)

at com.metamx.tranquility.kafka.KafkaMain.run(KafkaMain.java:122)

at com.metamx.tranquility.kafka.KafkaMain.main(KafkaMain.java:71)

at com.metamx.tranquility.distribution.DistributionMain$.main(DistributionMain.scala:37)

at com.metamx.tranquility.distribution.DistributionMain.main(DistributionMain.scala)


It is the same problem with example test.

Venkat Reddy Amalla

未读，
2016年2月24日 07:23:582016/2/24
收件人 Druid User
Hi,

I resolved the same exception by upgrading java7 to java-8. 
...

Gian Merlino

未读，
2016年2月24日 20:52:152016/2/24
收件人 druid...@googlegroups.com
Sorry, yes, this was a mistake in the original release of tranquility-kafka. It was intended to support java 7 too. The next release will support both java 7 and 8 (master already does). Sorry for the inconvenience.

Gian

--
You received this message because you are subscribed to the Google Groups "Druid User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to druid-user+...@googlegroups.com.
To post to this group, send email to druid...@googlegroups.com.
 To view this discussion on the web visit https://groups.google.com/d/msgid/druid-user/d8cd7d33-b87f-48d5-a52f-9c025386b55c%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

luo...@conew.com

未读，
2016年2月24日 21:29:352016/2/24
收件人 Druid User
Hi, Gian:

There are some exception when spark streaming ingest data with tranquility. 

在 2016年2月25日星期四 UTC+8上午9:52:15，Gian Merlino写道：
...

Gian Merlino

未读，
2016年2月24日 21:32:452016/2/24
收件人 druid...@googlegroups.com
yep, was just commenting there.

Gian

--
You received this message because you are subscribed to the Google Groups "Druid User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to druid-user+...@googlegroups.com.
To post to this group, send email to druid...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/druid-user/8c4eda9f-2753-4395-be3d-2ce9971d3ca6%40googlegroups.com.
回复全部
回复作者
转发
0 个新帖子