nohup: 忽略输入 2016-04-29 08:46:24,838 [main] INFO o.s.c.ConfigurationObjectFactory - Assigning value [2] for [consumer.numThreads] on [com.metamx.tranquility.kafka.model.PropertiesBasedKafkaConfig#getConsumerNumThreads()] 2016-04-29 08:46:24,847 [main] INFO o.s.c.ConfigurationObjectFactory - Assigning default value [(?!)] for [topicPattern] on [com.metamx.tranquility.kafka.model.PropertiesBasedKafkaConfig#getTopicPattern()] 2016-04-29 08:46:24,848 [main] INFO o.s.c.ConfigurationObjectFactory - Assigning value [tranquility-kafka] for [kafka.group.id] on [com.metamx.tranquility.kafka.model.PropertiesBasedKafkaConfig#getKafkaGroupId()] 2016-04-29 08:46:24,848 [main] INFO o.s.c.ConfigurationObjectFactory - Assigning value [localhost:2181] for [kafka.zookeeper.connect] on [com.metamx.tranquility.kafka.model.PropertiesBasedKafkaConfig#getKafkaZookeeperConnect()] 2016-04-29 08:46:24,848 [main] INFO o.s.c.ConfigurationObjectFactory - Assigning default value [false] for [useTopicAsDataSource] on [com.metamx.tranquility.kafka.model.PropertiesBasedKafkaConfig#useTopicAsDataSource()] 2016-04-29 08:46:24,849 [main] INFO o.s.c.ConfigurationObjectFactory - Assigning default value [1] for [topicPattern.priority] on [com.metamx.tranquility.kafka.model.PropertiesBasedKafkaConfig#getTopicPatternPriority()] 2016-04-29 08:46:24,849 [main] INFO o.s.c.ConfigurationObjectFactory - Assigning value [15000] for [commit.periodMillis] on [com.metamx.tranquility.kafka.model.PropertiesBasedKafkaConfig#getCommitPeriodMillis()] 2016-04-29 08:46:24,850 [main] INFO o.s.c.ConfigurationObjectFactory - Assigning default value [false] for [reportDropsAsExceptions] on [com.metamx.tranquility.kafka.model.PropertiesBasedKafkaConfig#reportDropsAsExceptions()] 2016-04-29 08:46:24,850 [main] INFO o.s.c.ConfigurationObjectFactory - Using method itself for [zookeeper.timeout] on [com.metamx.tranquility.config.PropertiesBasedConfig#zookeeperTimeout()] 2016-04-29 08:46:24,914 [main] INFO o.s.c.ConfigurationObjectFactory - Assigning value [druid/overlord] for [druid.selectors.indexing.serviceName] on [com.metamx.tranquility.config.PropertiesBasedConfig#druidIndexingServiceName()] 2016-04-29 08:46:24,915 [main] INFO o.s.c.ConfigurationObjectFactory - Using method itself for [task.partitions] on [com.metamx.tranquility.config.PropertiesBasedConfig#taskPartitions()] 2016-04-29 08:46:24,915 [main] INFO o.s.c.ConfigurationObjectFactory - Using method itself for [task.replicants] on [com.metamx.tranquility.config.PropertiesBasedConfig#taskReplicants()] 2016-04-29 08:46:24,915 [main] INFO o.s.c.ConfigurationObjectFactory - Using method itself for [task.warmingPeriod] on [com.metamx.tranquility.config.PropertiesBasedConfig#taskWarmingPeriod()] 2016-04-29 08:46:24,915 [main] INFO o.s.c.ConfigurationObjectFactory - Assigning value [localhost:2181] for [zookeeper.connect] on [com.metamx.tranquility.config.PropertiesBasedConfig#zookeeperConnect()] 2016-04-29 08:46:24,916 [main] INFO o.s.c.ConfigurationObjectFactory - Using method itself for [tranquility.maxBatchSize] on [com.metamx.tranquility.config.PropertiesBasedConfig#tranquilityMaxBatchSize()] 2016-04-29 08:46:24,919 [main] INFO o.s.c.ConfigurationObjectFactory - Using method itself for [tranquility.maxPendingBatches] on [com.metamx.tranquility.config.PropertiesBasedConfig#tranquilityMaxPendingBatches()] 2016-04-29 08:46:24,919 [main] INFO o.s.c.ConfigurationObjectFactory - Using method itself for [tranquility.lingerMillis] on [com.metamx.tranquility.config.PropertiesBasedConfig#tranquilityLingerMillis()] 2016-04-29 08:46:24,920 [main] INFO o.s.c.ConfigurationObjectFactory - Assigning value [/druid/discovery] for [druid.discovery.curator.path] on [com.metamx.tranquility.config.PropertiesBasedConfig#discoPath()] 2016-04-29 08:46:24,920 [main] INFO o.s.c.ConfigurationObjectFactory - Using method itself for [druidBeam.firehoseGracePeriod] on [com.metamx.tranquility.config.PropertiesBasedConfig#firehoseGracePeriod()] 2016-04-29 08:46:24,920 [main] INFO o.s.c.ConfigurationObjectFactory - Using method itself for [druidBeam.firehoseQuietPeriod] on [com.metamx.tranquility.config.PropertiesBasedConfig#firehoseQuietPeriod()] 2016-04-29 08:46:24,920 [main] INFO o.s.c.ConfigurationObjectFactory - Using method itself for [druidBeam.firehoseRetryPeriod] on [com.metamx.tranquility.config.PropertiesBasedConfig#firehoseRetryPeriod()] 2016-04-29 08:46:24,921 [main] INFO o.s.c.ConfigurationObjectFactory - Using method itself for [druidBeam.firehoseChunkSize] on [com.metamx.tranquility.config.PropertiesBasedConfig#firehoseChunkSize()] 2016-04-29 08:46:24,921 [main] INFO o.s.c.ConfigurationObjectFactory - Using method itself for [druidBeam.randomizeTaskId] on [com.metamx.tranquility.config.PropertiesBasedConfig#randomizeTaskId()] 2016-04-29 08:46:24,921 [main] INFO o.s.c.ConfigurationObjectFactory - Using method itself for [druidBeam.indexRetryPeriod] on [com.metamx.tranquility.config.PropertiesBasedConfig#indexRetryPeriod()] 2016-04-29 08:46:24,921 [main] INFO o.s.c.ConfigurationObjectFactory - Using method itself for [druidBeam.firehoseBufferSize] on [com.metamx.tranquility.config.PropertiesBasedConfig#firehoseBufferSize()] 2016-04-29 08:46:24,922 [main] INFO o.s.c.ConfigurationObjectFactory - Using method itself for [druidBeam.overlordLocator] on [com.metamx.tranquility.config.PropertiesBasedConfig#overlordLocator()] 2016-04-29 08:46:24,922 [main] INFO o.s.c.ConfigurationObjectFactory - Using method itself for [druidBeam.taskLocator] on [com.metamx.tranquility.config.PropertiesBasedConfig#taskLocator()] 2016-04-29 08:46:24,922 [main] INFO o.s.c.ConfigurationObjectFactory - Using method itself for [druidBeam.overlordPollPeriod] on [com.metamx.tranquility.config.PropertiesBasedConfig#overlordPollPeriod()] 2016-04-29 08:46:25,078 [main] INFO o.s.c.ConfigurationObjectFactory - Assigning value [2] for [consumer.numThreads] on [com.metamx.tranquility.kafka.model.PropertiesBasedKafkaConfig#getConsumerNumThreads()] 2016-04-29 08:46:25,079 [main] INFO o.s.c.ConfigurationObjectFactory - Assigning value [metrics] for [topicPattern] on [com.metamx.tranquility.kafka.model.PropertiesBasedKafkaConfig#getTopicPattern()] 2016-04-29 08:46:25,079 [main] INFO o.s.c.ConfigurationObjectFactory - Assigning value [tranquility-kafka] for [kafka.group.id] on [com.metamx.tranquility.kafka.model.PropertiesBasedKafkaConfig#getKafkaGroupId()] 2016-04-29 08:46:25,080 [main] INFO o.s.c.ConfigurationObjectFactory - Assigning value [localhost:2181] for [kafka.zookeeper.connect] on [com.metamx.tranquility.kafka.model.PropertiesBasedKafkaConfig#getKafkaZookeeperConnect()] 2016-04-29 08:46:25,080 [main] INFO o.s.c.ConfigurationObjectFactory - Assigning default value [false] for [useTopicAsDataSource] on [com.metamx.tranquility.kafka.model.PropertiesBasedKafkaConfig#useTopicAsDataSource()] 2016-04-29 08:46:25,080 [main] INFO o.s.c.ConfigurationObjectFactory - Assigning default value [1] for [topicPattern.priority] on [com.metamx.tranquility.kafka.model.PropertiesBasedKafkaConfig#getTopicPatternPriority()] 2016-04-29 08:46:25,081 [main] INFO o.s.c.ConfigurationObjectFactory - Assigning value [15000] for [commit.periodMillis] on [com.metamx.tranquility.kafka.model.PropertiesBasedKafkaConfig#getCommitPeriodMillis()] 2016-04-29 08:46:25,081 [main] INFO o.s.c.ConfigurationObjectFactory - Assigning default value [false] for [reportDropsAsExceptions] on [com.metamx.tranquility.kafka.model.PropertiesBasedKafkaConfig#reportDropsAsExceptions()] 2016-04-29 08:46:25,081 [main] INFO o.s.c.ConfigurationObjectFactory - Using method itself for [zookeeper.timeout] on [com.metamx.tranquility.config.PropertiesBasedConfig#zookeeperTimeout()] 2016-04-29 08:46:25,082 [main] INFO o.s.c.ConfigurationObjectFactory - Assigning value [druid/overlord] for [druid.selectors.indexing.serviceName] on [com.metamx.tranquility.config.PropertiesBasedConfig#druidIndexingServiceName()] 2016-04-29 08:46:25,082 [main] INFO o.s.c.ConfigurationObjectFactory - Assigning value [1] for [task.partitions] on [com.metamx.tranquility.config.PropertiesBasedConfig#taskPartitions()] 2016-04-29 08:46:25,082 [main] INFO o.s.c.ConfigurationObjectFactory - Assigning value [1] for [task.replicants] on [com.metamx.tranquility.config.PropertiesBasedConfig#taskReplicants()] 2016-04-29 08:46:25,082 [main] INFO o.s.c.ConfigurationObjectFactory - Using method itself for [task.warmingPeriod] on [com.metamx.tranquility.config.PropertiesBasedConfig#taskWarmingPeriod()] 2016-04-29 08:46:25,083 [main] INFO o.s.c.ConfigurationObjectFactory - Assigning value [localhost:2181] for [zookeeper.connect] on [com.metamx.tranquility.config.PropertiesBasedConfig#zookeeperConnect()] 2016-04-29 08:46:25,083 [main] INFO o.s.c.ConfigurationObjectFactory - Using method itself for [tranquility.maxBatchSize] on [com.metamx.tranquility.config.PropertiesBasedConfig#tranquilityMaxBatchSize()] 2016-04-29 08:46:25,083 [main] INFO o.s.c.ConfigurationObjectFactory - Using method itself for [tranquility.maxPendingBatches] on [com.metamx.tranquility.config.PropertiesBasedConfig#tranquilityMaxPendingBatches()] 2016-04-29 08:46:25,083 [main] INFO o.s.c.ConfigurationObjectFactory - Using method itself for [tranquility.lingerMillis] on [com.metamx.tranquility.config.PropertiesBasedConfig#tranquilityLingerMillis()] 2016-04-29 08:46:25,084 [main] INFO o.s.c.ConfigurationObjectFactory - Assigning value [/druid/discovery] for [druid.discovery.curator.path] on [com.metamx.tranquility.config.PropertiesBasedConfig#discoPath()] 2016-04-29 08:46:25,084 [main] INFO o.s.c.ConfigurationObjectFactory - Using method itself for [druidBeam.firehoseGracePeriod] on [com.metamx.tranquility.config.PropertiesBasedConfig#firehoseGracePeriod()] 2016-04-29 08:46:25,084 [main] INFO o.s.c.ConfigurationObjectFactory - Using method itself for [druidBeam.firehoseQuietPeriod] on [com.metamx.tranquility.config.PropertiesBasedConfig#firehoseQuietPeriod()] 2016-04-29 08:46:25,084 [main] INFO o.s.c.ConfigurationObjectFactory - Using method itself for [druidBeam.firehoseRetryPeriod] on [com.metamx.tranquility.config.PropertiesBasedConfig#firehoseRetryPeriod()] 2016-04-29 08:46:25,085 [main] INFO o.s.c.ConfigurationObjectFactory - Using method itself for [druidBeam.firehoseChunkSize] on [com.metamx.tranquility.config.PropertiesBasedConfig#firehoseChunkSize()] 2016-04-29 08:46:25,085 [main] INFO o.s.c.ConfigurationObjectFactory - Using method itself for [druidBeam.randomizeTaskId] on [com.metamx.tranquility.config.PropertiesBasedConfig#randomizeTaskId()] 2016-04-29 08:46:25,085 [main] INFO o.s.c.ConfigurationObjectFactory - Using method itself for [druidBeam.indexRetryPeriod] on [com.metamx.tranquility.config.PropertiesBasedConfig#indexRetryPeriod()] 2016-04-29 08:46:25,085 [main] INFO o.s.c.ConfigurationObjectFactory - Using method itself for [druidBeam.firehoseBufferSize] on [com.metamx.tranquility.config.PropertiesBasedConfig#firehoseBufferSize()] 2016-04-29 08:46:25,086 [main] INFO o.s.c.ConfigurationObjectFactory - Using method itself for [druidBeam.overlordLocator] on [com.metamx.tranquility.config.PropertiesBasedConfig#overlordLocator()] 2016-04-29 08:46:25,086 [main] INFO o.s.c.ConfigurationObjectFactory - Using method itself for [druidBeam.taskLocator] on [com.metamx.tranquility.config.PropertiesBasedConfig#taskLocator()] 2016-04-29 08:46:25,086 [main] INFO o.s.c.ConfigurationObjectFactory - Using method itself for [druidBeam.overlordPollPeriod] on [com.metamx.tranquility.config.PropertiesBasedConfig#overlordPollPeriod()] 2016-04-29 08:46:25,157 [main] WARN o.a.c.retry.ExponentialBackoffRetry - maxRetries too large (500). Pinning to 29 2016-04-29 08:46:25,158 [main] INFO c.m.t.kafka.writer.WriterController - Ready: [topicPattern] -> dataSource mappings: 2016-04-29 08:46:25,158 [main] INFO c.m.t.kafka.writer.WriterController - [metrics] -> metrics-kafka (priority: 1) 2016-04-29 08:46:25,261 [main] INFO kafka.utils.VerifiableProperties - Verifying properties 2016-04-29 08:46:25,268 [main] INFO kafka.utils.VerifiableProperties - Property auto.commit.enable is overridden to false 2016-04-29 08:46:25,268 [main] INFO kafka.utils.VerifiableProperties - Property group.id is overridden to tranquility-kafka 2016-04-29 08:46:25,268 [main] INFO kafka.utils.VerifiableProperties - Property zookeeper.connect is overridden to localhost:2181 2016-04-29 08:46:25,268 [main] INFO kafka.utils.VerifiableProperties - Property zookeeper.session.timeout.ms is overridden to 20000 2016-04-29 08:46:25,366 [main] INFO k.c.ZookeeperConsumerConnector - [tranquility-kafka_master-1461919585364-7f6a5b44], Connecting to zookeeper instance at localhost:2181 2016-04-29 08:46:25,380 [ZkClient-EventThread-12-localhost:2181] INFO org.I0Itec.zkclient.ZkEventThread - Starting ZkClient event thread. 2016-04-29 08:46:25,389 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT 2016-04-29 08:46:25,389 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:host.name=master 2016-04-29 08:46:25,389 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.version=1.7.0_79 2016-04-29 08:46:25,389 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.vendor=Oracle Corporation 2016-04-29 08:46:25,389 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.home=/usr/local/java/jdk1.7.0_79/jre 2016-04-29 08:46:25,389 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.class.path=/opt/druid-0.9.0/tranquility/lib/io.druid.tranquility-distribution-0.7.4.jar:/opt/druid-0.9.0/tranquility/lib/io.druid.tranquility-kafka-0.7.4.jar:/opt/druid-0.9.0/tranquility/lib/io.druid.tranquility-core-0.7.4.jar:/opt/druid-0.9.0/tranquility/lib/io.druid.tranquility-server-0.7.4.jar:/opt/druid-0.9.0/tranquility/lib/org.scala-lang.scala-library-2.11.7.jar:/opt/druid-0.9.0/tranquility/lib/com.metamx.scala-util_2.11-1.11.6.jar:/opt/druid-0.9.0/tranquility/lib/com.metamx.loglady_2.11-1.1.0-mmx.jar:/opt/druid-0.9.0/tranquility/lib/com.metamx.java-util-0.27.4.jar:/opt/druid-0.9.0/tranquility/lib/org.skife.config.config-magic-0.9.jar:/opt/druid-0.9.0/tranquility/lib/com.google.guava.guava-16.0.1.jar:/opt/druid-0.9.0/tranquility/lib/com.fasterxml.jackson.core.jackson-annotations-2.4.6.jar:/opt/druid-0.9.0/tranquility/lib/com.fasterxml.jackson.core.jackson-core-2.4.6.jar:/opt/druid-0.9.0/tranquility/lib/com.fasterxml.jackson.core.jackson-databind-2.4.6.jar:/opt/druid-0.9.0/tranquility/lib/net.sf.opencsv.opencsv-2.3.jar:/opt/druid-0.9.0/tranquility/lib/io.netty.netty-3.10.5.Final.jar:/opt/druid-0.9.0/tranquility/lib/com.metamx.emitter-0.3.3.jar:/opt/druid-0.9.0/tranquility/lib/javax.validation.validation-api-1.1.0.Final.jar:/opt/druid-0.9.0/tranquility/lib/com.metamx.server-metrics-0.2.6.jar:/opt/druid-0.9.0/tranquility/lib/commons-lang.commons-lang-2.6.jar:/opt/druid-0.9.0/tranquility/lib/org.scalaj.scalaj-time_2.11-0.5.jar:/opt/druid-0.9.0/tranquility/lib/org.yaml.snakeyaml-1.11.jar:/opt/druid-0.9.0/tranquility/lib/com.fasterxml.jackson.dataformat.jackson-dataformat-smile-2.4.6.jar:/opt/druid-0.9.0/tranquility/lib/com.fasterxml.jackson.datatype.jackson-datatype-joda-2.4.6.jar:/opt/druid-0.9.0/tranquility/lib/com.fasterxml.jackson.module.jackson-module-scala_2.11-2.4.5.jar:/opt/druid-0.9.0/tranquility/lib/org.scala-lang.scala-reflect-2.11.2.jar:/opt/druid-0.9.0/tranquility/lib/com.thoughtworks.paranamer.paranamer-2.6.jar:/opt/druid-0.9.0/tranquility/lib/com.google.code.findbugs.jsr305-2.0.1.jar:/opt/druid-0.9.0/tranquility/lib/mysql.mysql-connector-java-5.1.18.jar:/opt/druid-0.9.0/tranquility/lib/com.h2database.h2-1.3.158.jar:/opt/druid-0.9.0/tranquility/lib/c3p0.c3p0-0.9.1.2.jar:/opt/druid-0.9.0/tranquility/lib/jline.jline-0.9.94.jar:/opt/druid-0.9.0/tranquility/lib/junit.junit-3.8.1.jar:/opt/druid-0.9.0/tranquility/lib/org.apache.zookeeper.zookeeper-3.4.6.jar:/opt/druid-0.9.0/tranquility/lib/org.codehaus.jackson.jackson-mapper-asl-1.9.13.jar:/opt/druid-0.9.0/tranquility/lib/org.codehaus.jackson.jackson-core-asl-1.9.13.jar:/opt/druid-0.9.0/tranquility/lib/com.twitter.util-core_2.11-6.30.0.jar:/opt/druid-0.9.0/tranquility/lib/com.twitter.util-function_2.11-6.30.0.jar:/opt/druid-0.9.0/tranquility/lib/com.twitter.jsr166e-1.0.0.jar:/opt/druid-0.9.0/tranquility/lib/org.scala-lang.modules.scala-parser-combinators_2.11-1.0.4.jar:/opt/druid-0.9.0/tranquility/lib/com.twitter.finagle-core_2.11-6.31.0.jar:/opt/druid-0.9.0/tranquility/lib/com.twitter.util-app_2.11-6.30.0.jar:/opt/druid-0.9.0/tranquility/lib/com.twitter.util-registry_2.11-6.30.0.jar:/opt/druid-0.9.0/tranquility/lib/com.twitter.util-cache_2.11-6.30.0.jar:/opt/druid-0.9.0/tranquility/lib/com.twitter.util-codec_2.11-6.30.0.jar:/opt/druid-0.9.0/tranquility/lib/commons-codec.commons-codec-1.9.jar:/opt/druid-0.9.0/tranquility/lib/com.twitter.util-collection_2.11-6.30.0.jar:/opt/druid-0.9.0/tranquility/lib/javax.inject.javax.inject-1.jar:/opt/druid-0.9.0/tranquility/lib/commons-collections.commons-collections-3.2.1.jar:/opt/druid-0.9.0/tranquility/lib/com.twitter.util-hashing_2.11-6.30.0.jar:/opt/druid-0.9.0/tranquility/lib/com.twitter.util-jvm_2.11-6.30.0.jar:/opt/druid-0.9.0/tranquility/lib/com.twitter.util-lint_2.11-6.30.0.jar:/opt/druid-0.9.0/tranquility/lib/com.twitter.util-logging_2.11-6.30.0.jar:/opt/druid-0.9.0/tranquility/lib/com.twitter.util-stats_2.11-6.30.0.jar:/opt/druid-0.9.0/tranquility/lib/com.twitter.finagle-http_2.11-6.31.0.jar:/opt/druid-0.9.0/tranquility/lib/org.slf4j.slf4j-api-1.7.12.jar:/opt/druid-0.9.0/tranquility/lib/org.slf4j.jul-to-slf4j-1.7.12.jar:/opt/druid-0.9.0/tranquility/lib/commons-logging.commons-logging-1.1.3.jar:/opt/druid-0.9.0/tranquility/lib/io.druid.druid-server-0.8.2.jar:/opt/druid-0.9.0/tranquility/lib/io.druid.druid-processing-0.8.2.jar:/opt/druid-0.9.0/tranquility/lib/io.druid.druid-common-0.8.2.jar:/opt/druid-0.9.0/tranquility/lib/io.druid.druid-api-0.3.13.jar:/opt/druid-0.9.0/tranquility/lib/aopalliance.aopalliance-1.0.jar:/opt/druid-0.9.0/tranquility/lib/io.airlift.airline-0.7.jar:/opt/druid-0.9.0/tranquility/lib/com.google.code.findbugs.annotations-2.0.3.jar:/opt/druid-0.9.0/tranquility/lib/org.hibernate.hibernate-validator-5.1.3.Final.jar:/opt/druid-0.9.0/tranquility/lib/org.jboss.logging.jboss-logging-3.1.3.GA.jar:/opt/druid-0.9.0/tranquility/lib/com.fasterxml.classmate-1.0.0.jar:/opt/druid-0.9.0/tranquility/lib/commons-io.commons-io-2.0.1.jar:/opt/druid-0.9.0/tranquility/lib/org.apache.commons.commons-dbcp2-2.0.1.jar:/opt/druid-0.9.0/tranquility/lib/org.apache.commons.commons-pool2-2.2.jar:/opt/druid-0.9.0/tranquility/lib/commons-pool.commons-pool-1.6.jar:/opt/druid-0.9.0/tranquility/lib/javax.el.javax.el-api-3.0.0.jar:/opt/druid-0.9.0/tranquility/lib/com.fasterxml.jackson.datatype.jackson-datatype-guava-2.4.6.jar:/opt/druid-0.9.0/tranquility/lib/org.jdbi.jdbi-2.63.1.jar:/opt/druid-0.9.0/tranquility/lib/joda-time.joda-time-2.8.2.jar:/opt/druid-0.9.0/tranquility/lib/org.apache.logging.log4j.log4j-jul-2.3.jar:/opt/druid-0.9.0/tranquility/lib/org.slf4j.jcl-over-slf4j-1.7.12.jar:/opt/druid-0.9.0/tranquility/lib/com.metamx.bytebuffer-collections-0.2.1.jar:/opt/druid-0.9.0/tranquility/lib/com.metamx.extendedset-1.3.9.jar:/opt/druid-0.9.0/tranquility/lib/org.roaringbitmap.RoaringBitmap-0.5.4.jar:/opt/druid-0.9.0/tranquility/lib/com.ning.compress-lzf-1.0.3.jar:/opt/druid-0.9.0/tranquility/lib/com.google.protobuf.protobuf-java-2.5.0.jar:/opt/druid-0.9.0/tranquility/lib/com.ibm.icu.icu4j-4.8.1.jar:/opt/druid-0.9.0/tranquility/lib/org.mozilla.rhino-1.7R5.jar:/opt/druid-0.9.0/tranquility/lib/net.jpountz.lz4.lz4-1.3.0.jar:/opt/druid-0.9.0/tranquility/lib/org.mapdb.mapdb-1.0.8.jar:/opt/druid-0.9.0/tranquility/lib/io.druid.druid-aws-common-0.8.2.jar:/opt/druid-0.9.0/tranquility/lib/com.amazonaws.aws-java-sdk-1.10.21.jar:/opt/druid-0.9.0/tranquility/lib/com.amazonaws.aws-java-sdk-support-1.10.21.jar:/opt/druid-0.9.0/tranquility/lib/com.amazonaws.aws-java-sdk-core-1.10.21.jar:/opt/druid-0.9.0/tranquility/lib/org.apache.httpcomponents.httpclient-4.5.1.jar:/opt/druid-0.9.0/tranquility/lib/org.apache.httpcomponents.httpcore-4.4.3.jar:/opt/druid-0.9.0/tranquility/lib/com.amazonaws.aws-java-sdk-simpledb-1.10.21.jar:/opt/druid-0.9.0/tranquility/lib/com.amazonaws.aws-java-sdk-simpleworkflow-1.10.21.jar:/opt/druid-0.9.0/tranquility/lib/com.amazonaws.aws-java-sdk-storagegateway-1.10.21.jar:/opt/druid-0.9.0/tranquility/lib/com.amazonaws.aws-java-sdk-route53-1.10.21.jar:/opt/druid-0.9.0/tranquility/lib/com.amazonaws.aws-java-sdk-s3-1.10.21.jar:/opt/druid-0.9.0/tranquility/lib/com.amazonaws.aws-java-sdk-kms-1.10.21.jar:/opt/druid-0.9.0/tranquility/lib/com.amazonaws.aws-java-sdk-importexport-1.10.21.jar:/opt/druid-0.9.0/tranquility/lib/com.amazonaws.aws-java-sdk-sts-1.10.21.jar:/opt/druid-0.9.0/tranquility/lib/com.amazonaws.aws-java-sdk-sqs-1.10.21.jar:/opt/druid-0.9.0/tranquility/lib/com.amazonaws.aws-java-sdk-rds-1.10.21.jar:/opt/druid-0.9.0/tranquility/lib/com.amazonaws.aws-java-sdk-redshift-1.10.21.jar:/opt/druid-0.9.0/tranquility/lib/com.amazonaws.aws-java-sdk-elasticbeanstalk-1.10.21.jar:/opt/druid-0.9.0/tranquility/lib/com.amazonaws.aws-java-sdk-glacier-1.10.21.jar:/opt/druid-0.9.0/tranquility/lib/com.amazonaws.aws-java-sdk-sns-1.10.21.jar:/opt/druid-0.9.0/tranquility/lib/com.amazonaws.aws-java-sdk-iam-1.10.21.jar:/opt/druid-0.9.0/tranquility/lib/com.amazonaws.aws-java-sdk-datapipeline-1.10.21.jar:/opt/druid-0.9.0/tranquility/lib/com.amazonaws.aws-java-sdk-elasticloadbalancing-1.10.21.jar:/opt/druid-0.9.0/tranquility/lib/com.amazonaws.aws-java-sdk-emr-1.10.21.jar:/opt/druid-0.9.0/tranquility/lib/com.amazonaws.aws-java-sdk-elasticache-1.10.21.jar:/opt/druid-0.9.0/tranquility/lib/com.amazonaws.aws-java-sdk-elastictranscoder-1.10.21.jar:/opt/druid-0.9.0/tranquility/lib/com.amazonaws.aws-java-sdk-ec2-1.10.21.jar:/opt/druid-0.9.0/tranquility/lib/com.amazonaws.aws-java-sdk-dynamodb-1.10.21.jar:/opt/druid-0.9.0/tranquility/lib/com.amazonaws.aws-java-sdk-cloudtrail-1.10.21.jar:/opt/druid-0.9.0/tranquility/lib/com.amazonaws.aws-java-sdk-cloudwatch-1.10.21.jar:/opt/druid-0.9.0/tranquility/lib/com.amazonaws.aws-java-sdk-logs-1.10.21.jar:/opt/druid-0.9.0/tranquility/lib/com.amazonaws.aws-java-sdk-cognitoidentity-1.10.21.jar:/opt/druid-0.9.0/tranquility/lib/com.amazonaws.aws-java-sdk-cognitosync-1.10.21.jar:/opt/druid-0.9.0/tranquility/lib/com.amazonaws.aws-java-sdk-directconnect-1.10.21.jar:/opt/druid-0.9.0/tranquility/lib/com.amazonaws.aws-java-sdk-cloudformation-1.10.21.jar:/opt/druid-0.9.0/tranquility/lib/com.amazonaws.aws-java-sdk-cloudfront-1.10.21.jar:/opt/druid-0.9.0/tranquility/lib/com.amazonaws.aws-java-sdk-kinesis-1.10.21.jar:/opt/druid-0.9.0/tranquility/lib/com.amazonaws.aws-java-sdk-opsworks-1.10.21.jar:/opt/druid-0.9.0/tranquility/lib/com.amazonaws.aws-java-sdk-ses-1.10.21.jar:/opt/druid-0.9.0/tranquility/lib/com.amazonaws.aws-java-sdk-autoscaling-1.10.21.jar:/opt/druid-0.9.0/tranquility/lib/com.amazonaws.aws-java-sdk-cloudsearch-1.10.21.jar:/opt/druid-0.9.0/tranquility/lib/com.amazonaws.aws-java-sdk-cloudwatchmetrics-1.10.21.jar:/opt/druid-0.9.0/tranquility/lib/com.amazonaws.aws-java-sdk-swf-libraries-1.10.21.jar:/opt/druid-0.9.0/tranquility/lib/com.amazonaws.aws-java-sdk-codedeploy-1.10.21.jar:/opt/druid-0.9.0/tranquility/lib/com.amazonaws.aws-java-sdk-codepipeline-1.10.21.jar:/opt/druid-0.9.0/tranquility/lib/com.amazonaws.aws-java-sdk-config-1.10.21.jar:/opt/druid-0.9.0/tranquility/lib/com.amazonaws.aws-java-sdk-lambda-1.10.21.jar:/opt/druid-0.9.0/tranquility/lib/com.amazonaws.aws-java-sdk-ecs-1.10.21.jar:/opt/druid-0.9.0/tranquility/lib/com.amazonaws.aws-java-sdk-cloudhsm-1.10.21.jar:/opt/druid-0.9.0/tranquility/lib/com.amazonaws.aws-java-sdk-ssm-1.10.21.jar:/opt/druid-0.9.0/tranquility/lib/com.amazonaws.aws-java-sdk-workspaces-1.10.21.jar:/opt/druid-0.9.0/tranquility/lib/com.amazonaws.aws-java-sdk-machinelearning-1.10.21.jar:/opt/druid-0.9.0/tranquility/lib/com.amazonaws.aws-java-sdk-directory-1.10.21.jar:/opt/druid-0.9.0/tranquility/lib/com.amazonaws.aws-java-sdk-efs-1.10.21.jar:/opt/druid-0.9.0/tranquility/lib/com.amazonaws.aws-java-sdk-codecommit-1.10.21.jar:/opt/druid-0.9.0/tranquility/lib/com.amazonaws.aws-java-sdk-devicefarm-1.10.21.jar:/opt/druid-0.9.0/tranquility/lib/io.druid.druid-console-0.0.2.jar:/opt/druid-0.9.0/tranquility/lib/com.metamx.http-client-1.0.4.jar:/opt/druid-0.9.0/tranquility/lib/commons-cli.commons-cli-1.2.jar:/opt/druid-0.9.0/tranquility/lib/org.glassfish.javax.el-3.0.0.jar:/opt/druid-0.9.0/tranquility/lib/org.apache.curator.curator-framework-2.8.0.jar:/opt/druid-0.9.0/tranquility/lib/org.apache.curator.curator-client-2.8.0.jar:/opt/druid-0.9.0/tranquility/lib/org.apache.curator.curator-x-discovery-2.8.0.jar:/opt/druid-0.9.0/tranquility/lib/org.apache.curator.curator-recipes-2.8.0.jar:/opt/druid-0.9.0/tranquility/lib/com.fasterxml.jackson.jaxrs.jackson-jaxrs-json-provider-2.4.6.jar:/opt/druid-0.9.0/tranquility/lib/com.fasterxml.jackson.jaxrs.jackson-jaxrs-base-2.4.6.jar:/opt/druid-0.9.0/tranquility/lib/com.fasterxml.jackson.module.jackson-module-jaxb-annotations-2.4.6.jar:/opt/druid-0.9.0/tranquility/lib/com.fasterxml.jackson.jaxrs.jackson-jaxrs-smile-provider-2.4.6.jar:/opt/druid-0.9.0/tranquility/lib/com.sun.jersey.jersey-server-1.19.jar:/opt/druid-0.9.0/tranquility/lib/com.sun.jersey.jersey-core-1.19.jar:/opt/druid-0.9.0/tranquility/lib/javax.ws.rs.jsr311-api-1.1.1.jar:/opt/druid-0.9.0/tranquility/lib/com.sun.jersey.contribs.jersey-guice-1.19.jar:/opt/druid-0.9.0/tranquility/lib/org.eclipse.jetty.jetty-server-9.2.5.v20141112.jar:/opt/druid-0.9.0/tranquility/lib/javax.servlet.javax.servlet-api-3.1.0.jar:/opt/druid-0.9.0/tranquility/lib/org.eclipse.jetty.jetty-http-9.2.5.v20141112.jar:/opt/druid-0.9.0/tranquility/lib/org.eclipse.jetty.jetty-util-9.2.5.v20141112.jar:/opt/druid-0.9.0/tranquility/lib/org.eclipse.jetty.jetty-io-9.2.5.v20141112.jar:/opt/druid-0.9.0/tranquility/lib/org.eclipse.jetty.jetty-proxy-9.2.5.v20141112.jar:/opt/druid-0.9.0/tranquility/lib/org.eclipse.jetty.jetty-client-9.2.5.v20141112.jar:/opt/druid-0.9.0/tranquility/lib/io.tesla.aether.tesla-aether-0.0.5.jar:/opt/druid-0.9.0/tranquility/lib/org.eclipse.aether.aether-api-0.9.0.M2.jar:/opt/druid-0.9.0/tranquility/lib/org.eclipse.aether.aether-spi-0.9.0.M2.jar:/opt/druid-0.9.0/tranquility/lib/org.eclipse.aether.aether-util-0.9.0.M2.jar:/opt/druid-0.9.0/tranquility/lib/org.eclipse.aether.aether-impl-0.9.0.M2.jar:/opt/druid-0.9.0/tranquility/lib/org.eclipse.aether.aether-connector-file-0.9.0.M2.jar:/opt/druid-0.9.0/tranquility/lib/io.tesla.aether.aether-connector-okhttp-0.0.9.jar:/opt/druid-0.9.0/tranquility/lib/com.squareup.okhttp.okhttp-1.0.2.jar:/opt/druid-0.9.0/tranquility/lib/org.apache.maven.wagon.wagon-provider-api-2.4.jar:/opt/druid-0.9.0/tranquility/lib/org.codehaus.plexus.plexus-utils-3.0.15.jar:/opt/druid-0.9.0/tranquility/lib/org.apache.maven.maven-aether-provider-3.1.1.jar:/opt/druid-0.9.0/tranquility/lib/org.apache.maven.maven-model-3.1.1.jar:/opt/druid-0.9.0/tranquility/lib/org.apache.maven.maven-model-builder-3.1.1.jar:/opt/druid-0.9.0/tranquility/lib/org.codehaus.plexus.plexus-interpolation-1.19.jar:/opt/druid-0.9.0/tranquility/lib/org.apache.maven.maven-repository-metadata-3.1.1.jar:/opt/druid-0.9.0/tranquility/lib/org.apache.maven.maven-settings-builder-3.1.1.jar:/opt/druid-0.9.0/tranquility/lib/org.apache.maven.maven-settings-3.1.1.jar:/opt/druid-0.9.0/tranquility/lib/org.antlr.antlr4-runtime-4.0.jar:/opt/druid-0.9.0/tranquility/lib/org.abego.treelayout.org.abego.treelayout.core-1.0.1.jar:/opt/druid-0.9.0/tranquility/lib/net.spy.spymemcached-2.11.7.jar:/opt/druid-0.9.0/tranquility/lib/org.eclipse.jetty.jetty-servlet-9.2.5.v20141112.jar:/opt/druid-0.9.0/tranquility/lib/org.eclipse.jetty.jetty-security-9.2.5.v20141112.jar:/opt/druid-0.9.0/tranquility/lib/org.eclipse.jetty.jetty-servlets-9.2.5.v20141112.jar:/opt/druid-0.9.0/tranquility/lib/org.eclipse.jetty.jetty-continuation-9.2.5.v20141112.jar:/opt/druid-0.9.0/tranquility/lib/com.ircclouds.irc.irc-api-1.0-0014.jar:/opt/druid-0.9.0/tranquility/lib/com.maxmind.geoip2.geoip2-0.4.0.jar:/opt/druid-0.9.0/tranquility/lib/com.maxmind.maxminddb.maxminddb-0.2.0.jar:/opt/druid-0.9.0/tranquility/lib/com.google.http-client.google-http-client-1.15.0-rc.jar:/opt/druid-0.9.0/tranquility/lib/xpp3.xpp3-1.1.4c.jar:/opt/druid-0.9.0/tranquility/lib/com.google.http-client.google-http-client-jackson2-1.15.0-rc.jar:/opt/druid-0.9.0/tranquility/lib/org.apache.derby.derby-10.11.1.1.jar:/opt/druid-0.9.0/tranquility/lib/org.apache.derby.derbynet-10.11.1.1.jar:/opt/druid-0.9.0/tranquility/lib/org.apache.derby.derbyclient-10.11.1.1.jar:/opt/druid-0.9.0/tranquility/lib/com.google.inject.guice-4.0.jar:/opt/druid-0.9.0/tranquility/lib/com.google.inject.extensions.guice-servlet-4.0.jar:/opt/druid-0.9.0/tranquility/lib/com.google.inject.extensions.guice-multibindings-4.0.jar:/opt/druid-0.9.0/tranquility/lib/org.apache.kafka.kafka_2.11-0.8.2.2.jar:/opt/druid-0.9.0/tranquility/lib/com.yammer.metrics.metrics-core-2.2.0.jar:/opt/druid-0.9.0/tranquility/lib/net.sf.jopt-simple.jopt-simple-3.2.jar:/opt/druid-0.9.0/tranquility/lib/com.101tec.zkclient-0.3.jar:/opt/druid-0.9.0/tranquility/lib/org.apache.kafka.kafka-clients-0.8.2.2.jar:/opt/druid-0.9.0/tranquility/lib/org.xerial.snappy.snappy-java-1.1.1.7.jar:/opt/druid-0.9.0/tranquility/lib/ch.qos.logback.logback-core-1.1.2.jar:/opt/druid-0.9.0/tranquility/lib/ch.qos.logback.logback-classic-1.1.2.jar:/opt/druid-0.9.0/tranquility/lib/org.apache.logging.log4j.log4j-to-slf4j-2.4.jar:/opt/druid-0.9.0/tranquility/lib/org.apache.logging.log4j.log4j-api-2.4.jar:/opt/druid-0.9.0/tranquility/lib/org.slf4j.log4j-over-slf4j-1.7.12.jar:/opt/druid-0.9.0/tranquility/lib/org.scalatra.scalatra_2.11-2.3.1.jar:/opt/druid-0.9.0/tranquility/lib/org.scalatra.scalatra-common_2.11-2.3.1.jar:/opt/druid-0.9.0/tranquility/lib/org.clapper.grizzled-slf4j_2.11-1.0.2.jar:/opt/druid-0.9.0/tranquility/lib/org.scalatra.rl.rl_2.11-0.4.10.jar:/opt/druid-0.9.0/tranquility/lib/com.googlecode.juniversalchardet.juniversalchardet-1.0.3.jar:/opt/druid-0.9.0/tranquility/lib/eu.medsea.mimeutil.mime-util-2.1.3.jar:/opt/druid-0.9.0/tranquility/lib/org.joda.joda-convert-1.7.jar:/opt/druid-0.9.0/tranquility/lib/org.scala-lang.modules.scala-xml_2.11-1.0.3.jar 2016-04-29 08:46:25,389 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib 2016-04-29 08:46:25,389 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.io.tmpdir=/tmp 2016-04-29 08:46:25,389 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.compiler= 2016-04-29 08:46:25,389 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.name=Linux 2016-04-29 08:46:25,389 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.arch=amd64 2016-04-29 08:46:25,389 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.version=2.6.32-431.el6.x86_64 2016-04-29 08:46:25,389 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.name=root 2016-04-29 08:46:25,389 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.home=/root 2016-04-29 08:46:25,390 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.dir=/opt/druid-0.9.0 2016-04-29 08:46:25,391 [main] INFO org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=localhost:2181 sessionTimeout=20000 watcher=org.I0Itec.zkclient.ZkClient@58430cb2 2016-04-29 08:46:25,424 [main-SendThread(localhost:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error) 2016-04-29 08:46:25,430 [main-SendThread(localhost:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket connection established to localhost/127.0.0.1:2181, initiating session 2016-04-29 08:46:25,441 [main-SendThread(localhost:2181)] INFO org.apache.zookeeper.ClientCnxn - Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x154384946db00a9, negotiated timeout = 20000 2016-04-29 08:46:25,443 [main-EventThread] INFO org.I0Itec.zkclient.ZkClient - zookeeper state changed (SyncConnected) 2016-04-29 08:46:25,460 [main] INFO c.m.tranquility.kafka.KafkaConsumer - Kafka topic filter [(metrics)] 2016-04-29 08:46:25,511 [main] INFO k.c.ZookeeperConsumerConnector - [tranquility-kafka_master-1461919585364-7f6a5b44], begin registering consumer tranquility-kafka_master-1461919585364-7f6a5b44 in ZK 2016-04-29 08:46:25,554 [main] INFO k.c.ZookeeperConsumerConnector - [tranquility-kafka_master-1461919585364-7f6a5b44], end registering consumer tranquility-kafka_master-1461919585364-7f6a5b44 in ZK 2016-04-29 08:46:25,566 [tranquility-kafka_master-1461919585364-7f6a5b44_watcher_executor] INFO k.c.ZookeeperConsumerConnector - [tranquility-kafka_master-1461919585364-7f6a5b44], starting watcher executor thread for consumer tranquility-kafka_master-1461919585364-7f6a5b44 2016-04-29 08:46:25,606 [main] INFO k.c.ZookeeperConsumerConnector - [tranquility-kafka_master-1461919585364-7f6a5b44], begin rebalancing consumer tranquility-kafka_master-1461919585364-7f6a5b44 try #0 2016-04-29 08:46:25,946 [main] INFO k.consumer.ConsumerFetcherManager - [ConsumerFetcherManager-1461919585452] Stopping leader finder thread 2016-04-29 08:46:25,947 [main] INFO k.consumer.ConsumerFetcherManager - [ConsumerFetcherManager-1461919585452] Stopping all fetchers 2016-04-29 08:46:25,949 [main] INFO k.consumer.ConsumerFetcherManager - [ConsumerFetcherManager-1461919585452] All connections stopped 2016-04-29 08:46:25,950 [main] INFO k.c.ZookeeperConsumerConnector - [tranquility-kafka_master-1461919585364-7f6a5b44], Cleared all relevant queues for this fetcher 2016-04-29 08:46:25,953 [main] INFO k.c.ZookeeperConsumerConnector - [tranquility-kafka_master-1461919585364-7f6a5b44], Cleared the data chunks in all the consumer message iterators 2016-04-29 08:46:25,953 [main] INFO k.c.ZookeeperConsumerConnector - [tranquility-kafka_master-1461919585364-7f6a5b44], Committing all offsets after clearing the fetcher queues 2016-04-29 08:46:25,954 [main] INFO k.c.ZookeeperConsumerConnector - [tranquility-kafka_master-1461919585364-7f6a5b44], Releasing partition ownership 2016-04-29 08:46:26,028 [main] INFO kafka.consumer.RangeAssignor - Consumer tranquility-kafka_master-1461919585364-7f6a5b44 rebalancing the following partitions: ArrayBuffer(0) for topic metrics with consumers: List(tranquility-kafka_master-1461919585364-7f6a5b44-0, tranquility-kafka_master-1461919585364-7f6a5b44-1) 2016-04-29 08:46:26,030 [main] WARN kafka.consumer.RangeAssignor - No broker partitions consumed by consumer thread tranquility-kafka_master-1461919585364-7f6a5b44-1 for topic metrics 2016-04-29 08:46:26,032 [main] INFO kafka.consumer.RangeAssignor - tranquility-kafka_master-1461919585364-7f6a5b44-0 attempting to claim partition 0 2016-04-29 08:46:26,064 [main] INFO k.c.ZookeeperConsumerConnector - [tranquility-kafka_master-1461919585364-7f6a5b44], tranquility-kafka_master-1461919585364-7f6a5b44-0 successfully owned partition 0 for topic metrics 2016-04-29 08:46:26,085 [main] INFO k.c.ZookeeperConsumerConnector - [tranquility-kafka_master-1461919585364-7f6a5b44], Consumer tranquility-kafka_master-1461919585364-7f6a5b44 selected partitions : metrics:0: fetched offset = 317: consumed offset = 317 2016-04-29 08:46:26,088 [tranquility-kafka_master-1461919585364-7f6a5b44-leader-finder-thread] INFO k.c.ConsumerFetcherManager$LeaderFinderThread - [tranquility-kafka_master-1461919585364-7f6a5b44-leader-finder-thread], Starting 2016-04-29 08:46:26,089 [main] INFO k.c.ZookeeperConsumerConnector - [tranquility-kafka_master-1461919585364-7f6a5b44], end rebalancing consumer tranquility-kafka_master-1461919585364-7f6a5b44 try #0 2016-04-29 08:46:26,101 [main] INFO k.c.ZookeeperConsumerConnector - [tranquility-kafka_master-1461919585364-7f6a5b44], Creating topic event watcher for topics (metrics) 2016-04-29 08:46:26,113 [main] INFO k.c.ZookeeperConsumerConnector - [tranquility-kafka_master-1461919585364-7f6a5b44], Topics to consume = List(metrics) 2016-04-29 08:46:26,186 [tranquility-kafka_master-1461919585364-7f6a5b44-leader-finder-thread] INFO kafka.utils.VerifiableProperties - Verifying properties 2016-04-29 08:46:26,186 [tranquility-kafka_master-1461919585364-7f6a5b44-leader-finder-thread] INFO kafka.utils.VerifiableProperties - Property client.id is overridden to tranquility-kafka 2016-04-29 08:46:26,187 [tranquility-kafka_master-1461919585364-7f6a5b44-leader-finder-thread] INFO kafka.utils.VerifiableProperties - Property metadata.broker.list is overridden to master:9092,slave2:9092 2016-04-29 08:46:26,187 [tranquility-kafka_master-1461919585364-7f6a5b44-leader-finder-thread] INFO kafka.utils.VerifiableProperties - Property request.timeout.ms is overridden to 30000 2016-04-29 08:46:26,242 [tranquility-kafka_master-1461919585364-7f6a5b44-leader-finder-thread] INFO kafka.client.ClientUtils$ - Fetching metadata from broker id:2,host:slave2,port:9092 with correlation id 0 for 1 topic(s) Set(metrics) 2016-04-29 08:46:26,246 [tranquility-kafka_master-1461919585364-7f6a5b44-leader-finder-thread] INFO kafka.producer.SyncProducer - Connected to slave2:9092 for producing 2016-04-29 08:46:26,292 [tranquility-kafka_master-1461919585364-7f6a5b44-leader-finder-thread] INFO kafka.producer.SyncProducer - Disconnecting from slave2:9092 2016-04-29 08:46:26,318 [ConsumerFetcherThread-tranquility-kafka_master-1461919585364-7f6a5b44-0-2] INFO kafka.consumer.ConsumerFetcherThread - [ConsumerFetcherThread-tranquility-kafka_master-1461919585364-7f6a5b44-0-2], Starting 2016-04-29 08:46:26,320 [tranquility-kafka_master-1461919585364-7f6a5b44-leader-finder-thread] INFO k.consumer.ConsumerFetcherManager - [ConsumerFetcherManager-1461919585452] Added fetcher for partitions ArrayBuffer([[metrics,0], initOffset 317 to broker id:2,host:slave2,port:9092] ) 2016-04-29 08:46:26,370 [KafkaConsumer-1] INFO c.m.t.kafka.writer.WriterController - Creating EventWriter for topic [metrics] using dataSource [metrics-kafka] 2016-04-29 08:46:26,458 [KafkaConsumer-1] INFO o.a.c.f.imps.CuratorFrameworkImpl - Starting 2016-04-29 08:46:26,461 [KafkaConsumer-1] INFO org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=localhost:2181 sessionTimeout=60000 watcher=org.apache.curator.ConnectionState@4694ef64 2016-04-29 08:46:26,462 [KafkaConsumer-1-SendThread(localhost:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error) 2016-04-29 08:46:26,464 [KafkaConsumer-1-SendThread(localhost:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket connection established to localhost/127.0.0.1:2181, initiating session 2016-04-29 08:46:26,470 [KafkaConsumer-1-SendThread(localhost:2181)] INFO org.apache.zookeeper.ClientCnxn - Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x154384946db00aa, negotiated timeout = 40000 2016-04-29 08:46:26,478 [KafkaConsumer-1-EventThread] INFO o.a.c.f.state.ConnectionStateManager - State change: CONNECTED 2016-04-29 08:46:26,676 [KafkaConsumer-1] INFO c.m.t.finagle.FinagleRegistry - Adding resolver for scheme[disco]. 2016-04-29 08:46:27,211 [KafkaConsumer-1] INFO o.h.validator.internal.util.Version - HV000001: Hibernate Validator 5.1.3.Final 2016-04-29 08:46:27,656 [KafkaConsumer-1] INFO io.druid.guice.JsonConfigurator - Loaded class[class io.druid.guice.ExtensionsConfig] from props[druid.extensions.] as [ExtensionsConfig{searchCurrentClassloader=true, coordinates=[], defaultVersion='0.3.13', localRepository='/root/.m2/repository', remoteRepositories=[https://repo1.maven.org/maven2/, https://metamx.artifactoryonline.com/metamx/pub-libs-releases-local]}] 2016-04-29 08:46:28,283 [KafkaConsumer-1] INFO o.s.c.ConfigurationObjectFactory - Using method itself for [${base_path}.columnCache.sizeBytes] on [io.druid.query.DruidProcessingConfig#columnCacheSizeBytes()] 2016-04-29 08:46:28,284 [KafkaConsumer-1] INFO o.s.c.ConfigurationObjectFactory - Using method itself for [druid.computation.buffer.size, ${base_path}.buffer.sizeBytes] on [io.druid.query.DruidProcessingConfig#intermediateComputeSizeBytes()] 2016-04-29 08:46:28,284 [KafkaConsumer-1] INFO o.s.c.ConfigurationObjectFactory - Using method itself for [${base_path}.numThreads] on [io.druid.query.DruidProcessingConfig#getNumThreads()] 2016-04-29 08:46:28,284 [KafkaConsumer-1] INFO o.s.c.ConfigurationObjectFactory - Assigning default value [processing-%s] for [${base_path}.formatString] on [com.metamx.common.concurrent.ExecutorServiceConfig#getFormatString()] 2016-04-29 08:46:28,313 [KafkaConsumer-1] INFO io.druid.guice.JsonConfigurator - Loaded class[interface io.druid.segment.data.BitmapSerdeFactory] from props[druid.processing.bitmap.] as [ConciseBitmapSerdeFactory{}] 2016-04-29 08:46:28,439 [KafkaConsumer-1] INFO c.metamx.emitter.core.LoggingEmitter - Start: started [true] 2016-04-29 08:46:29,350 [ClusteredBeam-ZkFuturePool-1e419b48-1aa3-4cc4-9ba9-15dadf5e53e6] INFO c.m.tranquility.beam.ClusteredBeam - Creating new merged beam for identifier[druid:overlord/metrics-kafka] timestamp[2016-04-29T08:00:00.000Z] (target = 1, actual = 0) 2016-04-29 08:46:29,547 [ClusteredBeam-ZkFuturePool-1e419b48-1aa3-4cc4-9ba9-15dadf5e53e6] INFO com.metamx.common.scala.control$ - Creating druid indexing task (service = druid:overlord): { "type" : "index_realtime", "id" : "index_realtime_metrics-kafka_2016-04-29T08:00:00.000Z_0_0", "resource" : { "availabilityGroup" : "metrics-kafka-08-0000", "requiredCapacity" : 1 }, "spec" : { "dataSchema" : { "dataSource" : "metrics-kafka", "parser" : { "type" : "map", "parseSpec" : { "format" : "json", "timestampSpec" : { "column" : "timestamp", "format" : "auto", "missingValue" : null }, "dimensionsSpec" : { "dimensions" : [ "http_code", "http_method", "metricType", "page", "server", "timestamp", "unit", "value" ], "spatialDimensions" : [ ] } } }, "metricsSpec" : [ { "type" : "count", "name" : "count" }, { "type" : "doubleSum", "name" : "value_sum", "fieldName" : "value" }, { "type" : "doubleMin", "name" : "value_min", "fieldName" : "value" }, { "type" : "doubleMax", "name" : "value_max", "fieldName" : "value" } ], "granularitySpec" : { "type" : "uniform", "segmentGranularity" : "HOUR", "queryGranularity" : { "type" : "none" } } }, "ioConfig" : { "type" : "realtime", "plumber" : null, "firehose" : { "type" : "clipped", "interval" : "2016-04-29T08:00:00.000Z/2016-04-29T09:00:00.000Z", "delegate" : { "type" : "timed", "shutoffTime" : "2016-04-29T09:15:00.000Z", "delegate" : { "type" : "receiver", "serviceName" : "firehose:druid:overlord:metrics-kafka-08-0000-0000", "bufferSize" : 100000 } } } }, "tuningConfig" : { "shardSpec" : { "type" : "linear", "partitionNum" : 0 }, "rejectionPolicy" : { "type" : "none" }, "buildV9Directly" : false, "maxPendingPersists" : 0, "intermediatePersistPeriod" : "PT10M", "windowPeriod" : "PT10M", "type" : "realtime", "maxRowsInMemory" : "100000" } } } 四月 29, 2016 8:46:29 上午 com.twitter.finagle.Init$$anonfun$1 apply$mcV$sp 信息: Finagle version 6.31.0 (rev=50d3bb0eea5ad3ed332111d707184c80fed6a506) built at 20151203-164135 2016-04-29 08:46:29,962 [ClusteredBeam-ZkFuturePool-1e419b48-1aa3-4cc4-9ba9-15dadf5e53e6] INFO c.m.c.s.net.finagle.DiscoResolver - Updating instances for service[druid:overlord] to Set(ServiceInstance{name='druid:overlord', id='0a5c08c7-b1c0-49bc-a9a2-6d1cc7db9022', address='master', port=8090, sslPort=null, payload=null, registrationTimeUTC=1461919498604, serviceType=DYNAMIC, uriSpec=null}) 2016-04-29 08:46:30,073 [ClusteredBeam-ZkFuturePool-1e419b48-1aa3-4cc4-9ba9-15dadf5e53e6] INFO c.m.t.finagle.FinagleRegistry - Created client for service: disco!druid:overlord 2016-04-29 08:46:30,700 [finagle/netty3-1] ERROR com.metamx.common.scala.control$ - Non-retryable response for uri[/druid/indexer/v1/task] with content: {"error":"Instantiation of [simple type, class io.druid.segment.indexing.RealtimeTuningConfig] value failed: Failed to create directory within 10000 attempts (tried 1461919590607-0 to 1461919590607-9999)"} 2016-04-29 08:46:30,713 [ClusteredBeam-ZkFuturePool-1e419b48-1aa3-4cc4-9ba9-15dadf5e53e6] ERROR c.m.tranquility.beam.ClusteredBeam - Failed to update cluster state: druid:overlord/metrics-kafka com.metamx.tranquility.druid.IndexServicePermanentException: Service[druid:overlord] call failed with status: 400 Bad Request at com.metamx.tranquility.druid.IndexService$$anonfun$call$1$$anonfun$apply$17.apply(IndexService.scala:158) ~[io.druid.tranquility-core-0.7.4.jar:0.7.4] at com.metamx.tranquility.druid.IndexService$$anonfun$call$1$$anonfun$apply$17.apply(IndexService.scala:132) ~[io.druid.tranquility-core-0.7.4.jar:0.7.4] at com.twitter.util.Future$$anonfun$map$1$$anonfun$apply$6.apply(Future.scala:950) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Try$.apply(Try.scala:13) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$.apply(Future.scala:97) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$$anonfun$map$1.apply(Future.scala:950) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$$anonfun$map$1.apply(Future.scala:949) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.liftedTree1$1(Promise.scala:112) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.k(Promise.scala:112) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.apply(Promise.scala:122) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.apply(Promise.scala:103) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$$anon$1.run(Promise.scala:366) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler$Activation.run(Scheduler.scala:178) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler$Activation.submit(Scheduler.scala:136) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler.submit(Scheduler.scala:207) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.Scheduler$.submit(Scheduler.scala:92) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.runq(Promise.scala:350) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.updateIfEmpty(Promise.scala:721) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.update(Promise.scala:694) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.setValue(Promise.scala:670) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.AsyncQueue.offer(AsyncQueue.scala:111) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.finagle.netty3.transport.ChannelTransport.handleUpstream(ChannelTransport.scala:55) ~[com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.http.HttpContentDecoder.messageReceived(HttpContentDecoder.java:108) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:194) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.http.HttpClientCodec.handleUpstream(HttpClientCodec.java:92) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142) ~[io.netty.netty-3.10.5.Final.jar:na] at com.twitter.finagle.netty3.channel.ChannelStatsHandler.messageReceived(ChannelStatsHandler.scala:78) ~[com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0] at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142) ~[io.netty.netty-3.10.5.Final.jar:na] at com.twitter.finagle.netty3.channel.ChannelRequestStatsHandler.messageReceived(ChannelRequestStatsHandler.scala:35) ~[com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0] at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) ~[io.netty.netty-3.10.5.Final.jar:na] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_79] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_79] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79] 2016-04-29 08:46:30,727 [ClusteredBeam-ZkFuturePool-1e419b48-1aa3-4cc4-9ba9-15dadf5e53e6] WARN c.m.tranquility.beam.ClusteredBeam - Emitting alert: [anomaly] Failed to create merged beam: druid:overlord/metrics-kafka { } java.lang.IllegalStateException: Failed to save new beam for identifier[druid:overlord/metrics-kafka] timestamp[2016-04-29T08:00:00.000Z] at com.metamx.tranquility.beam.ClusteredBeam$$anonfun$2.applyOrElse(ClusteredBeam.scala:286) ~[io.druid.tranquility-core-0.7.4.jar:0.7.4] at com.metamx.tranquility.beam.ClusteredBeam$$anonfun$2.applyOrElse(ClusteredBeam.scala:283) ~[io.druid.tranquility-core-0.7.4.jar:0.7.4] at com.twitter.util.Future$$anonfun$rescue$1.apply(Future.scala:924) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$$anonfun$rescue$1.apply(Future.scala:922) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.liftedTree1$1(Promise.scala:112) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.k(Promise.scala:112) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.apply(Promise.scala:122) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.apply(Promise.scala:103) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$$anon$1.run(Promise.scala:366) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler$Activation.run(Scheduler.scala:178) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler$Activation.submit(Scheduler.scala:136) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler.submit(Scheduler.scala:207) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.Scheduler$.submit(Scheduler.scala:92) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.runq(Promise.scala:350) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.updateIfEmpty(Promise.scala:721) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.ExecutorServiceFuturePool$$anon$2.run(FuturePool.scala:107) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_79] at java.util.concurrent.FutureTask.run(FutureTask.java:262) [na:1.7.0_79] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_79] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_79] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79] Caused by: com.metamx.tranquility.druid.IndexServicePermanentException: Service[druid:overlord] call failed with status: 400 Bad Request at com.metamx.tranquility.druid.IndexService$$anonfun$call$1$$anonfun$apply$17.apply(IndexService.scala:158) ~[io.druid.tranquility-core-0.7.4.jar:0.7.4] at com.metamx.tranquility.druid.IndexService$$anonfun$call$1$$anonfun$apply$17.apply(IndexService.scala:132) ~[io.druid.tranquility-core-0.7.4.jar:0.7.4] at com.twitter.util.Future$$anonfun$map$1$$anonfun$apply$6.apply(Future.scala:950) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Try$.apply(Try.scala:13) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$.apply(Future.scala:97) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$$anonfun$map$1.apply(Future.scala:950) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$$anonfun$map$1.apply(Future.scala:949) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.liftedTree1$1(Promise.scala:112) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.k(Promise.scala:112) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.apply(Promise.scala:122) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.apply(Promise.scala:103) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$$anon$1.run(Promise.scala:366) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler$Activation.run(Scheduler.scala:178) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler$Activation.submit(Scheduler.scala:136) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler.submit(Scheduler.scala:207) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.Scheduler$.submit(Scheduler.scala:92) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.runq(Promise.scala:350) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.updateIfEmpty(Promise.scala:721) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.update(Promise.scala:694) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.setValue(Promise.scala:670) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.AsyncQueue.offer(AsyncQueue.scala:111) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.finagle.netty3.transport.ChannelTransport.handleUpstream(ChannelTransport.scala:55) ~[com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.http.HttpContentDecoder.messageReceived(HttpContentDecoder.java:108) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:194) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.http.HttpClientCodec.handleUpstream(HttpClientCodec.java:92) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142) ~[io.netty.netty-3.10.5.Final.jar:na] at com.twitter.finagle.netty3.channel.ChannelStatsHandler.messageReceived(ChannelStatsHandler.scala:78) ~[com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0] at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142) ~[io.netty.netty-3.10.5.Final.jar:na] at com.twitter.finagle.netty3.channel.ChannelRequestStatsHandler.messageReceived(ChannelRequestStatsHandler.scala:35) ~[com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0] at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) ~[io.netty.netty-3.10.5.Final.jar:na] ... 3 common frames omitted 2016-04-29 08:46:30,747 [ClusteredBeam-ZkFuturePool-1e419b48-1aa3-4cc4-9ba9-15dadf5e53e6] INFO c.metamx.emitter.core.LoggingEmitter - Event [{"feed":"alerts","timestamp":"2016-04-29T08:46:30.738Z","service":"tranquility","host":"localhost","severity":"anomaly","description":"Failed to create merged beam: druid:overlord/metrics-kafka","data":{"exceptionType":"java.lang.IllegalStateException","exceptionMessage":"Failed to save new beam for identifier[druid:overlord/metrics-kafka] timestamp[2016-04-29T08:00:00.000Z]","exceptionStackTrace":"java.lang.IllegalStateException: Failed to save new beam for identifier[druid:overlord/metrics-kafka] timestamp[2016-04-29T08:00:00.000Z]\n\tat com.metamx.tranquility.beam.ClusteredBeam$$anonfun$2.applyOrElse(ClusteredBeam.scala:286)\n\tat com.metamx.tranquility.beam.ClusteredBeam$$anonfun$2.applyOrElse(ClusteredBeam.scala:283)\n\tat com.twitter.util.Future$$anonfun$rescue$1.apply(Future.scala:924)\n\tat com.twitter.util.Future$$anonfun$rescue$1.apply(Future.scala:922)\n\tat com.twitter.util.Promise$Transformer.liftedTree1$1(Promise.scala:112)\n\tat com.twitter.util.Promise$Transformer.k(Promise.scala:112)\n\tat com.twitter.util.Promise$Transformer.apply(Promise.scala:122)\n\tat com.twitter.util.Promise$Transformer.apply(Promise.scala:103)\n\tat com.twitter.util.Promise$$anon$1.run(Promise.scala:366)\n\tat com.twitter.concurrent.LocalScheduler$Activation.run(Scheduler.scala:178)\n\tat com.twitter.concurrent.LocalScheduler$Activation.submit(Scheduler.scala:136)\n\tat com.twitter.concurrent.LocalScheduler.submit(Scheduler.scala:207)\n\tat com.twitter.concurrent.Scheduler$.submit(Scheduler.scala:92)\n\tat com.twitter.util.Promise.runq(Promise.scala:350)\n\tat com.twitter.util.Promise.updateIfEmpty(Promise.scala:721)\n\tat com.twitter.util.ExecutorServiceFuturePool$$anon$2.run(FuturePool.scala:107)\n\tat java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:262)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)\n\tat java.lang.Thread.run(Thread.java:745)\nCaused by: com.metamx.tranquility.druid.IndexServicePermanentException: Service[druid:overlord] call failed with status: 400 Bad Request\n\tat com.metamx.tranquility.druid.IndexService$$anonfun$call$1$$anonfun$apply$17.apply(IndexService.scala:158)\n\tat com.metamx.tranquility.druid.IndexService$$anonfun$call$1$$anonfun$apply$17.apply(IndexService.scala:132)\n\tat com.twitter.util.Future$$anonfun$map$1$$anonfun$apply$6.apply(Future.scala:950)\n\tat com.twitter.util.Try$.apply(Try.scala:13)\n\tat com.twitter.util.Future$.apply(Future.scala:97)\n\tat com.twitter.util.Future$$anonfun$map$1.apply(Future.scala:950)\n\tat com.twitter.util.Future$$anonfun$map$1.apply(Future.scala:949)\n\tat com.twitter.util.Promise$Transformer.liftedTree1$1(Promise.scala:112)\n\tat com.twitter.util.Promise$Transformer.k(Promise.scala:112)\n\tat com.twitter.util.Promise$Transformer.apply(Promise.scala:122)\n\tat com.twitter.util.Promise$Transformer.apply(Promise.scala:103)\n\tat com.twitter.util.Promise$$anon$1.run(Promise.scala:366)\n\tat com.twitter.concurrent.LocalScheduler$Activation.run(Scheduler.scala:178)\n\tat com.twitter.concurrent.LocalScheduler$Activation.submit(Scheduler.scala:136)\n\tat com.twitter.concurrent.LocalScheduler.submit(Scheduler.scala:207)\n\tat com.twitter.concurrent.Scheduler$.submit(Scheduler.scala:92)\n\tat com.twitter.util.Promise.runq(Promise.scala:350)\n\tat com.twitter.util.Promise.updateIfEmpty(Promise.scala:721)\n\tat com.twitter.util.Promise.update(Promise.scala:694)\n\tat com.twitter.util.Promise.setValue(Promise.scala:670)\n\tat com.twitter.concurrent.AsyncQueue.offer(AsyncQueue.scala:111)\n\tat com.twitter.finagle.netty3.transport.ChannelTransport.handleUpstream(ChannelTransport.scala:55)\n\tat org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)\n\tat org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)\n\tat org.jboss.netty.handler.codec.http.HttpContentDecoder.messageReceived(HttpContentDecoder.java:108)\n\tat org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)\n\tat org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)\n\tat org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)\n\tat org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)\n\tat org.jboss.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:194)\n\tat org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)\n\tat org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)\n\tat org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)\n\tat org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)\n\tat org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459)\n\tat org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536)\n\tat org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435)\n\tat org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)\n\tat org.jboss.netty.handler.codec.http.HttpClientCodec.handleUpstream(HttpClientCodec.java:92)\n\tat org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)\n\tat org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)\n\tat org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142)\n\tat com.twitter.finagle.netty3.channel.ChannelStatsHandler.messageReceived(ChannelStatsHandler.scala:78)\n\tat org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88)\n\tat org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)\n\tat org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)\n\tat org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142)\n\tat com.twitter.finagle.netty3.channel.ChannelRequestStatsHandler.messageReceived(ChannelRequestStatsHandler.scala:35)\n\tat org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88)\n\tat org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)\n\tat org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)\n\tat org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)\n\tat org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)\n\tat org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)\n\tat org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)\n\tat org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)\n\tat org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)\n\tat org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)\n\tat org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)\n\tat org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)\n\t... 3 more\n"}}] 2016-04-29 08:46:30,748 [ClusteredBeam-ZkFuturePool-1e419b48-1aa3-4cc4-9ba9-15dadf5e53e6] WARN c.m.t.tranquilizer.Tranquilizer - Failed to send 1 messages. java.lang.IllegalStateException: Failed to save new beam for identifier[druid:overlord/metrics-kafka] timestamp[2016-04-29T08:00:00.000Z] at com.metamx.tranquility.beam.ClusteredBeam$$anonfun$2.applyOrElse(ClusteredBeam.scala:286) ~[io.druid.tranquility-core-0.7.4.jar:0.7.4] at com.metamx.tranquility.beam.ClusteredBeam$$anonfun$2.applyOrElse(ClusteredBeam.scala:283) ~[io.druid.tranquility-core-0.7.4.jar:0.7.4] at com.twitter.util.Future$$anonfun$rescue$1.apply(Future.scala:924) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$$anonfun$rescue$1.apply(Future.scala:922) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.liftedTree1$1(Promise.scala:112) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.k(Promise.scala:112) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.apply(Promise.scala:122) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.apply(Promise.scala:103) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$$anon$1.run(Promise.scala:366) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler$Activation.run(Scheduler.scala:178) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler$Activation.submit(Scheduler.scala:136) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler.submit(Scheduler.scala:207) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.Scheduler$.submit(Scheduler.scala:92) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.runq(Promise.scala:350) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.updateIfEmpty(Promise.scala:721) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.ExecutorServiceFuturePool$$anon$2.run(FuturePool.scala:107) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_79] at java.util.concurrent.FutureTask.run(FutureTask.java:262) [na:1.7.0_79] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_79] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_79] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79] Caused by: com.metamx.tranquility.druid.IndexServicePermanentException: Service[druid:overlord] call failed with status: 400 Bad Request at com.metamx.tranquility.druid.IndexService$$anonfun$call$1$$anonfun$apply$17.apply(IndexService.scala:158) ~[io.druid.tranquility-core-0.7.4.jar:0.7.4] at com.metamx.tranquility.druid.IndexService$$anonfun$call$1$$anonfun$apply$17.apply(IndexService.scala:132) ~[io.druid.tranquility-core-0.7.4.jar:0.7.4] at com.twitter.util.Future$$anonfun$map$1$$anonfun$apply$6.apply(Future.scala:950) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Try$.apply(Try.scala:13) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$.apply(Future.scala:97) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$$anonfun$map$1.apply(Future.scala:950) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$$anonfun$map$1.apply(Future.scala:949) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.liftedTree1$1(Promise.scala:112) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.k(Promise.scala:112) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.apply(Promise.scala:122) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.apply(Promise.scala:103) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$$anon$1.run(Promise.scala:366) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler$Activation.run(Scheduler.scala:178) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler$Activation.submit(Scheduler.scala:136) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler.submit(Scheduler.scala:207) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.Scheduler$.submit(Scheduler.scala:92) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.runq(Promise.scala:350) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.updateIfEmpty(Promise.scala:721) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.update(Promise.scala:694) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.setValue(Promise.scala:670) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.AsyncQueue.offer(AsyncQueue.scala:111) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.finagle.netty3.transport.ChannelTransport.handleUpstream(ChannelTransport.scala:55) ~[com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.http.HttpContentDecoder.messageReceived(HttpContentDecoder.java:108) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:194) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.http.HttpClientCodec.handleUpstream(HttpClientCodec.java:92) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142) ~[io.netty.netty-3.10.5.Final.jar:na] at com.twitter.finagle.netty3.channel.ChannelStatsHandler.messageReceived(ChannelStatsHandler.scala:78) ~[com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0] at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142) ~[io.netty.netty-3.10.5.Final.jar:na] at com.twitter.finagle.netty3.channel.ChannelRequestStatsHandler.messageReceived(ChannelRequestStatsHandler.scala:35) ~[com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0] at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) ~[io.netty.netty-3.10.5.Final.jar:na] ... 3 common frames omitted 2016-04-29 08:46:30,765 [ClusteredBeam-ZkFuturePool-1e419b48-1aa3-4cc4-9ba9-15dadf5e53e6] INFO c.m.tranquility.beam.ClusteredBeam - Creating new merged beam for identifier[druid:overlord/metrics-kafka] timestamp[2016-04-29T08:00:00.000Z] (target = 1, actual = 0) 2016-04-29 08:46:30,773 [ClusteredBeam-ZkFuturePool-1e419b48-1aa3-4cc4-9ba9-15dadf5e53e6] INFO com.metamx.common.scala.control$ - Creating druid indexing task (service = druid:overlord): { "type" : "index_realtime", "id" : "index_realtime_metrics-kafka_2016-04-29T08:00:00.000Z_0_0", "resource" : { "availabilityGroup" : "metrics-kafka-08-0000", "requiredCapacity" : 1 }, "spec" : { "dataSchema" : { "dataSource" : "metrics-kafka", "parser" : { "type" : "map", "parseSpec" : { "format" : "json", "timestampSpec" : { "column" : "timestamp", "format" : "auto", "missingValue" : null }, "dimensionsSpec" : { "dimensions" : [ "http_code", "http_method", "metricType", "page", "server", "timestamp", "unit", "value" ], "spatialDimensions" : [ ] } } }, "metricsSpec" : [ { "type" : "count", "name" : "count" }, { "type" : "doubleSum", "name" : "value_sum", "fieldName" : "value" }, { "type" : "doubleMin", "name" : "value_min", "fieldName" : "value" }, { "type" : "doubleMax", "name" : "value_max", "fieldName" : "value" } ], "granularitySpec" : { "type" : "uniform", "segmentGranularity" : "HOUR", "queryGranularity" : { "type" : "none" } } }, "ioConfig" : { "type" : "realtime", "plumber" : null, "firehose" : { "type" : "clipped", "interval" : "2016-04-29T08:00:00.000Z/2016-04-29T09:00:00.000Z", "delegate" : { "type" : "timed", "shutoffTime" : "2016-04-29T09:15:00.000Z", "delegate" : { "type" : "receiver", "serviceName" : "firehose:druid:overlord:metrics-kafka-08-0000-0000", "bufferSize" : 100000 } } } }, "tuningConfig" : { "shardSpec" : { "type" : "linear", "partitionNum" : 0 }, "rejectionPolicy" : { "type" : "none" }, "buildV9Directly" : false, "maxPendingPersists" : 0, "intermediatePersistPeriod" : "PT10M", "windowPeriod" : "PT10M", "type" : "realtime", "maxRowsInMemory" : "100000" } } } 2016-04-29 08:46:30,813 [finagle/netty3-1] WARN c.m.tranquility.finagle.FutureRetry$ - Transient error, will try again in 13,860 ms com.metamx.tranquility.druid.IndexServiceTransientException: Service[druid:overlord] call failed with status: 500 Internal Server Error at com.metamx.tranquility.druid.IndexService$$anonfun$call$1$$anonfun$apply$17.apply(IndexService.scala:150) ~[io.druid.tranquility-core-0.7.4.jar:0.7.4] at com.metamx.tranquility.druid.IndexService$$anonfun$call$1$$anonfun$apply$17.apply(IndexService.scala:132) ~[io.druid.tranquility-core-0.7.4.jar:0.7.4] at com.twitter.util.Future$$anonfun$map$1$$anonfun$apply$6.apply(Future.scala:950) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Try$.apply(Try.scala:13) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$.apply(Future.scala:97) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$$anonfun$map$1.apply(Future.scala:950) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$$anonfun$map$1.apply(Future.scala:949) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.liftedTree1$1(Promise.scala:112) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.k(Promise.scala:112) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.apply(Promise.scala:122) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.apply(Promise.scala:103) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$$anon$1.run(Promise.scala:366) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler$Activation.run(Scheduler.scala:178) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler$Activation.submit(Scheduler.scala:136) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler.submit(Scheduler.scala:207) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.Scheduler$.submit(Scheduler.scala:92) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.runq(Promise.scala:350) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.updateIfEmpty(Promise.scala:721) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.update(Promise.scala:694) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.setValue(Promise.scala:670) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.AsyncQueue.offer(AsyncQueue.scala:111) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.finagle.netty3.transport.ChannelTransport.handleUpstream(ChannelTransport.scala:55) [com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.http.HttpContentDecoder.messageReceived(HttpContentDecoder.java:108) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:145) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.http.HttpClientCodec.handleUpstream(HttpClientCodec.java:92) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142) [io.netty.netty-3.10.5.Final.jar:na] at com.twitter.finagle.netty3.channel.ChannelStatsHandler.messageReceived(ChannelStatsHandler.scala:78) [com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0] at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142) [io.netty.netty-3.10.5.Final.jar:na] at com.twitter.finagle.netty3.channel.ChannelRequestStatsHandler.messageReceived(ChannelRequestStatsHandler.scala:35) [com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0] at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) [io.netty.netty-3.10.5.Final.jar:na] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_79] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_79] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79] 2016-04-29 08:46:44,712 [finagle/netty3-2] WARN c.m.tranquility.finagle.FutureRetry$ - Transient error, will try again in 27,515 ms com.metamx.tranquility.druid.IndexServiceTransientException: Service[druid:overlord] call failed with status: 500 Internal Server Error at com.metamx.tranquility.druid.IndexService$$anonfun$call$1$$anonfun$apply$17.apply(IndexService.scala:150) ~[io.druid.tranquility-core-0.7.4.jar:0.7.4] at com.metamx.tranquility.druid.IndexService$$anonfun$call$1$$anonfun$apply$17.apply(IndexService.scala:132) ~[io.druid.tranquility-core-0.7.4.jar:0.7.4] at com.twitter.util.Future$$anonfun$map$1$$anonfun$apply$6.apply(Future.scala:950) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Try$.apply(Try.scala:13) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$.apply(Future.scala:97) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$$anonfun$map$1.apply(Future.scala:950) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$$anonfun$map$1.apply(Future.scala:949) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.liftedTree1$1(Promise.scala:112) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.k(Promise.scala:112) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.apply(Promise.scala:122) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.apply(Promise.scala:103) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$$anon$1.run(Promise.scala:366) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler$Activation.run(Scheduler.scala:178) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler$Activation.submit(Scheduler.scala:136) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler.submit(Scheduler.scala:207) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.Scheduler$.submit(Scheduler.scala:92) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.runq(Promise.scala:350) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.updateIfEmpty(Promise.scala:721) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.update(Promise.scala:694) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.setValue(Promise.scala:670) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.AsyncQueue.offer(AsyncQueue.scala:111) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.finagle.netty3.transport.ChannelTransport.handleUpstream(ChannelTransport.scala:55) [com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.http.HttpContentDecoder.messageReceived(HttpContentDecoder.java:108) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:145) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.http.HttpClientCodec.handleUpstream(HttpClientCodec.java:92) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142) [io.netty.netty-3.10.5.Final.jar:na] at com.twitter.finagle.netty3.channel.ChannelStatsHandler.messageReceived(ChannelStatsHandler.scala:78) [com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0] at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142) [io.netty.netty-3.10.5.Final.jar:na] at com.twitter.finagle.netty3.channel.ChannelRequestStatsHandler.messageReceived(ChannelRequestStatsHandler.scala:35) [com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0] at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) [io.netty.netty-3.10.5.Final.jar:na] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_79] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_79] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79] 2016-04-29 08:47:12,255 [finagle/netty3-3] WARN c.m.tranquility.finagle.FutureRetry$ - Transient error, will try again in 47,428 ms com.metamx.tranquility.druid.IndexServiceTransientException: Service[druid:overlord] call failed with status: 500 Internal Server Error at com.metamx.tranquility.druid.IndexService$$anonfun$call$1$$anonfun$apply$17.apply(IndexService.scala:150) ~[io.druid.tranquility-core-0.7.4.jar:0.7.4] at com.metamx.tranquility.druid.IndexService$$anonfun$call$1$$anonfun$apply$17.apply(IndexService.scala:132) ~[io.druid.tranquility-core-0.7.4.jar:0.7.4] at com.twitter.util.Future$$anonfun$map$1$$anonfun$apply$6.apply(Future.scala:950) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Try$.apply(Try.scala:13) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$.apply(Future.scala:97) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$$anonfun$map$1.apply(Future.scala:950) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$$anonfun$map$1.apply(Future.scala:949) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.liftedTree1$1(Promise.scala:112) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.k(Promise.scala:112) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.apply(Promise.scala:122) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.apply(Promise.scala:103) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$$anon$1.run(Promise.scala:366) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler$Activation.run(Scheduler.scala:178) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler$Activation.submit(Scheduler.scala:136) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler.submit(Scheduler.scala:207) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.Scheduler$.submit(Scheduler.scala:92) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.runq(Promise.scala:350) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.updateIfEmpty(Promise.scala:721) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.update(Promise.scala:694) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.setValue(Promise.scala:670) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.AsyncQueue.offer(AsyncQueue.scala:111) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.finagle.netty3.transport.ChannelTransport.handleUpstream(ChannelTransport.scala:55) [com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.http.HttpContentDecoder.messageReceived(HttpContentDecoder.java:108) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:145) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.http.HttpClientCodec.handleUpstream(HttpClientCodec.java:92) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142) [io.netty.netty-3.10.5.Final.jar:na] at com.twitter.finagle.netty3.channel.ChannelStatsHandler.messageReceived(ChannelStatsHandler.scala:78) [com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0] at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142) [io.netty.netty-3.10.5.Final.jar:na] at com.twitter.finagle.netty3.channel.ChannelRequestStatsHandler.messageReceived(ChannelRequestStatsHandler.scala:35) [com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0] at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) [io.netty.netty-3.10.5.Final.jar:na] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_79] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_79] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79] 2016-04-29 08:47:59,702 [ClusteredBeam-ZkFuturePool-1e419b48-1aa3-4cc4-9ba9-15dadf5e53e6] ERROR c.m.tranquility.beam.ClusteredBeam - Failed to update cluster state: druid:overlord/metrics-kafka com.metamx.tranquility.druid.IndexServiceTransientException: Service[druid:overlord] call failed with status: 500 Internal Server Error at com.metamx.tranquility.druid.IndexService$$anonfun$call$1$$anonfun$apply$17.apply(IndexService.scala:150) ~[io.druid.tranquility-core-0.7.4.jar:0.7.4] at com.metamx.tranquility.druid.IndexService$$anonfun$call$1$$anonfun$apply$17.apply(IndexService.scala:132) ~[io.druid.tranquility-core-0.7.4.jar:0.7.4] at com.twitter.util.Future$$anonfun$map$1$$anonfun$apply$6.apply(Future.scala:950) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Try$.apply(Try.scala:13) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$.apply(Future.scala:97) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$$anonfun$map$1.apply(Future.scala:950) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$$anonfun$map$1.apply(Future.scala:949) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.liftedTree1$1(Promise.scala:112) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.k(Promise.scala:112) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.apply(Promise.scala:122) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.apply(Promise.scala:103) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$$anon$1.run(Promise.scala:366) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler$Activation.run(Scheduler.scala:178) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler$Activation.submit(Scheduler.scala:136) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler.submit(Scheduler.scala:207) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.Scheduler$.submit(Scheduler.scala:92) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.runq(Promise.scala:350) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.updateIfEmpty(Promise.scala:721) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.update(Promise.scala:694) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.setValue(Promise.scala:670) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.AsyncQueue.offer(AsyncQueue.scala:111) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.finagle.netty3.transport.ChannelTransport.handleUpstream(ChannelTransport.scala:55) ~[com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.http.HttpContentDecoder.messageReceived(HttpContentDecoder.java:108) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:145) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.http.HttpClientCodec.handleUpstream(HttpClientCodec.java:92) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142) ~[io.netty.netty-3.10.5.Final.jar:na] at com.twitter.finagle.netty3.channel.ChannelStatsHandler.messageReceived(ChannelStatsHandler.scala:78) ~[com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0] at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142) ~[io.netty.netty-3.10.5.Final.jar:na] at com.twitter.finagle.netty3.channel.ChannelRequestStatsHandler.messageReceived(ChannelRequestStatsHandler.scala:35) ~[com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0] at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) ~[io.netty.netty-3.10.5.Final.jar:na] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_79] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_79] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79] 2016-04-29 08:47:59,708 [ClusteredBeam-ZkFuturePool-1e419b48-1aa3-4cc4-9ba9-15dadf5e53e6] WARN c.m.tranquility.beam.ClusteredBeam - Emitting alert: [anomaly] Failed to create merged beam: druid:overlord/metrics-kafka { } java.lang.IllegalStateException: Failed to save new beam for identifier[druid:overlord/metrics-kafka] timestamp[2016-04-29T08:00:00.000Z] at com.metamx.tranquility.beam.ClusteredBeam$$anonfun$2.applyOrElse(ClusteredBeam.scala:286) ~[io.druid.tranquility-core-0.7.4.jar:0.7.4] at com.metamx.tranquility.beam.ClusteredBeam$$anonfun$2.applyOrElse(ClusteredBeam.scala:283) ~[io.druid.tranquility-core-0.7.4.jar:0.7.4] at com.twitter.util.Future$$anonfun$rescue$1.apply(Future.scala:924) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$$anonfun$rescue$1.apply(Future.scala:922) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.liftedTree1$1(Promise.scala:112) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.k(Promise.scala:112) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.apply(Promise.scala:122) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.apply(Promise.scala:103) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$$anon$1.run(Promise.scala:366) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler$Activation.run(Scheduler.scala:178) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler$Activation.submit(Scheduler.scala:136) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler.submit(Scheduler.scala:207) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.Scheduler$.submit(Scheduler.scala:92) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.runq(Promise.scala:350) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.updateIfEmpty(Promise.scala:721) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.ExecutorServiceFuturePool$$anon$2.run(FuturePool.scala:107) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_79] at java.util.concurrent.FutureTask.run(FutureTask.java:262) [na:1.7.0_79] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_79] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_79] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79] Caused by: com.metamx.tranquility.druid.IndexServiceTransientException: Service[druid:overlord] call failed with status: 500 Internal Server Error at com.metamx.tranquility.druid.IndexService$$anonfun$call$1$$anonfun$apply$17.apply(IndexService.scala:150) ~[io.druid.tranquility-core-0.7.4.jar:0.7.4] at com.metamx.tranquility.druid.IndexService$$anonfun$call$1$$anonfun$apply$17.apply(IndexService.scala:132) ~[io.druid.tranquility-core-0.7.4.jar:0.7.4] at com.twitter.util.Future$$anonfun$map$1$$anonfun$apply$6.apply(Future.scala:950) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Try$.apply(Try.scala:13) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$.apply(Future.scala:97) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$$anonfun$map$1.apply(Future.scala:950) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$$anonfun$map$1.apply(Future.scala:949) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.liftedTree1$1(Promise.scala:112) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.k(Promise.scala:112) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.apply(Promise.scala:122) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.apply(Promise.scala:103) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$$anon$1.run(Promise.scala:366) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler$Activation.run(Scheduler.scala:178) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler$Activation.submit(Scheduler.scala:136) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler.submit(Scheduler.scala:207) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.Scheduler$.submit(Scheduler.scala:92) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.runq(Promise.scala:350) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.updateIfEmpty(Promise.scala:721) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.update(Promise.scala:694) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.setValue(Promise.scala:670) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.AsyncQueue.offer(AsyncQueue.scala:111) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.finagle.netty3.transport.ChannelTransport.handleUpstream(ChannelTransport.scala:55) ~[com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.http.HttpContentDecoder.messageReceived(HttpContentDecoder.java:108) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:145) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.http.HttpClientCodec.handleUpstream(HttpClientCodec.java:92) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142) ~[io.netty.netty-3.10.5.Final.jar:na] at com.twitter.finagle.netty3.channel.ChannelStatsHandler.messageReceived(ChannelStatsHandler.scala:78) ~[com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0] at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142) ~[io.netty.netty-3.10.5.Final.jar:na] at com.twitter.finagle.netty3.channel.ChannelRequestStatsHandler.messageReceived(ChannelRequestStatsHandler.scala:35) ~[com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0] at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) ~[io.netty.netty-3.10.5.Final.jar:na] ... 3 common frames omitted 2016-04-29 08:47:59,710 [ClusteredBeam-ZkFuturePool-1e419b48-1aa3-4cc4-9ba9-15dadf5e53e6] INFO c.metamx.emitter.core.LoggingEmitter - Event [{"feed":"alerts","timestamp":"2016-04-29T08:47:59.710Z","service":"tranquility","host":"localhost","severity":"anomaly","description":"Failed to create merged beam: druid:overlord/metrics-kafka","data":{"exceptionType":"java.lang.IllegalStateException","exceptionMessage":"Failed to save new beam for identifier[druid:overlord/metrics-kafka] timestamp[2016-04-29T08:00:00.000Z]","exceptionStackTrace":"java.lang.IllegalStateException: Failed to save new beam for identifier[druid:overlord/metrics-kafka] timestamp[2016-04-29T08:00:00.000Z]\n\tat com.metamx.tranquility.beam.ClusteredBeam$$anonfun$2.applyOrElse(ClusteredBeam.scala:286)\n\tat com.metamx.tranquility.beam.ClusteredBeam$$anonfun$2.applyOrElse(ClusteredBeam.scala:283)\n\tat com.twitter.util.Future$$anonfun$rescue$1.apply(Future.scala:924)\n\tat com.twitter.util.Future$$anonfun$rescue$1.apply(Future.scala:922)\n\tat com.twitter.util.Promise$Transformer.liftedTree1$1(Promise.scala:112)\n\tat com.twitter.util.Promise$Transformer.k(Promise.scala:112)\n\tat com.twitter.util.Promise$Transformer.apply(Promise.scala:122)\n\tat com.twitter.util.Promise$Transformer.apply(Promise.scala:103)\n\tat com.twitter.util.Promise$$anon$1.run(Promise.scala:366)\n\tat com.twitter.concurrent.LocalScheduler$Activation.run(Scheduler.scala:178)\n\tat com.twitter.concurrent.LocalScheduler$Activation.submit(Scheduler.scala:136)\n\tat com.twitter.concurrent.LocalScheduler.submit(Scheduler.scala:207)\n\tat com.twitter.concurrent.Scheduler$.submit(Scheduler.scala:92)\n\tat com.twitter.util.Promise.runq(Promise.scala:350)\n\tat com.twitter.util.Promise.updateIfEmpty(Promise.scala:721)\n\tat com.twitter.util.ExecutorServiceFuturePool$$anon$2.run(FuturePool.scala:107)\n\tat java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:262)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)\n\tat java.lang.Thread.run(Thread.java:745)\nCaused by: com.metamx.tranquility.druid.IndexServiceTransientException: Service[druid:overlord] call failed with status: 500 Internal Server Error\n\tat com.metamx.tranquility.druid.IndexService$$anonfun$call$1$$anonfun$apply$17.apply(IndexService.scala:150)\n\tat com.metamx.tranquility.druid.IndexService$$anonfun$call$1$$anonfun$apply$17.apply(IndexService.scala:132)\n\tat com.twitter.util.Future$$anonfun$map$1$$anonfun$apply$6.apply(Future.scala:950)\n\tat com.twitter.util.Try$.apply(Try.scala:13)\n\tat com.twitter.util.Future$.apply(Future.scala:97)\n\tat com.twitter.util.Future$$anonfun$map$1.apply(Future.scala:950)\n\tat com.twitter.util.Future$$anonfun$map$1.apply(Future.scala:949)\n\tat com.twitter.util.Promise$Transformer.liftedTree1$1(Promise.scala:112)\n\tat com.twitter.util.Promise$Transformer.k(Promise.scala:112)\n\tat com.twitter.util.Promise$Transformer.apply(Promise.scala:122)\n\tat com.twitter.util.Promise$Transformer.apply(Promise.scala:103)\n\tat com.twitter.util.Promise$$anon$1.run(Promise.scala:366)\n\tat com.twitter.concurrent.LocalScheduler$Activation.run(Scheduler.scala:178)\n\tat com.twitter.concurrent.LocalScheduler$Activation.submit(Scheduler.scala:136)\n\tat com.twitter.concurrent.LocalScheduler.submit(Scheduler.scala:207)\n\tat com.twitter.concurrent.Scheduler$.submit(Scheduler.scala:92)\n\tat com.twitter.util.Promise.runq(Promise.scala:350)\n\tat com.twitter.util.Promise.updateIfEmpty(Promise.scala:721)\n\tat com.twitter.util.Promise.update(Promise.scala:694)\n\tat com.twitter.util.Promise.setValue(Promise.scala:670)\n\tat com.twitter.concurrent.AsyncQueue.offer(AsyncQueue.scala:111)\n\tat com.twitter.finagle.netty3.transport.ChannelTransport.handleUpstream(ChannelTransport.scala:55)\n\tat org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)\n\tat org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)\n\tat org.jboss.netty.handler.codec.http.HttpContentDecoder.messageReceived(HttpContentDecoder.java:108)\n\tat org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)\n\tat org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)\n\tat org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)\n\tat org.jboss.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:145)\n\tat org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)\n\tat org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)\n\tat org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)\n\tat org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)\n\tat org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459)\n\tat org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536)\n\tat org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435)\n\tat org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)\n\tat org.jboss.netty.handler.codec.http.HttpClientCodec.handleUpstream(HttpClientCodec.java:92)\n\tat org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)\n\tat org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)\n\tat org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142)\n\tat com.twitter.finagle.netty3.channel.ChannelStatsHandler.messageReceived(ChannelStatsHandler.scala:78)\n\tat org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88)\n\tat org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)\n\tat org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)\n\tat org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142)\n\tat com.twitter.finagle.netty3.channel.ChannelRequestStatsHandler.messageReceived(ChannelRequestStatsHandler.scala:35)\n\tat org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88)\n\tat org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)\n\tat org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)\n\tat org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)\n\tat org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)\n\tat org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)\n\tat org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)\n\tat org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)\n\tat org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)\n\tat org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)\n\tat org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)\n\tat org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)\n\t... 3 more\n"}}] 2016-04-29 08:47:59,711 [ClusteredBeam-ZkFuturePool-1e419b48-1aa3-4cc4-9ba9-15dadf5e53e6] WARN c.m.t.tranquilizer.Tranquilizer - Failed to send 1 messages. java.lang.IllegalStateException: Failed to save new beam for identifier[druid:overlord/metrics-kafka] timestamp[2016-04-29T08:00:00.000Z] at com.metamx.tranquility.beam.ClusteredBeam$$anonfun$2.applyOrElse(ClusteredBeam.scala:286) ~[io.druid.tranquility-core-0.7.4.jar:0.7.4] at com.metamx.tranquility.beam.ClusteredBeam$$anonfun$2.applyOrElse(ClusteredBeam.scala:283) ~[io.druid.tranquility-core-0.7.4.jar:0.7.4] at com.twitter.util.Future$$anonfun$rescue$1.apply(Future.scala:924) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$$anonfun$rescue$1.apply(Future.scala:922) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.liftedTree1$1(Promise.scala:112) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.k(Promise.scala:112) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.apply(Promise.scala:122) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.apply(Promise.scala:103) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$$anon$1.run(Promise.scala:366) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler$Activation.run(Scheduler.scala:178) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler$Activation.submit(Scheduler.scala:136) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler.submit(Scheduler.scala:207) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.Scheduler$.submit(Scheduler.scala:92) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.runq(Promise.scala:350) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.updateIfEmpty(Promise.scala:721) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.ExecutorServiceFuturePool$$anon$2.run(FuturePool.scala:107) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_79] at java.util.concurrent.FutureTask.run(FutureTask.java:262) [na:1.7.0_79] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_79] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_79] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79] Caused by: com.metamx.tranquility.druid.IndexServiceTransientException: Service[druid:overlord] call failed with status: 500 Internal Server Error at com.metamx.tranquility.druid.IndexService$$anonfun$call$1$$anonfun$apply$17.apply(IndexService.scala:150) ~[io.druid.tranquility-core-0.7.4.jar:0.7.4] at com.metamx.tranquility.druid.IndexService$$anonfun$call$1$$anonfun$apply$17.apply(IndexService.scala:132) ~[io.druid.tranquility-core-0.7.4.jar:0.7.4] at com.twitter.util.Future$$anonfun$map$1$$anonfun$apply$6.apply(Future.scala:950) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Try$.apply(Try.scala:13) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$.apply(Future.scala:97) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$$anonfun$map$1.apply(Future.scala:950) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$$anonfun$map$1.apply(Future.scala:949) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.liftedTree1$1(Promise.scala:112) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.k(Promise.scala:112) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.apply(Promise.scala:122) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.apply(Promise.scala:103) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$$anon$1.run(Promise.scala:366) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler$Activation.run(Scheduler.scala:178) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler$Activation.submit(Scheduler.scala:136) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler.submit(Scheduler.scala:207) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.Scheduler$.submit(Scheduler.scala:92) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.runq(Promise.scala:350) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.updateIfEmpty(Promise.scala:721) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.update(Promise.scala:694) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.setValue(Promise.scala:670) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.AsyncQueue.offer(AsyncQueue.scala:111) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.finagle.netty3.transport.ChannelTransport.handleUpstream(ChannelTransport.scala:55) ~[com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.http.HttpContentDecoder.messageReceived(HttpContentDecoder.java:108) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:145) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.http.HttpClientCodec.handleUpstream(HttpClientCodec.java:92) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142) ~[io.netty.netty-3.10.5.Final.jar:na] at com.twitter.finagle.netty3.channel.ChannelStatsHandler.messageReceived(ChannelStatsHandler.scala:78) ~[com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0] at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142) ~[io.netty.netty-3.10.5.Final.jar:na] at com.twitter.finagle.netty3.channel.ChannelRequestStatsHandler.messageReceived(ChannelRequestStatsHandler.scala:35) ~[com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0] at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) ~[io.netty.netty-3.10.5.Final.jar:na] ... 3 common frames omitted 2016-04-29 08:47:59,723 [ClusteredBeam-ZkFuturePool-1e419b48-1aa3-4cc4-9ba9-15dadf5e53e6] INFO c.m.tranquility.beam.ClusteredBeam - Creating new merged beam for identifier[druid:overlord/metrics-kafka] timestamp[2016-04-29T08:00:00.000Z] (target = 1, actual = 0) 2016-04-29 08:47:59,730 [ClusteredBeam-ZkFuturePool-1e419b48-1aa3-4cc4-9ba9-15dadf5e53e6] INFO com.metamx.common.scala.control$ - Creating druid indexing task (service = druid:overlord): { "type" : "index_realtime", "id" : "index_realtime_metrics-kafka_2016-04-29T08:00:00.000Z_0_0", "resource" : { "availabilityGroup" : "metrics-kafka-08-0000", "requiredCapacity" : 1 }, "spec" : { "dataSchema" : { "dataSource" : "metrics-kafka", "parser" : { "type" : "map", "parseSpec" : { "format" : "json", "timestampSpec" : { "column" : "timestamp", "format" : "auto", "missingValue" : null }, "dimensionsSpec" : { "dimensions" : [ "http_code", "http_method", "metricType", "page", "server", "timestamp", "unit", "value" ], "spatialDimensions" : [ ] } } }, "metricsSpec" : [ { "type" : "count", "name" : "count" }, { "type" : "doubleSum", "name" : "value_sum", "fieldName" : "value" }, { "type" : "doubleMin", "name" : "value_min", "fieldName" : "value" }, { "type" : "doubleMax", "name" : "value_max", "fieldName" : "value" } ], "granularitySpec" : { "type" : "uniform", "segmentGranularity" : "HOUR", "queryGranularity" : { "type" : "none" } } }, "ioConfig" : { "type" : "realtime", "plumber" : null, "firehose" : { "type" : "clipped", "interval" : "2016-04-29T08:00:00.000Z/2016-04-29T09:00:00.000Z", "delegate" : { "type" : "timed", "shutoffTime" : "2016-04-29T09:15:00.000Z", "delegate" : { "type" : "receiver", "serviceName" : "firehose:druid:overlord:metrics-kafka-08-0000-0000", "bufferSize" : 100000 } } } }, "tuningConfig" : { "shardSpec" : { "type" : "linear", "partitionNum" : 0 }, "rejectionPolicy" : { "type" : "none" }, "buildV9Directly" : false, "maxPendingPersists" : 0, "intermediatePersistPeriod" : "PT10M", "windowPeriod" : "PT10M", "type" : "realtime", "maxRowsInMemory" : "100000" } } } 2016-04-29 08:47:59,748 [finagle/netty3-1] WARN c.m.tranquility.finagle.FutureRetry$ - Transient error, will try again in 12,458 ms com.metamx.tranquility.druid.IndexServiceTransientException: Service[druid:overlord] call failed with status: 500 Internal Server Error at com.metamx.tranquility.druid.IndexService$$anonfun$call$1$$anonfun$apply$17.apply(IndexService.scala:150) ~[io.druid.tranquility-core-0.7.4.jar:0.7.4] at com.metamx.tranquility.druid.IndexService$$anonfun$call$1$$anonfun$apply$17.apply(IndexService.scala:132) ~[io.druid.tranquility-core-0.7.4.jar:0.7.4] at com.twitter.util.Future$$anonfun$map$1$$anonfun$apply$6.apply(Future.scala:950) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Try$.apply(Try.scala:13) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$.apply(Future.scala:97) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$$anonfun$map$1.apply(Future.scala:950) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$$anonfun$map$1.apply(Future.scala:949) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.liftedTree1$1(Promise.scala:112) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.k(Promise.scala:112) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.apply(Promise.scala:122) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.apply(Promise.scala:103) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$$anon$1.run(Promise.scala:366) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler$Activation.run(Scheduler.scala:178) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler$Activation.submit(Scheduler.scala:136) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler.submit(Scheduler.scala:207) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.Scheduler$.submit(Scheduler.scala:92) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.runq(Promise.scala:350) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.updateIfEmpty(Promise.scala:721) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.update(Promise.scala:694) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.setValue(Promise.scala:670) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.AsyncQueue.offer(AsyncQueue.scala:111) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.finagle.netty3.transport.ChannelTransport.handleUpstream(ChannelTransport.scala:55) [com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.http.HttpContentDecoder.messageReceived(HttpContentDecoder.java:108) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:145) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.http.HttpClientCodec.handleUpstream(HttpClientCodec.java:92) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142) [io.netty.netty-3.10.5.Final.jar:na] at com.twitter.finagle.netty3.channel.ChannelStatsHandler.messageReceived(ChannelStatsHandler.scala:78) [com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0] at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142) [io.netty.netty-3.10.5.Final.jar:na] at com.twitter.finagle.netty3.channel.ChannelRequestStatsHandler.messageReceived(ChannelRequestStatsHandler.scala:35) [com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0] at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) [io.netty.netty-3.10.5.Final.jar:na] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_79] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_79] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79] 2016-04-29 08:48:12,237 [finagle/netty3-2] WARN c.m.tranquility.finagle.FutureRetry$ - Transient error, will try again in 28,934 ms com.metamx.tranquility.druid.IndexServiceTransientException: Service[druid:overlord] call failed with status: 500 Internal Server Error at com.metamx.tranquility.druid.IndexService$$anonfun$call$1$$anonfun$apply$17.apply(IndexService.scala:150) ~[io.druid.tranquility-core-0.7.4.jar:0.7.4] at com.metamx.tranquility.druid.IndexService$$anonfun$call$1$$anonfun$apply$17.apply(IndexService.scala:132) ~[io.druid.tranquility-core-0.7.4.jar:0.7.4] at com.twitter.util.Future$$anonfun$map$1$$anonfun$apply$6.apply(Future.scala:950) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Try$.apply(Try.scala:13) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$.apply(Future.scala:97) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$$anonfun$map$1.apply(Future.scala:950) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$$anonfun$map$1.apply(Future.scala:949) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.liftedTree1$1(Promise.scala:112) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.k(Promise.scala:112) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.apply(Promise.scala:122) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.apply(Promise.scala:103) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$$anon$1.run(Promise.scala:366) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler$Activation.run(Scheduler.scala:178) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler$Activation.submit(Scheduler.scala:136) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler.submit(Scheduler.scala:207) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.Scheduler$.submit(Scheduler.scala:92) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.runq(Promise.scala:350) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.updateIfEmpty(Promise.scala:721) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.update(Promise.scala:694) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.setValue(Promise.scala:670) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.AsyncQueue.offer(AsyncQueue.scala:111) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.finagle.netty3.transport.ChannelTransport.handleUpstream(ChannelTransport.scala:55) [com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.http.HttpContentDecoder.messageReceived(HttpContentDecoder.java:108) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:145) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.http.HttpClientCodec.handleUpstream(HttpClientCodec.java:92) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142) [io.netty.netty-3.10.5.Final.jar:na] at com.twitter.finagle.netty3.channel.ChannelStatsHandler.messageReceived(ChannelStatsHandler.scala:78) [com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0] at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142) [io.netty.netty-3.10.5.Final.jar:na] at com.twitter.finagle.netty3.channel.ChannelRequestStatsHandler.messageReceived(ChannelRequestStatsHandler.scala:35) [com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0] at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) [io.netty.netty-3.10.5.Final.jar:na] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_79] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_79] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79] 2016-04-29 08:48:41,195 [finagle/netty3-3] WARN c.m.tranquility.finagle.FutureRetry$ - Transient error, will try again in 57,440 ms com.metamx.tranquility.druid.IndexServiceTransientException: Service[druid:overlord] call failed with status: 500 Internal Server Error at com.metamx.tranquility.druid.IndexService$$anonfun$call$1$$anonfun$apply$17.apply(IndexService.scala:150) ~[io.druid.tranquility-core-0.7.4.jar:0.7.4] at com.metamx.tranquility.druid.IndexService$$anonfun$call$1$$anonfun$apply$17.apply(IndexService.scala:132) ~[io.druid.tranquility-core-0.7.4.jar:0.7.4] at com.twitter.util.Future$$anonfun$map$1$$anonfun$apply$6.apply(Future.scala:950) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Try$.apply(Try.scala:13) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$.apply(Future.scala:97) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$$anonfun$map$1.apply(Future.scala:950) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$$anonfun$map$1.apply(Future.scala:949) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.liftedTree1$1(Promise.scala:112) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.k(Promise.scala:112) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.apply(Promise.scala:122) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.apply(Promise.scala:103) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$$anon$1.run(Promise.scala:366) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler$Activation.run(Scheduler.scala:178) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler$Activation.submit(Scheduler.scala:136) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler.submit(Scheduler.scala:207) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.Scheduler$.submit(Scheduler.scala:92) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.runq(Promise.scala:350) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.updateIfEmpty(Promise.scala:721) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.update(Promise.scala:694) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.setValue(Promise.scala:670) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.AsyncQueue.offer(AsyncQueue.scala:111) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.finagle.netty3.transport.ChannelTransport.handleUpstream(ChannelTransport.scala:55) [com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.http.HttpContentDecoder.messageReceived(HttpContentDecoder.java:108) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:145) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.http.HttpClientCodec.handleUpstream(HttpClientCodec.java:92) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142) [io.netty.netty-3.10.5.Final.jar:na] at com.twitter.finagle.netty3.channel.ChannelStatsHandler.messageReceived(ChannelStatsHandler.scala:78) [com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0] at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142) [io.netty.netty-3.10.5.Final.jar:na] at com.twitter.finagle.netty3.channel.ChannelRequestStatsHandler.messageReceived(ChannelRequestStatsHandler.scala:35) [com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0] at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) [io.netty.netty-3.10.5.Final.jar:na] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_79] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_79] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79] 2016-04-29 08:49:38,659 [ClusteredBeam-ZkFuturePool-1e419b48-1aa3-4cc4-9ba9-15dadf5e53e6] ERROR c.m.tranquility.beam.ClusteredBeam - Failed to update cluster state: druid:overlord/metrics-kafka com.metamx.tranquility.druid.IndexServiceTransientException: Service[druid:overlord] call failed with status: 500 Internal Server Error at com.metamx.tranquility.druid.IndexService$$anonfun$call$1$$anonfun$apply$17.apply(IndexService.scala:150) ~[io.druid.tranquility-core-0.7.4.jar:0.7.4] at com.metamx.tranquility.druid.IndexService$$anonfun$call$1$$anonfun$apply$17.apply(IndexService.scala:132) ~[io.druid.tranquility-core-0.7.4.jar:0.7.4] at com.twitter.util.Future$$anonfun$map$1$$anonfun$apply$6.apply(Future.scala:950) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Try$.apply(Try.scala:13) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$.apply(Future.scala:97) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$$anonfun$map$1.apply(Future.scala:950) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$$anonfun$map$1.apply(Future.scala:949) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.liftedTree1$1(Promise.scala:112) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.k(Promise.scala:112) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.apply(Promise.scala:122) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.apply(Promise.scala:103) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$$anon$1.run(Promise.scala:366) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler$Activation.run(Scheduler.scala:178) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler$Activation.submit(Scheduler.scala:136) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler.submit(Scheduler.scala:207) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.Scheduler$.submit(Scheduler.scala:92) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.runq(Promise.scala:350) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.updateIfEmpty(Promise.scala:721) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.update(Promise.scala:694) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.setValue(Promise.scala:670) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.AsyncQueue.offer(AsyncQueue.scala:111) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.finagle.netty3.transport.ChannelTransport.handleUpstream(ChannelTransport.scala:55) ~[com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.http.HttpContentDecoder.messageReceived(HttpContentDecoder.java:108) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:145) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.http.HttpClientCodec.handleUpstream(HttpClientCodec.java:92) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142) ~[io.netty.netty-3.10.5.Final.jar:na] at com.twitter.finagle.netty3.channel.ChannelStatsHandler.messageReceived(ChannelStatsHandler.scala:78) ~[com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0] at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142) ~[io.netty.netty-3.10.5.Final.jar:na] at com.twitter.finagle.netty3.channel.ChannelRequestStatsHandler.messageReceived(ChannelRequestStatsHandler.scala:35) ~[com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0] at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) ~[io.netty.netty-3.10.5.Final.jar:na] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_79] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_79] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79] 2016-04-29 08:49:38,664 [ClusteredBeam-ZkFuturePool-1e419b48-1aa3-4cc4-9ba9-15dadf5e53e6] WARN c.m.tranquility.beam.ClusteredBeam - Emitting alert: [anomaly] Failed to create merged beam: druid:overlord/metrics-kafka { } java.lang.IllegalStateException: Failed to save new beam for identifier[druid:overlord/metrics-kafka] timestamp[2016-04-29T08:00:00.000Z] at com.metamx.tranquility.beam.ClusteredBeam$$anonfun$2.applyOrElse(ClusteredBeam.scala:286) ~[io.druid.tranquility-core-0.7.4.jar:0.7.4] at com.metamx.tranquility.beam.ClusteredBeam$$anonfun$2.applyOrElse(ClusteredBeam.scala:283) ~[io.druid.tranquility-core-0.7.4.jar:0.7.4] at com.twitter.util.Future$$anonfun$rescue$1.apply(Future.scala:924) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$$anonfun$rescue$1.apply(Future.scala:922) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.liftedTree1$1(Promise.scala:112) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.k(Promise.scala:112) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.apply(Promise.scala:122) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.apply(Promise.scala:103) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$$anon$1.run(Promise.scala:366) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler$Activation.run(Scheduler.scala:178) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler$Activation.submit(Scheduler.scala:136) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler.submit(Scheduler.scala:207) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.Scheduler$.submit(Scheduler.scala:92) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.runq(Promise.scala:350) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.updateIfEmpty(Promise.scala:721) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.ExecutorServiceFuturePool$$anon$2.run(FuturePool.scala:107) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_79] at java.util.concurrent.FutureTask.run(FutureTask.java:262) [na:1.7.0_79] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_79] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_79] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79] Caused by: com.metamx.tranquility.druid.IndexServiceTransientException: Service[druid:overlord] call failed with status: 500 Internal Server Error at com.metamx.tranquility.druid.IndexService$$anonfun$call$1$$anonfun$apply$17.apply(IndexService.scala:150) ~[io.druid.tranquility-core-0.7.4.jar:0.7.4] at com.metamx.tranquility.druid.IndexService$$anonfun$call$1$$anonfun$apply$17.apply(IndexService.scala:132) ~[io.druid.tranquility-core-0.7.4.jar:0.7.4] at com.twitter.util.Future$$anonfun$map$1$$anonfun$apply$6.apply(Future.scala:950) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Try$.apply(Try.scala:13) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$.apply(Future.scala:97) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$$anonfun$map$1.apply(Future.scala:950) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$$anonfun$map$1.apply(Future.scala:949) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.liftedTree1$1(Promise.scala:112) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.k(Promise.scala:112) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.apply(Promise.scala:122) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.apply(Promise.scala:103) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$$anon$1.run(Promise.scala:366) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler$Activation.run(Scheduler.scala:178) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler$Activation.submit(Scheduler.scala:136) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler.submit(Scheduler.scala:207) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.Scheduler$.submit(Scheduler.scala:92) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.runq(Promise.scala:350) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.updateIfEmpty(Promise.scala:721) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.update(Promise.scala:694) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.setValue(Promise.scala:670) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.AsyncQueue.offer(AsyncQueue.scala:111) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.finagle.netty3.transport.ChannelTransport.handleUpstream(ChannelTransport.scala:55) ~[com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.http.HttpContentDecoder.messageReceived(HttpContentDecoder.java:108) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:145) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.http.HttpClientCodec.handleUpstream(HttpClientCodec.java:92) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142) ~[io.netty.netty-3.10.5.Final.jar:na] at com.twitter.finagle.netty3.channel.ChannelStatsHandler.messageReceived(ChannelStatsHandler.scala:78) ~[com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0] at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142) ~[io.netty.netty-3.10.5.Final.jar:na] at com.twitter.finagle.netty3.channel.ChannelRequestStatsHandler.messageReceived(ChannelRequestStatsHandler.scala:35) ~[com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0] at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) ~[io.netty.netty-3.10.5.Final.jar:na] ... 3 common frames omitted 2016-04-29 08:49:38,666 [ClusteredBeam-ZkFuturePool-1e419b48-1aa3-4cc4-9ba9-15dadf5e53e6] INFO c.metamx.emitter.core.LoggingEmitter - Event [{"feed":"alerts","timestamp":"2016-04-29T08:49:38.665Z","service":"tranquility","host":"localhost","severity":"anomaly","description":"Failed to create merged beam: druid:overlord/metrics-kafka","data":{"exceptionType":"java.lang.IllegalStateException","exceptionMessage":"Failed to save new beam for identifier[druid:overlord/metrics-kafka] timestamp[2016-04-29T08:00:00.000Z]","exceptionStackTrace":"java.lang.IllegalStateException: Failed to save new beam for identifier[druid:overlord/metrics-kafka] timestamp[2016-04-29T08:00:00.000Z]\n\tat com.metamx.tranquility.beam.ClusteredBeam$$anonfun$2.applyOrElse(ClusteredBeam.scala:286)\n\tat com.metamx.tranquility.beam.ClusteredBeam$$anonfun$2.applyOrElse(ClusteredBeam.scala:283)\n\tat com.twitter.util.Future$$anonfun$rescue$1.apply(Future.scala:924)\n\tat com.twitter.util.Future$$anonfun$rescue$1.apply(Future.scala:922)\n\tat com.twitter.util.Promise$Transformer.liftedTree1$1(Promise.scala:112)\n\tat com.twitter.util.Promise$Transformer.k(Promise.scala:112)\n\tat com.twitter.util.Promise$Transformer.apply(Promise.scala:122)\n\tat com.twitter.util.Promise$Transformer.apply(Promise.scala:103)\n\tat com.twitter.util.Promise$$anon$1.run(Promise.scala:366)\n\tat com.twitter.concurrent.LocalScheduler$Activation.run(Scheduler.scala:178)\n\tat com.twitter.concurrent.LocalScheduler$Activation.submit(Scheduler.scala:136)\n\tat com.twitter.concurrent.LocalScheduler.submit(Scheduler.scala:207)\n\tat com.twitter.concurrent.Scheduler$.submit(Scheduler.scala:92)\n\tat com.twitter.util.Promise.runq(Promise.scala:350)\n\tat com.twitter.util.Promise.updateIfEmpty(Promise.scala:721)\n\tat com.twitter.util.ExecutorServiceFuturePool$$anon$2.run(FuturePool.scala:107)\n\tat java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:262)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)\n\tat java.lang.Thread.run(Thread.java:745)\nCaused by: com.metamx.tranquility.druid.IndexServiceTransientException: Service[druid:overlord] call failed with status: 500 Internal Server Error\n\tat com.metamx.tranquility.druid.IndexService$$anonfun$call$1$$anonfun$apply$17.apply(IndexService.scala:150)\n\tat com.metamx.tranquility.druid.IndexService$$anonfun$call$1$$anonfun$apply$17.apply(IndexService.scala:132)\n\tat com.twitter.util.Future$$anonfun$map$1$$anonfun$apply$6.apply(Future.scala:950)\n\tat com.twitter.util.Try$.apply(Try.scala:13)\n\tat com.twitter.util.Future$.apply(Future.scala:97)\n\tat com.twitter.util.Future$$anonfun$map$1.apply(Future.scala:950)\n\tat com.twitter.util.Future$$anonfun$map$1.apply(Future.scala:949)\n\tat com.twitter.util.Promise$Transformer.liftedTree1$1(Promise.scala:112)\n\tat com.twitter.util.Promise$Transformer.k(Promise.scala:112)\n\tat com.twitter.util.Promise$Transformer.apply(Promise.scala:122)\n\tat com.twitter.util.Promise$Transformer.apply(Promise.scala:103)\n\tat com.twitter.util.Promise$$anon$1.run(Promise.scala:366)\n\tat com.twitter.concurrent.LocalScheduler$Activation.run(Scheduler.scala:178)\n\tat com.twitter.concurrent.LocalScheduler$Activation.submit(Scheduler.scala:136)\n\tat com.twitter.concurrent.LocalScheduler.submit(Scheduler.scala:207)\n\tat com.twitter.concurrent.Scheduler$.submit(Scheduler.scala:92)\n\tat com.twitter.util.Promise.runq(Promise.scala:350)\n\tat com.twitter.util.Promise.updateIfEmpty(Promise.scala:721)\n\tat com.twitter.util.Promise.update(Promise.scala:694)\n\tat com.twitter.util.Promise.setValue(Promise.scala:670)\n\tat com.twitter.concurrent.AsyncQueue.offer(AsyncQueue.scala:111)\n\tat com.twitter.finagle.netty3.transport.ChannelTransport.handleUpstream(ChannelTransport.scala:55)\n\tat org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)\n\tat org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)\n\tat org.jboss.netty.handler.codec.http.HttpContentDecoder.messageReceived(HttpContentDecoder.java:108)\n\tat org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)\n\tat org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)\n\tat org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)\n\tat org.jboss.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:145)\n\tat org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)\n\tat org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)\n\tat org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)\n\tat org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)\n\tat org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459)\n\tat org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536)\n\tat org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435)\n\tat org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)\n\tat org.jboss.netty.handler.codec.http.HttpClientCodec.handleUpstream(HttpClientCodec.java:92)\n\tat org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)\n\tat org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)\n\tat org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142)\n\tat com.twitter.finagle.netty3.channel.ChannelStatsHandler.messageReceived(ChannelStatsHandler.scala:78)\n\tat org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88)\n\tat org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)\n\tat org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)\n\tat org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142)\n\tat com.twitter.finagle.netty3.channel.ChannelRequestStatsHandler.messageReceived(ChannelRequestStatsHandler.scala:35)\n\tat org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88)\n\tat org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)\n\tat org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)\n\tat org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)\n\tat org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)\n\tat org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)\n\tat org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)\n\tat org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)\n\tat org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)\n\tat org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)\n\tat org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)\n\tat org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)\n\t... 3 more\n"}}] 2016-04-29 08:49:38,667 [ClusteredBeam-ZkFuturePool-1e419b48-1aa3-4cc4-9ba9-15dadf5e53e6] WARN c.m.t.tranquilizer.Tranquilizer - Failed to send 1 messages. java.lang.IllegalStateException: Failed to save new beam for identifier[druid:overlord/metrics-kafka] timestamp[2016-04-29T08:00:00.000Z] at com.metamx.tranquility.beam.ClusteredBeam$$anonfun$2.applyOrElse(ClusteredBeam.scala:286) ~[io.druid.tranquility-core-0.7.4.jar:0.7.4] at com.metamx.tranquility.beam.ClusteredBeam$$anonfun$2.applyOrElse(ClusteredBeam.scala:283) ~[io.druid.tranquility-core-0.7.4.jar:0.7.4] at com.twitter.util.Future$$anonfun$rescue$1.apply(Future.scala:924) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$$anonfun$rescue$1.apply(Future.scala:922) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.liftedTree1$1(Promise.scala:112) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.k(Promise.scala:112) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.apply(Promise.scala:122) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.apply(Promise.scala:103) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$$anon$1.run(Promise.scala:366) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler$Activation.run(Scheduler.scala:178) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler$Activation.submit(Scheduler.scala:136) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler.submit(Scheduler.scala:207) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.Scheduler$.submit(Scheduler.scala:92) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.runq(Promise.scala:350) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.updateIfEmpty(Promise.scala:721) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.ExecutorServiceFuturePool$$anon$2.run(FuturePool.scala:107) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_79] at java.util.concurrent.FutureTask.run(FutureTask.java:262) [na:1.7.0_79] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_79] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_79] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79] Caused by: com.metamx.tranquility.druid.IndexServiceTransientException: Service[druid:overlord] call failed with status: 500 Internal Server Error at com.metamx.tranquility.druid.IndexService$$anonfun$call$1$$anonfun$apply$17.apply(IndexService.scala:150) ~[io.druid.tranquility-core-0.7.4.jar:0.7.4] at com.metamx.tranquility.druid.IndexService$$anonfun$call$1$$anonfun$apply$17.apply(IndexService.scala:132) ~[io.druid.tranquility-core-0.7.4.jar:0.7.4] at com.twitter.util.Future$$anonfun$map$1$$anonfun$apply$6.apply(Future.scala:950) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Try$.apply(Try.scala:13) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$.apply(Future.scala:97) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$$anonfun$map$1.apply(Future.scala:950) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$$anonfun$map$1.apply(Future.scala:949) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.liftedTree1$1(Promise.scala:112) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.k(Promise.scala:112) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.apply(Promise.scala:122) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.apply(Promise.scala:103) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$$anon$1.run(Promise.scala:366) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler$Activation.run(Scheduler.scala:178) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler$Activation.submit(Scheduler.scala:136) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler.submit(Scheduler.scala:207) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.Scheduler$.submit(Scheduler.scala:92) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.runq(Promise.scala:350) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.updateIfEmpty(Promise.scala:721) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.update(Promise.scala:694) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.setValue(Promise.scala:670) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.AsyncQueue.offer(AsyncQueue.scala:111) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.finagle.netty3.transport.ChannelTransport.handleUpstream(ChannelTransport.scala:55) ~[com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.http.HttpContentDecoder.messageReceived(HttpContentDecoder.java:108) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:145) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.http.HttpClientCodec.handleUpstream(HttpClientCodec.java:92) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142) ~[io.netty.netty-3.10.5.Final.jar:na] at com.twitter.finagle.netty3.channel.ChannelStatsHandler.messageReceived(ChannelStatsHandler.scala:78) ~[com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0] at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142) ~[io.netty.netty-3.10.5.Final.jar:na] at com.twitter.finagle.netty3.channel.ChannelRequestStatsHandler.messageReceived(ChannelRequestStatsHandler.scala:35) ~[com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0] at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) ~[io.netty.netty-3.10.5.Final.jar:na] ... 3 common frames omitted 2016-04-29 08:49:38,677 [ClusteredBeam-ZkFuturePool-1e419b48-1aa3-4cc4-9ba9-15dadf5e53e6] INFO c.m.tranquility.beam.ClusteredBeam - Creating new merged beam for identifier[druid:overlord/metrics-kafka] timestamp[2016-04-29T08:00:00.000Z] (target = 1, actual = 0) 2016-04-29 08:49:38,684 [ClusteredBeam-ZkFuturePool-1e419b48-1aa3-4cc4-9ba9-15dadf5e53e6] INFO com.metamx.common.scala.control$ - Creating druid indexing task (service = druid:overlord): { "type" : "index_realtime", "id" : "index_realtime_metrics-kafka_2016-04-29T08:00:00.000Z_0_0", "resource" : { "availabilityGroup" : "metrics-kafka-08-0000", "requiredCapacity" : 1 }, "spec" : { "dataSchema" : { "dataSource" : "metrics-kafka", "parser" : { "type" : "map", "parseSpec" : { "format" : "json", "timestampSpec" : { "column" : "timestamp", "format" : "auto", "missingValue" : null }, "dimensionsSpec" : { "dimensions" : [ "http_code", "http_method", "metricType", "page", "server", "timestamp", "unit", "value" ], "spatialDimensions" : [ ] } } }, "metricsSpec" : [ { "type" : "count", "name" : "count" }, { "type" : "doubleSum", "name" : "value_sum", "fieldName" : "value" }, { "type" : "doubleMin", "name" : "value_min", "fieldName" : "value" }, { "type" : "doubleMax", "name" : "value_max", "fieldName" : "value" } ], "granularitySpec" : { "type" : "uniform", "segmentGranularity" : "HOUR", "queryGranularity" : { "type" : "none" } } }, "ioConfig" : { "type" : "realtime", "plumber" : null, "firehose" : { "type" : "clipped", "interval" : "2016-04-29T08:00:00.000Z/2016-04-29T09:00:00.000Z", "delegate" : { "type" : "timed", "shutoffTime" : "2016-04-29T09:15:00.000Z", "delegate" : { "type" : "receiver", "serviceName" : "firehose:druid:overlord:metrics-kafka-08-0000-0000", "bufferSize" : 100000 } } } }, "tuningConfig" : { "shardSpec" : { "type" : "linear", "partitionNum" : 0 }, "rejectionPolicy" : { "type" : "none" }, "buildV9Directly" : false, "maxPendingPersists" : 0, "intermediatePersistPeriod" : "PT10M", "windowPeriod" : "PT10M", "type" : "realtime", "maxRowsInMemory" : "100000" } } } 2016-04-29 08:49:38,698 [finagle/netty3-1] WARN c.m.tranquility.finagle.FutureRetry$ - Transient error, will try again in 13,260 ms com.metamx.tranquility.druid.IndexServiceTransientException: Service[druid:overlord] call failed with status: 500 Internal Server Error at com.metamx.tranquility.druid.IndexService$$anonfun$call$1$$anonfun$apply$17.apply(IndexService.scala:150) ~[io.druid.tranquility-core-0.7.4.jar:0.7.4] at com.metamx.tranquility.druid.IndexService$$anonfun$call$1$$anonfun$apply$17.apply(IndexService.scala:132) ~[io.druid.tranquility-core-0.7.4.jar:0.7.4] at com.twitter.util.Future$$anonfun$map$1$$anonfun$apply$6.apply(Future.scala:950) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Try$.apply(Try.scala:13) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$.apply(Future.scala:97) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$$anonfun$map$1.apply(Future.scala:950) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$$anonfun$map$1.apply(Future.scala:949) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.liftedTree1$1(Promise.scala:112) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.k(Promise.scala:112) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.apply(Promise.scala:122) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.apply(Promise.scala:103) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$$anon$1.run(Promise.scala:366) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler$Activation.run(Scheduler.scala:178) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler$Activation.submit(Scheduler.scala:136) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler.submit(Scheduler.scala:207) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.Scheduler$.submit(Scheduler.scala:92) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.runq(Promise.scala:350) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.updateIfEmpty(Promise.scala:721) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.update(Promise.scala:694) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.setValue(Promise.scala:670) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.AsyncQueue.offer(AsyncQueue.scala:111) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.finagle.netty3.transport.ChannelTransport.handleUpstream(ChannelTransport.scala:55) [com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.http.HttpContentDecoder.messageReceived(HttpContentDecoder.java:108) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:145) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.http.HttpClientCodec.handleUpstream(HttpClientCodec.java:92) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142) [io.netty.netty-3.10.5.Final.jar:na] at com.twitter.finagle.netty3.channel.ChannelStatsHandler.messageReceived(ChannelStatsHandler.scala:78) [com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0] at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142) [io.netty.netty-3.10.5.Final.jar:na] at com.twitter.finagle.netty3.channel.ChannelRequestStatsHandler.messageReceived(ChannelRequestStatsHandler.scala:35) [com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0] at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) [io.netty.netty-3.10.5.Final.jar:na] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_79] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_79] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79] 2016-04-29 08:49:51,978 [finagle/netty3-2] WARN c.m.tranquility.finagle.FutureRetry$ - Transient error, will try again in 28,549 ms com.metamx.tranquility.druid.IndexServiceTransientException: Service[druid:overlord] call failed with status: 500 Internal Server Error at com.metamx.tranquility.druid.IndexService$$anonfun$call$1$$anonfun$apply$17.apply(IndexService.scala:150) ~[io.druid.tranquility-core-0.7.4.jar:0.7.4] at com.metamx.tranquility.druid.IndexService$$anonfun$call$1$$anonfun$apply$17.apply(IndexService.scala:132) ~[io.druid.tranquility-core-0.7.4.jar:0.7.4] at com.twitter.util.Future$$anonfun$map$1$$anonfun$apply$6.apply(Future.scala:950) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Try$.apply(Try.scala:13) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$.apply(Future.scala:97) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$$anonfun$map$1.apply(Future.scala:950) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$$anonfun$map$1.apply(Future.scala:949) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.liftedTree1$1(Promise.scala:112) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.k(Promise.scala:112) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.apply(Promise.scala:122) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.apply(Promise.scala:103) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$$anon$1.run(Promise.scala:366) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler$Activation.run(Scheduler.scala:178) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler$Activation.submit(Scheduler.scala:136) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler.submit(Scheduler.scala:207) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.Scheduler$.submit(Scheduler.scala:92) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.runq(Promise.scala:350) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.updateIfEmpty(Promise.scala:721) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.update(Promise.scala:694) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.setValue(Promise.scala:670) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.AsyncQueue.offer(AsyncQueue.scala:111) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.finagle.netty3.transport.ChannelTransport.handleUpstream(ChannelTransport.scala:55) [com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.http.HttpContentDecoder.messageReceived(HttpContentDecoder.java:108) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:145) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.http.HttpClientCodec.handleUpstream(HttpClientCodec.java:92) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142) [io.netty.netty-3.10.5.Final.jar:na] at com.twitter.finagle.netty3.channel.ChannelStatsHandler.messageReceived(ChannelStatsHandler.scala:78) [com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0] at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142) [io.netty.netty-3.10.5.Final.jar:na] at com.twitter.finagle.netty3.channel.ChannelRequestStatsHandler.messageReceived(ChannelRequestStatsHandler.scala:35) [com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0] at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) [io.netty.netty-3.10.5.Final.jar:na] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_79] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_79] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79] 2016-04-29 08:50:20,549 [finagle/netty3-3] WARN c.m.tranquility.finagle.FutureRetry$ - Transient error, will try again in 53,770 ms com.metamx.tranquility.druid.IndexServiceTransientException: Service[druid:overlord] call failed with status: 500 Internal Server Error at com.metamx.tranquility.druid.IndexService$$anonfun$call$1$$anonfun$apply$17.apply(IndexService.scala:150) ~[io.druid.tranquility-core-0.7.4.jar:0.7.4] at com.metamx.tranquility.druid.IndexService$$anonfun$call$1$$anonfun$apply$17.apply(IndexService.scala:132) ~[io.druid.tranquility-core-0.7.4.jar:0.7.4] at com.twitter.util.Future$$anonfun$map$1$$anonfun$apply$6.apply(Future.scala:950) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Try$.apply(Try.scala:13) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$.apply(Future.scala:97) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$$anonfun$map$1.apply(Future.scala:950) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$$anonfun$map$1.apply(Future.scala:949) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.liftedTree1$1(Promise.scala:112) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.k(Promise.scala:112) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.apply(Promise.scala:122) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.apply(Promise.scala:103) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$$anon$1.run(Promise.scala:366) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler$Activation.run(Scheduler.scala:178) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler$Activation.submit(Scheduler.scala:136) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler.submit(Scheduler.scala:207) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.Scheduler$.submit(Scheduler.scala:92) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.runq(Promise.scala:350) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.updateIfEmpty(Promise.scala:721) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.update(Promise.scala:694) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.setValue(Promise.scala:670) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.AsyncQueue.offer(AsyncQueue.scala:111) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.finagle.netty3.transport.ChannelTransport.handleUpstream(ChannelTransport.scala:55) [com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.http.HttpContentDecoder.messageReceived(HttpContentDecoder.java:108) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:145) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.http.HttpClientCodec.handleUpstream(HttpClientCodec.java:92) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142) [io.netty.netty-3.10.5.Final.jar:na] at com.twitter.finagle.netty3.channel.ChannelStatsHandler.messageReceived(ChannelStatsHandler.scala:78) [com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0] at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142) [io.netty.netty-3.10.5.Final.jar:na] at com.twitter.finagle.netty3.channel.ChannelRequestStatsHandler.messageReceived(ChannelRequestStatsHandler.scala:35) [com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0] at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) [io.netty.netty-3.10.5.Final.jar:na] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_79] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_79] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79] 2016-04-29 08:51:14,346 [ClusteredBeam-ZkFuturePool-1e419b48-1aa3-4cc4-9ba9-15dadf5e53e6] ERROR c.m.tranquility.beam.ClusteredBeam - Failed to update cluster state: druid:overlord/metrics-kafka com.metamx.tranquility.druid.IndexServiceTransientException: Service[druid:overlord] call failed with status: 500 Internal Server Error at com.metamx.tranquility.druid.IndexService$$anonfun$call$1$$anonfun$apply$17.apply(IndexService.scala:150) ~[io.druid.tranquility-core-0.7.4.jar:0.7.4] at com.metamx.tranquility.druid.IndexService$$anonfun$call$1$$anonfun$apply$17.apply(IndexService.scala:132) ~[io.druid.tranquility-core-0.7.4.jar:0.7.4] at com.twitter.util.Future$$anonfun$map$1$$anonfun$apply$6.apply(Future.scala:950) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Try$.apply(Try.scala:13) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$.apply(Future.scala:97) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$$anonfun$map$1.apply(Future.scala:950) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$$anonfun$map$1.apply(Future.scala:949) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.liftedTree1$1(Promise.scala:112) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.k(Promise.scala:112) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.apply(Promise.scala:122) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.apply(Promise.scala:103) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$$anon$1.run(Promise.scala:366) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler$Activation.run(Scheduler.scala:178) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler$Activation.submit(Scheduler.scala:136) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler.submit(Scheduler.scala:207) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.Scheduler$.submit(Scheduler.scala:92) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.runq(Promise.scala:350) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.updateIfEmpty(Promise.scala:721) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.update(Promise.scala:694) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.setValue(Promise.scala:670) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.AsyncQueue.offer(AsyncQueue.scala:111) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.finagle.netty3.transport.ChannelTransport.handleUpstream(ChannelTransport.scala:55) ~[com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.http.HttpContentDecoder.messageReceived(HttpContentDecoder.java:108) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:145) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.http.HttpClientCodec.handleUpstream(HttpClientCodec.java:92) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142) ~[io.netty.netty-3.10.5.Final.jar:na] at com.twitter.finagle.netty3.channel.ChannelStatsHandler.messageReceived(ChannelStatsHandler.scala:78) ~[com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0] at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142) ~[io.netty.netty-3.10.5.Final.jar:na] at com.twitter.finagle.netty3.channel.ChannelRequestStatsHandler.messageReceived(ChannelRequestStatsHandler.scala:35) ~[com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0] at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) ~[io.netty.netty-3.10.5.Final.jar:na] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_79] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_79] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79] 2016-04-29 08:51:14,353 [ClusteredBeam-ZkFuturePool-1e419b48-1aa3-4cc4-9ba9-15dadf5e53e6] WARN c.m.tranquility.beam.ClusteredBeam - Emitting alert: [anomaly] Failed to create merged beam: druid:overlord/metrics-kafka { } java.lang.IllegalStateException: Failed to save new beam for identifier[druid:overlord/metrics-kafka] timestamp[2016-04-29T08:00:00.000Z] at com.metamx.tranquility.beam.ClusteredBeam$$anonfun$2.applyOrElse(ClusteredBeam.scala:286) ~[io.druid.tranquility-core-0.7.4.jar:0.7.4] at com.metamx.tranquility.beam.ClusteredBeam$$anonfun$2.applyOrElse(ClusteredBeam.scala:283) ~[io.druid.tranquility-core-0.7.4.jar:0.7.4] at com.twitter.util.Future$$anonfun$rescue$1.apply(Future.scala:924) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$$anonfun$rescue$1.apply(Future.scala:922) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.liftedTree1$1(Promise.scala:112) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.k(Promise.scala:112) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.apply(Promise.scala:122) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.apply(Promise.scala:103) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$$anon$1.run(Promise.scala:366) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler$Activation.run(Scheduler.scala:178) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler$Activation.submit(Scheduler.scala:136) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler.submit(Scheduler.scala:207) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.Scheduler$.submit(Scheduler.scala:92) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.runq(Promise.scala:350) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.updateIfEmpty(Promise.scala:721) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.ExecutorServiceFuturePool$$anon$2.run(FuturePool.scala:107) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_79] at java.util.concurrent.FutureTask.run(FutureTask.java:262) [na:1.7.0_79] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_79] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_79] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79] Caused by: com.metamx.tranquility.druid.IndexServiceTransientException: Service[druid:overlord] call failed with status: 500 Internal Server Error at com.metamx.tranquility.druid.IndexService$$anonfun$call$1$$anonfun$apply$17.apply(IndexService.scala:150) ~[io.druid.tranquility-core-0.7.4.jar:0.7.4] at com.metamx.tranquility.druid.IndexService$$anonfun$call$1$$anonfun$apply$17.apply(IndexService.scala:132) ~[io.druid.tranquility-core-0.7.4.jar:0.7.4] at com.twitter.util.Future$$anonfun$map$1$$anonfun$apply$6.apply(Future.scala:950) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Try$.apply(Try.scala:13) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$.apply(Future.scala:97) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$$anonfun$map$1.apply(Future.scala:950) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$$anonfun$map$1.apply(Future.scala:949) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.liftedTree1$1(Promise.scala:112) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.k(Promise.scala:112) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.apply(Promise.scala:122) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.apply(Promise.scala:103) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$$anon$1.run(Promise.scala:366) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler$Activation.run(Scheduler.scala:178) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler$Activation.submit(Scheduler.scala:136) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler.submit(Scheduler.scala:207) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.Scheduler$.submit(Scheduler.scala:92) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.runq(Promise.scala:350) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.updateIfEmpty(Promise.scala:721) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.update(Promise.scala:694) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.setValue(Promise.scala:670) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.AsyncQueue.offer(AsyncQueue.scala:111) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.finagle.netty3.transport.ChannelTransport.handleUpstream(ChannelTransport.scala:55) ~[com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.http.HttpContentDecoder.messageReceived(HttpContentDecoder.java:108) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:145) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.http.HttpClientCodec.handleUpstream(HttpClientCodec.java:92) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142) ~[io.netty.netty-3.10.5.Final.jar:na] at com.twitter.finagle.netty3.channel.ChannelStatsHandler.messageReceived(ChannelStatsHandler.scala:78) ~[com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0] at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142) ~[io.netty.netty-3.10.5.Final.jar:na] at com.twitter.finagle.netty3.channel.ChannelRequestStatsHandler.messageReceived(ChannelRequestStatsHandler.scala:35) ~[com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0] at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) ~[io.netty.netty-3.10.5.Final.jar:na] ... 3 common frames omitted 2016-04-29 08:51:14,356 [ClusteredBeam-ZkFuturePool-1e419b48-1aa3-4cc4-9ba9-15dadf5e53e6] INFO c.metamx.emitter.core.LoggingEmitter - Event [{"feed":"alerts","timestamp":"2016-04-29T08:51:14.355Z","service":"tranquility","host":"localhost","severity":"anomaly","description":"Failed to create merged beam: druid:overlord/metrics-kafka","data":{"exceptionType":"java.lang.IllegalStateException","exceptionMessage":"Failed to save new beam for identifier[druid:overlord/metrics-kafka] timestamp[2016-04-29T08:00:00.000Z]","exceptionStackTrace":"java.lang.IllegalStateException: Failed to save new beam for identifier[druid:overlord/metrics-kafka] timestamp[2016-04-29T08:00:00.000Z]\n\tat com.metamx.tranquility.beam.ClusteredBeam$$anonfun$2.applyOrElse(ClusteredBeam.scala:286)\n\tat com.metamx.tranquility.beam.ClusteredBeam$$anonfun$2.applyOrElse(ClusteredBeam.scala:283)\n\tat com.twitter.util.Future$$anonfun$rescue$1.apply(Future.scala:924)\n\tat com.twitter.util.Future$$anonfun$rescue$1.apply(Future.scala:922)\n\tat com.twitter.util.Promise$Transformer.liftedTree1$1(Promise.scala:112)\n\tat com.twitter.util.Promise$Transformer.k(Promise.scala:112)\n\tat com.twitter.util.Promise$Transformer.apply(Promise.scala:122)\n\tat com.twitter.util.Promise$Transformer.apply(Promise.scala:103)\n\tat com.twitter.util.Promise$$anon$1.run(Promise.scala:366)\n\tat com.twitter.concurrent.LocalScheduler$Activation.run(Scheduler.scala:178)\n\tat com.twitter.concurrent.LocalScheduler$Activation.submit(Scheduler.scala:136)\n\tat com.twitter.concurrent.LocalScheduler.submit(Scheduler.scala:207)\n\tat com.twitter.concurrent.Scheduler$.submit(Scheduler.scala:92)\n\tat com.twitter.util.Promise.runq(Promise.scala:350)\n\tat com.twitter.util.Promise.updateIfEmpty(Promise.scala:721)\n\tat com.twitter.util.ExecutorServiceFuturePool$$anon$2.run(FuturePool.scala:107)\n\tat java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:262)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)\n\tat java.lang.Thread.run(Thread.java:745)\nCaused by: com.metamx.tranquility.druid.IndexServiceTransientException: Service[druid:overlord] call failed with status: 500 Internal Server Error\n\tat com.metamx.tranquility.druid.IndexService$$anonfun$call$1$$anonfun$apply$17.apply(IndexService.scala:150)\n\tat com.metamx.tranquility.druid.IndexService$$anonfun$call$1$$anonfun$apply$17.apply(IndexService.scala:132)\n\tat com.twitter.util.Future$$anonfun$map$1$$anonfun$apply$6.apply(Future.scala:950)\n\tat com.twitter.util.Try$.apply(Try.scala:13)\n\tat com.twitter.util.Future$.apply(Future.scala:97)\n\tat com.twitter.util.Future$$anonfun$map$1.apply(Future.scala:950)\n\tat com.twitter.util.Future$$anonfun$map$1.apply(Future.scala:949)\n\tat com.twitter.util.Promise$Transformer.liftedTree1$1(Promise.scala:112)\n\tat com.twitter.util.Promise$Transformer.k(Promise.scala:112)\n\tat com.twitter.util.Promise$Transformer.apply(Promise.scala:122)\n\tat com.twitter.util.Promise$Transformer.apply(Promise.scala:103)\n\tat com.twitter.util.Promise$$anon$1.run(Promise.scala:366)\n\tat com.twitter.concurrent.LocalScheduler$Activation.run(Scheduler.scala:178)\n\tat com.twitter.concurrent.LocalScheduler$Activation.submit(Scheduler.scala:136)\n\tat com.twitter.concurrent.LocalScheduler.submit(Scheduler.scala:207)\n\tat com.twitter.concurrent.Scheduler$.submit(Scheduler.scala:92)\n\tat com.twitter.util.Promise.runq(Promise.scala:350)\n\tat com.twitter.util.Promise.updateIfEmpty(Promise.scala:721)\n\tat com.twitter.util.Promise.update(Promise.scala:694)\n\tat com.twitter.util.Promise.setValue(Promise.scala:670)\n\tat com.twitter.concurrent.AsyncQueue.offer(AsyncQueue.scala:111)\n\tat com.twitter.finagle.netty3.transport.ChannelTransport.handleUpstream(ChannelTransport.scala:55)\n\tat org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)\n\tat org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)\n\tat org.jboss.netty.handler.codec.http.HttpContentDecoder.messageReceived(HttpContentDecoder.java:108)\n\tat org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)\n\tat org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)\n\tat org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)\n\tat org.jboss.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:145)\n\tat org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)\n\tat org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)\n\tat org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)\n\tat org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)\n\tat org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459)\n\tat org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536)\n\tat org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435)\n\tat org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)\n\tat org.jboss.netty.handler.codec.http.HttpClientCodec.handleUpstream(HttpClientCodec.java:92)\n\tat org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)\n\tat org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)\n\tat org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142)\n\tat com.twitter.finagle.netty3.channel.ChannelStatsHandler.messageReceived(ChannelStatsHandler.scala:78)\n\tat org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88)\n\tat org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)\n\tat org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)\n\tat org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142)\n\tat com.twitter.finagle.netty3.channel.ChannelRequestStatsHandler.messageReceived(ChannelRequestStatsHandler.scala:35)\n\tat org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88)\n\tat org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)\n\tat org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)\n\tat org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)\n\tat org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)\n\tat org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)\n\tat org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)\n\tat org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)\n\tat org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)\n\tat org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)\n\tat org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)\n\tat org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)\n\t... 3 more\n"}}] 2016-04-29 08:51:14,358 [ClusteredBeam-ZkFuturePool-1e419b48-1aa3-4cc4-9ba9-15dadf5e53e6] WARN c.m.t.tranquilizer.Tranquilizer - Failed to send 1 messages. java.lang.IllegalStateException: Failed to save new beam for identifier[druid:overlord/metrics-kafka] timestamp[2016-04-29T08:00:00.000Z] at com.metamx.tranquility.beam.ClusteredBeam$$anonfun$2.applyOrElse(ClusteredBeam.scala:286) ~[io.druid.tranquility-core-0.7.4.jar:0.7.4] at com.metamx.tranquility.beam.ClusteredBeam$$anonfun$2.applyOrElse(ClusteredBeam.scala:283) ~[io.druid.tranquility-core-0.7.4.jar:0.7.4] at com.twitter.util.Future$$anonfun$rescue$1.apply(Future.scala:924) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$$anonfun$rescue$1.apply(Future.scala:922) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.liftedTree1$1(Promise.scala:112) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.k(Promise.scala:112) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.apply(Promise.scala:122) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.apply(Promise.scala:103) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$$anon$1.run(Promise.scala:366) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler$Activation.run(Scheduler.scala:178) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler$Activation.submit(Scheduler.scala:136) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler.submit(Scheduler.scala:207) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.Scheduler$.submit(Scheduler.scala:92) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.runq(Promise.scala:350) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.updateIfEmpty(Promise.scala:721) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.ExecutorServiceFuturePool$$anon$2.run(FuturePool.scala:107) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_79] at java.util.concurrent.FutureTask.run(FutureTask.java:262) [na:1.7.0_79] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_79] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_79] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79] Caused by: com.metamx.tranquility.druid.IndexServiceTransientException: Service[druid:overlord] call failed with status: 500 Internal Server Error at com.metamx.tranquility.druid.IndexService$$anonfun$call$1$$anonfun$apply$17.apply(IndexService.scala:150) ~[io.druid.tranquility-core-0.7.4.jar:0.7.4] at com.metamx.tranquility.druid.IndexService$$anonfun$call$1$$anonfun$apply$17.apply(IndexService.scala:132) ~[io.druid.tranquility-core-0.7.4.jar:0.7.4] at com.twitter.util.Future$$anonfun$map$1$$anonfun$apply$6.apply(Future.scala:950) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Try$.apply(Try.scala:13) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$.apply(Future.scala:97) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$$anonfun$map$1.apply(Future.scala:950) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$$anonfun$map$1.apply(Future.scala:949) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.liftedTree1$1(Promise.scala:112) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.k(Promise.scala:112) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.apply(Promise.scala:122) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.apply(Promise.scala:103) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$$anon$1.run(Promise.scala:366) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler$Activation.run(Scheduler.scala:178) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler$Activation.submit(Scheduler.scala:136) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler.submit(Scheduler.scala:207) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.Scheduler$.submit(Scheduler.scala:92) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.runq(Promise.scala:350) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.updateIfEmpty(Promise.scala:721) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.update(Promise.scala:694) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.setValue(Promise.scala:670) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.AsyncQueue.offer(AsyncQueue.scala:111) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.finagle.netty3.transport.ChannelTransport.handleUpstream(ChannelTransport.scala:55) ~[com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.http.HttpContentDecoder.messageReceived(HttpContentDecoder.java:108) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:145) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.http.HttpClientCodec.handleUpstream(HttpClientCodec.java:92) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142) ~[io.netty.netty-3.10.5.Final.jar:na] at com.twitter.finagle.netty3.channel.ChannelStatsHandler.messageReceived(ChannelStatsHandler.scala:78) ~[com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0] at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142) ~[io.netty.netty-3.10.5.Final.jar:na] at com.twitter.finagle.netty3.channel.ChannelRequestStatsHandler.messageReceived(ChannelRequestStatsHandler.scala:35) ~[com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0] at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) ~[io.netty.netty-3.10.5.Final.jar:na] ... 3 common frames omitted 2016-04-29 08:51:14,384 [ClusteredBeam-ZkFuturePool-1e419b48-1aa3-4cc4-9ba9-15dadf5e53e6] INFO c.m.tranquility.beam.ClusteredBeam - Creating new merged beam for identifier[druid:overlord/metrics-kafka] timestamp[2016-04-29T08:00:00.000Z] (target = 1, actual = 0) 2016-04-29 08:51:14,394 [ClusteredBeam-ZkFuturePool-1e419b48-1aa3-4cc4-9ba9-15dadf5e53e6] INFO com.metamx.common.scala.control$ - Creating druid indexing task (service = druid:overlord): { "type" : "index_realtime", "id" : "index_realtime_metrics-kafka_2016-04-29T08:00:00.000Z_0_0", "resource" : { "availabilityGroup" : "metrics-kafka-08-0000", "requiredCapacity" : 1 }, "spec" : { "dataSchema" : { "dataSource" : "metrics-kafka", "parser" : { "type" : "map", "parseSpec" : { "format" : "json", "timestampSpec" : { "column" : "timestamp", "format" : "auto", "missingValue" : null }, "dimensionsSpec" : { "dimensions" : [ "http_code", "http_method", "metricType", "page", "server", "timestamp", "unit", "value" ], "spatialDimensions" : [ ] } } }, "metricsSpec" : [ { "type" : "count", "name" : "count" }, { "type" : "doubleSum", "name" : "value_sum", "fieldName" : "value" }, { "type" : "doubleMin", "name" : "value_min", "fieldName" : "value" }, { "type" : "doubleMax", "name" : "value_max", "fieldName" : "value" } ], "granularitySpec" : { "type" : "uniform", "segmentGranularity" : "HOUR", "queryGranularity" : { "type" : "none" } } }, "ioConfig" : { "type" : "realtime", "plumber" : null, "firehose" : { "type" : "clipped", "interval" : "2016-04-29T08:00:00.000Z/2016-04-29T09:00:00.000Z", "delegate" : { "type" : "timed", "shutoffTime" : "2016-04-29T09:15:00.000Z", "delegate" : { "type" : "receiver", "serviceName" : "firehose:druid:overlord:metrics-kafka-08-0000-0000", "bufferSize" : 100000 } } } }, "tuningConfig" : { "shardSpec" : { "type" : "linear", "partitionNum" : 0 }, "rejectionPolicy" : { "type" : "none" }, "buildV9Directly" : false, "maxPendingPersists" : 0, "intermediatePersistPeriod" : "PT10M", "windowPeriod" : "PT10M", "type" : "realtime", "maxRowsInMemory" : "100000" } } } 2016-04-29 08:51:14,409 [finagle/netty3-1] WARN c.m.tranquility.finagle.FutureRetry$ - Transient error, will try again in 14,664 ms com.metamx.tranquility.druid.IndexServiceTransientException: Service[druid:overlord] call failed with status: 500 Internal Server Error at com.metamx.tranquility.druid.IndexService$$anonfun$call$1$$anonfun$apply$17.apply(IndexService.scala:150) ~[io.druid.tranquility-core-0.7.4.jar:0.7.4] at com.metamx.tranquility.druid.IndexService$$anonfun$call$1$$anonfun$apply$17.apply(IndexService.scala:132) ~[io.druid.tranquility-core-0.7.4.jar:0.7.4] at com.twitter.util.Future$$anonfun$map$1$$anonfun$apply$6.apply(Future.scala:950) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Try$.apply(Try.scala:13) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$.apply(Future.scala:97) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$$anonfun$map$1.apply(Future.scala:950) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$$anonfun$map$1.apply(Future.scala:949) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.liftedTree1$1(Promise.scala:112) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.k(Promise.scala:112) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.apply(Promise.scala:122) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.apply(Promise.scala:103) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$$anon$1.run(Promise.scala:366) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler$Activation.run(Scheduler.scala:178) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler$Activation.submit(Scheduler.scala:136) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler.submit(Scheduler.scala:207) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.Scheduler$.submit(Scheduler.scala:92) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.runq(Promise.scala:350) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.updateIfEmpty(Promise.scala:721) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.update(Promise.scala:694) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.setValue(Promise.scala:670) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.AsyncQueue.offer(AsyncQueue.scala:111) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.finagle.netty3.transport.ChannelTransport.handleUpstream(ChannelTransport.scala:55) [com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.http.HttpContentDecoder.messageReceived(HttpContentDecoder.java:108) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:145) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.http.HttpClientCodec.handleUpstream(HttpClientCodec.java:92) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142) [io.netty.netty-3.10.5.Final.jar:na] at com.twitter.finagle.netty3.channel.ChannelStatsHandler.messageReceived(ChannelStatsHandler.scala:78) [com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0] at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142) [io.netty.netty-3.10.5.Final.jar:na] at com.twitter.finagle.netty3.channel.ChannelRequestStatsHandler.messageReceived(ChannelRequestStatsHandler.scala:35) [com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0] at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) [io.netty.netty-3.10.5.Final.jar:na] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_79] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_79] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79] 2016-04-29 08:51:29,103 [finagle/netty3-2] WARN c.m.tranquility.finagle.FutureRetry$ - Transient error, will try again in 29,646 ms com.metamx.tranquility.druid.IndexServiceTransientException: Service[druid:overlord] call failed with status: 500 Internal Server Error at com.metamx.tranquility.druid.IndexService$$anonfun$call$1$$anonfun$apply$17.apply(IndexService.scala:150) ~[io.druid.tranquility-core-0.7.4.jar:0.7.4] at com.metamx.tranquility.druid.IndexService$$anonfun$call$1$$anonfun$apply$17.apply(IndexService.scala:132) ~[io.druid.tranquility-core-0.7.4.jar:0.7.4] at com.twitter.util.Future$$anonfun$map$1$$anonfun$apply$6.apply(Future.scala:950) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Try$.apply(Try.scala:13) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$.apply(Future.scala:97) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$$anonfun$map$1.apply(Future.scala:950) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$$anonfun$map$1.apply(Future.scala:949) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.liftedTree1$1(Promise.scala:112) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.k(Promise.scala:112) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.apply(Promise.scala:122) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.apply(Promise.scala:103) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$$anon$1.run(Promise.scala:366) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler$Activation.run(Scheduler.scala:178) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler$Activation.submit(Scheduler.scala:136) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler.submit(Scheduler.scala:207) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.Scheduler$.submit(Scheduler.scala:92) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.runq(Promise.scala:350) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.updateIfEmpty(Promise.scala:721) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.update(Promise.scala:694) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.setValue(Promise.scala:670) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.AsyncQueue.offer(AsyncQueue.scala:111) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.finagle.netty3.transport.ChannelTransport.handleUpstream(ChannelTransport.scala:55) [com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.http.HttpContentDecoder.messageReceived(HttpContentDecoder.java:108) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:145) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.http.HttpClientCodec.handleUpstream(HttpClientCodec.java:92) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142) [io.netty.netty-3.10.5.Final.jar:na] at com.twitter.finagle.netty3.channel.ChannelStatsHandler.messageReceived(ChannelStatsHandler.scala:78) [com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0] at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142) [io.netty.netty-3.10.5.Final.jar:na] at com.twitter.finagle.netty3.channel.ChannelRequestStatsHandler.messageReceived(ChannelRequestStatsHandler.scala:35) [com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0] at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) [io.netty.netty-3.10.5.Final.jar:na] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_79] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_79] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79] 2016-04-29 08:51:58,783 [finagle/netty3-3] WARN c.m.tranquility.finagle.FutureRetry$ - Transient error, will try again in 54,675 ms com.metamx.tranquility.druid.IndexServiceTransientException: Service[druid:overlord] call failed with status: 500 Internal Server Error at com.metamx.tranquility.druid.IndexService$$anonfun$call$1$$anonfun$apply$17.apply(IndexService.scala:150) ~[io.druid.tranquility-core-0.7.4.jar:0.7.4] at com.metamx.tranquility.druid.IndexService$$anonfun$call$1$$anonfun$apply$17.apply(IndexService.scala:132) ~[io.druid.tranquility-core-0.7.4.jar:0.7.4] at com.twitter.util.Future$$anonfun$map$1$$anonfun$apply$6.apply(Future.scala:950) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Try$.apply(Try.scala:13) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$.apply(Future.scala:97) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$$anonfun$map$1.apply(Future.scala:950) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$$anonfun$map$1.apply(Future.scala:949) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.liftedTree1$1(Promise.scala:112) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.k(Promise.scala:112) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.apply(Promise.scala:122) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.apply(Promise.scala:103) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$$anon$1.run(Promise.scala:366) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler$Activation.run(Scheduler.scala:178) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler$Activation.submit(Scheduler.scala:136) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler.submit(Scheduler.scala:207) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.Scheduler$.submit(Scheduler.scala:92) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.runq(Promise.scala:350) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.updateIfEmpty(Promise.scala:721) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.update(Promise.scala:694) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.setValue(Promise.scala:670) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.AsyncQueue.offer(AsyncQueue.scala:111) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.finagle.netty3.transport.ChannelTransport.handleUpstream(ChannelTransport.scala:55) [com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.http.HttpContentDecoder.messageReceived(HttpContentDecoder.java:108) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:145) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.http.HttpClientCodec.handleUpstream(HttpClientCodec.java:92) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142) [io.netty.netty-3.10.5.Final.jar:na] at com.twitter.finagle.netty3.channel.ChannelStatsHandler.messageReceived(ChannelStatsHandler.scala:78) [com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0] at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142) [io.netty.netty-3.10.5.Final.jar:na] at com.twitter.finagle.netty3.channel.ChannelRequestStatsHandler.messageReceived(ChannelRequestStatsHandler.scala:35) [com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0] at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) [io.netty.netty-3.10.5.Final.jar:na] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_79] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_79] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79] 2016-04-29 08:52:53,486 [ClusteredBeam-ZkFuturePool-1e419b48-1aa3-4cc4-9ba9-15dadf5e53e6] ERROR c.m.tranquility.beam.ClusteredBeam - Failed to update cluster state: druid:overlord/metrics-kafka com.metamx.tranquility.druid.IndexServiceTransientException: Service[druid:overlord] call failed with status: 500 Internal Server Error at com.metamx.tranquility.druid.IndexService$$anonfun$call$1$$anonfun$apply$17.apply(IndexService.scala:150) ~[io.druid.tranquility-core-0.7.4.jar:0.7.4] at com.metamx.tranquility.druid.IndexService$$anonfun$call$1$$anonfun$apply$17.apply(IndexService.scala:132) ~[io.druid.tranquility-core-0.7.4.jar:0.7.4] at com.twitter.util.Future$$anonfun$map$1$$anonfun$apply$6.apply(Future.scala:950) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Try$.apply(Try.scala:13) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$.apply(Future.scala:97) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$$anonfun$map$1.apply(Future.scala:950) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$$anonfun$map$1.apply(Future.scala:949) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.liftedTree1$1(Promise.scala:112) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.k(Promise.scala:112) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.apply(Promise.scala:122) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.apply(Promise.scala:103) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$$anon$1.run(Promise.scala:366) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler$Activation.run(Scheduler.scala:178) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler$Activation.submit(Scheduler.scala:136) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler.submit(Scheduler.scala:207) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.Scheduler$.submit(Scheduler.scala:92) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.runq(Promise.scala:350) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.updateIfEmpty(Promise.scala:721) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.update(Promise.scala:694) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.setValue(Promise.scala:670) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.AsyncQueue.offer(AsyncQueue.scala:111) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.finagle.netty3.transport.ChannelTransport.handleUpstream(ChannelTransport.scala:55) ~[com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.http.HttpContentDecoder.messageReceived(HttpContentDecoder.java:108) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:145) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.http.HttpClientCodec.handleUpstream(HttpClientCodec.java:92) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142) ~[io.netty.netty-3.10.5.Final.jar:na] at com.twitter.finagle.netty3.channel.ChannelStatsHandler.messageReceived(ChannelStatsHandler.scala:78) ~[com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0] at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142) ~[io.netty.netty-3.10.5.Final.jar:na] at com.twitter.finagle.netty3.channel.ChannelRequestStatsHandler.messageReceived(ChannelRequestStatsHandler.scala:35) ~[com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0] at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) ~[io.netty.netty-3.10.5.Final.jar:na] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_79] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_79] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79] 2016-04-29 08:52:53,491 [ClusteredBeam-ZkFuturePool-1e419b48-1aa3-4cc4-9ba9-15dadf5e53e6] WARN c.m.tranquility.beam.ClusteredBeam - Emitting alert: [anomaly] Failed to create merged beam: druid:overlord/metrics-kafka { } java.lang.IllegalStateException: Failed to save new beam for identifier[druid:overlord/metrics-kafka] timestamp[2016-04-29T08:00:00.000Z] at com.metamx.tranquility.beam.ClusteredBeam$$anonfun$2.applyOrElse(ClusteredBeam.scala:286) ~[io.druid.tranquility-core-0.7.4.jar:0.7.4] at com.metamx.tranquility.beam.ClusteredBeam$$anonfun$2.applyOrElse(ClusteredBeam.scala:283) ~[io.druid.tranquility-core-0.7.4.jar:0.7.4] at com.twitter.util.Future$$anonfun$rescue$1.apply(Future.scala:924) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$$anonfun$rescue$1.apply(Future.scala:922) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.liftedTree1$1(Promise.scala:112) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.k(Promise.scala:112) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.apply(Promise.scala:122) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.apply(Promise.scala:103) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$$anon$1.run(Promise.scala:366) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler$Activation.run(Scheduler.scala:178) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler$Activation.submit(Scheduler.scala:136) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler.submit(Scheduler.scala:207) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.Scheduler$.submit(Scheduler.scala:92) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.runq(Promise.scala:350) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.updateIfEmpty(Promise.scala:721) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.ExecutorServiceFuturePool$$anon$2.run(FuturePool.scala:107) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_79] at java.util.concurrent.FutureTask.run(FutureTask.java:262) [na:1.7.0_79] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_79] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_79] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79] Caused by: com.metamx.tranquility.druid.IndexServiceTransientException: Service[druid:overlord] call failed with status: 500 Internal Server Error at com.metamx.tranquility.druid.IndexService$$anonfun$call$1$$anonfun$apply$17.apply(IndexService.scala:150) ~[io.druid.tranquility-core-0.7.4.jar:0.7.4] at com.metamx.tranquility.druid.IndexService$$anonfun$call$1$$anonfun$apply$17.apply(IndexService.scala:132) ~[io.druid.tranquility-core-0.7.4.jar:0.7.4] at com.twitter.util.Future$$anonfun$map$1$$anonfun$apply$6.apply(Future.scala:950) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Try$.apply(Try.scala:13) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$.apply(Future.scala:97) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$$anonfun$map$1.apply(Future.scala:950) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$$anonfun$map$1.apply(Future.scala:949) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.liftedTree1$1(Promise.scala:112) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.k(Promise.scala:112) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.apply(Promise.scala:122) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.apply(Promise.scala:103) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$$anon$1.run(Promise.scala:366) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler$Activation.run(Scheduler.scala:178) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler$Activation.submit(Scheduler.scala:136) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler.submit(Scheduler.scala:207) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.Scheduler$.submit(Scheduler.scala:92) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.runq(Promise.scala:350) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.updateIfEmpty(Promise.scala:721) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.update(Promise.scala:694) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.setValue(Promise.scala:670) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.AsyncQueue.offer(AsyncQueue.scala:111) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.finagle.netty3.transport.ChannelTransport.handleUpstream(ChannelTransport.scala:55) ~[com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.http.HttpContentDecoder.messageReceived(HttpContentDecoder.java:108) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:145) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.http.HttpClientCodec.handleUpstream(HttpClientCodec.java:92) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142) ~[io.netty.netty-3.10.5.Final.jar:na] at com.twitter.finagle.netty3.channel.ChannelStatsHandler.messageReceived(ChannelStatsHandler.scala:78) ~[com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0] at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142) ~[io.netty.netty-3.10.5.Final.jar:na] at com.twitter.finagle.netty3.channel.ChannelRequestStatsHandler.messageReceived(ChannelRequestStatsHandler.scala:35) ~[com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0] at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) ~[io.netty.netty-3.10.5.Final.jar:na] ... 3 common frames omitted 2016-04-29 08:52:53,494 [ClusteredBeam-ZkFuturePool-1e419b48-1aa3-4cc4-9ba9-15dadf5e53e6] INFO c.metamx.emitter.core.LoggingEmitter - Event [{"feed":"alerts","timestamp":"2016-04-29T08:52:53.493Z","service":"tranquility","host":"localhost","severity":"anomaly","description":"Failed to create merged beam: druid:overlord/metrics-kafka","data":{"exceptionType":"java.lang.IllegalStateException","exceptionMessage":"Failed to save new beam for identifier[druid:overlord/metrics-kafka] timestamp[2016-04-29T08:00:00.000Z]","exceptionStackTrace":"java.lang.IllegalStateException: Failed to save new beam for identifier[druid:overlord/metrics-kafka] timestamp[2016-04-29T08:00:00.000Z]\n\tat com.metamx.tranquility.beam.ClusteredBeam$$anonfun$2.applyOrElse(ClusteredBeam.scala:286)\n\tat com.metamx.tranquility.beam.ClusteredBeam$$anonfun$2.applyOrElse(ClusteredBeam.scala:283)\n\tat com.twitter.util.Future$$anonfun$rescue$1.apply(Future.scala:924)\n\tat com.twitter.util.Future$$anonfun$rescue$1.apply(Future.scala:922)\n\tat com.twitter.util.Promise$Transformer.liftedTree1$1(Promise.scala:112)\n\tat com.twitter.util.Promise$Transformer.k(Promise.scala:112)\n\tat com.twitter.util.Promise$Transformer.apply(Promise.scala:122)\n\tat com.twitter.util.Promise$Transformer.apply(Promise.scala:103)\n\tat com.twitter.util.Promise$$anon$1.run(Promise.scala:366)\n\tat com.twitter.concurrent.LocalScheduler$Activation.run(Scheduler.scala:178)\n\tat com.twitter.concurrent.LocalScheduler$Activation.submit(Scheduler.scala:136)\n\tat com.twitter.concurrent.LocalScheduler.submit(Scheduler.scala:207)\n\tat com.twitter.concurrent.Scheduler$.submit(Scheduler.scala:92)\n\tat com.twitter.util.Promise.runq(Promise.scala:350)\n\tat com.twitter.util.Promise.updateIfEmpty(Promise.scala:721)\n\tat com.twitter.util.ExecutorServiceFuturePool$$anon$2.run(FuturePool.scala:107)\n\tat java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:262)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)\n\tat java.lang.Thread.run(Thread.java:745)\nCaused by: com.metamx.tranquility.druid.IndexServiceTransientException: Service[druid:overlord] call failed with status: 500 Internal Server Error\n\tat com.metamx.tranquility.druid.IndexService$$anonfun$call$1$$anonfun$apply$17.apply(IndexService.scala:150)\n\tat com.metamx.tranquility.druid.IndexService$$anonfun$call$1$$anonfun$apply$17.apply(IndexService.scala:132)\n\tat com.twitter.util.Future$$anonfun$map$1$$anonfun$apply$6.apply(Future.scala:950)\n\tat com.twitter.util.Try$.apply(Try.scala:13)\n\tat com.twitter.util.Future$.apply(Future.scala:97)\n\tat com.twitter.util.Future$$anonfun$map$1.apply(Future.scala:950)\n\tat com.twitter.util.Future$$anonfun$map$1.apply(Future.scala:949)\n\tat com.twitter.util.Promise$Transformer.liftedTree1$1(Promise.scala:112)\n\tat com.twitter.util.Promise$Transformer.k(Promise.scala:112)\n\tat com.twitter.util.Promise$Transformer.apply(Promise.scala:122)\n\tat com.twitter.util.Promise$Transformer.apply(Promise.scala:103)\n\tat com.twitter.util.Promise$$anon$1.run(Promise.scala:366)\n\tat com.twitter.concurrent.LocalScheduler$Activation.run(Scheduler.scala:178)\n\tat com.twitter.concurrent.LocalScheduler$Activation.submit(Scheduler.scala:136)\n\tat com.twitter.concurrent.LocalScheduler.submit(Scheduler.scala:207)\n\tat com.twitter.concurrent.Scheduler$.submit(Scheduler.scala:92)\n\tat com.twitter.util.Promise.runq(Promise.scala:350)\n\tat com.twitter.util.Promise.updateIfEmpty(Promise.scala:721)\n\tat com.twitter.util.Promise.update(Promise.scala:694)\n\tat com.twitter.util.Promise.setValue(Promise.scala:670)\n\tat com.twitter.concurrent.AsyncQueue.offer(AsyncQueue.scala:111)\n\tat com.twitter.finagle.netty3.transport.ChannelTransport.handleUpstream(ChannelTransport.scala:55)\n\tat org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)\n\tat org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)\n\tat org.jboss.netty.handler.codec.http.HttpContentDecoder.messageReceived(HttpContentDecoder.java:108)\n\tat org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)\n\tat org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)\n\tat org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)\n\tat org.jboss.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:145)\n\tat org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)\n\tat org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)\n\tat org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)\n\tat org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)\n\tat org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459)\n\tat org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536)\n\tat org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435)\n\tat org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)\n\tat org.jboss.netty.handler.codec.http.HttpClientCodec.handleUpstream(HttpClientCodec.java:92)\n\tat org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)\n\tat org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)\n\tat org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142)\n\tat com.twitter.finagle.netty3.channel.ChannelStatsHandler.messageReceived(ChannelStatsHandler.scala:78)\n\tat org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88)\n\tat org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)\n\tat org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)\n\tat org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142)\n\tat com.twitter.finagle.netty3.channel.ChannelRequestStatsHandler.messageReceived(ChannelRequestStatsHandler.scala:35)\n\tat org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88)\n\tat org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)\n\tat org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)\n\tat org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)\n\tat org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)\n\tat org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)\n\tat org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)\n\tat org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)\n\tat org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)\n\tat org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)\n\tat org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)\n\tat org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)\n\t... 3 more\n"}}] 2016-04-29 08:52:53,495 [ClusteredBeam-ZkFuturePool-1e419b48-1aa3-4cc4-9ba9-15dadf5e53e6] WARN c.m.t.tranquilizer.Tranquilizer - Failed to send 1 messages. java.lang.IllegalStateException: Failed to save new beam for identifier[druid:overlord/metrics-kafka] timestamp[2016-04-29T08:00:00.000Z] at com.metamx.tranquility.beam.ClusteredBeam$$anonfun$2.applyOrElse(ClusteredBeam.scala:286) ~[io.druid.tranquility-core-0.7.4.jar:0.7.4] at com.metamx.tranquility.beam.ClusteredBeam$$anonfun$2.applyOrElse(ClusteredBeam.scala:283) ~[io.druid.tranquility-core-0.7.4.jar:0.7.4] at com.twitter.util.Future$$anonfun$rescue$1.apply(Future.scala:924) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$$anonfun$rescue$1.apply(Future.scala:922) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.liftedTree1$1(Promise.scala:112) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.k(Promise.scala:112) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.apply(Promise.scala:122) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.apply(Promise.scala:103) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$$anon$1.run(Promise.scala:366) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler$Activation.run(Scheduler.scala:178) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler$Activation.submit(Scheduler.scala:136) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler.submit(Scheduler.scala:207) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.Scheduler$.submit(Scheduler.scala:92) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.runq(Promise.scala:350) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.updateIfEmpty(Promise.scala:721) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.ExecutorServiceFuturePool$$anon$2.run(FuturePool.scala:107) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_79] at java.util.concurrent.FutureTask.run(FutureTask.java:262) [na:1.7.0_79] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_79] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_79] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79] Caused by: com.metamx.tranquility.druid.IndexServiceTransientException: Service[druid:overlord] call failed with status: 500 Internal Server Error at com.metamx.tranquility.druid.IndexService$$anonfun$call$1$$anonfun$apply$17.apply(IndexService.scala:150) ~[io.druid.tranquility-core-0.7.4.jar:0.7.4] at com.metamx.tranquility.druid.IndexService$$anonfun$call$1$$anonfun$apply$17.apply(IndexService.scala:132) ~[io.druid.tranquility-core-0.7.4.jar:0.7.4] at com.twitter.util.Future$$anonfun$map$1$$anonfun$apply$6.apply(Future.scala:950) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Try$.apply(Try.scala:13) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$.apply(Future.scala:97) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$$anonfun$map$1.apply(Future.scala:950) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$$anonfun$map$1.apply(Future.scala:949) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.liftedTree1$1(Promise.scala:112) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.k(Promise.scala:112) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.apply(Promise.scala:122) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.apply(Promise.scala:103) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$$anon$1.run(Promise.scala:366) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler$Activation.run(Scheduler.scala:178) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler$Activation.submit(Scheduler.scala:136) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler.submit(Scheduler.scala:207) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.Scheduler$.submit(Scheduler.scala:92) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.runq(Promise.scala:350) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.updateIfEmpty(Promise.scala:721) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.update(Promise.scala:694) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.setValue(Promise.scala:670) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.AsyncQueue.offer(AsyncQueue.scala:111) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.finagle.netty3.transport.ChannelTransport.handleUpstream(ChannelTransport.scala:55) ~[com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.http.HttpContentDecoder.messageReceived(HttpContentDecoder.java:108) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:145) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.http.HttpClientCodec.handleUpstream(HttpClientCodec.java:92) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142) ~[io.netty.netty-3.10.5.Final.jar:na] at com.twitter.finagle.netty3.channel.ChannelStatsHandler.messageReceived(ChannelStatsHandler.scala:78) ~[com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0] at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142) ~[io.netty.netty-3.10.5.Final.jar:na] at com.twitter.finagle.netty3.channel.ChannelRequestStatsHandler.messageReceived(ChannelRequestStatsHandler.scala:35) ~[com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0] at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) ~[io.netty.netty-3.10.5.Final.jar:na] ... 3 common frames omitted 2016-04-29 08:52:53,508 [ClusteredBeam-ZkFuturePool-1e419b48-1aa3-4cc4-9ba9-15dadf5e53e6] INFO c.m.tranquility.beam.ClusteredBeam - Creating new merged beam for identifier[druid:overlord/metrics-kafka] timestamp[2016-04-29T08:00:00.000Z] (target = 1, actual = 0) 2016-04-29 08:52:53,514 [ClusteredBeam-ZkFuturePool-1e419b48-1aa3-4cc4-9ba9-15dadf5e53e6] INFO com.metamx.common.scala.control$ - Creating druid indexing task (service = druid:overlord): { "type" : "index_realtime", "id" : "index_realtime_metrics-kafka_2016-04-29T08:00:00.000Z_0_0", "resource" : { "availabilityGroup" : "metrics-kafka-08-0000", "requiredCapacity" : 1 }, "spec" : { "dataSchema" : { "dataSource" : "metrics-kafka", "parser" : { "type" : "map", "parseSpec" : { "format" : "json", "timestampSpec" : { "column" : "timestamp", "format" : "auto", "missingValue" : null }, "dimensionsSpec" : { "dimensions" : [ "http_code", "http_method", "metricType", "page", "server", "timestamp", "unit", "value" ], "spatialDimensions" : [ ] } } }, "metricsSpec" : [ { "type" : "count", "name" : "count" }, { "type" : "doubleSum", "name" : "value_sum", "fieldName" : "value" }, { "type" : "doubleMin", "name" : "value_min", "fieldName" : "value" }, { "type" : "doubleMax", "name" : "value_max", "fieldName" : "value" } ], "granularitySpec" : { "type" : "uniform", "segmentGranularity" : "HOUR", "queryGranularity" : { "type" : "none" } } }, "ioConfig" : { "type" : "realtime", "plumber" : null, "firehose" : { "type" : "clipped", "interval" : "2016-04-29T08:00:00.000Z/2016-04-29T09:00:00.000Z", "delegate" : { "type" : "timed", "shutoffTime" : "2016-04-29T09:15:00.000Z", "delegate" : { "type" : "receiver", "serviceName" : "firehose:druid:overlord:metrics-kafka-08-0000-0000", "bufferSize" : 100000 } } } }, "tuningConfig" : { "shardSpec" : { "type" : "linear", "partitionNum" : 0 }, "rejectionPolicy" : { "type" : "none" }, "buildV9Directly" : false, "maxPendingPersists" : 0, "intermediatePersistPeriod" : "PT10M", "windowPeriod" : "PT10M", "type" : "realtime", "maxRowsInMemory" : "100000" } } } 2016-04-29 08:52:53,533 [finagle/netty3-1] WARN c.m.tranquility.finagle.FutureRetry$ - Transient error, will try again in 15,158 ms com.metamx.tranquility.druid.IndexServiceTransientException: Service[druid:overlord] call failed with status: 500 Internal Server Error at com.metamx.tranquility.druid.IndexService$$anonfun$call$1$$anonfun$apply$17.apply(IndexService.scala:150) ~[io.druid.tranquility-core-0.7.4.jar:0.7.4] at com.metamx.tranquility.druid.IndexService$$anonfun$call$1$$anonfun$apply$17.apply(IndexService.scala:132) ~[io.druid.tranquility-core-0.7.4.jar:0.7.4] at com.twitter.util.Future$$anonfun$map$1$$anonfun$apply$6.apply(Future.scala:950) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Try$.apply(Try.scala:13) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$.apply(Future.scala:97) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$$anonfun$map$1.apply(Future.scala:950) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$$anonfun$map$1.apply(Future.scala:949) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.liftedTree1$1(Promise.scala:112) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.k(Promise.scala:112) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.apply(Promise.scala:122) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.apply(Promise.scala:103) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$$anon$1.run(Promise.scala:366) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler$Activation.run(Scheduler.scala:178) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler$Activation.submit(Scheduler.scala:136) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler.submit(Scheduler.scala:207) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.Scheduler$.submit(Scheduler.scala:92) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.runq(Promise.scala:350) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.updateIfEmpty(Promise.scala:721) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.update(Promise.scala:694) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.setValue(Promise.scala:670) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.AsyncQueue.offer(AsyncQueue.scala:111) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.finagle.netty3.transport.ChannelTransport.handleUpstream(ChannelTransport.scala:55) [com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.http.HttpContentDecoder.messageReceived(HttpContentDecoder.java:108) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:145) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.http.HttpClientCodec.handleUpstream(HttpClientCodec.java:92) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142) [io.netty.netty-3.10.5.Final.jar:na] at com.twitter.finagle.netty3.channel.ChannelStatsHandler.messageReceived(ChannelStatsHandler.scala:78) [com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0] at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142) [io.netty.netty-3.10.5.Final.jar:na] at com.twitter.finagle.netty3.channel.ChannelRequestStatsHandler.messageReceived(ChannelRequestStatsHandler.scala:35) [com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0] at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) [io.netty.netty-3.10.5.Final.jar:na] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_79] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_79] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79] 2016-04-29 08:53:08,718 [finagle/netty3-2] WARN c.m.tranquility.finagle.FutureRetry$ - Transient error, will try again in 18,720 ms com.metamx.tranquility.druid.IndexServiceTransientException: Service[druid:overlord] call failed with status: 500 Internal Server Error at com.metamx.tranquility.druid.IndexService$$anonfun$call$1$$anonfun$apply$17.apply(IndexService.scala:150) ~[io.druid.tranquility-core-0.7.4.jar:0.7.4] at com.metamx.tranquility.druid.IndexService$$anonfun$call$1$$anonfun$apply$17.apply(IndexService.scala:132) ~[io.druid.tranquility-core-0.7.4.jar:0.7.4] at com.twitter.util.Future$$anonfun$map$1$$anonfun$apply$6.apply(Future.scala:950) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Try$.apply(Try.scala:13) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$.apply(Future.scala:97) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$$anonfun$map$1.apply(Future.scala:950) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$$anonfun$map$1.apply(Future.scala:949) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.liftedTree1$1(Promise.scala:112) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.k(Promise.scala:112) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.apply(Promise.scala:122) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.apply(Promise.scala:103) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$$anon$1.run(Promise.scala:366) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler$Activation.run(Scheduler.scala:178) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler$Activation.submit(Scheduler.scala:136) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler.submit(Scheduler.scala:207) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.Scheduler$.submit(Scheduler.scala:92) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.runq(Promise.scala:350) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.updateIfEmpty(Promise.scala:721) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.update(Promise.scala:694) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.setValue(Promise.scala:670) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.AsyncQueue.offer(AsyncQueue.scala:111) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.finagle.netty3.transport.ChannelTransport.handleUpstream(ChannelTransport.scala:55) [com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.http.HttpContentDecoder.messageReceived(HttpContentDecoder.java:108) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:145) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.http.HttpClientCodec.handleUpstream(HttpClientCodec.java:92) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142) [io.netty.netty-3.10.5.Final.jar:na] at com.twitter.finagle.netty3.channel.ChannelStatsHandler.messageReceived(ChannelStatsHandler.scala:78) [com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0] at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142) [io.netty.netty-3.10.5.Final.jar:na] at com.twitter.finagle.netty3.channel.ChannelRequestStatsHandler.messageReceived(ChannelRequestStatsHandler.scala:35) [com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0] at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) [io.netty.netty-3.10.5.Final.jar:na] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_79] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_79] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79] 2016-04-29 08:53:27,468 [finagle/netty3-3] WARN c.m.tranquility.finagle.FutureRetry$ - Transient error, will try again in 35,347 ms com.metamx.tranquility.druid.IndexServiceTransientException: Service[druid:overlord] call failed with status: 500 Internal Server Error at com.metamx.tranquility.druid.IndexService$$anonfun$call$1$$anonfun$apply$17.apply(IndexService.scala:150) ~[io.druid.tranquility-core-0.7.4.jar:0.7.4] at com.metamx.tranquility.druid.IndexService$$anonfun$call$1$$anonfun$apply$17.apply(IndexService.scala:132) ~[io.druid.tranquility-core-0.7.4.jar:0.7.4] at com.twitter.util.Future$$anonfun$map$1$$anonfun$apply$6.apply(Future.scala:950) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Try$.apply(Try.scala:13) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$.apply(Future.scala:97) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$$anonfun$map$1.apply(Future.scala:950) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$$anonfun$map$1.apply(Future.scala:949) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.liftedTree1$1(Promise.scala:112) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.k(Promise.scala:112) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.apply(Promise.scala:122) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.apply(Promise.scala:103) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$$anon$1.run(Promise.scala:366) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler$Activation.run(Scheduler.scala:178) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler$Activation.submit(Scheduler.scala:136) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler.submit(Scheduler.scala:207) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.Scheduler$.submit(Scheduler.scala:92) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.runq(Promise.scala:350) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.updateIfEmpty(Promise.scala:721) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.update(Promise.scala:694) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.setValue(Promise.scala:670) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.AsyncQueue.offer(AsyncQueue.scala:111) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.finagle.netty3.transport.ChannelTransport.handleUpstream(ChannelTransport.scala:55) [com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.http.HttpContentDecoder.messageReceived(HttpContentDecoder.java:108) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:145) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.http.HttpClientCodec.handleUpstream(HttpClientCodec.java:92) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142) [io.netty.netty-3.10.5.Final.jar:na] at com.twitter.finagle.netty3.channel.ChannelStatsHandler.messageReceived(ChannelStatsHandler.scala:78) [com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0] at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142) [io.netty.netty-3.10.5.Final.jar:na] at com.twitter.finagle.netty3.channel.ChannelRequestStatsHandler.messageReceived(ChannelRequestStatsHandler.scala:35) [com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0] at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) [io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) [io.netty.netty-3.10.5.Final.jar:na] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_79] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_79] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79] 2016-04-29 08:54:02,848 [ClusteredBeam-ZkFuturePool-1e419b48-1aa3-4cc4-9ba9-15dadf5e53e6] ERROR c.m.tranquility.beam.ClusteredBeam - Failed to update cluster state: druid:overlord/metrics-kafka com.metamx.tranquility.druid.IndexServiceTransientException: Service[druid:overlord] call failed with status: 500 Internal Server Error at com.metamx.tranquility.druid.IndexService$$anonfun$call$1$$anonfun$apply$17.apply(IndexService.scala:150) ~[io.druid.tranquility-core-0.7.4.jar:0.7.4] at com.metamx.tranquility.druid.IndexService$$anonfun$call$1$$anonfun$apply$17.apply(IndexService.scala:132) ~[io.druid.tranquility-core-0.7.4.jar:0.7.4] at com.twitter.util.Future$$anonfun$map$1$$anonfun$apply$6.apply(Future.scala:950) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Try$.apply(Try.scala:13) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$.apply(Future.scala:97) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$$anonfun$map$1.apply(Future.scala:950) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$$anonfun$map$1.apply(Future.scala:949) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.liftedTree1$1(Promise.scala:112) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.k(Promise.scala:112) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.apply(Promise.scala:122) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.apply(Promise.scala:103) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$$anon$1.run(Promise.scala:366) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler$Activation.run(Scheduler.scala:178) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler$Activation.submit(Scheduler.scala:136) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler.submit(Scheduler.scala:207) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.Scheduler$.submit(Scheduler.scala:92) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.runq(Promise.scala:350) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.updateIfEmpty(Promise.scala:721) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.update(Promise.scala:694) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.setValue(Promise.scala:670) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.AsyncQueue.offer(AsyncQueue.scala:111) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.finagle.netty3.transport.ChannelTransport.handleUpstream(ChannelTransport.scala:55) ~[com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.http.HttpContentDecoder.messageReceived(HttpContentDecoder.java:108) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:145) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.http.HttpClientCodec.handleUpstream(HttpClientCodec.java:92) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142) ~[io.netty.netty-3.10.5.Final.jar:na] at com.twitter.finagle.netty3.channel.ChannelStatsHandler.messageReceived(ChannelStatsHandler.scala:78) ~[com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0] at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142) ~[io.netty.netty-3.10.5.Final.jar:na] at com.twitter.finagle.netty3.channel.ChannelRequestStatsHandler.messageReceived(ChannelRequestStatsHandler.scala:35) ~[com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0] at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) ~[io.netty.netty-3.10.5.Final.jar:na] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_79] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_79] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79] 2016-04-29 08:54:02,853 [ClusteredBeam-ZkFuturePool-1e419b48-1aa3-4cc4-9ba9-15dadf5e53e6] WARN c.m.tranquility.beam.ClusteredBeam - Emitting alert: [anomaly] Failed to create merged beam: druid:overlord/metrics-kafka { } java.lang.IllegalStateException: Failed to save new beam for identifier[druid:overlord/metrics-kafka] timestamp[2016-04-29T08:00:00.000Z] at com.metamx.tranquility.beam.ClusteredBeam$$anonfun$2.applyOrElse(ClusteredBeam.scala:286) ~[io.druid.tranquility-core-0.7.4.jar:0.7.4] at com.metamx.tranquility.beam.ClusteredBeam$$anonfun$2.applyOrElse(ClusteredBeam.scala:283) ~[io.druid.tranquility-core-0.7.4.jar:0.7.4] at com.twitter.util.Future$$anonfun$rescue$1.apply(Future.scala:924) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$$anonfun$rescue$1.apply(Future.scala:922) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.liftedTree1$1(Promise.scala:112) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.k(Promise.scala:112) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.apply(Promise.scala:122) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.apply(Promise.scala:103) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$$anon$1.run(Promise.scala:366) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler$Activation.run(Scheduler.scala:178) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler$Activation.submit(Scheduler.scala:136) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler.submit(Scheduler.scala:207) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.Scheduler$.submit(Scheduler.scala:92) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.runq(Promise.scala:350) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.updateIfEmpty(Promise.scala:721) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.ExecutorServiceFuturePool$$anon$2.run(FuturePool.scala:107) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_79] at java.util.concurrent.FutureTask.run(FutureTask.java:262) [na:1.7.0_79] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_79] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_79] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79] Caused by: com.metamx.tranquility.druid.IndexServiceTransientException: Service[druid:overlord] call failed with status: 500 Internal Server Error at com.metamx.tranquility.druid.IndexService$$anonfun$call$1$$anonfun$apply$17.apply(IndexService.scala:150) ~[io.druid.tranquility-core-0.7.4.jar:0.7.4] at com.metamx.tranquility.druid.IndexService$$anonfun$call$1$$anonfun$apply$17.apply(IndexService.scala:132) ~[io.druid.tranquility-core-0.7.4.jar:0.7.4] at com.twitter.util.Future$$anonfun$map$1$$anonfun$apply$6.apply(Future.scala:950) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Try$.apply(Try.scala:13) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$.apply(Future.scala:97) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$$anonfun$map$1.apply(Future.scala:950) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$$anonfun$map$1.apply(Future.scala:949) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.liftedTree1$1(Promise.scala:112) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.k(Promise.scala:112) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.apply(Promise.scala:122) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.apply(Promise.scala:103) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$$anon$1.run(Promise.scala:366) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler$Activation.run(Scheduler.scala:178) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler$Activation.submit(Scheduler.scala:136) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler.submit(Scheduler.scala:207) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.Scheduler$.submit(Scheduler.scala:92) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.runq(Promise.scala:350) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.updateIfEmpty(Promise.scala:721) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.update(Promise.scala:694) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.setValue(Promise.scala:670) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.AsyncQueue.offer(AsyncQueue.scala:111) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.finagle.netty3.transport.ChannelTransport.handleUpstream(ChannelTransport.scala:55) ~[com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.http.HttpContentDecoder.messageReceived(HttpContentDecoder.java:108) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:145) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.http.HttpClientCodec.handleUpstream(HttpClientCodec.java:92) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142) ~[io.netty.netty-3.10.5.Final.jar:na] at com.twitter.finagle.netty3.channel.ChannelStatsHandler.messageReceived(ChannelStatsHandler.scala:78) ~[com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0] at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142) ~[io.netty.netty-3.10.5.Final.jar:na] at com.twitter.finagle.netty3.channel.ChannelRequestStatsHandler.messageReceived(ChannelRequestStatsHandler.scala:35) ~[com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0] at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) ~[io.netty.netty-3.10.5.Final.jar:na] ... 3 common frames omitted 2016-04-29 08:54:02,855 [ClusteredBeam-ZkFuturePool-1e419b48-1aa3-4cc4-9ba9-15dadf5e53e6] INFO c.metamx.emitter.core.LoggingEmitter - Event [{"feed":"alerts","timestamp":"2016-04-29T08:54:02.855Z","service":"tranquility","host":"localhost","severity":"anomaly","description":"Failed to create merged beam: druid:overlord/metrics-kafka","data":{"exceptionType":"java.lang.IllegalStateException","exceptionMessage":"Failed to save new beam for identifier[druid:overlord/metrics-kafka] timestamp[2016-04-29T08:00:00.000Z]","exceptionStackTrace":"java.lang.IllegalStateException: Failed to save new beam for identifier[druid:overlord/metrics-kafka] timestamp[2016-04-29T08:00:00.000Z]\n\tat com.metamx.tranquility.beam.ClusteredBeam$$anonfun$2.applyOrElse(ClusteredBeam.scala:286)\n\tat com.metamx.tranquility.beam.ClusteredBeam$$anonfun$2.applyOrElse(ClusteredBeam.scala:283)\n\tat com.twitter.util.Future$$anonfun$rescue$1.apply(Future.scala:924)\n\tat com.twitter.util.Future$$anonfun$rescue$1.apply(Future.scala:922)\n\tat com.twitter.util.Promise$Transformer.liftedTree1$1(Promise.scala:112)\n\tat com.twitter.util.Promise$Transformer.k(Promise.scala:112)\n\tat com.twitter.util.Promise$Transformer.apply(Promise.scala:122)\n\tat com.twitter.util.Promise$Transformer.apply(Promise.scala:103)\n\tat com.twitter.util.Promise$$anon$1.run(Promise.scala:366)\n\tat com.twitter.concurrent.LocalScheduler$Activation.run(Scheduler.scala:178)\n\tat com.twitter.concurrent.LocalScheduler$Activation.submit(Scheduler.scala:136)\n\tat com.twitter.concurrent.LocalScheduler.submit(Scheduler.scala:207)\n\tat com.twitter.concurrent.Scheduler$.submit(Scheduler.scala:92)\n\tat com.twitter.util.Promise.runq(Promise.scala:350)\n\tat com.twitter.util.Promise.updateIfEmpty(Promise.scala:721)\n\tat com.twitter.util.ExecutorServiceFuturePool$$anon$2.run(FuturePool.scala:107)\n\tat java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:262)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)\n\tat java.lang.Thread.run(Thread.java:745)\nCaused by: com.metamx.tranquility.druid.IndexServiceTransientException: Service[druid:overlord] call failed with status: 500 Internal Server Error\n\tat com.metamx.tranquility.druid.IndexService$$anonfun$call$1$$anonfun$apply$17.apply(IndexService.scala:150)\n\tat com.metamx.tranquility.druid.IndexService$$anonfun$call$1$$anonfun$apply$17.apply(IndexService.scala:132)\n\tat com.twitter.util.Future$$anonfun$map$1$$anonfun$apply$6.apply(Future.scala:950)\n\tat com.twitter.util.Try$.apply(Try.scala:13)\n\tat com.twitter.util.Future$.apply(Future.scala:97)\n\tat com.twitter.util.Future$$anonfun$map$1.apply(Future.scala:950)\n\tat com.twitter.util.Future$$anonfun$map$1.apply(Future.scala:949)\n\tat com.twitter.util.Promise$Transformer.liftedTree1$1(Promise.scala:112)\n\tat com.twitter.util.Promise$Transformer.k(Promise.scala:112)\n\tat com.twitter.util.Promise$Transformer.apply(Promise.scala:122)\n\tat com.twitter.util.Promise$Transformer.apply(Promise.scala:103)\n\tat com.twitter.util.Promise$$anon$1.run(Promise.scala:366)\n\tat com.twitter.concurrent.LocalScheduler$Activation.run(Scheduler.scala:178)\n\tat com.twitter.concurrent.LocalScheduler$Activation.submit(Scheduler.scala:136)\n\tat com.twitter.concurrent.LocalScheduler.submit(Scheduler.scala:207)\n\tat com.twitter.concurrent.Scheduler$.submit(Scheduler.scala:92)\n\tat com.twitter.util.Promise.runq(Promise.scala:350)\n\tat com.twitter.util.Promise.updateIfEmpty(Promise.scala:721)\n\tat com.twitter.util.Promise.update(Promise.scala:694)\n\tat com.twitter.util.Promise.setValue(Promise.scala:670)\n\tat com.twitter.concurrent.AsyncQueue.offer(AsyncQueue.scala:111)\n\tat com.twitter.finagle.netty3.transport.ChannelTransport.handleUpstream(ChannelTransport.scala:55)\n\tat org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)\n\tat org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)\n\tat org.jboss.netty.handler.codec.http.HttpContentDecoder.messageReceived(HttpContentDecoder.java:108)\n\tat org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)\n\tat org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)\n\tat org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)\n\tat org.jboss.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:145)\n\tat org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)\n\tat org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)\n\tat org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)\n\tat org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)\n\tat org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459)\n\tat org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536)\n\tat org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435)\n\tat org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)\n\tat org.jboss.netty.handler.codec.http.HttpClientCodec.handleUpstream(HttpClientCodec.java:92)\n\tat org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)\n\tat org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)\n\tat org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142)\n\tat com.twitter.finagle.netty3.channel.ChannelStatsHandler.messageReceived(ChannelStatsHandler.scala:78)\n\tat org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88)\n\tat org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)\n\tat org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)\n\tat org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142)\n\tat com.twitter.finagle.netty3.channel.ChannelRequestStatsHandler.messageReceived(ChannelRequestStatsHandler.scala:35)\n\tat org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88)\n\tat org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)\n\tat org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)\n\tat org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)\n\tat org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)\n\tat org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)\n\tat org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)\n\tat org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)\n\tat org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)\n\tat org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)\n\tat org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)\n\tat org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)\n\t... 3 more\n"}}] 2016-04-29 08:54:02,856 [ClusteredBeam-ZkFuturePool-1e419b48-1aa3-4cc4-9ba9-15dadf5e53e6] WARN c.m.t.tranquilizer.Tranquilizer - Failed to send 20 messages. java.lang.IllegalStateException: Failed to save new beam for identifier[druid:overlord/metrics-kafka] timestamp[2016-04-29T08:00:00.000Z] at com.metamx.tranquility.beam.ClusteredBeam$$anonfun$2.applyOrElse(ClusteredBeam.scala:286) ~[io.druid.tranquility-core-0.7.4.jar:0.7.4] at com.metamx.tranquility.beam.ClusteredBeam$$anonfun$2.applyOrElse(ClusteredBeam.scala:283) ~[io.druid.tranquility-core-0.7.4.jar:0.7.4] at com.twitter.util.Future$$anonfun$rescue$1.apply(Future.scala:924) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$$anonfun$rescue$1.apply(Future.scala:922) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.liftedTree1$1(Promise.scala:112) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.k(Promise.scala:112) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.apply(Promise.scala:122) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.apply(Promise.scala:103) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$$anon$1.run(Promise.scala:366) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler$Activation.run(Scheduler.scala:178) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler$Activation.submit(Scheduler.scala:136) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler.submit(Scheduler.scala:207) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.Scheduler$.submit(Scheduler.scala:92) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.runq(Promise.scala:350) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.updateIfEmpty(Promise.scala:721) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.ExecutorServiceFuturePool$$anon$2.run(FuturePool.scala:107) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_79] at java.util.concurrent.FutureTask.run(FutureTask.java:262) [na:1.7.0_79] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_79] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_79] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79] Caused by: com.metamx.tranquility.druid.IndexServiceTransientException: Service[druid:overlord] call failed with status: 500 Internal Server Error at com.metamx.tranquility.druid.IndexService$$anonfun$call$1$$anonfun$apply$17.apply(IndexService.scala:150) ~[io.druid.tranquility-core-0.7.4.jar:0.7.4] at com.metamx.tranquility.druid.IndexService$$anonfun$call$1$$anonfun$apply$17.apply(IndexService.scala:132) ~[io.druid.tranquility-core-0.7.4.jar:0.7.4] at com.twitter.util.Future$$anonfun$map$1$$anonfun$apply$6.apply(Future.scala:950) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Try$.apply(Try.scala:13) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$.apply(Future.scala:97) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$$anonfun$map$1.apply(Future.scala:950) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$$anonfun$map$1.apply(Future.scala:949) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.liftedTree1$1(Promise.scala:112) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.k(Promise.scala:112) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.apply(Promise.scala:122) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.apply(Promise.scala:103) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$$anon$1.run(Promise.scala:366) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler$Activation.run(Scheduler.scala:178) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler$Activation.submit(Scheduler.scala:136) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler.submit(Scheduler.scala:207) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.Scheduler$.submit(Scheduler.scala:92) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.runq(Promise.scala:350) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.updateIfEmpty(Promise.scala:721) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.update(Promise.scala:694) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.setValue(Promise.scala:670) [com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.AsyncQueue.offer(AsyncQueue.scala:111) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.finagle.netty3.transport.ChannelTransport.handleUpstream(ChannelTransport.scala:55) ~[com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.http.HttpContentDecoder.messageReceived(HttpContentDecoder.java:108) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:145) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.http.HttpClientCodec.handleUpstream(HttpClientCodec.java:92) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142) ~[io.netty.netty-3.10.5.Final.jar:na] at com.twitter.finagle.netty3.channel.ChannelStatsHandler.messageReceived(ChannelStatsHandler.scala:78) ~[com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0] at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142) ~[io.netty.netty-3.10.5.Final.jar:na] at com.twitter.finagle.netty3.channel.ChannelRequestStatsHandler.messageReceived(ChannelRequestStatsHandler.scala:35) ~[com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0] at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) ~[io.netty.netty-3.10.5.Final.jar:na] ... 3 common frames omitted 2016-04-29 08:54:02,860 [KafkaConsumer-CommitThread] ERROR c.m.tranquility.kafka.KafkaConsumer - Commit thread failed! java.lang.IllegalStateException: Failed to save new beam for identifier[druid:overlord/metrics-kafka] timestamp[2016-04-29T08:00:00.000Z] at com.metamx.tranquility.beam.ClusteredBeam$$anonfun$2.applyOrElse(ClusteredBeam.scala:286) ~[io.druid.tranquility-core-0.7.4.jar:0.7.4] at com.metamx.tranquility.beam.ClusteredBeam$$anonfun$2.applyOrElse(ClusteredBeam.scala:283) ~[io.druid.tranquility-core-0.7.4.jar:0.7.4] at com.twitter.util.Future$$anonfun$rescue$1.apply(Future.scala:924) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$$anonfun$rescue$1.apply(Future.scala:922) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.liftedTree1$1(Promise.scala:112) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.k(Promise.scala:112) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.apply(Promise.scala:122) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.apply(Promise.scala:103) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$$anon$1.run(Promise.scala:366) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler$Activation.run(Scheduler.scala:178) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler$Activation.submit(Scheduler.scala:136) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler.submit(Scheduler.scala:207) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.Scheduler$.submit(Scheduler.scala:92) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.runq(Promise.scala:350) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.updateIfEmpty(Promise.scala:721) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.ExecutorServiceFuturePool$$anon$2.run(FuturePool.scala:107) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) ~[na:1.7.0_79] at java.util.concurrent.FutureTask.run(FutureTask.java:262) ~[na:1.7.0_79] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[na:1.7.0_79] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ~[na:1.7.0_79] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79] Caused by: com.metamx.tranquility.druid.IndexServicePermanentException: Service[druid:overlord] call failed with status: 400 Bad Request at com.metamx.tranquility.druid.IndexService$$anonfun$call$1$$anonfun$apply$17.apply(IndexService.scala:158) ~[io.druid.tranquility-core-0.7.4.jar:0.7.4] at com.metamx.tranquility.druid.IndexService$$anonfun$call$1$$anonfun$apply$17.apply(IndexService.scala:132) ~[io.druid.tranquility-core-0.7.4.jar:0.7.4] at com.twitter.util.Future$$anonfun$map$1$$anonfun$apply$6.apply(Future.scala:950) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Try$.apply(Try.scala:13) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$.apply(Future.scala:97) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$$anonfun$map$1.apply(Future.scala:950) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Future$$anonfun$map$1.apply(Future.scala:949) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.liftedTree1$1(Promise.scala:112) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.k(Promise.scala:112) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.apply(Promise.scala:122) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$Transformer.apply(Promise.scala:103) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise$$anon$1.run(Promise.scala:366) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler$Activation.run(Scheduler.scala:178) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler$Activation.submit(Scheduler.scala:136) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.LocalScheduler.submit(Scheduler.scala:207) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.Scheduler$.submit(Scheduler.scala:92) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.runq(Promise.scala:350) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.updateIfEmpty(Promise.scala:721) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.update(Promise.scala:694) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.util.Promise.setValue(Promise.scala:670) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.concurrent.AsyncQueue.offer(AsyncQueue.scala:111) ~[com.twitter.util-core_2.11-6.30.0.jar:6.30.0] at com.twitter.finagle.netty3.transport.ChannelTransport.handleUpstream(ChannelTransport.scala:55) ~[com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.http.HttpContentDecoder.messageReceived(HttpContentDecoder.java:108) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:194) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.handler.codec.http.HttpClientCodec.handleUpstream(HttpClientCodec.java:92) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142) ~[io.netty.netty-3.10.5.Final.jar:na] at com.twitter.finagle.netty3.channel.ChannelStatsHandler.messageReceived(ChannelStatsHandler.scala:78) ~[com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0] at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142) ~[io.netty.netty-3.10.5.Final.jar:na] at com.twitter.finagle.netty3.channel.ChannelRequestStatsHandler.messageReceived(ChannelRequestStatsHandler.scala:35) ~[com.twitter.finagle-core_2.11-6.31.0.jar:6.31.0] at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) ~[io.netty.netty-3.10.5.Final.jar:na] at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) ~[io.netty.netty-3.10.5.Final.jar:na] ... 3 common frames omitted 2016-04-29 08:54:02,861 [KafkaConsumer-CommitThread] INFO c.m.tranquility.kafka.KafkaConsumer - Shutting down - attempting to flush buffers and commit final offsets 2016-04-29 08:54:02,862 [KafkaConsumer-CommitThread] INFO k.c.ZookeeperConsumerConnector - [tranquility-kafka_master-1461919585364-7f6a5b44], ZKConsumerConnector shutting down 2016-04-29 08:54:02,876 [KafkaConsumer-CommitThread] INFO k.c.ZookeeperTopicEventWatcher - Shutting down topic event watcher. 2016-04-29 08:54:02,877 [KafkaConsumer-CommitThread] INFO k.consumer.ConsumerFetcherManager - [ConsumerFetcherManager-1461919585452] Stopping leader finder thread 2016-04-29 08:54:02,877 [KafkaConsumer-CommitThread] INFO k.c.ConsumerFetcherManager$LeaderFinderThread - [tranquility-kafka_master-1461919585364-7f6a5b44-leader-finder-thread], Shutting down 2016-04-29 08:54:02,878 [KafkaConsumer-CommitThread] INFO k.c.ConsumerFetcherManager$LeaderFinderThread - [tranquility-kafka_master-1461919585364-7f6a5b44-leader-finder-thread], Shutdown completed 2016-04-29 08:54:02,878 [tranquility-kafka_master-1461919585364-7f6a5b44-leader-finder-thread] INFO k.c.ConsumerFetcherManager$LeaderFinderThread - [tranquility-kafka_master-1461919585364-7f6a5b44-leader-finder-thread], Stopped 2016-04-29 08:54:02,878 [KafkaConsumer-CommitThread] INFO k.consumer.ConsumerFetcherManager - [ConsumerFetcherManager-1461919585452] Stopping all fetchers 2016-04-29 08:54:02,879 [KafkaConsumer-CommitThread] INFO kafka.consumer.ConsumerFetcherThread - [ConsumerFetcherThread-tranquility-kafka_master-1461919585364-7f6a5b44-0-2], Shutting down 2016-04-29 08:54:02,904 [ConsumerFetcherThread-tranquility-kafka_master-1461919585364-7f6a5b44-0-2] INFO kafka.consumer.SimpleConsumer - Reconnect due to socket error: java.nio.channels.ClosedByInterruptException 2016-04-29 08:54:02,905 [ConsumerFetcherThread-tranquility-kafka_master-1461919585364-7f6a5b44-0-2] INFO kafka.consumer.ConsumerFetcherThread - [ConsumerFetcherThread-tranquility-kafka_master-1461919585364-7f6a5b44-0-2], Stopped 2016-04-29 08:54:02,905 [KafkaConsumer-CommitThread] INFO kafka.consumer.ConsumerFetcherThread - [ConsumerFetcherThread-tranquility-kafka_master-1461919585364-7f6a5b44-0-2], Shutdown completed 2016-04-29 08:54:02,905 [KafkaConsumer-CommitThread] INFO k.consumer.ConsumerFetcherManager - [ConsumerFetcherManager-1461919585452] All connections stopped 2016-04-29 08:54:02,907 [ZkClient-EventThread-12-localhost:2181] INFO org.I0Itec.zkclient.ZkEventThread - Terminate ZkClient event thread. 2016-04-29 08:54:02,914 [KafkaConsumer-CommitThread] INFO org.apache.zookeeper.ZooKeeper - Session: 0x154384946db00a9 closed 2016-04-29 08:54:02,914 [main-EventThread] INFO org.apache.zookeeper.ClientCnxn - EventThread shut down 2016-04-29 08:54:02,915 [KafkaConsumer-CommitThread] INFO k.c.ZookeeperConsumerConnector - [tranquility-kafka_master-1461919585364-7f6a5b44], ZKConsumerConnector shutdown completed in 51 ms Exception in thread "KafkaConsumer-CommitThread" java.lang.IllegalStateException: Failed to save new beam for identifier[druid:overlord/metrics-kafka] timestamp[2016-04-29T08:00:00.000Z] at com.metamx.tranquility.beam.ClusteredBeam$$anonfun$2.applyOrElse(ClusteredBeam.scala:286) at com.metamx.tranquility.beam.ClusteredBeam$$anonfun$2.applyOrElse(ClusteredBeam.scala:283) at com.twitter.util.Future$$anonfun$rescue$1.apply(Future.scala:924) at com.twitter.util.Future$$anonfun$rescue$1.apply(Future.scala:922) at com.twitter.util.Promise$Transformer.liftedTree1$1(Promise.scala:112) at com.twitter.util.Promise$Transformer.k(Promise.scala:112) at com.twitter.util.Promise$Transformer.apply(Promise.scala:122) at com.twitter.util.Promise$Transformer.apply(Promise.scala:103) at com.twitter.util.Promise$$anon$1.run(Promise.scala:366) at com.twitter.concurrent.LocalScheduler$Activation.run(Scheduler.scala:178) at com.twitter.concurrent.LocalScheduler$Activation.submit(Scheduler.scala:136) at com.twitter.concurrent.LocalScheduler.submit(Scheduler.scala:207) at com.twitter.concurrent.Scheduler$.submit(Scheduler.scala:92) at com.twitter.util.Promise.runq(Promise.scala:350) at com.twitter.util.Promise.updateIfEmpty(Promise.scala:721) at com.twitter.util.ExecutorServiceFuturePool$$anon$2.run(FuturePool.scala:107) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: com.metamx.tranquility.druid.IndexServicePermanentException: Service[druid:overlord] call failed with status: 400 Bad Request at com.metamx.tranquility.druid.IndexService$$anonfun$call$1$$anonfun$apply$17.apply(IndexService.scala:158) at com.metamx.tranquility.druid.IndexService$$anonfun$call$1$$anonfun$apply$17.apply(IndexService.scala:132) at com.twitter.util.Future$$anonfun$map$1$$anonfun$apply$6.apply(Future.scala:950) at com.twitter.util.Try$.apply(Try.scala:13) at com.twitter.util.Future$.apply(Future.scala:97) at com.twitter.util.Future$$anonfun$map$1.apply(Future.scala:950) at com.twitter.util.Future$$anonfun$map$1.apply(Future.scala:949) at com.twitter.util.Promise$Transformer.liftedTree1$1(Promise.scala:112) at com.twitter.util.Promise$Transformer.k(Promise.scala:112) at com.twitter.util.Promise$Transformer.apply(Promise.scala:122) at com.twitter.util.Promise$Transformer.apply(Promise.scala:103) at com.twitter.util.Promise$$anon$1.run(Promise.scala:366) at com.twitter.concurrent.LocalScheduler$Activation.run(Scheduler.scala:178) at com.twitter.concurrent.LocalScheduler$Activation.submit(Scheduler.scala:136) at com.twitter.concurrent.LocalScheduler.submit(Scheduler.scala:207) at com.twitter.concurrent.Scheduler$.submit(Scheduler.scala:92) at com.twitter.util.Promise.runq(Promise.scala:350) at com.twitter.util.Promise.updateIfEmpty(Promise.scala:721) at com.twitter.util.Promise.update(Promise.scala:694) at com.twitter.util.Promise.setValue(Promise.scala:670) at com.twitter.concurrent.AsyncQueue.offer(AsyncQueue.scala:111) at com.twitter.finagle.netty3.transport.ChannelTransport.handleUpstream(ChannelTransport.scala:55) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) at org.jboss.netty.handler.codec.http.HttpContentDecoder.messageReceived(HttpContentDecoder.java:108) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) at org.jboss.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:194) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) at org.jboss.netty.handler.codec.http.HttpClientCodec.handleUpstream(HttpClientCodec.java:92) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) at org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142) at com.twitter.finagle.netty3.channel.ChannelStatsHandler.messageReceived(ChannelStatsHandler.scala:78) at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) at org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142) at com.twitter.finagle.netty3.channel.ChannelRequestStatsHandler.messageReceived(ChannelRequestStatsHandler.scala:35) at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) ... 3 more 2016-04-29 08:54:03,643 [tranquility-kafka_master-1461919585364-7f6a5b44_watcher_executor] INFO k.c.ZookeeperConsumerConnector - [tranquility-kafka_master-1461919585364-7f6a5b44], stopping watcher executor thread for consumer tranquility-kafka_master-1461919585364-7f6a5b44 2016-04-29 08:54:03,645 [Thread-4] INFO c.metamx.tranquility.kafka.KafkaMain - Initiating shutdown...