tranquility error : twitter.finagle.NoBrokersAvailableException: No hosts are available for disco!ov

Harshada Madav

unread,

Nov 2, 2016, 8:01:24 AM11/2/16

to Druid User

overlord on server1 and its config is as follows

--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

druid.host=c14-og2a-hadoop-druid-2.srv.media.net

druid.port=8090

druid.service=overlord

druid.indexer.queue.startDelay=PT120S

druid.indexer.runner.type=local

druid.indexer.storage.type=metadata

druid.indexer.logs.type=file

druid.monitoring.monitors=["com.metamx.metrics.SysMonitor","com.metamx.metrics.JvmMonitor"]

druid.indexer.fork.property.druid.processing.numThreads=2

druid.indexer.fork.property.druid.computation.buffer.size=1000000000

--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

and i am getting following error

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

2016-11-02 11:58:07,975 [KafkaConsumer-2] INFO o.h.validator.internal.util.Version - HV000001: Hibernate Validator 5.1.3.Final

2016-11-02 11:58:08,279 [KafkaConsumer-2] INFO io.druid.guice.JsonConfigurator - Loaded class[class io.druid.guice.ExtensionsConfig] from props[druid.extensions.] as [ExtensionsConfig{searchCurrentClassloader=true, directory='extensions', hadoopDependenciesDir='hadoop-dependencies', loadList=null}]

2016-11-02 11:58:08,708 [KafkaConsumer-2] INFO c.metamx.emitter.core.LoggingEmitter - Start: started [true]

2016-11-02 11:58:09,369 [ClusteredBeam-ZkFuturePool-971b0c7f-8939-4c3e-a599-0f5afb177240] INFO c.m.tranquility.beam.ClusteredBeam - Creating new merged beam for identifier[overlord/addclick_stream_tr] timestamp[2016-11-02T11:00:00.000Z] (target = 1, actual = 0)

2016-11-02 11:58:09,418 [ClusteredBeam-ZkFuturePool-971b0c7f-8939-4c3e-a599-0f5afb177240] WARN c.m.tranquility.druid.DruidBeamMaker - DruidTuning key[rejectionPolicy] for task[index_realtime_addclick_stream_tr_2016-11-02T11:00:00.000Z_0_0] overridden from[Map(type -> messageTime)] to[Map(type -> none)].

2016-11-02 11:58:09,537 [ClusteredBeam-ZkFuturePool-971b0c7f-8939-4c3e-a599-0f5afb177240] INFO com.metamx.common.scala.control$ - Creating druid indexing task (service = overlord): {

"type" : "index_realtime",

"id" : "index_realtime_addclick_stream_tr_2016-11-02T11:00:00.000Z_0_0",

"resource" : {

"availabilityGroup" : "addclick_stream_tr-2016-11-02T11:00:00.000Z-0000",

"requiredCapacity" : 1

},

"spec" : {

"dataSchema" : {

"dataSource" : "addclick_stream_tr",

"parser" : {

"type" : "map",

"parseSpec" : {

"format" : "json",

"timestampSpec" : {

"column" : "time_stamp",

"format" : "millis",

"missingValue" : null

},

"dimensionsSpec" : {

"dimensions" : [ "accept_language_header", "ad_desc", "ad_display_url", "adepth", "ad_position", "ad_provider_url", "ads_return_count", "ad_title", "ad_type", "ad_variation", "article_page_data", "article_strategy_used", "article_subtype", "base_concept", "base_concept2", "bidder_id", "bucket_id", "called_in_frame", "channel_id", "channel_name", "channel_name2", "channel_name3", "click_status", "cookie_support", "country_code", "creative_id", "cropped_image_id", "customer_id", "ddepth", "device_type_id", "domain_name", "first_displayed_keyword_type_id", "http_referer", "ip_address", "ip_ad_hash", "ip_useragent_hash", "is_adp", "is_https", "is_monetise", "is_proxy", "javascript_support", "kbb_page_data", "keyword_id", "keyword_position_id", "keyword_strategy_used", "keyword_term", "keyword_test_data", "keyword_type_id", "landing_page_first_displayed_keyword_type_id", "landing_page_framework_page_id", "landing_page_keyword_tag", "landing_page_type_id", "language_header", "lineitem_id", "local_timezone", "logging_server_ip_address", "mouse_x_pos", "mouse_y_pos", "original_referer", "page_type_id", "partner_id", "portfolio_id", "provider_account_id", "provider_api_id", "provider_group_id", "provider_id", "publisher_url", "refering_portfolio_id", "rendered_ad_unit_type", "rtb_test_data", "screen_height", "screen_width", "sde", "serp_template_id", "site_link_position", "site_link_title", "sub_bidder_id", "template_id", "template_size_id", "user_agent", "user_agent_group_id", "view_status", "webserver_ip_address", "yahoo_d2r", "yahoo_keyword_term", "yahoo_market_place", "spam_rank", "audit_spam_flag", "cf_type", "cf_score" ],

"spatialDimensions" : [ ]

}

},

"metricsSpec" : [ {

"type" : "count",

"name" : "count"

}, {

"type" : "hyperUnique",

"name" : "DistinctVisitors",

"fieldName" : "visitor_id"

}, {

"type" : "doubleSum",

"name" : "net_bid",

"fieldName" : "net_bid"

}, {

"type" : "doubleSum",

"name" : "actual_cpc",

"fieldName" : "actual_cpc"

}, {

"type" : "doubleSum",

"name" : "net_total_revenue",

"fieldName" : "net_total_revenue"

} ],

"granularitySpec" : {

"type" : "uniform",

"segmentGranularity" : "HOUR",

"queryGranularity" : {

"type" : "duration",

"duration" : 60000,

"origin" : "1970-01-01T00:00:00.000Z"

}

},

"ioConfig" : {

"type" : "realtime",

"plumber" : null,

"firehose" : {

"type" : "clipped",

"interval" : "2016-11-02T11:00:00.000Z/2016-11-02T12:00:00.000Z",

"delegate" : {

"type" : "timed",

"shutoffTime" : "2016-11-02T12:15:00.000Z",

"delegate" : {

"type" : "receiver",

"serviceName" : "firehose:overlord:addclick_stream_tr-011-0000-0000",

"bufferSize" : 100000

}

},

"tuningConfig" : {

"shardSpec" : {

"type" : "linear",

"partitionNum" : 0

},

"rejectionPolicy" : {

"type" : "none"

},

"buildV9Directly" : false,

"maxPendingPersists" : 0,

"intermediatePersistPeriod" : "PT5M",

"windowPeriod" : "PT10M",

"type" : "realtime",

"maxRowsInMemory" : "300000"

}

Nov 02, 2016 11:58:09 AM com.twitter.finagle.Init$$anonfun$1 apply$mcV$sp

INFO: Finagle version 6.31.0 (rev=50d3bb0eea5ad3ed332111d707184c80fed6a506) built at 20151203-164135

2016-11-02 11:58:09,796 [ClusteredBeam-ZkFuturePool-971b0c7f-8939-4c3e-a599-0f5afb177240] INFO c.m.c.s.net.finagle.DiscoResolver - Updating instances for service[overlord] to Set()

2016-11-02 11:58:09,872 [ClusteredBeam-ZkFuturePool-971b0c7f-8939-4c3e-a599-0f5afb177240] INFO c.m.t.finagle.FinagleRegistry - Created client for service: disco!overlord

2016-11-02 11:58:09,930 [ClusteredBeam-ZkFuturePool-971b0c7f-8939-4c3e-a599-0f5afb177240] WARN c.m.tranquility.finagle.FutureRetry$ - Transient error, will try again in 13,573 ms

com.twitter.finagle.NoBrokersAvailableException: No hosts are available for disco!overlord, Dtab.base=[], Dtab.local=[]

at com.twitter.finagle.NoStacktrace(Unknown Source) ~[na:na]

--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

and there is no middle manager service on any server...

David Lim

unread,

Nov 2, 2016, 12:52:53 PM11/2/16

to Druid User

Hey Harshada,

The exception means that Tranquility can't communicate with the indexing task to send data to it. It looks like you set the overlord's druid.service=overlord, in which case you need to make sure your config for Tranquility has druid.selectors.indexing.serviceName=overlord. Also make sure your druid.discovery.curator.path is the same for Tranquility and the rest of your Druid nodes.

Your last statement seems to indicate that you don't have any middle managers running. In general you should have at least one middle manager to use the indexing service (you can use druid.indexer.runner.type=local without middle managers for testing, but production systems should use druid.indexer.runner.type=remote with middle managers).

Message has been deleted

Harshada Madav

unread,

Nov 3, 2016, 4:13:40 AM11/3/16

to Druid User

hello @david,

what i've done now is as follows,

overlord config is (on server No.1)

------------------------------------------------------------------------------------------------

druid.host=server1

druid.port=8090

druid.service=overlord

druid.indexer.queue.startDelay=PT120S

druid.indexer.runner.type=remote

druid.indexer.storage.type=metadata

druid.indexer.logs.type=file

druid.monitoring.monitors=["com.metamx.metrics.SysMonitor","com.metamx.metrics.JvmMonitor","io.druid.server.metrics.EventReceiverFirehoseMonitor"]

druid.indexer.fork.property.druid.processing.numThreads=2

druid.indexer.fork.property.druid.computation.buffer.size=1000000000

------------------------------------------------------------------------------------------------

middle manager config is (on server no.1 )

------------------------------------------------------------------------------------------------

druid.service=druid/middleManager

druid.port=8091

# Number of tasks per middleManager

druid.worker.capacity=3

# Task launch parameters

druid.indexer.runner.javaOpts=-server -Xmx2g -Duser.timezone=UTC -Dfile.encoding=UTF-8 -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager

druid.indexer.task.baseTaskDir=var/druid/task

# HTTP server threads

druid.server.http.numThreads=25

# Processing threads and buffers

druid.processing.buffer.sizeBytes=536870912

druid.processing.numThreads=2

# Hadoop indexing

druid.indexer.task.hadoopWorkingPath=var/druid/hadoop-tmp

druid.indexer.task.defaultHadoopCoordinates=["org.apache.hadoop:hadoop-client:2.3.0"]

------------------------------------------------------------------------------------------------

overlord config is (on server no.2)

------------------------------------------------------------------------------------------------

druid.host=server2

druid.port=8090

druid.service=overlord

druid.indexer.queue.startDelay=PT120S

druid.indexer.runner.type=local

druid.indexer.storage.type=metadata

druid.indexer.logs.type=file

druid.monitoring.monitors=["com.metamx.metrics.SysMonitor","com.metamx.metrics.JvmMonitor"]

druid.indexer.fork.property.druid.processing.numThreads=2

druid.indexer.fork.property.druid.computation.buffer.size=1000000000

------------------------------------------------------------------------------------------------

and my json file is :

------------------------------------------------------------------------------------------------

{

"dataSources": [

{

"spec": {

"dataSchema": {

"parser": {

"type": "string",

"parseSpec": {

"timestampSpec": {

"format": "yyyy-MM-dd HH:mm:ss",

"column": "time_stamp"

},

"dimensionsSpec": {

"dimensions": [

--------------------

]

},

"format": "json"

}

},

"dataSource": "............",

"granularitySpec": {

"segmentGranularity": "HOUR",

"type": "uniform",

"queryGranularity": "MINUTE"

},

"metricsSpec": [

----------------------

]

},

"tuningConfig": {

"maxRowsInMemory": "300000",

"type": "realtime",

"windowPeriod": "PT10M",

"intermediatePersistPeriod": "PT5M",

"rejectionPolicy": {

"type": "messageTime"

}

},

"properties": {

"topicPattern.priority": "1",

"topicPattern": "c[0-9]*\\.mn.acm.*",

"task.partitions": "1"

}

],

"properties": {

"zookeeper.connect": "localhost:2181",

"zookeeper.timeout": "PT20S",

"druid.selectors.indexing.serviceName": "overlord",

"druid.discovery.curator.path": "/data/realtime/basePersist",

"kafka.zookeeper.connect": "....................",

"kafka.group.id": "tranquility-consumer",

"kafka.offsets.storage": "kafka",

"reportDropsAsExceptions": "false",

"consumer.numThreads": "4"

}

------------------------------------------------------------------------------------------------

and i am running tranquility on sever no 1 but still i am getting same error..

that druid.discovery.curator.path is same for tranquility and druid also.

is there any issue with 2 overlords or else..?

or do i shut down overlord service on server no 1 and make druid.indexer.runner.type as remote on server no 2 and then i run my tranquility on server no 1 with middle manager..
will this work..?

Harshada Madav

unread,

Nov 3, 2016, 4:20:37 AM11/3/16

to Druid User

and one more thing is there

in my tranquility config i am setting druid.selectors.indexing.serviceName=overlord and when i submit/run it it shows like as follows :

-------------------------------------------

"ioConfig" : {

"type" : "realtime",

"plumber" : null,

"firehose" : {

"type" : "clipped",

"interval" : "2016-11-03T07:00:00.000Z/2016-11-03T08:00:00.000Z",

"delegate" : {

"type" : "timed",

"shutoffTime" : "2016-11-03T08:15:00.000Z",

"delegate" : {

"type" : "receiver",

"serviceName" : "firehose:overlord:addclick_stream_tr-007-0000-0000",

"bufferSize" : 100000

}

-------------------------------------------

same thing is happening with even with timestampSpec also

my config has following :

"timestampSpec": {

"format": "yyyy-MM-dd HH:mm:ss",

"column": "time_stamp"

}

and when i run tranquility logs it shows :

"timestampSpec" : {

"column" : "time_stamp",

"format" : "millis",

"missingValue" : null

},

why this is so..?

David Lim

unread,

Nov 3, 2016, 4:29:10 PM11/3/16

to Druid User

Hi Harshada,

Your value for druid.discovery.curator.path looks suspect. You set it to '/data/realtime/basePersist' in your Tranquility spec, did you also set it to this value in your common.runtime.properties? If you didn't, the default value is '/druid/discovery' and you should set this value in your Tranquility spec.

As for your questions about the config for the realtime indexing task, the serviceName for the task will not (and should not) be the same as the serviceName you provide for your overlord, and Tranquility will parse your event using whatever timestampSpec you provide (i.e. format: yyyy-MM-dd HH:mm:ss) but it looks like it sends it to Druid as millis so you shouldn't be concerned about that.

Harshada Madav

unread,

Nov 4, 2016, 5:26:58 AM11/4/16

to Druid User

hii Devid,

thanks for help.. i changed druid.discovery.curator.path to /druid/discovery and it worked .. thanks again

Reply all

Reply to author

Forward