2016-02-25 11:34:30,588 [ClusteredBeam-ZkFuturePool-8b83a847-703c-4976-a6c0-1b92e86bac9b] INFO c.m.tranquility.beam.ClusteredBeam - Creating new merged beam foridentifier[druid:overlord/pageviews] timestamp[2016-02-25T11:00:00.000Z] (target = 1, actual = 0)
2016-02-25 11:34:30,724 [ClusteredBeam-ZkFuturePool-8b83a847-703c-4976-a6c0-1b92e86bac9b] INFO com.metamx.common.scala.control$ - Creating druid indexing task (service= druid:overlord): {
"type" : "index_realtime",
"id" : "index_realtime_pageviews_2016-02-25T11:00:00.000Z_0_0",
"resource" : {
"availabilityGroup" : "pageviews-11-0000",
"requiredCapacity" : 1
},
"spec" : {
"dataSchema" : {
"dataSource" : "pageviews",
"parser" : {
"type" : "map",
"parseSpec" : {
"format" : "json",
"timestampSpec" : {
"column" : "time_stamp",
"format" : "yyyy-MM-dd HH:mm:ss",
"missingValue" : null
},
"dimensionsSpec" : {
"dimensions" : [ "ad_type", "ad_variation"],
"spatialDimensions" : [ ]
}
}
},
"metricsSpec" : [ {
"type" : "count",
"name" : "count"
} ],
"granularitySpec" : {
"type" : "uniform",
"segmentGranularity" : "HOUR",
"queryGranularity" : {
"type" : "none"
}
}
},
"ioConfig" : {
"type" : "realtime",
"plumber" : null,
"firehose" : {
"type" : "clipped",
"interval" : "2016-02-25T11:00:00.000Z/2016-02-25T12:00:00.000Z",
"delegate" : {
"type" : "timed",
"shutoffTime" : "2016-02-25T12:15:00.000Z",
"delegate" : {
"type" : "receiver",
"serviceName" : "firehose:druid:overlord:pageviews-11-0000-0000",
"bufferSize" : 100000
}
}
}
},
"tuningConfig" : {
"shardSpec" : {
"type" : "linear",
"partitionNum" : 0
},
"rejectionPolicy" : {
"type" : "none"
},
"buildV9Directly" : false,
"maxPendingPersists" : 0,
"intermediatePersistPeriod" : "PT10M",
"windowPeriod" : "PT10M",
"type" : "realtime",
"maxRowsInMemory" : 100000
}
}
}
Feb 25, 2016 11:34:30 AM com.twitter.finagle.Init$$anonfun$1 apply$mcV$sp
INFO: Finagle version 6.31.0 (rev=50d3bb0eea5ad3ed332111d707184c80fed6a506) built at 20151203-164135
2016-02-25 11:34:30,958 [ClusteredBeam-ZkFuturePool-8b83a847-703c-4976-a6c0-1b92e86bac9b] INFO c.m.c.s.net.finagle.DiscoResolver - Updating instances forservice[druid:overlord] to Set()
2016-02-25 11:34:31,040 [ClusteredBeam-ZkFuturePool-8b83a847-703c-4976-a6c0-1b92e86bac9b] INFO c.m.t.finagle.FinagleRegistry - Created client for service:druid:overlord
2016-02-25 11:34:31,120 [ClusteredBeam-ZkFuturePool-8b83a847-703c-4976-a6c0-1b92e86bac9b] WARN c.m.tranquility.finagle.FutureRetry$ - Transient error, will try again in9,703 ms
com.twitter.finagle.NoBrokersAvailableException: No hosts are available for druid:overlord, Dtab.base=[], Dtab.local=[]
at com.twitter.finagle.NoStacktrace(Unknown Source) ~[na:na]
2016-02-25 11:34:40,839 [Hashed wheel timer #1] WARN c.m.tranquility.finagle.FutureRetry$ - Transient error, will try again in 23,565 ms
com.twitter.finagle.NoBrokersAvailableException: No hosts are available for druid:overlord, Dtab.base=[], Dtab.local=[] at com.twitter.finagle.NoStacktrace(Unknown Source) ~[na:na]
Can somebody help me with using tranquility for data ingestion from kafka to druid without using storm in a very basic setup?I've these four nodes up : coordinator, broker, historical and overlord.Also i have Tranquility kafka application running using this :bin/tranquility kafka -configFile conf/tranquility-kafka.json.Now when i insert data in a kafka topic mentioned in tranquility-kafka.json, i'm getting the error :
2016-02-25 11:34:30,588 [ClusteredBeam-ZkFuturePool-8b83a847-703c-4976-a6c0-1b92e86bac9b] INFO c.m.tranquility.beam.ClusteredBeam - Creating new merged beam for identifier[druid:overlord/pageviews] timestamp[2016-02-25T11:00:00.000Z] (target = 1, actual = 0)
2016-02-25 11:34:30,724 [ClusteredBeam-ZkFuturePool-8b83a847-703c-4976-a6c0-1b92e86bac9b] INFO com.metamx.common.scala.control$ - Creating druid indexing task (service = druid:overlord): {
"type" : "index_realtime",
"id" : "index_realtime_pageviews_2016-02-25T11:00:00.000Z_0_0",
"resource" : {
"availabilityGroup" : "pageviews-11-0000",
"requiredCapacity" : 1
},
"spec" : {
"dataSchema" : {
"dataSource" : "pageviews",
"parser" : {
"type" : "map",
"parseSpec" : {
"format" : "json",
"timestampSpec" : {
"column" : "time_stamp",
"format" : "yyyy-MM-dd HH:mm:ss",
"missingValue" : null
},
"dimensionsSpec" : {
"dimensions" : [ "ad_type", "ad_variation", "article_basis", "article_basis_2", "article_id", "article_page_data", "article_position", "article_strategy_used", "article_test_data", "bca", "bidder_data", "bidder_id", "bidder_offset", "bidder_position_id", "bucket_id", "canonical_hash", "canonical_url_hash", "channel_id", "channel_name", "channel_name2", "channel_name3", "container_creative_id", "country_code", "creative_id", "customer_id", "device_type_id", "domain_name", "dynamic_multiply_factor", "first_displayed_keyword_type_id", "geo", "http_referer", "ip_address", "ip_useragent_hash", "is_adp", "is_container", "is_https", "is_monetise", "is_proxy", "kbb_call_timeout", "kbb_page_data", "keyword_strategy_used", "keyword_term", "keyword_type_id", "landing_page_first_displayed_keyword_type_id", "landing_page_framework_page_id", "landing_page_keyword_tag", "landing_page_type_id", "lineitem_id", "logging_server_ip_address", "multiply_factor", "opp_id", "original_referer", "original_request_url", "page_type_id", "partner_id", "port", "portfolio_id", "provider10_ads_return_count", "provider10_id", "provider11_id", "provider12_ads_return_count", "provider12_id", "provider13_id", "provider15_ads_return_count", "provider17_id", "provider18_id", "publisher_url", "publisher_url_hash", "refering_domain", "refering_portfolio_id", "rtb_test_data", "serp_template_id", "slot_id", "sub_bidder_id", "template_id", "template_size_id", "test_data", "time_spent_on_last_page", "user_agent", "user_agent_group_id", "viewFlag", "view_id", "view_status", "visit_id", "visitor_id", "webserver_ip_address", "yahoo_keyword_term" ],
2016-02-25 11:34:30,958 [ClusteredBeam-ZkFuturePool-8b83a847-703c-4976-a6c0-1b92e86bac9b] INFO c.m.c.s.net.finagle.DiscoResolver - Updating instances for service[druid:overlord] to Set()
2016-02-25 11:34:31,040 [ClusteredBeam-ZkFuturePool-8b83a847-703c-4976-a6c0-1b92e86bac9b] INFO c.m.t.finagle.FinagleRegistry - Created client for service: druid:overlord
2016-02-25 11:34:31,120 [ClusteredBeam-ZkFuturePool-8b83a847-703c-4976-a6c0-1b92e86bac9b] WARN c.m.tranquility.finagle.FutureRetry$ - Transient error, will try again in 9,703 ms
com.twitter.finagle.NoBrokersAvailableException: No hosts are available for druid:overlord, Dtab.base=[], Dtab.local=[]
at com.twitter.finagle.NoStacktrace(Unknown Source) ~[na:na]
2016-02-25 11:34:40,839 [Hashed wheel timer #1] WARN c.m.tranquility.finagle.FutureRetry$ - Transient error, will try again in 23,565 ms
com.twitter.finagle.NoBrokersAvailableException: No hosts are available for druid:overlord, Dtab.base=[], Dtab.local=[]
at com.twitter.finagle.NoStacktrace(Unknown Source) ~[na:na]
--
You received this message because you are subscribed to the Google Groups "Druid User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to druid-user+...@googlegroups.com.
To post to this group, send email to druid...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/druid-user/599b52af-124b-45fc-834b-0e1b80359b26%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Hi Gian,
By changing the druid.service in overlord config to “druid/overlord” (default was "overlord") and “druid.selectors.indexing.serviceName” to “druid/overlord” (default was “overlord”) the above errors are removed and indexing task begins.
Thank you so much for your help ! :)
- Priyanka