Trouble ingesting data from kafka to druid without using storm

118 views
Skip to first unread message

Priyanka Tanvani

unread,
Feb 25, 2016, 6:43:01 AM2/25/16
to Druid User
Can somebody help me with using tranquility for data ingestion from kafka to druid without using storm in a very basic setup?
I've these four nodes up : coordinator, broker, historical and overlord. 
Also i have Tranquility kafka application running using this : 

bin/tranquility kafka -configFile conf/tranquility-kafka.json.

Now when i insert data in a kafka topic mentioned in tranquility-kafka.json, i'm getting the error :

2016-02-25 11:34:30,588 [ClusteredBeam-ZkFuturePool-8b83a847-703c-4976-a6c0-1b92e86bac9b] INFO  c.m.tranquility.beam.ClusteredBeam - Creating new merged beam foridentifier[druid:overlord/pageviews] timestamp[2016-02-25T11:00:00.000Z] (target = 1, actual = 0)



2016-02-25 11:34:30,724 [ClusteredBeam-ZkFuturePool-8b83a847-703c-4976-a6c0-1b92e86bac9b] INFO  com.metamx.common.scala.control$ - Creating druid indexing task (service= druid:overlord): {

 
"type" : "index_realtime",

 
"id" : "index_realtime_pageviews_2016-02-25T11:00:00.000Z_0_0",

 
"resource" : {

   
"availabilityGroup" : "pageviews-11-0000",

   
"requiredCapacity" : 1

 
},

 
"spec" : {

   
"dataSchema" : {

     
"dataSource" : "pageviews",

     
"parser" : {

       
"type" : "map",

       
"parseSpec" : {

         
"format" : "json",

         
"timestampSpec" : {

           
"column" : "time_stamp",

           
"format" : "yyyy-MM-dd HH:mm:ss",

           
"missingValue" : null

         
},

         
"dimensionsSpec" : {

           
"dimensions" : [ "ad_type", "ad_variation"],

           
"spatialDimensions" : [ ]

         
}

       
}

     
},

     
"metricsSpec" : [ {

       
"type" : "count",

       
"name" : "count"

     
} ],

     
"granularitySpec" : {

       
"type" : "uniform",

       
"segmentGranularity" : "HOUR",

       
"queryGranularity" : {

         
"type" : "none"

       
}

     
}

   
},

   
"ioConfig" : {

     
"type" : "realtime",

     
"plumber" : null,

     
"firehose" : {

       
"type" : "clipped",

       
"interval" : "2016-02-25T11:00:00.000Z/2016-02-25T12:00:00.000Z",

       
"delegate" : {

         
"type" : "timed",

         
"shutoffTime" : "2016-02-25T12:15:00.000Z",

         
"delegate" : {

           
"type" : "receiver",

           
"serviceName" : "firehose:druid:overlord:pageviews-11-0000-0000",

           
"bufferSize" : 100000

         
}

       
}

     
}

   
},

   
"tuningConfig" : {

     
"shardSpec" : {

       
"type" : "linear",

       
"partitionNum" : 0

     
},

     
"rejectionPolicy" : {

       
"type" : "none"

     
},

     
"buildV9Directly" : false,

     
"maxPendingPersists" : 0,

     
"intermediatePersistPeriod" : "PT10M",

     
"windowPeriod" : "PT10M",

     
"type" : "realtime",

     
"maxRowsInMemory" : 100000

   
}

 
}

}

Feb 25, 2016 11:34:30 AM com.twitter.finagle.Init$$anonfun$1 apply$mcV$sp

INFO
: Finagle version 6.31.0 (rev=50d3bb0eea5ad3ed332111d707184c80fed6a506) built at 20151203-164135

2016-02-25 11:34:30,958 [ClusteredBeam-ZkFuturePool-8b83a847-703c-4976-a6c0-1b92e86bac9b] INFO  c.m.c.s.net.finagle.DiscoResolver - Updating instances forservice[druid:overlord] to Set()

2016-02-25 11:34:31,040 [ClusteredBeam-ZkFuturePool-8b83a847-703c-4976-a6c0-1b92e86bac9b] INFO  c.m.t.finagle.FinagleRegistry - Created client for service:druid:overlord

2016-02-25 11:34:31,120 [ClusteredBeam-ZkFuturePool-8b83a847-703c-4976-a6c0-1b92e86bac9b] WARN  c.m.tranquility.finagle.FutureRetry$ - Transient error, will try again in9,703 ms

com
.twitter.finagle.NoBrokersAvailableException: No hosts are available for druid:overlord, Dtab.base=[], Dtab.local=[]

 at com
.twitter.finagle.NoStacktrace(Unknown Source) ~[na:na]

2016-02-25 11:34:40,839 [Hashed wheel timer #1] WARN  c.m.tranquility.finagle.FutureRetry$ - Transient error, will try again in 23,565 ms

com
.twitter.finagle.NoBrokersAvailableException: No hosts are available for druid:overlord, Dtab.base=[], Dtab.local=[] at com.twitter.finagle.NoStacktrace(Unknown Source) ~[na:na]


Gian Merlino

unread,
Feb 25, 2016, 5:55:51 PM2/25/16
to druid...@googlegroups.com
Hey Priyanka,

If tranquility-kafka is not finding the overlord, try checking these things:

1) your overlord is up and running
2) your overlord's druid.service matches what Tranquility is looking for (in your case- "druid/overlord", which is also the default)
3) your tranquility-kafka "zookeeper.connect" should match the ZooKeeper server used by Druid
4) your tranquility-kafka "kafka.zookeeper.connect" should match the ZooKeeper server used by Kafka

Gian

On Thu, Feb 25, 2016 at 3:36 AM, Priyanka Tanvani <priyank...@gmail.com> wrote:
Can somebody help me with using tranquility for data ingestion from kafka to druid without using storm in a very basic setup?
I've these four nodes up : coordinator, broker, historical and overlord. 
Also i have Tranquility kafka application running using this : 

bin/tranquility kafka -configFile conf/tranquility-kafka.json.

Now when i insert data in a kafka topic mentioned in tranquility-kafka.json, i'm getting the error :




2016-02-25 11:34:30,588 [ClusteredBeam-ZkFuturePool-8b83a847-703c-4976-a6c0-1b92e86bac9b] INFO  c.m.tranquility.beam.ClusteredBeam - Creating new merged beam for identifier[druid:overlord/pageviews] timestamp[2016-02-25T11:00:00.000Z] (target = 1, actual = 0)
2016-02-25 11:34:30,724 [ClusteredBeam-ZkFuturePool-8b83a847-703c-4976-a6c0-1b92e86bac9b] INFO  com.metamx.common.scala.control$ - Creating druid indexing task (service = druid:overlord): {

 
"type" : "index_realtime",
 
"id" : "index_realtime_pageviews_2016-02-25T11:00:00.000Z_0_0",
 
"resource" : {
   
"availabilityGroup" : "pageviews-11-0000",
   
"requiredCapacity" : 1
 
},
 
"spec" : {
   
"dataSchema" : {
     
"dataSource" : "pageviews",
     
"parser" : {
       
"type" : "map",
       
"parseSpec" : {
         
"format" : "json",
         
"timestampSpec" : {
           
"column" : "time_stamp",
           
"format" : "yyyy-MM-dd HH:mm:ss",
           
"missingValue" : null
         
},
         
"dimensionsSpec" : {

           
"dimensions" : [ "ad_type", "ad_variation", "article_basis", "article_basis_2", "article_id", "article_page_data", "article_position", "article_strategy_used", "article_test_data", "bca", "bidder_data", "bidder_id", "bidder_offset", "bidder_position_id", "bucket_id", "canonical_hash", "canonical_url_hash", "channel_id", "channel_name", "channel_name2", "channel_name3", "container_creative_id", "country_code", "creative_id", "customer_id", "device_type_id", "domain_name", "dynamic_multiply_factor", "first_displayed_keyword_type_id", "geo", "http_referer", "ip_address", "ip_useragent_hash", "is_adp", "is_container", "is_https", "is_monetise", "is_proxy", "kbb_call_timeout", "kbb_page_data", "keyword_strategy_used", "keyword_term", "keyword_type_id", "landing_page_first_displayed_keyword_type_id", "landing_page_framework_page_id", "landing_page_keyword_tag", "landing_page_type_id", "lineitem_id", "logging_server_ip_address", "multiply_factor", "opp_id", "original_referer", "original_request_url", "page_type_id", "partner_id", "port", "portfolio_id", "provider10_ads_return_count", "provider10_id", "provider11_id", "provider12_ads_return_count", "provider12_id", "provider13_id", "provider15_ads_return_count", "provider17_id", "provider18_id", "publisher_url", "publisher_url_hash", "refering_domain", "refering_portfolio_id", "rtb_test_data", "serp_template_id", "slot_id", "sub_bidder_id", "template_id", "template_size_id", "test_data", "time_spent_on_last_page", "user_agent", "user_agent_group_id", "viewFlag", "view_id", "view_status", "visit_id", "visitor_id", "webserver_ip_address", "yahoo_keyword_term" ],
2016-02-25 11:34:30,958 [ClusteredBeam-ZkFuturePool-8b83a847-703c-4976-a6c0-1b92e86bac9b] INFO  c.m.c.s.net.finagle.DiscoResolver - Updating instances for service[druid:overlord] to Set()

2016-02-25 11:34:31,040 [ClusteredBeam-ZkFuturePool-8b83a847-703c-4976-a6c0-1b92e86bac9b] INFO  c.m.t.finagle.FinagleRegistry - Created client for service: druid:overlord
2016-02-25 11:34:31,120 [ClusteredBeam-ZkFuturePool-8b83a847-703c-4976-a6c0-1b92e86bac9b] WARN  c.m.tranquility.finagle.FutureRetry$ - Transient error, will try again in 9,703 ms
com
.twitter.finagle.NoBrokersAvailableException: No hosts are available for druid:overlord, Dtab.base=[], Dtab.local=[]

 at com
.twitter.finagle.NoStacktrace(Unknown Source) ~[na:na]
2016-02-25 11:34:40,839 [Hashed wheel timer #1] WARN  c.m.tranquility.finagle.FutureRetry$ - Transient error, will try again in 23,565 ms
com
.twitter.finagle.NoBrokersAvailableException: No hosts are available for druid:overlord, Dtab.base=[], Dtab.local=[]
 at com
.twitter.finagle.NoStacktrace(Unknown Source) ~[na:na]


--
You received this message because you are subscribed to the Google Groups "Druid User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to druid-user+...@googlegroups.com.
To post to this group, send email to druid...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/druid-user/599b52af-124b-45fc-834b-0e1b80359b26%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Message has been deleted
Message has been deleted
Message has been deleted
Message has been deleted

Priyanka Tanvani

unread,
Feb 26, 2016, 3:51:37 AM2/26/16
to Druid User

Hi Gian,

By changing the druid.service in overlord config to “druid/overlord” (default was "overlord") and “druid.selectors.indexing.serviceName” to “druid/overlord” (default was “overlord”) the above errors are removed and indexing task begins.

Thank you so much for your help ! :)


- Priyanka

Pritesh Damani

unread,
Oct 27, 2016, 11:39:41 PM10/27/16
to Druid User
Hi Priyanka,

This is a rather old discussion, but I am facing the exact issue. However, my service names are the same.. Is there anything else you had done to fix the issue?

Thanks,

Fangjin Yang

unread,
Nov 4, 2016, 6:01:04 PM11/4/16
to Druid User
Hi Pritesh, can you paste the error you are hitting?
Reply all
Reply to author
Forward
0 new messages