No hosts are available for disco!druid:overlord, Dtab.base=[], Dtab.local=[]

[Pritesh Damani]

Hello,

We have a cluster setup with remote Middle Managers. Everything is running perfectly, when it comes to regular druid jobs. 

Druid version we are using is 0.9.1.1. 

When I try to connect tranquility to this setup, I am getting the error below. Exact same versions and config on my local machine work. 

Any pointers would be appreciated.  

2016-10-28 02:47:00,841 [ClusteredBeam-ZkFuturePool-3a9e12f5-c01b-4820-9060-7715e45fcfb7] INFO  c.m.t.finagle.FinagleRegistry - Created client for service: disco!druid:overlord

2016-10-28 02:47:00,898 [ClusteredBeam-ZkFuturePool-3a9e12f5-c01b-4820-9060-7715e45fcfb7] WARN  c.m.tranquility.finagle.FutureRetry$ - Transient error, will try again in 13,593 ms

com.twitter.finagle.NoBrokersAvailableException: No hosts are available for disco!druid:overlord, Dtab.base=[], Dtab.local=[]

at com.twitter.finagle.NoStacktrace(Unknown Source) ~[na:na]

[David Lim]

Hey Pritesh,

That error means that Tranquility can't find the indexing tasks to send events to. This could be happening for a number of reasons, but I'd first check:

a) That you have available worker capacity for the required number of tasks from Tranquility
b) That the Tranquility config has the same zookeeper.connect as Druid
c) That druid.discovery.curator.path is the same as what your Druid nodes use
d) That druid.selectors.indexing.serviceName matches the druid.service name of your overlord.

[David Lim]

I saw your post in the other thread so I thought I'd add that if all these are set correctly, are there any exceptions or other interesting logs in your overlord console?

[Pritesh Damani]

Hi David,

 

I went through your items. It is the discovery that was wrong! That fixed it! Thank you.

 

Do you have a recommended config I should use?

consumer.numThreads ?

Also, what decides the number of slots that are using for indexing on the indexing service. Currently it is only using one and there are many available. I am afraid it might not be able to keep up..

--
You received this message because you are subscribed to a topic in the Google Groups "Druid User" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/druid-user/DKGsyq_HW2o/unsubscribe.
To unsubscribe from this group and all its topics, send an email to druid-user+...@googlegroups.com.
To post to this group, send email to druid...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/druid-user/6a3b4c9a-7254-45eb-9439-fd7fbd0118fd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Pritesh Damani]

David,

 

One more thing, from pivot I am seeing this error.. Seen this before?

 

Cluster 'druid' could not introspect 'realtime because: Failure getting results from[http://xxxxx/druid/v2/] because of [org.jboss.netty.channel.ChannelException: Faulty channel in resource pool]

 

 

From: <druid...@googlegroups.com> on behalf of David Lim <davi...@imply.io>

Reply-To: <druid...@googlegroups.com>
Date: Friday, October 28, 2016 at 1:50 PM
To: Druid User <druid...@googlegroups.com>

--

[David Lim]

Hey Pritesh,

Glad to hear things are working. For consumer.numThreads, the default (numProcessors - 1) should work reasonably well. It doesn't really make sense to go above numProcessors and it might make sense to decrease it if you have other processes running on the same machine sharing CPU time.

The number of slots used is configured by setting task.partitions. It definitely makes sense to increase the number of partitions to handle your ingestion throughput.

As for the Pivot issue, I'm not sure what's causing it. Are there any interesting logs in your broker? I would try restarting the broker and restarting Pivot and see if that helps. If you're still seeing it, you should post your question as a separate topic, perhaps in the Imply user group.

[Pritesh Damani]

Ok thank you. 

Regards,
Pritesh Damani
CTO, GoWatchIt.
(sent from mobile phone)

On Fri, Oct 28, 2016 at 4:58 PM -0400, "Pritesh Damani" <pri...@gowatchit.com> wrote:

David,

 

One more thing, from pivot I am seeing this error.. Seen this before?

 

Cluster 'druid' could not introspect 'realtime because: Failure getting results from[http://xxxxx/druid/v2/] because of [org.jboss.netty.channel.ChannelException: Faulty channel in resource pool]

 

 

From: <druid...@googlegroups.com> on behalf of David Lim <davi...@imply.io>

Reply-To: <druid...@googlegroups.com>
Date: Friday, October 28, 2016 at 1:50 PM
To: Druid User <druid...@googlegroups.com>

--

[Anzee Wagh]

I also went through these kind of errors.

I was setting up druid database to store data coming from Kafka.

I get solved this by following settings in my ingestion spec file of tranquility 

"properties" : {

    "zookeeper.connect" : "localhost(client ip):2181",     

    "druid.discovery.curator.path" : "/druid/discovery",

    "druid.selectors.indexing.serviceName" : "druid/overlord",

    "commit.periodMillis" : "15000",

    "consumer.numThreads" : "2",

    "kafka.zookeeper.connect" : "server_ip:2181",

    "kafka.group.id" : "tranquility-kafka"

  }

tranquility and druid both should be on same zookeeper.