Connect consumers, bootstrap servers and zookeeper

[Matthew Burgess]

Hi all,

I'm a little confused about how to get our Kafka consumers to reliably connect to a broker cluster.

Let's assume the following:

3x broker+zookeeper nodes: broker01, broker02, broker03. Kafka broker is on port 9092, zookeeper is on port 2181

1 worker node (to be configured with 3 consumer tasks): worker01

Using 'bootstrap.servers=broker01:9092,broker02:9092,broker03:9092' is not ideal, as that requires all 3 brokers in the cluster to be available when the worker comes up (I think).  Probability wise you're better off specifying just a single node here; if a single node fails and you've specified 3 nodes in the list, you're guaranteed to fail to start the worker 100% of the time.  If you specify a single node here and you have a single node fail on you there's only a 33.33% chance that it's the one you've specified and therefore causes you to fail.  Does anyone have any thoughts on what the optimum strategy here is? To me the current behaviour of 'fail if any of the bootstrap.servers is unavailable' is counterintuitive.  I'd have expected it to try each in turn and as long as any one of them is available but I don't think that's what was happening during our testing.

Once a connection to one of the bootstrap.servers is made, my understanding is that metadata on that broker is used to work out the full cluster membership, and that the workers never need to connect to zookeeper at all.  If that's the case then the example docker images we're using specify CONNECT_ZOOKEEPER_CONNECT (https://github.com/confluentinc/cp-docker-images/blob/5.2.1-post/examples/cp-all-in-one/docker-compose.yml#L80), which is what has caused a bit more confusion on my side.  The thing is that I don't see 'zookeeper.connect' listed as a valid config option on https://docs.confluent.io/current/connect/references/allconfigs.html#kconnect-long-worker-configs, so I don't think that line is actually doing anything useful?  Is that line a red-herring? If so, I don't mind raising a PR that removes it.

Thanks for any clarification you can provide.

Kind Regards,

Matt

[Matthew Burgess]

On Monday, April 15, 2019 at 5:32:14 PM UTC+1, Matthew Burgess wrote:

Hi all,

I'm a little confused about how to get our Kafka consumers to reliably connect to a broker cluster.

Let's assume the following:

3x broker+zookeeper nodes: broker01, broker02, broker03. Kafka broker is on port 9092, zookeeper is on port 2181

1 worker node (to be configured with 3 consumer tasks): worker01

Using 'bootstrap.servers=broker01:9092,broker02:9092,broker03:9092' is not ideal, as that requires all 3 brokers in the cluster to be available when the worker comes up (I think).  Probability wise you're better off specifying just a single node here; if a single node fails and you've specified 3 nodes in the list, you're guaranteed to fail to start the worker 100% of the time.  If you specify a single node here and you have a single node fail on you there's only a 33.33% chance that it's the one you've specified and therefore causes you to fail.  Does anyone have any thoughts on what the optimum strategy here is? To me the current behaviour of 'fail if any of the bootstrap.servers is unavailable' is counterintuitive.  I'd have expected it to try each in turn and as long as any one of them is available but I don't think that's what was happening during our testing.

Well, all I can say is I've no idea what was going on yesterday...yet!  Just confirmed on my local docker setup that the behaviour I'm after is exactly what happens; the behaviour of CUB (https://github.com/confluentinc/confluent-docker-utils/blob/master/confluent/docker_utils/cub.py#L113) does exactly what I'm after. So I can specify as many or as few brokers as I want in BOOTSTRAP_SERVERS and as long as any one of them up, CUB will succeed and the container, worker and connector will come up.

 

Once a connection to one of the bootstrap.servers is made, my understanding is that metadata on that broker is used to work out the full cluster membership, and that the workers never need to connect to zookeeper at all.  If that's the case then the example docker images we're using specify CONNECT_ZOOKEEPER_CONNECT (https://github.com/confluentinc/cp-docker-images/blob/5.2.1-post/examples/cp-all-in-one/docker-compose.yml#L80), which is what has caused a bit more confusion on my side.  The thing is that I don't see 'zookeeper.connect' listed as a valid config option on https://docs.confluent.io/current/connect/references/allconfigs.html#kconnect-long-worker-configs, so I don't think that line is actually doing anything useful?  Is that line a red-herring? If so, I don't mind raising a PR that removes it.

Zookeeper doesn't appear to be needed for at least the initial container bringup.  However, I'd be interested to hear if there are any situations where I will need my consumers to be able to reach Zookeeper or whether any references to it in the code and docs are just historic references.

Kind Regards,

Matt.