Duplicate assignment of Broker ID when creating kafka cluster in 10.2.0

2,298 views
Skip to first unread message

Eugene Dvorkin

unread,
Jul 20, 2017, 2:35:19 PM7/20/17
to Confluent Platform
I am using ansible to create a cluster of 3 brokers. I do not specify any broker id, relying on automatically assigned broker id.During startup, I am receiving an error message in kafka logs:

[2017-07-18 04:11:54,940] FATAL [Kafka Server 1002], Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)

java.lang.RuntimeException:

A broker is already registered on the path /brokers/ids/1002.

 This probably indicates that you either have configured a brokerid that is already in use, or else you have shutdown this broker and restarted it faster than the zookeeper timeout so it appears to be re-registering.

    at kafka.utils.ZkUtils.registerBrokerInZk(ZkUtils.scala:375)

    at kafka.utils.ZkUtils.registerBrokerInZk(ZkUtils.scala:361)

    at kafka.server.KafkaHealthcheck.register(KafkaHealthcheck.scala:71)

    at kafka.server.KafkaHealthcheck.startup(KafkaHealthcheck.scala:51)


I saw couple of servers with the same id. To fix the issue I updated matadata file with next available id (1003) and restarted the server. 


My question is why this happening? should brokers lock in zookeeper to prevent this situation? I have 10.2.0. Anything I can do so this not happening?/

Thanks


SJ

unread,
Nov 27, 2017, 5:10:48 AM11/27/17
to Confluent Platform

Hi Eugene,

I am facing the same issue, How did you resolve it then.
I have configured zookeeper, kafka as linux services so when i reboot the system they should automatically get restarted.
But sometimes both kafka and zookeeper get started sometimes kafka is not starting and issue is same as you faced.
Please help.

Best Regards,
Shubham Jain

Eugene

unread,
Nov 28, 2017, 8:39:10 AM11/28/17
to confluent...@googlegroups.com
In my case I think the reason was I have 3 zookeepers running with the exhibitor. The problem is that zookeeper ensemble takes some time to setup. the moment I started kafka, not all zookeeper nodes joined the ensemble yet, so it was a split brain situation. One kafka nodes connected to a zookeeper not yet in the ensemble. When I start waiting and checking until all zookeepers are in the ensemble, (using exhibitors API) I did not saw this error much. So wait until you get zookeeper up completely.  another approach is to use server's IP hash as a broker ID. This was also working in my containers. Just make sure you don't restart kafka too fast, let zookeeper remove the node first. 

--
You received this message because you are subscribed to a topic in the Google Groups "Confluent Platform" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/confluent-platform/c-tabzzxhAo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent-platform@googlegroups.com.
 To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/92cd23ff-e23e-4a39-bd7f-3548faff5ffd%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.



--
Eugene Dvorkin

SHUBHAM JAIN

unread,
Nov 28, 2017, 9:50:01 AM11/28/17
to confluent...@googlegroups.com
Thanks Eugen for your inputs...
Merely the situation was same for our side.
We resolved it by adding delay between start and stop order of zookeeper, kafka, kafka rest service.
Thus First all zookeepers get started and then delay of 1 minutes and then all kafka brokers and delay of 1 minute and then all kafka rest proxy servers.


To unsubscribe from this group and all its topics, send an email to confluent-platform+unsubscribe@googlegroups.com.

To post to this group, send email to confluent-platform@googlegroups.com.
--
Eugene Dvorkin

--
You received this message because you are subscribed to a topic in the Google Groups "Confluent Platform" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/confluent-platform/c-tabzzxhAo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent-platform@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/CACiatOoGv8HqKiBO8-MOdTzrN8x_nhASAoJfz3GEhQZrGQjh8Q%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.



--
Reply all
Reply to author
Forward
0 new messages