Kafka Producer not sending message when change broker.id= 1.

[Kuldeep Poonia]

Hi all,

I am new to Kafka. I am trying to send messages using multiple brokers. Every thing is fine when I set broker.id = 0 in my server.properties file. But when changed broker.id to 1 i.e when set broker.id = 1 then producer not send messages. I am working on window.

1. Create a topic using command

.\bin\windows\kafka-topics.bat --create --zookeeper localhost:2181 --replication-factor 2 --partitions 1 --topic test2

2. Running kafka server using command

.\bin\windows\kafka-server-start.bat .\config\server.properties

3. --Create Producere

.\bin\windows\kafka-console-producer.bat --broker-list localhost:9092 --topic test2

4. --Create Consumer

.\bin\windows\kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic test2

Please help me where I am doing something wrong.

[Roger Hoover]

Hi Kuldeep,

It sounds like you created a topic "test2" while you had a broker alive with id = 0.  Then you shutdown broker 0 and brought up broker 1.

The problem is that the leader (and only replica) for your topic was on broker 0 and the cluster metadata stored in ZooKeeper still reflects that.  The producer cannot send messages to topic "test2" because the leader for that topic (broker 0) is not available and broker 1 does not know that it has a fully copy of broker 0's filesystem and could take over as leader of topic2.

A better test would be for you to bring up two brokers at the same time and create a topic with replication-factor 2.  Then if you shut one of them down, you will see leadership transfer to the other broker.

Hope that helps.

Roger

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent-platform@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/6472da04-3f4c-46bf-b82b-fbd0fa23f77e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Kuldeep Poonia]

Hi Rogar,

Thanks for your reply but still problem not resolved. As per your reply, I start two server with brokerid 0 and brokerid 1 and then create a new topic with the command

.\bin\windows\kafka-topics.bat --create --zookeeper localhost:2181 --replication-factor 2 --partitions 1 --topic test6

When I start up server with broker id 0 and 1 or down server with broker id 1 then messages are sending successfully but when down server with broker id 0 then it will showing me the error

ERROR Error when sending message to topic test6 with key: null, value: 9 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)

org.apache.kafka.common.errors.TimeoutException: Expiring 1 record(s) for test6-0: 1510 ms has passed since batch creation plus linger time

and  sometime it display ERROR Error when sending message to topic test6 with key: null, value: 4 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)

org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms.

however the describe command

.\bin\windows\kafka-topics.bat --describe --zookeeper localhost:2181 --topic test6  showing that leader is 1. 

On Tuesday, 7 March 2017 11:03:23 UTC+5:30, Roger Hoover wrote:

Hi Kuldeep,

It sounds like you created a topic "test2" while you had a broker alive with id = 0.  Then you shutdown broker 0 and brought up broker 1.

The problem is that the leader (and only replica) for your topic was on broker 0 and the cluster metadata stored in ZooKeeper still reflects that.  The producer cannot send messages to topic "test2" because the leader for that topic (broker 0) is not available and broker 1 does not know that it has a fully copy of broker 0's filesystem and could take over as leader of topic2.

A better test would be for you to bring up two brokers at the same time and create a topic with replication-factor 2.  Then if you shut one of them down, you will see leadership transfer to the other broker.

Hope that helps.

Roger

On Mon, Mar 6, 2017 at 1:48 AM, Kuldeep Poonia <kuldeep...@darcl.com> wrote:

Hi all,

I am new to Kafka. I am trying to send messages using multiple brokers. Every thing is fine when I set broker.id = 0 in my server.properties file. But when changed broker.id to 1 i.e when set broker.id = 1 then producer not send messages. I am working on window.

1. Create a topic using command

.\bin\windows\kafka-topics.bat --create --zookeeper localhost:2181 --replication-factor 2 --partitions 1 --topic test2

2. Running kafka server using command

.\bin\windows\kafka-server-start.bat .\config\server.properties

3. --Create Producere

.\bin\windows\kafka-console-producer.bat --broker-list localhost:9092 --topic test2

4. --Create Consumer

.\bin\windows\kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic test2

Please help me where I am doing something wrong.

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.

[Roger Hoover]

Kuldeep,

What did you use for the `bootstrap.servers` config in the producer?  It will need to have a live broker in the list to be able to do the metadata request to figure out where the leader is.

Cheers,

Roger

To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsubscribe@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/6472da04-3f4c-46bf-b82b-fbd0fa23f77e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.

To post to this group, send email to confluent-platform@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/74c3cade-f88e-46b0-b080-e99e61cc19ef%40googlegroups.com.

[Kuldeep Poonia]

Hi Roger,

By default it is bootstrap.servers=localhost:9092

To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/6472da04-3f4c-46bf-b82b-fbd0fa23f77e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.

[Roger Hoover]

I'd suggest adding both broker to that list.

Hi Roger,

To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsubscribe@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/6472da04-3f4c-46bf-b82b-fbd0fa23f77e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.

To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsubscribe@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/74c3cade-f88e-46b0-b080-e99e61cc19ef%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.

To post to this group, send email to confluent-platform@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/5d978362-6ac5-4880-ac01-bf9ca0e4aa38%40googlegroups.com.