Cannot read messages using a specific group

[Major OPPO]

Hi,

One of our brokers (Kafka 1.1.1-cp1) went down and when it came back up the consumer of one topic couldn’t read using the original consumer group.

This doesn’t output anything:

bin/kafka-console-consumer --bootstrap-server server --topic ASSETS --group group_name

However, if I change the name of the group it does output data:

bin/kafka-console-consumer --bootstrap-server server --topic ASSETS --group group_name_renamed

I tried to use the kafka-consumer-groups command here is the output:

bin/kafka-consumer-groups --bootstrap-server server --describe --group group_name
Error: Executing consumer group command failed due to The consumer group command timed out while waiting for group to initialize

The group_name consumer group is not running in the cluster. What is noteworthy is that the consumer group group_name cannot be reset by the CLI (to either latest or earliest) and is still not working way past the retention period of all the topics (from the consumers with latest or earliest auto.offset.reset).

Here is the output:

bin/kafka-consumer-groups --bootstrap-server server --group group_name --reset-offsets --to-latest --execute
Error: Executing consumer group command failed due to The consumer group command timed out while waiting for group to initialize

I set the log level to INFO and here are the logs.

I’m not sure why those logs say that the Group coordinator is unavailable or invalid because the brokers are all up (including the ones with the ip specified in the log) and just changing the consumer group makes it work. If anybody had an idea, that would be very helpful. (also committing the offsets to Zookeeper works, so I'm wondering if there is a bug with __Consummer_Offset)

Thank you for your help!

kafka-console-consumer --bootstrap-server ip1:9092,ip2:9092 --topic ASSETS --group ASSETS_PROCESSOR_TOPOLOGY

[2018-12-26 15:47:15,243] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$)
[2018-12-26 15:47:15,387] INFO ConsumerConfig values:
    auto.commit.interval.ms = 5000
    auto.offset.reset = latest
    bootstrap.servers = [ip1:9092, ip2:9092]
    check.crcs = true
    client.id =
    connections.max.idle.ms = 540000
    default.api.timeout.ms = 60000
    enable.auto.commit = true
    exclude.internal.topics = true
    fetch.max.bytes = 52428800
    fetch.max.wait.ms = 500
    fetch.min.bytes = 1
    group.id = ASSETS_PROCESSOR_TOPOLOGY
    heartbeat.interval.ms = 3000
    interceptor.classes = []
    internal.leave.group.on.close = true
    isolation.level = read_uncommitted
    key.deserializer = class org.apache.kafka.common.serialization.ByteArrayDeserializer
    max.partition.fetch.bytes = 1048576
    max.poll.interval.ms = 300000
    max.poll.records = 500
    metadata.max.age.ms = 300000
    metric.reporters = []
    metrics.num.samples = 2
    metrics.recording.level = INFO
    metrics.sample.window.ms = 30000
    partition.assignment.strategy = [class org.apache.kafka.clients.consumer.RangeAssignor]
    receive.buffer.bytes = 65536
    reconnect.backoff.max.ms = 1000
    reconnect.backoff.ms = 50
    request.timeout.ms = 30000
    retry.backoff.ms = 100
    sasl.client.callback.handler.class = null
    sasl.jaas.config = null
    sasl.kerberos.kinit.cmd = /usr/bin/kinit
    sasl.kerberos.min.time.before.relogin = 60000
    sasl.kerberos.service.name = null
    sasl.kerberos.ticket.renew.jitter = 0.05
    sasl.kerberos.ticket.renew.window.factor = 0.8
    sasl.login.callback.handler.class = null
    sasl.login.class = null
    sasl.login.refresh.buffer.seconds = 300
    sasl.login.refresh.min.period.seconds = 60
    sasl.login.refresh.window.factor = 0.8
    sasl.login.refresh.window.jitter = 0.05
    sasl.mechanism = GSSAPI
    security.protocol = PLAINTEXT
    send.buffer.bytes = 131072
    session.timeout.ms = 10000
    ssl.cipher.suites = null
    ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
    ssl.endpoint.identification.algorithm = https
    ssl.key.password = null
    ssl.keymanager.algorithm = SunX509
    ssl.keystore.location = null
    ssl.keystore.password = null
    ssl.keystore.type = JKS
    ssl.protocol = TLS
    ssl.provider = null
    ssl.secure.random.implementation = null
    ssl.trustmanager.algorithm = PKIX
    ssl.truststore.location = null
    ssl.truststore.password = null
    ssl.truststore.type = JKS
    value.deserializer = class org.apache.kafka.common.serialization.ByteArrayDeserializer
 (org.apache.kafka.clients.consumer.ConsumerConfig)
[2018-12-26 15:47:15,579] INFO Kafka version : 2.0.1-cp1 (org.apache.kafka.common.utils.AppInfoParser)
[2018-12-26 15:47:15,579] INFO Kafka commitId : 3d167ab3fdad2e73 (org.apache.kafka.common.utils.AppInfoParser)
[2018-12-26 15:47:16,711] INFO Cluster ID: AkXF_gdZS5eVEEYbvUiyRg (org.apache.kafka.clients.Metadata)
[2018-12-26 15:47:16,712] INFO [Consumer clientId=consumer-1, groupId=ASSETS_PROCESSOR_TOPOLOGY] Discovered group coordinator ip_address1:9092 (id: 2147483643 rack: null) (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2018-12-26 15:47:16,713] INFO [Consumer clientId=consumer-1, groupId=ASSETS_PROCESSOR_TOPOLOGY] Revoking previously assigned partitions [] (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)
[2018-12-26 15:47:16,714] INFO [Consumer clientId=consumer-1, groupId=ASSETS_PROCESSOR_TOPOLOGY] (Re-)joining group (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2018-12-26 15:47:17,351] INFO [Consumer clientId=consumer-1, groupId=ASSETS_PROCESSOR_TOPOLOGY] Group coordinator ip_address1:9092 (id: 2147483643 rack: null) is unavailable or invalid, will attempt rediscovery (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2018-12-26 15:47:18,189] INFO [Consumer clientId=consumer-1, groupId=ASSETS_PROCESSOR_TOPOLOGY] Discovered group coordinator ip_address2:9092 (id: 2147483645 rack: null) (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2018-12-26 15:47:18,190] INFO [Consumer clientId=consumer-1, groupId=ASSETS_PROCESSOR_TOPOLOGY] (Re-)joining group (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2018-12-26 15:47:19,501] INFO [Consumer clientId=consumer-1, groupId=ASSETS_PROCESSOR_TOPOLOGY] Group coordinator ip_address2:9092 (id: 2147483645 rack: null) is unavailable or invalid, will attempt rediscovery (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2018-12-26 15:47:19,723] INFO [Consumer clientId=consumer-1, groupId=ASSETS_PROCESSOR_TOPOLOGY] Discovered group coordinator ip_address2:9092 (id: 2147483645 rack: null) (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2018-12-26 15:47:19,723] INFO [Consumer clientId=consumer-1, groupId=ASSETS_PROCESSOR_TOPOLOGY] (Re-)joining group (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2018-12-26 15:47:20,079] INFO [Consumer clientId=consumer-1, groupId=ASSETS_PROCESSOR_TOPOLOGY] Group coordinator ip_address2:9092 (id: 2147483645 rack: null) is unavailable or invalid, will attempt rediscovery (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2018-12-26 15:47:20,304] INFO [Consumer clientId=consumer-1, groupId=ASSETS_PROCESSOR_TOPOLOGY] Discovered group coordinator ip_address2:9092 (id: 2147483645 rack: null) (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2018-12-26 15:47:20,305] INFO [Consumer clientId=consumer-1, groupId=ASSETS_PROCESSOR_TOPOLOGY] (Re-)joining group (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2018-12-26 15:47:20,661] INFO [Consumer clientId=consumer-1, groupId=ASSETS_PROCESSOR_TOPOLOGY] Group coordinator ip_address2:9092 (id: 2147483645 rack: null) is unavailable or invalid, will attempt rediscovery (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2018-12-26 15:47:20,885] INFO [Consumer clientId=consumer-1, groupId=ASSETS_PROCESSOR_TOPOLOGY] Discovered group coordinator ip_address2:9092 (id: 2147483645 rack: null) (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2018-12-26 15:47:20,885] INFO [Consumer clientId=consumer-1, groupId=ASSETS_PROCESSOR_TOPOLOGY] (Re-)joining group (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2018-12-26 15:47:21,240] INFO [Consumer clientId=consumer-1, groupId=ASSETS_PROCESSOR_TOPOLOGY] Group coordinator ip_address2:9092 (id: 2147483645 rack: null) is unavailable or invalid, will attempt rediscovery (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2018-12-26 15:47:21,465] INFO [Consumer clientId=consumer-1, groupId=ASSETS_PROCESSOR_TOPOLOGY] Discovered group coordinator ip_address2:9092 (id: 2147483645 rack: null) (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2018-12-26 15:47:21,465] INFO [Consumer clientId=consumer-1, groupId=ASSETS_PROCESSOR_TOPOLOGY] (Re-)joining group (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2018-12-26 15:47:21,820] INFO [Consumer clientId=consumer-1, groupId=ASSETS_PROCESSOR_TOPOLOGY] Group coordinator ip_address2:9092 (id: 2147483645 rack: null) is unavailable or invalid, will attempt rediscovery (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2018-12-26 15:47:22,046] INFO [Consumer clientId=consumer-1, groupId=ASSETS_PROCESSOR_TOPOLOGY] Discovered group coordinator ip_address2:9092 (id: 2147483645 rack: null) (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2018-12-26 15:47:22,046] INFO [Consumer clientId=consumer-1, groupId=ASSETS_PROCESSOR_TOPOLOGY] (Re-)joining group (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2018-12-26 15:47:22,403] INFO [Consumer clientId=consumer-1, groupId=ASSETS_PROCESSOR_TOPOLOGY] Group coordinator ip_address2:9092 (id: 2147483645 rack: null) is unavailable or invalid, will attempt rediscovery (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2018-12-26 15:47:22,622] INFO [Consumer clientId=consumer-1, groupId=ASSETS_PROCESSOR_TOPOLOGY] Discovered group coordinator ip_address2:9092 (id: 2147483645 rack: null) (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2018-12-26 15:47:22,623] INFO [Consumer clientId=consumer-1, groupId=ASSETS_PROCESSOR_TOPOLOGY] (Re-)joining group (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2018-12-26 15:47:22,976] INFO [Consumer clientId=consumer-1, groupId=ASSETS_PROCESSOR_TOPOLOGY] Group coordinator ip_address2:9092 (id: 2147483645 rack: null) is unavailable or invalid, will attempt rediscovery (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2018-12-26 15:47:23,198] INFO [Consumer clientId=consumer-1, groupId=ASSETS_PROCESSOR_TOPOLOGY] Discovered group coordinator ip_address2:9092 (id: 2147483645 rack: null) (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2018-12-26 15:47:23,199] INFO [Consumer clientId=consumer-1, groupId=ASSETS_PROCESSOR_TOPOLOGY] (Re-)joining group (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2018-12-26 15:47:23,551] INFO [Consumer clientId=consumer-1, groupId=ASSETS_PROCESSOR_TOPOLOGY] Group coordinator ip_address2:9092 (id: 2147483645 rack: null) is unavailable or invalid, will attempt rediscovery (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)

[Andreas Sittler]

You've got a Confluent support contract?

I'd propose to contact them.

[marco]

Hi,

I got exactly problem here, did you resolve your issue yet? Thanks.

Marco

在 2019年1月4日星期五 UTC-6上午8:32:17，Major OPPO写道：