Kafka Streams org.apache.kafka.common.errors.RecordTooLargeException

[Levani Kokhreidze]

Hello, I’m experimenting in running Kafka Stream against the topics with 100 partitions. When I start 1st streaming application, during the rebalance phase it just goes dead and I see following error:

org.apache.kafka.common.KafkaException: Unexpected error from SyncGroup: The server experienced an unexpected error when processing the request

        at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$SyncGroupResponseHandler.handle(AbstractCoordinator.java:638) ~[kafka-clients-2.0.0.jar!/:na]

        at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$SyncGroupResponseHandler.handle(AbstractCoordinator.java:611) ~[kafka-clients-2.0.0.jar!/:na]

        at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:894) ~[kafka-clients-2.0.0.jar!/:na]

        at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:874) ~[kafka-clients-2.0.0.jar!/:na]

        at org.apache.kafka.clients.consumer.internals.RequestFuture$1.onSuccess(RequestFuture.java:204) ~[kafka-clients-2.0.0.jar!/:na]

        at org.apache.kafka.clients.consumer.internals.RequestFuture.fireSuccess(RequestFuture.java:167) ~[kafka-clients-2.0.0.jar!/:na]

        at org.apache.kafka.clients.consumer.internals.RequestFuture.complete(RequestFuture.java:127) ~[kafka-clients-2.0.0.jar!/:na]

        at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler.fireCompletion(ConsumerNetworkClient.java:586) ~[kafka-clients-2.0.0.jar!/:na]

        at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.firePendingCompletedRequests(ConsumerNetworkClient.java:400) ~[kafka-clients-2.0.0.jar!/:na]

        at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:303) ~[kafka-clients-2.0.0.jar!/:na]

        at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:242) ~[kafka-clients-2.0.0.jar!/:na]

        at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:218) ~[kafka-clients-2.0.0.jar!/:na]

        at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:413) ~[kafka-clients-2.0.0.jar!/:na]

        at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:352) ~[kafka-clients-2.0.0.jar!/:na]

        at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:337) ~[kafka-clients-2.0.0.jar!/:na]

        at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:333) ~[kafka-clients-2.0.0.jar!/:na]

        at org.apache.kafka.clients.consumer.KafkaConsumer.updateAssignmentMetadataIfNeeded(KafkaConsumer.java:1218) ~[kafka-clients-2.0.0.jar!/:na]

        at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1175) ~[kafka-clients-2.0.0.jar!/:na]

        at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1154) ~[kafka-clients-2.0.0.jar!/:na]

        at org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:861) ~[kafka-streams-2.0.0.jar!/:na]

        at org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:810) ~[kafka-streams-2.0.0.jar!/:na]

        at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:767) ~[kafka-streams-2.0.0.jar!/:na]

        at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:736) ~[kafka-streams-2.0.0.jar!/:na]

In broker I see following error:

[2018-09-14 13:14:58,759] ERROR [GroupMetadataManager brokerId=1002] Appending metadata message for group common-streaming-pipeline generation 1 failed due to org.apache.kafka.common.errors.RecordTooLargeException, returning UNKNOWN error code to the client (kafka.coordinator.group.GroupMetadataManager)

Increasing message.max.bytes solved the problem initially, but if I add more stream processing apps, during the rebalancing error appears again. Any ideas?

[Levani Kokhreidze]

Based on my investigation this failure correlates with number of processing threads in consumer group. If i decrease number of threads, it works fine. 

[Matthias J. Sax]

It also depends on the number of topic partition you consume from. It's
a known issue and WIP to fix it:
https://issues.apache.org/jira/browse/KAFKA-7149

Atm, increasing the `max.message.size` configuration is the only workaround.


-Matthias

> Increasing *message.max.bytes *solved the problem initially, but if

> I add more stream processing apps, during the rebalancing error
> appears again. Any ideas?
>

> --
> You received this message because you are subscribed to the Google
> Groups "Confluent Platform" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to confluent-platf...@googlegroups.com
> <mailto:confluent-platf...@googlegroups.com>.
> To post to this group, send email to confluent...@googlegroups.com
> <mailto:confluent...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/confluent-platform/8c3cbc5f-6aa4-4618-868b-a5e26edf2b00%40googlegroups.com
> <https://groups.google.com/d/msgid/confluent-platform/8c3cbc5f-6aa4-4618-868b-a5e26edf2b00%40googlegroups.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout.

[Des Sindatry]

Hello Matthias,

The logs only show me the error and not the size that rejected. 

2018-10-31 22:01:26,657 ERROR kafka.coordinator.group.GroupMetadataManager: [GroupMetadataManager brokerId=154] Appending metadata message for group prod-mkt-analytic-reporting-results generation 35 failed due to org.apache.kafka.common.errors.RecordTooLargeException, returning UNKNOWN error code to the client

Is there a way to know the size and increase it ?

> an email to confluent-platform+unsub...@googlegroups.com
> <mailto:confluent-platform+unsub...@googlegroups.com>.

[Matthias J. Sax]

The easiest way might be to register a `ProductionExceptionHandler` and
log the message size.

-Matthias

> > an email to confluent-platf...@googlegroups.com
> <javascript:>
> > <mailto:confluent-platf...@googlegroups.com
> <javascript:>>.

> > To post to this group, send email to confluent...@googlegroups.com

> <javascript:>
> > <mailto:confluent...@googlegroups.com <javascript:>>.

> > To view this discussion on the web visit
> >
> https://groups.google.com/d/msgid/confluent-platform/8c3cbc5f-6aa4-4618-868b-a5e26edf2b00%40googlegroups.com
> <https://groups.google.com/d/msgid/confluent-platform/8c3cbc5f-6aa4-4618-868b-a5e26edf2b00%40googlegroups.com>
>
> >

> <https://groups.google.com/d/msgid/confluent-platform/8c3cbc5f-6aa4-4618-868b-a5e26edf2b00%40googlegroups.com?utm_medium=email&utm_source=footer

> <https://groups.google.com/d/msgid/confluent-platform/8c3cbc5f-6aa4-4618-868b-a5e26edf2b00%40googlegroups.com?utm_medium=email&utm_source=footer>>.
>
> > For more options, visit https://groups.google.com/d/optout

> <https://groups.google.com/d/optout>.

>
> --
> You received this message because you are subscribed to the Google
> Groups "Confluent Platform" group.
> To unsubscribe from this group and stop receiving emails from it, send

> an email to confluent-platf...@googlegroups.com
> <mailto:confluent-platf...@googlegroups.com>.

> To post to this group, send email to confluent...@googlegroups.com
> <mailto:confluent...@googlegroups.com>.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/confluent-platform/bee4741e-09fc-4f26-9f8d-33ab21c370c8%40googlegroups.com
> <https://groups.google.com/d/msgid/confluent-platform/bee4741e-09fc-4f26-9f8d-33ab21c370c8%40googlegroups.com?utm_medium=email&utm_source=footer>.