Re: KafkaStream topology Error

702 views
Skip to first unread message

Matthias J. Sax

unread,
Feb 14, 2017, 9:12:07 PM2/14/17
to confluent...@googlegroups.com
It seems, that your client did time out. This can happen, if processing
takes to long.

Try to increase consumer configuration value "session.timeout.ms" via
StreamsConfig. Default value is 10000.

Cf. http://kafka.apache.org/documentation/#newconsumerconfigs


-Matthias


On 2/14/17 6:12 AM, Rodrigo Monteiro wrote:
> Hi,
>
> I have a application with some KafkaStream topology, they run normally
> around 12 hours, in the night one of the topology stop throwing this error
>
> org.apache.kafka.streams.errors.StreamsException: stream-thread
> [StreamThread-13] Failed to rebalance at
> org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:410)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:242)
> Caused by: org.apache.kafka.clients.consumer.CommitFailedException:
> Commit cannot be completed since the group has already rebalanced and
> assigned the partitions to another member. This means that the time
> between subsequent calls to poll() was longer than the configured
> max.poll.interval.ms, which typically implies that the poll loop is
> spending too much time message processing. You can address this either
> by increasing the session timeout or by reducing the maximum size of
> batches returned in poll() with max.poll.records. at
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.sendOffsetCommitRequest(ConsumerCoordinator.java:600)
> at
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.commitOffsetsSync(ConsumerCoordinator.java:498)
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.commitSync(KafkaConsumer.java:1104)
> at
> org.apache.kafka.streams.processor.internals.StreamTask.commitOffsets(StreamTask.java:297)
> at
> org.apache.kafka.streams.processor.internals.StreamThread$3.apply(StreamThread.java:359)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.performOnAllTasks(StreamThread.java:328)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.commitOffsets(StreamThread.java:355)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.shutdownTasksAndState(StreamThread.java:297)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.access$900(StreamThread.java:69)
> at
> org.apache.kafka.streams.processor.internals.StreamThread$1.onPartitionsRevoked(StreamThread.java:143)
> at
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinPrepare(ConsumerCoordinator.java:336)
> at
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:303)
> at
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:277)
> at
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:259)
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1013)
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:979)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:407)
> ... 1 common frames omitted
>
> I restart my application and the same topology stop after 10 minutes
>
> anybody with the same problem?
>
> --
> You received this message because you are subscribed to the Google
> Groups "Confluent Platform" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to confluent-platf...@googlegroups.com
> <mailto:confluent-platf...@googlegroups.com>.
> To post to this group, send email to confluent...@googlegroups.com
> <mailto:confluent...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/confluent-platform/a6344a7f-5fa1-473d-96fa-4d4a9215f8ea%40googlegroups.com
> <https://groups.google.com/d/msgid/confluent-platform/a6344a7f-5fa1-473d-96fa-4d4a9215f8ea%40googlegroups.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout.

signature.asc

Rodrigo Monteiro

unread,
Feb 15, 2017, 12:08:11 PM2/15/17
to Confluent Platform
yesterday I changed some configuration

- default values
max.poll.records = 1000

- my changes
max.poll.records = 100

the topology not stop after 10min, I think I finally find the solution...but

today in the morning when I saw the logs, the topology stop in 7am...

it's stay alive around 12 hours after stop

Sabarish Sasidharan

unread,
Feb 15, 2017, 12:13:23 PM2/15/17
to confluent...@googlegroups.com
Last time I checked Kafka Streams framework seemed to override max.poll.records configuration.

Yes you can increase the max poll interval and the session time outs.

Regards
Sab

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent-platform@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/bdb1ae1e-6eff-4567-a324-b4eab4bb9ca1%40googlegroups.com.

Matthias J. Sax

unread,
Feb 15, 2017, 1:05:17 PM2/15/17
to confluent...@googlegroups.com
Not sure.

Could it be, that your processing gets slower over time? Maybe
collecting Metrics gave put some light on this...

-Matthias

On 2/15/17 9:13 AM, Sabarish Sasidharan wrote:
> Last time I checked Kafka Streams framework seemed to override
> max.poll.records configuration.
>
> Yes you can increase the max poll interval and the session time outs.
>
> Regards
> Sab
>
> On 15 Feb 2017 10:38 p.m., "Rodrigo Monteiro" <rodr...@gmail.com
> <mailto:rodr...@gmail.com>> wrote:
>
> yesterday I changed some configuration
>
> - default values
> heartbeat.interval.ms <http://heartbeat.interval.ms> = 3000
> max.poll.interval.ms <http://max.poll.interval.ms> = 300000
> max.poll.records = 1000
> session.timeout.ms <http://session.timeout.ms> = 10000
>
> - my changes
> heartbeat.interval.ms <http://heartbeat.interval.ms> = 10000
> max.poll.interval.ms <http://max.poll.interval.ms> = 300000
> max.poll.records = 100
> session.timeout.ms <http://session.timeout.ms> = 30000
>
> the topology not stop after 10min, I think I finally find the
> solution...but
>
> today in the morning when I saw the logs, the topology stop in 7am...
>
> it's stay alive around 12 hours after stop
>
>
> Em quarta-feira, 15 de fevereiro de 2017 00:12:07 UTC-2, Matthias J.
> Sax escreveu:
>
> It seems, that your client did time out. This can happen, if
> processing
> takes to long.
>
> Try to increase consumer configuration value "session.timeout.ms
> <http://session.timeout.ms>" via
> StreamsConfig. Default value is 10000.
>
> Cf. http://kafka.apache.org/documentation/#newconsumerconfigs
> <http://kafka.apache.org/documentation/#newconsumerconfigs>
>
>
> -Matthias
>
>
> On 2/14/17 6:12 AM, Rodrigo Monteiro wrote:
> > Hi,
> >
> > I have a application with some KafkaStream topology, they run
> normally
> > around 12 hours, in the night one of the topology stop
> throwing this error
> >
> > org.apache.kafka.streams.errors.StreamsException: stream-thread
> > [StreamThread-13] Failed to rebalance at
> >
> org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:410)
>
> > at
> >
> org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:242)
>
> > Caused by:
> org.apache.kafka.clients.consumer.CommitFailedException:
> > Commit cannot be completed since the group has already
> rebalanced and
> > assigned the partitions to another member. This means that the
> time
> > between subsequent calls to poll() was longer than the configured
> > max.poll.interval.ms <http://max.poll.interval.ms>, which
> typically implies that the poll loop is
> > spending too much time message processing. You can address
> this either
> > by increasing the session timeout or by reducing the maximum
> size of
> > batches returned in poll() with max.poll.records. at
> >
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.sendOffsetCommitRequest(ConsumerCoordinator.java:600)
>
> > at
> >
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.commitOffsetsSync(ConsumerCoordinator.java:498)
>
> > at
> >
> org.apache.kafka.clients.consumer.KafkaConsumer.commitSync(KafkaConsumer.java:1104)
>
> > at
> >
> org.apache.kafka.streams.processor.internals.StreamTask.commitOffsets(StreamTask.java:297)
>
> > at
> >
> org.apache.kafka.streams.processor.internals.StreamThread$3.apply(StreamThread.java:359)
>
> > at
> > org.apache.kafka.streams.processor.internals.StreamThread.pe
> <http://ssor.internals.StreamThread.pe>rformOnAllTasks(StreamThread.java:328)
>
> > at
> >
> org.apache.kafka.streams.processor.internals.StreamThread.commitOffsets(StreamThread.java:355)
>
> > at
> > org.apache.kafka.streams.processor.internals.StreamThread.sh
> <http://ssor.internals.StreamThread.sh>utdownTasksAndState(StreamThread.java:297)
> > an email to confluent-platf...@googlegroups.com
> > <mailto:confluent-platf...@googlegroups.com>.
> > To post to this group, send email to
> confluent...@googlegroups.com
> > <mailto:confluent...@googlegroups.com>.
> > To view this discussion on the web visit
> >
> https://groups.google.com/d/msgid/confluent-platform/a6344a7f-5fa1-473d-96fa-4d4a9215f8ea%40googlegroups.com
> <https://groups.google.com/d/msgid/confluent-platform/a6344a7f-5fa1-473d-96fa-4d4a9215f8ea%40googlegroups.com>
>
> >
> <https://groups.google.com/d/msgid/confluent-platform/a6344a7f-5fa1-473d-96fa-4d4a9215f8ea%40googlegroups.com?utm_medium=email&utm_source=footer
> <https://groups.google.com/d/msgid/confluent-platform/a6344a7f-5fa1-473d-96fa-4d4a9215f8ea%40googlegroups.com?utm_medium=email&utm_source=footer>>.
>
> > For more options, visit https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>.
>
> --
> You received this message because you are subscribed to the Google
> Groups "Confluent Platform" group.
> To unsubscribe from this group and stop receiving emails from it,
> send an email to confluent-platf...@googlegroups.com
> <mailto:confluent-platf...@googlegroups.com>.
> To post to this group, send email to
> confluent...@googlegroups.com
> <mailto:confluent...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/confluent-platform/bdb1ae1e-6eff-4567-a324-b4eab4bb9ca1%40googlegroups.com
> <https://groups.google.com/d/msgid/confluent-platform/bdb1ae1e-6eff-4567-a324-b4eab4bb9ca1%40googlegroups.com?utm_medium=email&utm_source=footer>.
>
> For more options, visit https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>.
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "Confluent Platform" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to confluent-platf...@googlegroups.com
> <mailto:confluent-platf...@googlegroups.com>.
> To post to this group, send email to confluent...@googlegroups.com
> <mailto:confluent...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/confluent-platform/CAOTij-7zcG1bCT34HmTbXAzoFtgTHZVD49GQY7hWouqKQBpOKA%40mail.gmail.com
> <https://groups.google.com/d/msgid/confluent-platform/CAOTij-7zcG1bCT34HmTbXAzoFtgTHZVD49GQY7hWouqKQBpOKA%40mail.gmail.com?utm_medium=email&utm_source=footer>.
signature.asc

Matthias J. Sax

unread,
Feb 15, 2017, 1:09:40 PM2/15/17
to confluent...@googlegroups.com
Have a look here: http://kafka.apache.org/documentation/#monitoring
signature.asc
Reply all
Reply to author
Forward
0 new messages