akka-kafka-stream "Offset Storage in Kafka" question/clarification

[Drew Kutcharian]

Hi,

In the "Offset Storage in Kafka” on http://doc.akka.io/docs/akka-stream-kafka/current/consumer.html#offset-storage-in-kafka there is the following example:

.groupedWithin(10, 5.seconds)

.map(group => group.foldLeft(CommittableOffsetBatch.empty) { (batch, elem) => batch.updated(elem) })

.mapAsync(3)(_.commitScaladsl())

Wouldn’t the ".mapAsync(3)(_.commitScaladsl())” cause the offsets to be committed to Kafka out of order? Shouldn’t it be "mapAsync(1)”?

Best,

Drew

[Patrik Nordwall]

The commit commands are emitted to the underlying KafkaConsumer in right order. It is waiting for the results of the commits that are done in parallel, and will possibly increase total throughput. We observed better throughput in performance tests.

It would be great to clarify this in docs. You are not the first one to ask. Would you like to open a pull request?

/Patrik

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

[Arun]

Thanks Patrik.

So am I safe to use mapAsync greater than 1 (>1) , still my offset will be stored correctly?

what will happen if my stream fail/crash which offset will be processed..

Thanks,

Arun

[Patrik Nordwall]

On Thu, Nov 24, 2016 at 4:51 AM, Arun <sethi...@gmail.com> wrote:

Thanks Patrik.

So am I safe to use mapAsync greater than 1 (>1) , still my offset will be stored correctly?

yes

 

what will happen if my stream fail/crash which offset will be processed..

it will start from what the offset that was stored in Kafka, i.e. it's at-least-once delivery

 

To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscribe@googlegroups.com.

To post to this group, send email to akka...@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

--

Patrik Nordwall
Akka Tech Lead
Lightbend -  Reactive apps on the JVM
Twitter: @patriknw