[Akka Streams] At-Least-Once-Delivery behaviour with Kafka and Cassandra

[Juan José Vázquez Delgado]

Hello everyone,

I'm starting with Akka Streams and so far everything is going well. However, I have met with a use case that I don't know how to approach. The scenario is a stream with an ActorPublisher as a source that is consuming messages from Kafka and a subscriber as a sink that updates a Cassandra table.

Kafka ~> some mapping operations ~> Cassandra

The point is that I'd like to explicitly confirm to Kafka every time a message has been successfully processed and inserted into Cassandra so that I could re-read the message in case a disaster happens and the service fails, i.e. some kind of at least once delivery behaviour. How could I approach this in terms of Akka Streams?. Is a supported scenario?.

It's true that I always can configure the Kafka consumer with auto-commit behaviour but I'd rather take control of how I'm reading the messages.

Thanks in advance for your help.

[Akka Team]

Hi Huan,

What you need is a back-channel from the Cassandra Sink to the Kafka Source. In Akka Streams the way to solve this is to create an explicit backward channel with streams.

For example if you have an async Kafka API that gives back the result of your read as a Future, together with some metadata usable for acking, you can model it like this (pseudocode, I don't know how Kafka API looks like):

val kafkaReader: Flow[AckId, (MyData, AckId), Unit] = Flow[AckId].mapAsync(parallelism = 4) { ack =>
  kafka.acknowledge(ack)

  kafka.readAsync() // Returns Future[(MayData, AckId)]

}

val cassandraWriter: Flow[(MyData, AckId), AckId, Unit] = Flow[(MayData, AckId)].mapAsync(parallelism = 4) { case (data, ackid) =>

  cassandra.writeAsync(data).map(_ => ackid) // Returns Future[AckId]

}

kafkaReader.join(cassandraWriter).run() // Close the loop and start

The above code does not handle failed acknowledgments, you need to add that handling yourself (for example instead of sending Ackid from cassandra Flow, you send a Success(Ackid) or Failure(AckId)). Also, the above cycle will not run, because there are no initial acknowledgments in the loop to start it. You can do this by adding 4 dummy acknowledgments:

kafkaReader.join(cassandraWriter ++ Source(List(dummyAck1, ...))).run()

(or you can alternatively concat kafkaReader with a Source that injects 4 reads from Kafka)

-Endre

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

--

Akka Team

Typesafe - Reactive apps on the JVM

Blog: letitcrash.com
Twitter: @akkateam

[Juan José Vázquez Delgado]

Thanks for your answer. I'm currently in the middle of something and haven't had the opportunity to check this out. I'll have a look into your suggestion ASAP, but it seems it could be the way to go. I'll update this thread as soon as I test the solution. Thanks!.

BR, 

Juanjo.