[akka-stream] how to use akka-stream in a multi-instance environment

58 views
Skip to first unread message

Patrick Li

unread,
Nov 27, 2015, 10:54:00 PM11/27/15
to Akka User List
Hi,

I have an application (based on Play 2/akka) using akka-stream to stream data from one service (e.g. Twitter) to another. The application is stateless, so I plan to have two instances of the app running with a load balancer in the front to scale it out as well as have redundancy. Everything works well except in this case, two streams are created, 1 per app, and this duplicates the data being sent to the sink.

I read the documentation as well as some threads here, that akka-stream at the moment does not support remote actors, so I cannot leverage akka-cluster's singleton actor. Are there any workaround or other approach I can have two apps running but only 1 stream running (or 2 streams but only 1 active)? Preferably, if one stream or the app dies, the other app can start a new stream and keep going.

Thanks
Patrick

matheus...@gmail.com

unread,
Nov 28, 2015, 12:46:23 PM11/28/15
to Akka User List
You can use kafka to send messages do multiple akka stream consumers. Kafka is a pub subs infrastructure. It can balance load between multiple consumers and has high performance.
http://kafka.apache.org/

Basically you define a topic where messages are insert into and consumed. One topic has N partitions and the partition are shared between multple consumers.

Patrick Li

unread,
Nov 30, 2015, 1:23:58 AM11/30/15
to Akka User List
So, if I understand correctly, are you saying, the incoming data from 3rd party service is put into a Kafka topic, akka streams will use the Kafka topic as the source, and Kafka will ensure only one stream will consume the data at any given time?
Reply all
Reply to author
Forward
0 new messages