Backpressure for fan-in designs: pull vs push

[Evan Chan]

Dear friends,

I've read and grok the back-pressure story around Akka streams, which is pull-based.  I have also seen examples of this with fixed topologies that tend to be relatively linear or are fan-out: for example, a coordinator farming work out to worker nodes.   For these examples, I can see that it would work well.  Each worker issues a pull request (haha, no pun intended) for more work from one master.

Now, consider the case where I may have multiple flows or pipelines feeding in to one I/O sink, which does something like a mapAsync (feeds to futures doing I/O).  This is much more of a fan-in pattern.
The multiplicity of the pipelines changes dynamically.

For example, let's say that each pipeline ingests one customer's data, and it is simpler to keep one pipeline per customer.  As a new customer comes in, then we want to start a pipeline for them, and it should feed into the same I/O sink.  If there are more customers/pipelines, one would expect backpressure so that each flow slows down, which is expected.

Implementing a pull backpressure in this case seems much more complicated.  Does Akka streams handle dynamic registration / materialization?   In order for the sink to know about each pipeline, each pipeline must somehow register itself with the sink, and the sink must then send pull requests to each pipeline.   Or, would the sink only send pull requests to some random subset of pipelines, if it is more heavily loaded?

I can think of a way to have push, but still have backpressure, even over networks.

-Evan

[Roland Kuhn]

Hi Evan,

first off, we describe Reactive Streams as being dynamic push–pull: due to the batching, a fast consumer can keep the demand at the producer always big enough such that the system behaves exactly like a push-model.

Concerning merge strategies we offer a rather limited set at the moment plus the ability to write your own, and each of the input ports of the merging stage will have a (small) buffer that it tries to keep filled by signaling demand to its Publisher. As such you can implement any scheme that you like, but currently we do not support dynamic merging in the sense of allowing streams to come and go. I do agree that this is a must-have, though.

Regards,

Roland

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

Dr. Roland Kuhn
Akka Tech Lead
Typesafe – Reactive apps on the JVM.
twitter: @rolandkuhn

[Evan Chan]

Roland,

Thanks.  Do you guys have a time line for the dynamic merging?  Also for acknowledgement, or is that just through integration with persistent actors?

-Evan

"Never doubt that a small group of thoughtful, committed citizens can change the world" - M. Mead

You received this message because you are subscribed to a topic in the Google Groups "Akka User List" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/akka-user/iEALqdlScoQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to akka-user+...@googlegroups.com.