[akka-stream] demand doesn't follow arguments for for grouped or take

[Simon Schäfer]

I just tried the following (on 2.0-M1):

scala> Source(1 to 100).map{i ⇒ println(i); i}.grouped(10).runWith(Sink.head)
1
2
3
4
5
6
7
8
9
10
11
res10: scala.concurrent.Future[scala.collection.immutable.Seq[Int]] = scala.concurrent.impl.Promise$DefaultPromise@2df79942
12
13
14
15
16

scala> 17
18
19
20
21
22
23
24
res10.onComplete(println)
Success(Vector(1, 2, 3, 4, 5, 6, 7, 8, 9, 10))

A similar behavior can be seen for the take function:

scala> Source(1 to 100).map{i ⇒ println(i); i}.take(10).runWith(Sink.fold(Vector[Int]())(_ :+ _))
...


As one can see, the stream processes more elements than it has to. It is not a problem for me, but I would like to know if this behavior is spec'ed anywhere? Shouldn't the stream be able to say, just by looking at the arguments for grouped, take and the sink, how many elements should be produced?

[Endre Varga]

Hi Simon,

This documentation section explains this in more detail: http://doc.akka.io/docs/akka-stream-and-http-experimental/2.0-M1/scala/stream-rate.html

Btw, Sink.head actually does not request more than one, however this is not transitive and cannot be. Grouped cannot possibly know how many elements a downstream ever needs, so it tries to prefetch and exploit concurrency.

-Endre

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

[Simon Schäfer]

On 11/12/2015 06:57 PM, Endre Varga wrote:

Hi Simon,

This documentation section explains this in more detail: http://doc.akka.io/docs/akka-stream-and-http-experimental/2.0-M1/scala/stream-rate.html

Ok, I didn't come to that point of the documentation so far. I'm going to have a look, thanks!

Btw, Sink.head actually does not request more than one, however this is not transitive and cannot be. Grouped cannot possibly know how many elements a downstream ever needs, so it tries to prefetch and exploit concurrency.

Why not? Doesn't it have to wait for a demand as all other components have to do too? I mean Sink.head has a demand of 1, shouldn't grouped get this demand and then forward a demand of 1*N, where N is the argument to grouped?

You received this message because you are subscribed to a topic in the Google Groups "Akka User List" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/akka-user/m4V-AsScuak/unsubscribe.
To unsubscribe from this group and all its topics, send an email to akka-user+...@googlegroups.com.

[Endre Varga]

On Thu, Nov 12, 2015 at 7:09 PM, Simon Schäfer <ma...@antoras.de> wrote:

On 11/12/2015 06:57 PM, Endre Varga wrote:

Hi Simon,

This documentation section explains this in more detail: http://doc.akka.io/docs/akka-stream-and-http-experimental/2.0-M1/scala/stream-rate.html

Ok, I didn't come to that point of the documentation so far. I'm going to have a look, thanks!

Btw, Sink.head actually does not request more than one, however this is not transitive and cannot be. Grouped cannot possibly know how many elements a downstream ever needs, so it tries to prefetch and exploit concurrency.

Why not? Doesn't it have to wait for a demand as all other components have to do too? I mean Sink.head has a demand of 1, shouldn't grouped get this demand and then forward a demand of 1*N, where N is the argument to grouped?

No, demand is not forwarded, that does not work in general, especially in the face of graph enabled stages (how would merge work?). Demand is strictly local to a connection. If grouped gets a demand of 1, then it knows that it can only emit one, but that does not dictate how much it tries to get from upstream. What if the sink is not a Sink.head but a network transport that signals demand one-by-one once the bytes are written to the network? In this case grouped would better off aggregating elements in the background. Just because a stage demanded 1, it does not mean it will not demand later more. 

-Endre

[Simon Schäfer]

On 11/12/2015 07:15 PM, Endre Varga wrote:

On Thu, Nov 12, 2015 at 7:09 PM, Simon Schäfer <ma...@antoras.de> wrote:

On 11/12/2015 06:57 PM, Endre Varga wrote:

Hi Simon,

This documentation section explains this in more detail: http://doc.akka.io/docs/akka-stream-and-http-experimental/2.0-M1/scala/stream-rate.html

Ok, I didn't come to that point of the documentation so far. I'm going to have a look, thanks!

Btw, Sink.head actually does not request more than one, however this is not transitive and cannot be. Grouped cannot possibly know how many elements a downstream ever needs, so it tries to prefetch and exploit concurrency.

Why not? Doesn't it have to wait for a demand as all other components have to do too? I mean Sink.head has a demand of 1, shouldn't grouped get this demand and then forward a demand of 1*N, where N is the argument to grouped?

No, demand is not forwarded, that does not work in general, especially in the face of graph enabled stages (how would merge work?). Demand is strictly local to a connection. If grouped gets a demand of 1, then it knows that it can only emit one, but that does not dictate how much it tries to get from upstream. What if the sink is not a Sink.head but a network transport that signals demand one-by-one once the bytes are written to the network? In this case grouped would better off aggregating elements in the background. Just because a stage demanded 1, it does not mean it will not demand later more.

Ok, I understand now. Thanks for the explanation.