Remoting transport back-pressure

[Luben Karavelov]

Hi,

I have the following push dataflow using remote actors (netty.tcp transport) over the network:

SourceActor -> ProcessingActor ~~ network ~~>  CollectingActor

The problem is that if the network is relatively slow the processing actor successfully sends the messages but they get all buffered inside the same JVM until it blows with OOM. I am looking how to slow down the SourceActor if the upstream network is slower than the production rate. I have put bounded mailbox between the Source and Processing actor but it does not help because the ProcessingActor is not slowed by the network transport.

I looked at the code and it looks the problem arises because the EndpointWriter actor (that is kind of a proxy between the local actors and the transport as far as I understand) buffers the data to be sent in itself.

So here are my questions:

1. Is there a way to create a back-pressure between the network transport and the sender?

2. If there is no way and I have to create an application level flow-control over the network (between the Processing and Collecting actors) what will be the best way to slow down the SourceActor? Should I extend the flow control down to it? Or is there a way to slow down the reception of messages coming from the SourceActor so that the bounded mailbox will slow it down (I will still need the other messages to be delivered for flow-control signalization)?

Thanks in advance for any advises and suggestions,
luben

[Michael Frank]

i think you will need to implement your own acking protocol.

SourceActor sends something like PushMessage to ProcessingActor, which enriches the message with some processing, then sends to the remote CollectingActor.  CollectingActor then responds to ProcessingActor with AckMessage.  ProcessingActor forwards the AckMessage to SourceActor.  Once SourceActor receives AckMessage it sends the next PushMessage.

I don't think bounded mailboxes are a good solution, because they have a fixed capacity, and that capacity is not related to your network capacity.  an acking protocol will get you better network utilization, and be more tolerant/performant with respect to changing network conditions.

-Michael

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

[Любен]

Hi Michael,

I have taken another approach. I already had a Send/Ack flow control over the network. So I extended it to use the 'ask' and wait for the Ack from the collecting actor in order to exersice back-pressure to the source. Using blocking and bounded queue will also block the source after the queue is full.

Thanks for the advice,

luben

You received this message because you are subscribed to a topic in the Google Groups "Akka User List" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/akka-user/UYojosSS298/unsubscribe.
To unsubscribe from this group and all its topics, send an email to akka-user+...@googlegroups.com.

[Guido Medina]

Hi Luben,

There are other types of mailboxes good for that (depending on your requirements) you have bounded non-blocking and I'm not 100% sure but I think I read somewhere that in Akka streams you have a bounded chopping head sink, say you don't care about head messages that have taken too long to process and only the last newest 10k, etc.

HTH,

Guido.

[Любен]

Hi Guido,

Thanks for the pointer to the chopping head mailbox. I will take a look, sounds really interesting. I am already using different bounded mailboxes between the Source and the Processing actor depending on the semanitcs of the computation: if the source is shared between multiple processing actors (in the cases where getting the results on-time is more important than processing all the data) I use non-blocking mailbox. If the source is not shared and I care about processing all the input data I use blocking mailbox to slow-down the source.

Regards,

luben