Message Ordering in Akka

[Joseph Noir]

Hi,

Akka has a interesting page [1] about reliability for exchanged messages. After looking a bit into the topic, I understand reasoning behind choosing at-most-once delivery as a default.

What I still struggle with is the message order between two actors. Is this based on the ordering provided by TCP or does Akka reorder messages received (within a certain interval) by itself?

With regard to reliability, this article [2] is cited, which also argues that message order has to be tracked on the application layer if needed. Why is message order important or ensured per default?

Thanks
Joseph


[1] http://doc.akka.io/docs/akka/current/general/message-delivery-reliability.html
[2] http://www.infoq.com/articles/no-reliable-messaging

[Guido Medina]

With one connection, as soon as the message gets to the other side and it is put into the target mailbox the promised akka ordering documented rules are covered with that (it can be proven mathematically or by logic)

[Guido Medina]

Sorry I need to elaborate more: akka has a strong use of single-consumer mailboxes and one actor can only be a single producer so a message send from a sender will always be "before" the next message, and for a destination that message will always come after a previous message for a recipient, it sounds repetitive but the concept is in itself a bit recursive....the happens before rules are hence implied to be true, by mathematical implication.

[Joseph Noir]

On a local node, yes, sure. But how does this work in a distributed setup?

[Guido Medina]

Imagine there is a dedicated mailbox per remote TCP connection, example:

• Assume you have node 1, node 2 and node 3
• node 1 has a single consumer multiple producer mailbox dedicated for node 2 and another for node 3.
• that same for the other nodes.

So if many actors from node 1 to node 2, each actor individually can only queue its messages sequentially, interleaving messages will only happen between different senders but with the same sender.

A different rule is applied to the recipient, you deliver the messages that arrive sequentially to their respective mailbox interleaving the ones that the sender interleaved but only between different senders.

Akka team correct me if my assumptions are wrong.

Guido.

[Joseph Noir]

Imagine there is a dedicated mailbox per remote TCP connection [...]

Is the assumption that messages are always exchanged via TCP?

So if many actors from node 1 to node 2, each actor individually can only queue its messages sequentially, interleaving messages will only happen between different senders but with the same sender. 

A different rule is applied to the recipient, you deliver the messages that arrive sequentially to their respective mailbox interleaving the ones that the sender interleaved but only between different senders.

Is there more than one queue step involved on the recipient side -- one queue per connection and then a mailbox for each actor? This would allow reordering in the "connection" specific queues before passing messages to the mailboxes.

Thanks for taking the time to the discuss this with me!

Joseph

[Roland Kuhn]

Yes, messages are enqueued in the remoting layer such that the original ordering constraint it upheld, then transmitted in sequence and delivered to recipient mailboxes in that same sequence as well (synchronously—beware of bounded mailboxes with push timeout > 0).

In short, we rely upon the local message ordering guarantees to transitively implement the remote extension of it.

Regards,

Roland

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

Dr. Roland Kuhn
Akka Tech Lead
Typesafe – Reactive apps on the JVM.
twitter: @rolandkuhn

[Joseph Noir]

Ok. If that is the case, is the reason reliability is not guaranteed that mailboxes may be full or nodes may crash?

Thanks,

Joseph

[Roland Kuhn]

This email comes without context: what exactly are you referring to?

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

[Joseph Noir]

This email comes without context: what exactly are you referring to?

Sorry, I'll try to clarify my question.

 

Yes, messages are enqueued in the remoting layer such that the original ordering constraint it upheld, then transmitted in sequence and delivered to recipient mailboxes in that same sequence as well (synchronously—beware of bounded mailboxes with push timeout > 0).

I assumed that your "Yes" referred to my comment on TCP being used to ensure ordering when delivering messages over the network. TCP itself ensures reliable delivery as well as maintaining the order of messages. However, the documentation [1] specifies message delivery to be at-most-once, i.e., messages can be lost. Why is the ordering ensured by TCP, when the delivery is not? 

Ok. If that is the case, is the reason reliability is not guaranteed that mailboxes may be full or nodes may crash?

Hopefully, this clarified the question, i.e., even if the message is delivered to the network buffer it may still not be passed to an actor because the node may crash or mailboxes may be full.

Joseph

[1] http://doc.akka.io/docs/akka/current/general/message-delivery-reliability.html

[Endre Varga]

On Thu, Feb 4, 2016 at 2:52 PM, Joseph Noir <josep...@gmail.com> wrote:

This email comes without context: what exactly are you referring to?

Sorry, I'll try to clarify my question.

 

Yes, messages are enqueued in the remoting layer such that the original ordering constraint it upheld, then transmitted in sequence and delivered to recipient mailboxes in that same sequence as well (synchronously—beware of bounded mailboxes with push timeout > 0).

I assumed that your "Yes" referred to my comment on TCP being used to ensure ordering when delivering messages over the network. TCP itself ensures reliable delivery as well as maintaining the order of messages. However, the documentation [1] specifies message delivery to be at-most-once, i.e., messages can be lost. Why is the ordering ensured by TCP, when the delivery is not? 

Because connections might get severed in which case we have no clue how much of the messages have been actually delivered or not, TCP only knows the last ACK, but if the ACK does not arrive for later messages because the connection has been lost then there are no guarantees about them.

Also, ordering is not ensured by TCP alone, there are great pains taken in remoting that ordering is respected *across* TCP connections where reconnects happen.

On top of this, messages can get lost if messages are sent faster than the TCP connection can handle, forcing remoting to drop buffers to avoid OOME.

-Endre

 

Ok. If that is the case, is the reason reliability is not guaranteed that mailboxes may be full or nodes may crash?

Hopefully, this clarified the question, i.e., even if the message is delivered to the network buffer it may still not be passed to an actor because the node may crash or mailboxes may be full.

Joseph

[1] http://doc.akka.io/docs/akka/current/general/message-delivery-reliability.html

--

[Joseph Noir]

Thank you for the clarification!

Joseph