[akka-persistent]: Refine the logic regarding when channel messages should be re-delivered

[dong]

I'm seeing in my application a lot of channels message redelivery when the overall load of the system is high.

Is it a good idea to re-deliver channel message this way:

1. record the latest confirmed time [ of any message]
2. re-deliver the message only iff (current_timestamp - latest_confirmed_timestamp) > timeout1  && current_timestamp - the_message's_delivery_timestamp > timeout_2)

I may missed many details, but you get the idea.

-Daniel

[Björn Antonsson]

Hi Daniel,

Are you talking about getting RedeliveryFailure messages when your load is high? The Channel should handle redelivery for you. If you  have a lot of redeliveries happening, then maybe your redeliver interval is set too tight, or you are simply overloading your system.

B/

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

-- 

Björn Antonsson

Typesafe – Reactive Apps on the JVM

twitter: @bantonsson

JOIN US. REGISTER TODAY!

Scala

Days

June 16th-18th,

Berlin

[Dong Wang]

I se the interval to 10  seconds. Does my suggestion make any sense to you?

You received this message because you are subscribed to a topic in the Google Groups "Akka User List" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/akka-user/g47kdVSOqMw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to akka-user+...@googlegroups.com.

[Björn Antonsson]

Hi Daniel,

I'm not sure where your logic would be used. Are you using transient Channels and a Processor or are you using PersistentChannels? If you are using a PersistentChannel then you could limit the maximum number of outstanding messages via the pendingConfirmationsMax setting.

A redeliverInterval setting of 10 seconds seems fine, but what is your redeliverMax setting. Are you flooding the channel and your code would be a back off in the sender? You only need to resend from the sender if you get a redelivery failure message from the channel.

Is the channel destination remote or local? Do you see any remoting issues?

B/

[王东 Dong Wang]

On May 14, 2014, at 2:32 PM, Björn Antonsson <bjorn.a...@typesafe.com> wrote:

Hi Daniel,

I'm not sure where your logic would be used. Are you using transient Channels and a Processor or are you using PersistentChannels? If you are using a PersistentChannel then you could limit the maximum number of outstanding messages via the pendingConfirmationsMax setting.

We are using transient channels with a processor.

A redeliverInterval setting of 10 seconds seems fine, but what is your redeliverMax setting. Are you flooding the channel and your code would be a back off in the sender? You only need to resend from the sender if you get a redelivery failure message from the channel.

We use the default value of redeliverMax setting. We saw what we saw during a load testing, which means we don't back off senders at all.

[Björn Antonsson]

Hi Daniel,

You still haven't explained if there are redelivery failures or just redeliveries, and if the destination is local or remote.

If you see redeliveries or failures while the the destination is alive and healthy, then you are flooding your destination. That is a flow control issue that the channel can't solve for you in a generic way. You need to add some throttling in the form of high/low watermarks, work pulling and or fan-out.

The PersistenChannel will limit the number of messages in flight for you. It might work for bursts of traffic, but if you continuously send more messages than the destination can handle you will eventually overflow.

B/

[王东 Dong Wang]

Sorry for that. It's just redeliveries, a lot of redeliveries, we didn't really encounter redelivery failures. 

Thank you for the suggestions regarding throttling. We surely haven't done that yet.

I just want to share with you and the akka community that when our load is high, we can achieve a QPS of 10000 (of course this is very application specific) but when the load is 5x higher, the QPS dropped significantly to 200. And to me it was caused by a lot of redeliveries. Maybe the redelivery mechanism can be enhanced a little bit by implementing my little idea :) 

[Björn Antonsson]

Hi,

I still don't think that it's the job of the Transient channel to do this throttling for you. Have you experimented with your proposed change? Does it really help? What stops you from running out of memory by just growing the number of messages that are in flight indefinitely?

B/