Akka and ack-ing messages after successful processing

[Drew Kutcharian]

Hi Guys,

I wanted to see what would the best approach to processing messages where I would need to ACK messages after they have successfully been processed. In case of a failure, I should not ACK the message.

Basically I have a single producer that produces messages sequentially and a bunch of consumer actors that are consuming these messages. The producer needs to know that a message was processed successfully so it can ACK it, otherwise it has to send the message to a different actor.

Thanks,

Drew

[√iktor Ҡlang]

Hi Drew!

def receive = {

   case message => 

      doWork(message)

      sender ! message.acknowledgement()

}

Hope that helps!

Cheers,

√

 

Thanks,

Drew

--
>>>>>>>>>>      Read the docs: http://akka.io/docs/
>>>>>>>>>>      Check the FAQ: http://akka.io/faq/
>>>>>>>>>>      Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To post to this group, send email to akka...@googlegroups.com.
To unsubscribe from this group, send email to akka-user+...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user?hl=en.

--
Viktor Klang

Akka Tech Lead

Typesafe - The software stack for applications that scale

Twitter: @viktorklang

[Jan Ziniewicz]

Hi,

  I am just trying to solve very strange problem which occurs in my system. I am writing in this thread because I have a very similar use case. 

Basically, I have one P(roducer) and one C(onsumer) running on separate machines in the same local network. OS: Ubuntu 12.04., Akka version: 2.0.3.

P sends messages to C, C immediately responds to P. C & P can talk like this for days without any problems. 

Every 7 to 10 days strange things start to happen. P stops receiving confirmation messages from C. However, C still receives messages from P! 

There are no other traces of error in my system nor any anomalies in the machine. All munin / zabbix graphs are flat etc.

Now, I came up with list of possible solutions. Please let me know if any of them makes sense or maybe you have any better ideas which could be helpful.

1. akka.remote.netty.read-timeout = 10s # default is 0s == off

2. P will send poisonPill to itself if it doesn't receive message from C for some time.

3. P will send poisonPill to C if it doesn't receive message from C for some time.

I 'll be happy to answer your questions if you have any.

Thanks!

--

☀ Jan | http://jan.ziniewi.cz

[√iktor Ҡlang]

Hi Jan!

On Tue, Oct 30, 2012 at 12:19 PM, Jan Ziniewicz <janek.z...@gmail.com> wrote:

Hi,

  I am just trying to solve very strange problem which occurs in my system. I am writing in this thread because I have a very similar use case. 

Basically, I have one P(roducer) and one C(onsumer) running on separate machines in the same local network. OS: Ubuntu 12.04., Akka version: 2.0.3.

P sends messages to C, C immediately responds to P. C & P can talk like this for days without any problems. 

Every 7 to 10 days strange things start to happen. P stops receiving confirmation messages from C. However, C still receives messages from P! 

There are no other traces of error in my system nor any anomalies in the machine. All munin / zabbix graphs are flat etc.

Now, I came up with list of possible solutions. Please let me know if any of them makes sense or maybe you have any better ideas which could be helpful.

1. akka.remote.netty.read-timeout = 10s # default is 0s == off

Use 0s here. What do do you have any other connection timeouts set?

 

2. P will send poisonPill to itself if it doesn't receive message from C for some time. 

3. P will send poisonPill to C if it doesn't receive message from C for some time.

You might want to look into getting a Typesafe Subscription to get access to Support + the Console so you can see the activity.

Let me know if that is interesting and I can send over more details.

Cheers,

√

[Jan Ziniewicz]

W dniu wtorek, 30 października 2012 12:27:33 UTC+1 użytkownik √ napisał:

Hi Jan!

On Tue, Oct 30, 2012 at 12:19 PM, Jan Ziniewicz <janek.z...@gmail.com> wrote:

Hi,

  I am just trying to solve very strange problem which occurs in my system. I am writing in this thread because I have a very similar use case. 

Basically, I have one P(roducer) and one C(onsumer) running on separate machines in the same local network. OS: Ubuntu 12.04., Akka version: 2.0.3.

P sends messages to C, C immediately responds to P. C & P can talk like this for days without any problems. 

Every 7 to 10 days strange things start to happen. P stops receiving confirmation messages from C. However, C still receives messages from P! 

There are no other traces of error in my system nor any anomalies in the machine. All munin / zabbix graphs are flat etc.

Now, I came up with list of possible solutions. Please let me know if any of them makes sense or maybe you have any better ideas which could be helpful.

1. akka.remote.netty.read-timeout = 10s # default is 0s == off

Use 0s here. What do do you have any other connection timeouts set?

All connections timeouts are set to default values.

 

 

2. P will send poisonPill to itself if it doesn't receive message from C for some time. 

3. P will send poisonPill to C if it doesn't receive message from C for some time.

You might want to look into getting a Typesafe Subscription to get access to Support + the Console so you can see the activity.

Let me know if that is interesting and I can send over more details.

I definitely won't be able to afford it, sorry ;)

[√iktor Ҡlang]

On Tue, Oct 30, 2012 at 12:34 PM, Jan Ziniewicz <janek.z...@gmail.com> wrote:

W dniu wtorek, 30 października 2012 12:27:33 UTC+1 użytkownik √ napisał:

Hi Jan!

On Tue, Oct 30, 2012 at 12:19 PM, Jan Ziniewicz <janek.z...@gmail.com> wrote:

Hi,

  I am just trying to solve very strange problem which occurs in my system. I am writing in this thread because I have a very similar use case. 

Basically, I have one P(roducer) and one C(onsumer) running on separate machines in the same local network. OS: Ubuntu 12.04., Akka version: 2.0.3.

P sends messages to C, C immediately responds to P. C & P can talk like this for days without any problems. 

Every 7 to 10 days strange things start to happen. P stops receiving confirmation messages from C. However, C still receives messages from P! 

There are no other traces of error in my system nor any anomalies in the machine. All munin / zabbix graphs are flat etc.

Now, I came up with list of possible solutions. Please let me know if any of them makes sense or maybe you have any better ideas which could be helpful.

1. akka.remote.netty.read-timeout = 10s # default is 0s == off

Use 0s here. What do do you have any other connection timeouts set?

All connections timeouts are set to default values.

Do you log the remote lifecycle events? If so, what do they say?

 

 

 

2. P will send poisonPill to itself if it doesn't receive message from C for some time. 

3. P will send poisonPill to C if it doesn't receive message from C for some time.

You might want to look into getting a Typesafe Subscription to get access to Support + the Console so you can see the activity.

Let me know if that is interesting and I can send over more details.

I definitely won't be able to afford it, sorry ;)

Ok

Cheers,

√

[Jan Ziniewicz]

W dniu wtorek, 30 października 2012 12:38:25 UTC+1 użytkownik √ napisał:

On Tue, Oct 30, 2012 at 12:34 PM, Jan Ziniewicz <janek.z...@gmail.com> wrote:

W dniu wtorek, 30 października 2012 12:27:33 UTC+1 użytkownik √ napisał:

Hi Jan!

On Tue, Oct 30, 2012 at 12:19 PM, Jan Ziniewicz <janek.z...@gmail.com> wrote:

Hi,

  I am just trying to solve very strange problem which occurs in my system. I am writing in this thread because I have a very similar use case. 

Basically, I have one P(roducer) and one C(onsumer) running on separate machines in the same local network. OS: Ubuntu 12.04., Akka version: 2.0.3.

P sends messages to C, C immediately responds to P. C & P can talk like this for days without any problems. 

Every 7 to 10 days strange things start to happen. P stops receiving confirmation messages from C. However, C still receives messages from P! 

There are no other traces of error in my system nor any anomalies in the machine. All munin / zabbix graphs are flat etc.

Now, I came up with list of possible solutions. Please let me know if any of them makes sense or maybe you have any better ideas which could be helpful.

1. akka.remote.netty.read-timeout = 10s # default is 0s == off

Use 0s here. What do do you have any other connection timeouts set?

All connections timeouts are set to default values.

Do you log the remote lifecycle events? If so, what do they say?

Well, system seemed to be stable enough so I changed log level to INFO. Now, I reverted back to DEBUG.

Generally, I suppose that problem lies somewhere in linux kernel or hardware and not in akka. I noticed a similar bug a few years ago in another system (nginx <-> tomcat).

Thus, I am looking for some method of restarting tcp connections beteween P & C without restarting jvms. It seems to be an easier, faster (and cheaper) task than looking for bug. 

[Drew Kutcharian]

Would this guarantee at-least-once handling of the messages? And I guess I would need to somehow keep track of all these messages on the producer right? Wouldn't ask be a better approach?

[Evan Chan]

One approach which is more scalable is to keep an incrementing counter of messages on the producer which is the "last successfully sent" counter.   You also have a "current message" counter which increments too.    So the flow goes like:

* Send message with current counter

* consumer receives message, sends ack with message counter ID

* producer receives ack, updates "last successfully sent" counter

If you don't receive acks after a while, you can re-send messages starting from the "last successfully sent".   The consumer(s) must deal with de-dup.  This guarantees at least once.

This approach works better in conjunction with something like a message queue which supports rewinding messages, like Kafka.

-Evan

--
Evan Chan
Senior Software Engineer | 
e...@ooyala.com | (650) 996-4600
www.ooyala.com | blog | @ooyala

[Drew Kutcharian]

Thanks Evan. I was actually looking at using Akka Futures for this. Which one would be the better approach?

[Evan Chan]

I wonder how competitive the Futures version would be.  You could do a Future.sequence() on a batch of messages, and then it all succeeds or fails, then you can resend the batch to another destination.  The downside is you need to keep a whole bunch of futures open....

The counter approach is used by Kafka + Storm for at least once / exactly once processing.

-Evan

[Drew Kutcharian]

Well I'm not sure if the counter approach will work for my use case (or maybe I'm not getting it). What I'm trying to do is:

1. Consume N messages from a Queue

2. Launch N child actors to process those messages

3. Get the results of the children as two collections (successful messages and failed messages)

4. Put the failed messages in a retry queue

5. Commit the transaction so these N messages get removed from the queue

7. Go back to step 1

-- Drew