Re: [akka-user] Akka cluster: how to manage pending message in mailbox if a member goes down

[Akka Team]

Hi Olivier!

Can you give a little more detail? Who takes the message from JMS, a dedicated server? What you call "agents" are your worker servers? Do the workers send the results directly to the external applications or is there some entity that collects and forwards them?

-Endre

On Tue, Jun 4, 2013 at 4:12 PM, Olivier Martin <ekk...@gmail.com> wrote:

Hi all

I need to process some messages from a JMS store, do some logic with this message and notify them to some external applications.
In order to ensure scalability, i am starting several actor systems on different servers.
JMS reading is consistent so there is no problem on this part.
The problem i reach is from the akka part: I need to be able to redirect pending work (pending messages in agent mailbox) from one server to a running server member if the current one is going down (akka system shutdown).
Should i share mailbox between agents across the cluster? Should i listen to member events and enqueue message in another agent mailbox?
Thanks for your help

Olivier

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://akka.io/faq/
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--

Akka Team

Typesafe - The software stack for applications that scale

Blog: letitcrash.com
Twitter: @akkateam

[Akka Team]

Hi Olivier!

To be able to handoff pending jobs you will need an entity that keeps track of these jobs -- otherwise if a worker crashes and cannot handoff data itself, the jobs will be lost. A worker that shuts down gracefully can notify the bookkeeper entity what was the last job it executed so it can send the unfinished ones to another worker. In the case of worker crashes you cannot be ever sure what was the last successful job and what was lost, so you have to resubmit from the last known good one. Your systems have to recognize duplicates though, and ignore them.

-Endre

On Tue, Jun 4, 2013 at 5:29 PM, Olivier Martin <ekk...@gmail.com> wrote:

Hi endre
JMS is consumed by a java component outside akka system (a mdb in weblogic). The MDB is getting a reference to akka system thanks to spring.
What i call an agents is either an actor doing the business logic (some filtering, some enrichment on the data) or a notification to one or several different applications (i am using dispatcher to managed parallel calls)

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://akka.io/faq/
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[√iktor Ҡlang]

That sounds very fragile, in the event of a crash or network failure there is no way to gracefully handoff.

Instead I'd recommend to take from the queue without acking the JMS message before it's processed. Then you'll have to have a deadline for how long between consume and ack so that un-acked messages are eventually processed.

Does that make sense?

Cheers,

√

On Wed, Jun 5, 2013 at 9:48 AM, Olivier Martin <ekk...@gmail.com> wrote:

Hi endre

Thanks for your inputs.
For nice shutdown, if a cluster member goes down, the worker should stop the JMS consumer, send pending work from mailbox to other cluster members and then stop. Am i right?
I am thinking of using a shared durable mailbox between the same type of actor over the cluster. Do you think it is a good idea?

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://akka.io/faq/
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--

Viktor Klang
Director of Engineering

Typesafe

Twitter: @viktorklang

[Akka Team]

Hi!

Viktor was quicker than me :) Yes, the problem is that the sender and the worker has to both agree that the worker goes down and no messages should be delivered to it. This can never be done completely reliably. They also have to agree on that the worker has returned all of its pending jobs, which is basically the same problem.

-Endre