[Akka-persistence 2.4] How to retry failed message

[Anders Båtstrand]

Dear group

I am looking at Akka Persistence 2.4.

Sometimes, we get write timeout against Cassandra, and we want to do a retry. Using the new error handler system and BackoffSupervisor, the actor is restartet as expected. However, I am not able to understand how I can retry the message.

In onPersistFailure i get the message i tried to persist. That I can not use, since that is not the same message as was sent to the actor. I could persist it again, perhaps, but them the callback with side-effects will not be called.

Do I have to do event sourcing, or is there some way of getting the message (command) that was sent to the actor when it generated an event that failed to persist?

Best regards,

Anders Båtstrand

[mathe...@sagaranatech.com]

You can use ack. For example, the component that sends the message to you persistent actor will wait an acknowledge response. Your persistent actor will send this ack after sucessful persist the correspondent event. If the PA no send back response after a time, the component retry send the message. It leads to an at-leat-once-delivery behaviour and your persistent actor can receive unordered or duplicated messages.

[Anders Båtstrand]

Thank you!

That would actually solve the problem in the case I am working on
right now. The sender is an actor reading from a topic, and can check
for the ack.

It would be optimal for me if I could retry the failing message BEFORE
processing the next messages, but some degree of out-of-order is there
anyway.

Anders

> --
>>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>>> Check the FAQ:
>>>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
> ---
> You received this message because you are subscribed to a topic in the
> Google Groups "Akka User List" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/akka-user/Gp0659Dkfxk/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> akka-user+...@googlegroups.com.
> To post to this group, send email to akka...@googlegroups.com.
> Visit this group at http://groups.google.com/group/akka-user.
> For more options, visit https://groups.google.com/d/optout.

[mathe...@sagaranatech.com]

You can implement this behaviour send a sequence number together your message to the persistent actor and control if the persistent actor is receiving the expected sequential number. You can use AtLeasOnceDelivery trait to generate this sequential number.
http://doc.akka.io/docs/akka/2.4.0-RC2/scala/persistence.html#At-Least-Once_Delivery
In resume this trait generate sequence numbers to messages send and has a confirmDelivery method to confirm that a message was received for the destination. The important detail is if this actor send messages to many destinations, each destination is receive message with sequence number gap.
Ordered processing with akka persistence is doable, but only use it if you needs because is a bit complex implement.
I hope it help you.

[Anders Båtstrand]

Thank you for the detailed answer!

I will accept potentially more out-of-order messages.

Am I correct to also skip the BackoffSupervisor, and just use resume()? Restarting the actor seems to serve no purpuse now. With resume() the actor can still do other stuff.

Best regards,

Anders

[mathe...@sagaranatech.com]

Yes, resume actor will serve your proposal.

[Patrik Nordwall]

Regarding resume, if the persist fails the PA will unconditionally be stopped, so there is no supervision involved (no resume) for persist failures. What the BackoffSupervisor does is to start the PA again after exponential backoff delay.


/Patrik

--

>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---

You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.

[Anders Båtstrand]

Ah, so there is no way to avoid the recovery process, which then would hit Cassandra right after each timeout?

When the actor is stopped, I got dead letters in my test (messages sendt to the BackoffSupervisor ). Is that correct, or should the mailbox still be there, accepting messages?

Trying to make out how to migrate to Akka 2.4! :-)

[Patrik Nordwall]

Messages that are already in the PA mailbox will be dropped (dead letters) when the actor is stopped.

The BackoffSupervisor also drops messages during the backoff, and as discussed earlier some kind of resending is needed to make sure that all messages make it at-least-once.

Read about the reason for stopping in the docs http://doc.akka.io/docs/akka/2.4.0/java/persistence.html#Failures