Remote DeathWatch and comms failure

[Alistair George]

If I set up a watch on a remote actor (one on a remote actor system) and the network between me and the remote system fails, I get a Terminated message almost immediately. In fact, the remote actor hasn't terminated, and I can still use the ActorRef to send messages to it once comms are restored. (However, if comms fail a second time I don't get a second Terminated message.)

"Terminated" and "lost contact" are rather different states, and may need different handling. Does anyone know of a reliable way I can distinguish these? 

Thanks

Alistair

[Björn Antonsson]

Hi Alistair,

On Thursday, 16 January 2014 at 09:30, Alistair George wrote:

If I set up a watch on a remote actor (one on a remote actor system) and the network between me and the remote system fails, I get a Terminated message almost immediately. In fact, the remote actor hasn't terminated, and I can still use the ActorRef to send messages to it once comms are restored. (However, if comms fail a second time I don't get a second Terminated message.)

"Terminated" and "lost contact" are rather different states, and may need different handling. Does anyone know of a reliable way I can distinguish these? 

Which version of Akka are you using?

When you say that the network fails, what do you mean? How long is it “failed”?

What does the log say?

B/

 

Thanks

Alistair

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://akka.io/faq/
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/groups/opt_out.

-- 

Björn Antonsson

Typesafe – Reactive Apps on the JVM

twitter: @bantonsson

[Alistair George]

On Thursday, January 16, 2014 8:42:45 AM UTC, Björn Antonsson wrote:

Hi Alistair,

On Thursday, 16 January 2014 at 09:30, Alistair George wrote:

If I set up a watch on a remote actor (one on a remote actor system) and the network between me and the remote system fails, I get a Terminated message almost immediately. In fact, the remote actor hasn't terminated, and I can still use the ActorRef to send messages to it once comms are restored. (However, if comms fail a second time I don't get a second Terminated message.)

"Terminated" and "lost contact" are rather different states, and may need different handling. Does anyone know of a reliable way I can distinguish these? 

Which version of Akka are you using?

2.2.3

 

When you say that the network fails, what do you mean? How long is it “failed”?

I can simulate a failure is any number of ways - ifdown, unplug connections, power down the router. In each case I get the Terminated message within about 10 secs (I suspect this would depend on the heartbeat settings, but I haven't tried that experiment).

What does the log say?

Pretty much what you'd expect:

 [WARN] [01/17/2014 11:40:59.127] [RemoteSource-akka.actor.default-dispatcher-15] [akka://RemoteSource/system/remote-watcher] Detected unreachable: [akka.tcp://RemoteTarget@rho-w:2560]

[INFO] [01/17/2014 11:40:59.162] [RemoteSource-akka.actor.default-dispatcher-4] [Remoting] Address [akka.tcp://RemoteTarget@rho-w:2560] is now quarantined, all messages to this address will be delivered to dead letters.

.. app logging of Terminated message

[INFO] [01/17/2014 11:40:59.215] [RemoteSource-akka.actor.default-dispatcher-6] [akka://RemoteSource/user/source] Terminated(Terminated(Actor[akka.tcp://RemoteTarget@rho-w:2560/user/target#-319450745])) addressTerminated=true existenceConfirmed=false

[Akka Team]

Hi Alistair,

On Thu, Jan 16, 2014 at 9:30 AM, Alistair George <alistai...@gmail.com> wrote:

If I set up a watch on a remote actor (one on a remote actor system) and the network between me and the remote system fails, I get a Terminated message almost immediately. In fact, the remote actor hasn't terminated,

That does not matter. If you use remote DeathWatch, and one of the systems gets unreachable for enough time it will eventually fire Terminated for all the watched actors on the remote system and then quarantines that system so it never comes back again. The deathwatch failure detector (akka.remote.watch-failure-detector) settings controls how sensitive is this decision. If you think that a 1 hour unreachability should be not considered terminal, then you should configure those settings correspondingly.

 

and I can still use the ActorRef to send messages to it once comms are restored. (However, if comms fail a second time I don't get a second Terminated message.)

This is because we made the mistakes in 2.2.x:

 - we made quarantine times configurable

 - we set it to a low value, 60 seconds

After the quarantine elapses the systems can communicate again, regardless of the Terminated message, probably this is what you observed -- and this is exactly why quarantine in 2.3 is permanent.

 

"Terminated" and "lost contact" are rather different states, and may need different handling. Does anyone know of a reliable way I can distinguish these? 

DeathWatch sends Terminated in the case the remote system is in "lost contact" state for a long time. How long is that time is configurable by the DeathWatch failure detector. "lost contact" events are generated as remote lifecycle events, but I don't recommend using those directly. Message send supposed to be lossy, you can track reachability in your user layer by some heartbeating mechanism if you want it.

Btw, there is another failure detector (akka.remote.transport-failure-detector) that monitors the health of network connections, but it does not generate Terminated events, only reconnect attempts.

In 2.3 clustering will differentiate between UNREACHABLE events (which can heal) from removals. You probably want to use those features instead of plain remoting.

 

Thanks

Alistair

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://akka.io/faq/
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/groups/opt_out.

--

Akka Team

Typesafe - The software stack for applications that scale

Blog: letitcrash.com
Twitter: @akkateam

[Alistair George]

Hi Akka,

Thanks for the reply. One question: if (in 2.3) a remote actor system becomes permanently quarantined, what do I have to do to re-establish communication once the comms problem is fixed? Do I have to restart the remote actor system? Or the local one? Or both? 

Cheers

Alistair

[Akka Team]

Hi Alistair,

On Tue, Jan 21, 2014 at 8:38 AM, Alistair George <alistai...@gmail.com> wrote:

Hi Akka,

Thanks for the reply. One question: if (in 2.3) a remote actor system becomes permanently quarantined, what do I have to do to re-establish communication once the comms problem is fixed?

First of all, quarantining is a state where it is not considered just a communications problem but the remote system is declared dead (it is declared, not proven since all we know that it does not reply). Short communication failures do not trigger quarantine (what is considered short is configurable).

 

Do I have to restart the remote actor system? Or the local one? Or both?

From the remoting viewpoint it does not matter which one you restart. Obviously if one of the systems genuinely crashed then that is the one to be restarted, otherwise it is application specific.

-Endre

[√iktor Ҡlang]

"At some point you just have to move on..."

[Alistair George]

On Tuesday, January 21, 2014 8:44:17 AM UTC, Akka Team wrote:

Hi Alistair,

On Tue, Jan 21, 2014 at 8:38 AM, Alistair George <alistai...@gmail.com> wrote:

Hi Akka,

Thanks for the reply. One question: if (in 2.3) a remote actor system becomes permanently quarantined, what do I have to do to re-establish communication once the comms problem is fixed?

First of all, quarantining is a state where it is not considered just a communications problem but the remote system is declared dead (it is declared, not proven since all we know that it does not reply). Short communication failures do not trigger quarantine (what is considered short is configurable).

 

Do I have to restart the remote actor system? Or the local one? Or both?

From the remoting viewpoint it does not matter which one you restart. Obviously if one of the systems genuinely crashed then that is the one to be restarted, otherwise it is application specific.

I'm not sure this is desirable behaviour. I shouldn't have to restart a process just to recover from a comms failure. After all, nothing in the process has failed, and it may be providing services to other clients that have not suffered any comms failure. They shouldn't have to take the impact of a restart.

One of the strengths of Akka is that it doesn't pretend to do things that can't be done in a distributed context - this is essential for transparent distribution. One of this things you can't do distributed is give reliable, timely notification of a remote event, such as actor termination, and I don't think Akka should try.

What I'd prefer is this:

• Reconnect attempts should continue indefinitely.
• The DeathWatch protocol should be extended to include (possibly multiple) Reachable/Unreachable events.
• Terminate should only be delivered when the remote actor system is reachable and asserts that the watched actor does not exist. This might never happen: an actor might stay in an unreachable state forever.

I realise I can emulate this by setting the timeout before quarantine to be effectively infinite, and adding my own facility to detect reachability and termination, but this isn't trivial. I'd prefer this behaviour to be available out of the box, for both practical and conceptual reasons.

Just my $.02

Cheers

Alistair 

[√iktor Ҡlang]

We respectfully disagree then. Akka's design in this regard is highly deliberate.

[Akka Team]

Hi Alistair,

I'm not sure this is desirable behaviour. I shouldn't have to restart a process just to recover from a comms failure. After all, nothing in the process has failed, and it may be providing services to other clients that have not suffered any comms failure. They shouldn't have to take the impact of a restart.

No, restart is not needed to recover from a communications failure, it is needed to recover from the quarantined state.

 

One of the strengths of Akka is that it doesn't pretend to do things that can't be done in a distributed context - this is essential for transparent distribution. One of this things you can't do distributed is give reliable, timely notification of a remote event, such as actor termination, and I don't think Akka should try.

What I'd prefer is this:

• Reconnect attempts should continue indefinitely.

This is actually what happens. Quarantining only happens if you use remote DeathWatch. If this is not the feature you want, you should not use remote deathwatch. The contract of remote DeathWatch is that it eventually produces a Terminated message if the remote actor is dead, at the cost of false positives (i.e. actor is live but remote system was not responding for a long time). You can tune the false positive rate by setting the watch failure detector. You can even set it to trigger after one year.

• The DeathWatch protocol should be extended to include (possibly multiple) Reachable/Unreachable events.

This is not the purpose of DeathWatch, but you can implement such a thing on your own by sending heartbeats between the related actors. In clustering there is support for Unreachable/Reachable events.

• Terminate should only be delivered when the remote actor system is reachable and asserts that the watched actor does not exist. This might never happen: an actor might stay in an unreachable state forever.

While that is a useful feature, it is not the purpose of DeathWatch. Clustering has the feature you want, and in general, clustering is recommended over plain remoting.

 

I realise I can emulate this by setting the timeout before quarantine to be effectively infinite, and adding my own facility to detect reachability and termination, but this isn't trivial. I'd prefer this behaviour to be available out of the box, for both practical and conceptual reasons.

There is always the akka-contrib area for contributions :)

In general, I understand your use case, but DeathWatch does not support it. Many users would be surprised when they watch a remote actor, and kill the node running the system, but no Terminated messages would be generated. And they would argue that is wrong for practical and conceptual reasons :)

-Endre

[√iktor Ҡlang]

And it also introduces the need for unbounded buffering, i.e. memory leaks.

Cheers,

√

———————

Viktor Klang

Chief Architect - Typesafe

Twitter: @viktorklang

[Alistair George]

I don't see where the unbounded buffer is needed. I'd be grateful for a bit of explanation, especially since it looks like I'm going to have to implement this stuff :)

[√iktor Ҡlang]

NodeA & NodeB are communicating
NodeB disappears (not acking)
NodeA still has things it needs to propagate to NodeB (watching actors on that node etc), so they need to be buffered, also, if there are ordering requirements it means that other things may not be transmitted/received before other things, so more thing get buffered. At which point does NodeA either A) drop the buffer or B) throw OOME?

Cheers,

√

[Alistair George]

My plan for this was to have a proxy for NodeB watch state in NodeA. In normal (connected) operation it just remembers the current watch states (Actor Ax is/isn't watching Actor By) and passes the messages on to NodeB. If disconnected it just remembers the watch state. On reconnect, it sends a snapshot of the state to NodeB.

Because it's remembering state rather than messages it stays bounded. Need to make sure that "isn't watching" states get pruned, but that's just the usual sequence number/ack stuff (if disconnected, we can prune them immediately, because they don't form part of the snapshot).

Cheers

Alistair

[√iktor Ҡlang]

On Tue, Jan 21, 2014 at 7:50 PM, Alistair George <alistai...@gmail.com> wrote:

My plan for this was to have a proxy for NodeB watch state in NodeA. In normal (connected) operation it just remembers the current watch states (Actor Ax is/isn't watching Actor By) and passes the messages on to NodeB. If disconnected it just remembers the watch state. On reconnect, it sends a snapshot of the state to NodeB.

You still need to keep track of who watches (or wants to watch) who forever (in the case of a perpetually lost node)

 

Because it's remembering state rather than messages it stays bounded. Need to make sure that "isn't watching" states get pruned, but that's just the usual sequence number/ack stuff (if disconnected, we can prune them immediately, because they don't form part of the snapshot).

It's not unbounded: extrapolate over time as Nodes{N} join and then become silent, all watches need to be kept indefinitely = leak.

Cheers,

√

[Alistair George]

OK, so I think we're agreed that storage per remote node is bounded. The next question is whether the number of remote nodes is bounded.

• In theory, yes: there are only a finite number of IP:port combinations
• In practice no: it's a very big number
• In my use case yes: I'm dealing with a fixed set of nodes

I can see that the possibility of new nodes continuously connecting and falling silent does require quarantine.I'd be inclined to make this explicit in the DeathWatch protocol, giving message types Reachable, Unreachable, Terminated and Quarantined. However, I appreciate that you don't agree.

My takeaway from this is that, for my application, I need to set akka.remote.watch-failure-detector to have an effectively infinite timeout and use application heartbeats to detect reachability.

Cheers

Alistair

[√iktor Ҡlang]

Hi Alistair,

On Wed, Jan 22, 2014 at 9:31 AM, Alistair George <alistai...@gmail.com> wrote:

OK, so I think we're agreed that storage per remote node is bounded.

No, as number of actors is not bounded. (You can go OOM from creating enough actors locally)

 

The next question is whether the number of remote nodes is bounded.

• In theory, yes: there are only a finite number of IP:port combinations
• In practice no: it's a very big number
• In my use case yes: I'm dealing with a fixed set of nodes

You're forgetting the most important factor: time.

 

I can see that the possibility of new nodes continuously connecting and falling silent does require quarantine.I'd be inclined to make this explicit in the DeathWatch protocol, giving message types Reachable, Unreachable, Terminated and Quarantined. However, I appreciate that you don't agree.

What would be the value of providing these?

 

My takeaway from this is that, for my application, I need to set akka.remote.watch-failure-detector to have an effectively infinite timeout and use application heartbeats to detect reachability.

reachability is always a function of the past.

[Akka Team]

Hi Alistair,

On Wed, Jan 22, 2014 at 9:31 AM, Alistair George <alistai...@gmail.com> wrote:

OK, so I think we're agreed that storage per remote node is bounded. The next question is whether the number of remote nodes is bounded.

• In theory, yes: there are only a finite number of IP:port combinations
• In practice no: it's a very big number
• In my use case yes: I'm dealing with a fixed set of nodes

I can see that the possibility of new nodes continuously connecting and falling silent does require quarantine.I'd be inclined to make this explicit in the DeathWatch protocol, giving message types Reachable, Unreachable, Terminated and Quarantined. However, I appreciate that you don't agree.

My takeaway from this is that, for my application, I need to set akka.remote.watch-failure-detector to have an effectively infinite timeout and use application heartbeats to detect reachability.

Yes, a large watch-failure-detector acceptable-heartbeat-pause is most likely what you want. As for effective infinite, it depends. Do you consider a system healthy if it had not responded in 1 hour? 1 day? 1 week? When you find your psychological limit (well, most likely the limit that makes sense from the operations viewpoint), you should use that value :)

-Endre

[Alistair George]

On Wednesday, January 22, 2014 10:04:53 AM UTC, Akka Team wrote:

Hi Alistair,

On Wed, Jan 22, 2014 at 9:31 AM, Alistair George <alistai...@gmail.com> wrote:

OK, so I think we're agreed that storage per remote node is bounded. The next question is whether the number of remote nodes is bounded.

• In theory, yes: there are only a finite number of IP:port combinations
• In practice no: it's a very big number
• In my use case yes: I'm dealing with a fixed set of nodes

I can see that the possibility of new nodes continuously connecting and falling silent does require quarantine.I'd be inclined to make this explicit in the DeathWatch protocol, giving message types Reachable, Unreachable, Terminated and Quarantined. However, I appreciate that you don't agree.

My takeaway from this is that, for my application, I need to set akka.remote.watch-failure-detector to have an effectively infinite timeout and use application heartbeats to detect reachability.

Yes, a large watch-failure-detector acceptable-heartbeat-pause is most likely what you want. As for effective infinite, it depends. Do you consider a system healthy if it had not responded in 1 hour? 1 day? 1 week? When you find your psychological limit (well, most likely the limit that makes sense from the operations viewpoint), you should use that value :)

In my current use case it would be 1 week, since the system gets restarted at the weekend. If things aren't restarted fairly often people forget how the startup works, and can't fix it when it breaks :)

[Alistair George]

Hi Victor,

On Wednesday, January 22, 2014 9:13:26 AM UTC, √ wrote:

Hi Alistair,

On Wed, Jan 22, 2014 at 9:31 AM, Alistair George <alistai...@gmail.com> wrote:

OK, so I think we're agreed that storage per remote node is bounded.

No, as number of actors is not bounded. (You can go OOM from creating enough actors locally)

True, but I'm not sure how this is relevant to a discussion of the differential behaviour between connected and disconnected states.  

 

The next question is whether the number of remote nodes is bounded.

• In theory, yes: there are only a finite number of IP:port combinations
• In practice no: it's a very big number
• In my use case yes: I'm dealing with a fixed set of nodes

You're forgetting the most important factor: time.

 

There must be some misunderstanding here. When I talk of a node I mean an actor system that exposes a particular endpoint (IP:port, say). If the process that is listening on that endpoint is stopped and restarted, this does not (in my terminology) create a new node. The resources that refer to that endpoint do not need to build up indefinitely over restarts.

Of course, it could be that resources are building up indefinitely in normal operation, but again, I don't see the relevance of that question to the handling of disconnection.

 

I can see that the possibility of new nodes continuously connecting and falling silent does require quarantine.I'd be inclined to make this explicit in the DeathWatch protocol, giving message types Reachable, Unreachable, Terminated and Quarantined. However, I appreciate that you don't agree.

What would be the value of providing these?

 

Reachability notifications (such as are commonly derived from heartbeats) are used to provide notifications to operators and users of potential failures, and to initiate fallback behaviour (failover, say).

I think even you agree that Terminated has its uses.

As for Quarantined, I have two concerns about the proposed implementation of quarantining in 2.3:

• There is (if I understand correctly) no way of recovering from a quarantined state, short of a process restart.
• There's no programmatic notification of it, so no way to initiate recovery automatically.

The Quarantined message addresses the second of those. 

Cheers

Alistair

[√iktor Ҡlang]

Hi Alistair,

On Thu, Jan 23, 2014 at 8:12 AM, Alistair George <alistai...@gmail.com> wrote:

Hi Victor,

On Wednesday, January 22, 2014 9:13:26 AM UTC, √ wrote:

Hi Alistair,

On Wed, Jan 22, 2014 at 9:31 AM, Alistair George <alistai...@gmail.com> wrote:

OK, so I think we're agreed that storage per remote node is bounded.

No, as number of actors is not bounded. (You can go OOM from creating enough actors locally)

True, but I'm not sure how this is relevant to a discussion of the differential behaviour between connected and disconnected states.  

 

The next question is whether the number of remote nodes is bounded.

• In theory, yes: there are only a finite number of IP:port combinations
• In practice no: it's a very big number
• In my use case yes: I'm dealing with a fixed set of nodes

You're forgetting the most important factor: time.

 

There must be some misunderstanding here. When I talk of a node I mean an actor system that exposes a particular endpoint (IP:port, say). If the process that is listening on that endpoint is stopped and restarted, this does not (in my terminology) create a new node. The resources that refer to that endpoint do not need to build up indefinitely over restarts.

Then you are using other terminology than Akka. Every instantiation of an ActorSystem has it's own identity, just a any other object. Listening to a specific socket is unrelated.

 

Of course, it could be that resources are building up indefinitely in normal operation, but again, I don't see the relevance of that question to the handling of disconnection.

Do you agree that only running the JVM Garbage Collector every week instead of every minute would affect the stability of programs?

 

 

I can see that the possibility of new nodes continuously connecting and falling silent does require quarantine.I'd be inclined to make this explicit in the DeathWatch protocol, giving message types Reachable, Unreachable, Terminated and Quarantined. However, I appreciate that you don't agree.

What would be the value of providing these?

 

Reachability notifications (such as are commonly derived from heartbeats) are used to provide notifications to operators and users of potential failures, and to initiate fallback behaviour (failover, say).

Reachability is always a function of the past, so if you get an unreachable notification you don't know that the next message is a notification that it is Reachable again, and even with that information, you don't know how much time passed since it was Reachable and if it is Reachable when you attempt to send to it again.

 

I think even you agree that Terminated has its uses.

Yes, because it is final.

 

As for Quarantined, I have two concerns about the proposed implementation of quarantining in 2.3:

• There is (if I understand correctly) no way of recovering from a quarantined state, short of a process restart.

Once an ActorSystem has been declared dead by the others, it cannot come back. No zombie nodes.

 

• There's no programmatic notification of it, so no way to initiate recovery automatically.

You initiate recovery when you get the Terminated message.

 

The Quarantined message addresses the second of those. 

Terminated too.

Cheers,

√

[Akka Team]

Hi Alistair,

I think even you agree that Terminated has its uses.

As for Quarantined, I have two concerns about the proposed implementation of quarantining in 2.3:

• There is (if I understand correctly) no way of recovering from a quarantined state, short of a process restart.

This is exactly how it should work though, since once you signaled Terminated, then the other side should never be able to come back again even if it was a false positive -- otherwise the contract that Terminated is the "final breath" of an actor would be violated. It is a final decision, we declared the other system as dead (and should never come back) and made sure that it remains dead for our purposes. Restart is enough, since all new actors on the same path as the previous one will have a different UID.

The purpose of Quarantine is to be final, it is not an unreachability event but the final "get-rid-of-a-very-suspicious-remote-system-for-good" decision. Cluster has more fine grained support since it can mark nodes as unreachable as a first warning which is recoverable.

• There's no programmatic notification of it, so no way to initiate recovery automatically.

There is an akka.remote.QuarantinedEvent event published on the event stream when quarantining happens.

-Endre

[Alistair George]

Hi Victor,

This has been an interesting conversation, and I've learned some things about Akka that I needed to know. However, I don't think we're going to make further progress.

I've decided what to do about quarantine (avoid it) and reachability (use heartbeats), and these will meet my use case. I fear that more time spent on this risks giving the impression that I don't have enough work to do, so I won't be making any more posts on this topic. I'm sure I'll have plenty of other questions in the future.

Thanks again for your help.

Cheers

Alistair

[√iktor Ҡlang]

Happy hAkking!