how to properly deal with repeating "akka.remote.EndpointAssociationException: Association failed[...]"?

2,137 views
Skip to first unread message

Eniot Nacram

unread,
Feb 24, 2014, 10:50:49 AM2/24/14
to akka...@googlegroups.com
Hi all

I'm using akka 2.2.3 in a game prototype where a player can remotely connect to an existing world.
I'm having trouble properly managing the sudden deconnection of players.
The following happens:
1) The world (an actor) is started
2) A player (an actor) looks up the world remotely and they start exchanging messages
3) Everything is fine
4) The player actor dies
5) Some actors on the world side keep trying to send messages to the player side, hence some errors happen on the world side.
6) The deathwatch on the world side realizes the player is dead and the world discards the dead player (ie: no more messages sent to him or expected from him)
7) A continous flow of EndpointAssociationException happens in the world logs.

I would very much like to know:
1) how / where to handle the errors happening at step 5
2) correctly handle and stop the continous flow of errors happening at step 7
Thanks in advance for any tips or pointers.

Here's a log of this case (lines starting and ending with "---" are my application's logs edited for clarity)

[INFO] [02/24/2014 16:13:21.414] [Main-akka.actor.default-dispatcher-3] [Remoting] Starting remoting
[INFO] [02/24/2014 16:13:21.554] [Main-akka.actor.default-dispatcher-3] [Remoting] Remoting started; listening on addresses :[akka.tcp://plesio...@127.0.0.1:25852]

--- here the player actor joins the world ---
--- here everything is fine ---
--- here the player actor is killed ---

[INFO] [02/24/2014 16:13:34.924] [plesio-system-akka.actor.default-dispatcher-3] [akka://plesio-system/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fplesio-system%40127.0.0.1%3A56227-1] Message [akka.remote.transport.AssociationHandle$Disassociated] from Actor[akka://plesio-system/deadLetters] to Actor[akka://plesio-system/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fplesio-system%40127.0.0.1%3A56227-1#841387870] was not delivered. [1] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[INFO] [02/24/2014 16:13:34.933] [plesio-system-akka.actor.default-dispatcher-5] [akka://plesio-system/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fplesio-system%40127.0.0.1%3A56227-1] Message [akka.remote.transport.ActorTransportAdapter$DisassociateUnderlying] from Actor[akka://plesio-system/deadLetters] to Actor[akka://plesio-system/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fplesio-system%40127.0.0.1%3A56227-1#841387870] was not delivered. [2] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[ERROR] [02/24/2014 16:13:35.952] [plesio-system-akka.actor.default-dispatcher-20] [akka://plesio-system/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fthinclient%40127.0.0.1%3A25853-0/endpointWriter] AssociationError [akka.tcp://plesio...@127.0.0.1:25852] -> [akka.tcp://thinc...@127.0.0.1:25853]: Error [Association failed with [akka.tcp://thinc...@127.0.0.1:25853]] [
akka.remote.EndpointAssociationException: Association failed with [akka.tcp://thinc...@127.0.0.1:25853]
Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: no further information: /127.0.0.1:25853
]
[ERROR] [02/24/2014 16:13:36.958] [plesio-system-akka.actor.default-dispatcher-18] [akka://plesio-system/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fthinclient%40127.0.0.1%3A25853-0/endpointWriter] AssociationError [akka.tcp://plesio...@127.0.0.1:25852] -> [akka.tcp://thinc...@127.0.0.1:25853]: Error [Association failed with [akka.tcp://thinc...@127.0.0.1:25853]] [
akka.remote.EndpointAssociationException: Association failed with [akka.tcp://thinc...@127.0.0.1:25853]
Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: no further information: /127.0.0.1:25853
]
[ERROR] [02/24/2014 16:13:37.965] [plesio-system-akka.actor.default-dispatcher-20] [akka://plesio-system/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fthinclient%40127.0.0.1%3A25853-0/endpointWriter] AssociationError [akka.tcp://plesio...@127.0.0.1:25852] -> [akka.tcp://thinc...@127.0.0.1:25853]: Error [Association failed with [akka.tcp://thinc...@127.0.0.1:25853]] [
akka.remote.EndpointAssociationException: Association failed with [akka.tcp://thinc...@127.0.0.1:25853]
Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: no further information: /127.0.0.1:25853
]
[INFO] [02/24/2014 16:13:37.970] [plesio-system-akka.actor.default-dispatcher-20] [akka://plesio-system/deadLetters] Message [engine.Ticker$Tick] from Actor[akka://plesio-system/user/director/ticker#-1791557503] to Actor[akka://plesio-system/deadLetters] was not delivered. [3] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[INFO] [02/24/2014 16:13:37.970] [plesio-system-akka.actor.default-dispatcher-20] [akka://plesio-system/deadLetters] Message [engine.World$SunAngle] from Actor[akka://plesio-system/user/director/world#815835863] to Actor[akka://plesio-system/deadLetters] was not delivered. [4] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[INFO] [02/24/2014 16:13:37.970] [plesio-system-akka.actor.default-dispatcher-20] [akka://plesio-system/deadLetters] Message [engine.entity.Player$LookingAt] from Actor[akka://plesio-system/user/director/world#815835863] to Actor[akka://plesio-system/deadLetters] was not delivered. [5] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[INFO] [02/24/2014 16:13:37.970] [plesio-system-akka.actor.default-dispatcher-20] [akka://plesio-system/deadLetters] Message [engine.Ticker$Tick] from Actor[akka://plesio-system/user/director/ticker#-1791557503] to Actor[akka://plesio-system/deadLetters] was not delivered. [6] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[INFO] [02/24/2014 16:13:37.970] [plesio-system-akka.actor.default-dispatcher-20] [akka://plesio-system/deadLetters] Message [engine.World$SunAngle] from Actor[akka://plesio-system/user/director/world#815835863] to Actor[akka://plesio-system/deadLetters] was not delivered. [7] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[INFO] [02/24/2014 16:13:37.971] [plesio-system-akka.actor.default-dispatcher-20] [akka://plesio-system/deadLetters] Message [engine.entity.Player$LookingAt] from Actor[akka://plesio-system/user/director/world#815835863] to Actor[akka://plesio-system/deadLetters] was not delivered. [8] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[INFO] [02/24/2014 16:13:37.971] [plesio-system-akka.actor.default-dispatcher-20] [akka://plesio-system/deadLetters] Message [engine.Ticker$Tick] from Actor[akka://plesio-system/user/director/ticker#-1791557503] to Actor[akka://plesio-system/deadLetters] was not delivered. [9] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[INFO] [02/24/2014 16:13:37.971] [plesio-system-akka.actor.default-dispatcher-20] [akka://plesio-system/deadLetters] Message [engine.entity.Player$LookingAt] from Actor[akka://plesio-system/user/director/world#815835863] to Actor[akka://plesio-system/deadLetters] was not delivered. [10] dead letters encountered, no more dead letters will be logged. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[ERROR] [02/24/2014 16:13:39.015] [plesio-system-akka.actor.default-dispatcher-19] [akka://plesio-system/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fthinclient%40127.0.0.1%3A25853-0/endpointWriter] AssociationError [akka.tcp://plesio...@127.0.0.1:25852] -> [akka.tcp://thinc...@127.0.0.1:25853]: Error [Association failed with [akka.tcp://thinc...@127.0.0.1:25853]] [
akka.remote.EndpointAssociationException: Association failed with [akka.tcp://thinc...@127.0.0.1:25853]
Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: no further information: /127.0.0.1:25853
]
[ERROR] [02/24/2014 16:13:40.026] [plesio-system-akka.actor.default-dispatcher-2] [akka://plesio-system/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fthinclient%40127.0.0.1%3A25853-0/endpointWriter] AssociationError [akka.tcp://plesio...@127.0.0.1:25852] -> [akka.tcp://thinc...@127.0.0.1:25853]: Error [Association failed with [akka.tcp://thinc...@127.0.0.1:25853]] [
akka.remote.EndpointAssociationException: Association failed with [akka.tcp://thinc...@127.0.0.1:25853]
Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: no further information: /127.0.0.1:25853
]
[WARN] [02/24/2014 16:13:40.577] [plesio-system-akka.actor.default-dispatcher-18] [akka://plesio-system/system/remote-watcher] Detected unreachable: [akka.tcp://thinc...@127.0.0.1:25853]
[INFO] [02/24/2014 16:13:40.578] [plesio-system-akka.actor.default-dispatcher-3] [Remoting] Address [akka.tcp://thinc...@127.0.0.1:25853] is now quarantined, all messages to this address will be delivered to dead letters.

--- here the DeathWatch realized the remote actor (player) is dead and is removed from the world ---

[ERROR] [02/24/2014 16:14:41.617] [plesio-system-akka.actor.default-dispatcher-15] [akka://plesio-system/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fthinclient%40127.0.0.1%3A25853-1/endpointWriter] AssociationError [akka.tcp://plesio...@127.0.0.1:25852] -> [akka.tcp://thinc...@127.0.0.1:25853]: Error [Association failed with [akka.tcp://thinc...@127.0.0.1:25853]] [
akka.remote.EndpointAssociationException: Association failed with [akka.tcp://thinc...@127.0.0.1:25853]
Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: no further information: /127.0.0.1:25853
]
[ERROR] [02/24/2014 16:14:42.621] [plesio-system-akka.actor.default-dispatcher-14] [akka://plesio-system/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fthinclient%40127.0.0.1%3A25853-1/endpointWriter] AssociationError [akka.tcp://plesio...@127.0.0.1:25852] -> [akka.tcp://thinc...@127.0.0.1:25853]: Error [Association failed with [akka.tcp://thinc...@127.0.0.1:25853]] [
akka.remote.EndpointAssociationException: Association failed with [akka.tcp://thinc...@127.0.0.1:25853]
Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: no further information: /127.0.0.1:25853
]

--- errors like the last 2 keep flowing endlessly, with a different dispatcher id ---

Eniot Nacram

unread,
Feb 24, 2014, 11:05:24 AM2/24/14
to akka...@googlegroups.com
please also note the errors at step 7 dont start immediately, but after a minute or so (as seen in the logs)

- markus

Le lundi 24 février 2014 16:50:49 UTC+1, Eniot Nacram a écrit :
Hi all

I'm using akka 2.2.3 in a game prototype where a player can remotely connect to an existing world.
I'm having trouble properly managing the sudden deconnection of players.
The following happens:
1) The world (an actor) is started
2) A player (an actor) looks up the world remotely and they start exchanging messages
3) Everything is fine
4) The player actor dies
5) Some actors on the world side keep trying to send messages to the player side, hence some errors happen on the world side.
6) The deathwatch on the world side realizes the player is dead and the world discards the dead player (ie: no more messages sent to him or expected from him)
7) A continous flow of EndpointAssociationException happens in the world logs.

I would very much like to know:
1) how / where to handle the errors happening at step 5
2) correctly handle and stop the continous flow of errors happening at step 7
Thanks in advance for any tips or pointers.

Here's a log of this case (lines starting and ending with "---" are my application's logs edited for clarity)

[INFO] [02/24/2014 16:13:21.414] [Main-akka.actor.default-dispatcher-3] [Remoting] Starting remoting
[INFO] [02/24/2014 16:13:21.554] [Main-akka.actor.default-dispatcher-3] [Remoting] Remoting started; listening on addresses :[akka.tcp://plesio-system@127.0.0.1:25852]

--- here the player actor joins the world ---
--- here everything is fine ---
--- here the player actor is killed ---

[INFO] [02/24/2014 16:13:34.924] [plesio-system-akka.actor.default-dispatcher-3] [akka://plesio-system/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fplesio-system%40127.0.0.1%3A56227-1] Message [akka.remote.transport.AssociationHandle$Disassociated] from Actor[akka://plesio-system/deadLetters] to Actor[akka://plesio-system/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fplesio-system%40127.0.0.1%3A56227-1#841387870] was not delivered. [1] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[INFO] [02/24/2014 16:13:34.933] [plesio-system-akka.actor.default-dispatcher-5] [akka://plesio-system/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fplesio-system%40127.0.0.1%3A56227-1] Message [akka.remote.transport.ActorTransportAdapter$DisassociateUnderlying] from Actor[akka://plesio-system/deadLetters] to Actor[akka://plesio-system/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fplesio-system%40127.0.0.1%3A56227-1#841387870] was not delivered. [2] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[ERROR] [02/24/2014 16:13:35.952] [plesio-system-akka.actor.default-dispatcher-20] [akka://plesio-system/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fthinclient%40127.0.0.1%3A25853-0/endpointWriter] AssociationError [akka.tcp://plesio-system@127.0.0.1:25852] -> [akka.tcp://thinclient@127.0.0.1:25853]: Error [Association failed with [akka.tcp://thinclient@127.0.0.1:25853]] [
akka.remote.EndpointAssociationException: Association failed with [akka.tcp://thinclient@127.0.0.1:25853]
Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: no further information: /127.0.0.1:25853
]
[ERROR] [02/24/2014 16:13:36.958] [plesio-system-akka.actor.default-dispatcher-18] [akka://plesio-system/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fthinclient%40127.0.0.1%3A25853-0/endpointWriter] AssociationError [akka.tcp://plesio-system@127.0.0.1:25852] -> [akka.tcp://thinclient@127.0.0.1:25853]: Error [Association failed with [akka.tcp://thinclient@127.0.0.1:25853]] [
akka.remote.EndpointAssociationException: Association failed with [akka.tcp://thinclient@127.0.0.1:25853]
Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: no further information: /127.0.0.1:25853
]
[ERROR] [02/24/2014 16:13:37.965] [plesio-system-akka.actor.default-dispatcher-20] [akka://plesio-system/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fthinclient%40127.0.0.1%3A25853-0/endpointWriter] AssociationError [akka.tcp://plesio-system@127.0.0.1:25852] -> [akka.tcp://thinclient@127.0.0.1:25853]: Error [Association failed with [akka.tcp://thinclient@127.0.0.1:25853]] [
akka.remote.EndpointAssociationException: Association failed with [akka.tcp://thinclient@127.0.0.1:25853]
Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: no further information: /127.0.0.1:25853
]
[INFO] [02/24/2014 16:13:37.970] [plesio-system-akka.actor.default-dispatcher-20] [akka://plesio-system/deadLetters] Message [engine.Ticker$Tick] from Actor[akka://plesio-system/user/director/ticker#-1791557503] to Actor[akka://plesio-system/deadLetters] was not delivered. [3] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[INFO] [02/24/2014 16:13:37.970] [plesio-system-akka.actor.default-dispatcher-20] [akka://plesio-system/deadLetters] Message [engine.World$SunAngle] from Actor[akka://plesio-system/user/director/world#815835863] to Actor[akka://plesio-system/deadLetters] was not delivered. [4] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[INFO] [02/24/2014 16:13:37.970] [plesio-system-akka.actor.default-dispatcher-20] [akka://plesio-system/deadLetters] Message [engine.entity.Player$LookingAt] from Actor[akka://plesio-system/user/director/world#815835863] to Actor[akka://plesio-system/deadLetters] was not delivered. [5] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[INFO] [02/24/2014 16:13:37.970] [plesio-system-akka.actor.default-dispatcher-20] [akka://plesio-system/deadLetters] Message [engine.Ticker$Tick] from Actor[akka://plesio-system/user/director/ticker#-1791557503] to Actor[akka://plesio-system/deadLetters] was not delivered. [6] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[INFO] [02/24/2014 16:13:37.970] [plesio-system-akka.actor.default-dispatcher-20] [akka://plesio-system/deadLetters] Message [engine.World$SunAngle] from Actor[akka://plesio-system/user/director/world#815835863] to Actor[akka://plesio-system/deadLetters] was not delivered. [7] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[INFO] [02/24/2014 16:13:37.971] [plesio-system-akka.actor.default-dispatcher-20] [akka://plesio-system/deadLetters] Message [engine.entity.Player$LookingAt] from Actor[akka://plesio-system/user/director/world#815835863] to Actor[akka://plesio-system/deadLetters] was not delivered. [8] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[INFO] [02/24/2014 16:13:37.971] [plesio-system-akka.actor.default-dispatcher-20] [akka://plesio-system/deadLetters] Message [engine.Ticker$Tick] from Actor[akka://plesio-system/user/director/ticker#-1791557503] to Actor[akka://plesio-system/deadLetters] was not delivered. [9] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[INFO] [02/24/2014 16:13:37.971] [plesio-system-akka.actor.default-dispatcher-20] [akka://plesio-system/deadLetters] Message [engine.entity.Player$LookingAt] from Actor[akka://plesio-system/user/director/world#815835863] to Actor[akka://plesio-system/deadLetters] was not delivered. [10] dead letters encountered, no more dead letters will be logged. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[ERROR] [02/24/2014 16:13:39.015] [plesio-system-akka.actor.default-dispatcher-19] [akka://plesio-system/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fthinclient%40127.0.0.1%3A25853-0/endpointWriter] AssociationError [akka.tcp://plesio-system@127.0.0.1:25852] -> [akka.tcp://thinclient@127.0.0.1:25853]: Error [Association failed with [akka.tcp://thinclient@127.0.0.1:25853]] [
akka.remote.EndpointAssociationException: Association failed with [akka.tcp://thinclient@127.0.0.1:25853]
Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: no further information: /127.0.0.1:25853
]
[ERROR] [02/24/2014 16:13:40.026] [plesio-system-akka.actor.default-dispatcher-2] [akka://plesio-system/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fthinclient%40127.0.0.1%3A25853-0/endpointWriter] AssociationError [akka.tcp://plesio-system@127.0.0.1:25852] -> [akka.tcp://thinclient@127.0.0.1:25853]: Error [Association failed with [akka.tcp://thinclient@127.0.0.1:25853]] [
akka.remote.EndpointAssociationException: Association failed with [akka.tcp://thinclient@127.0.0.1:25853]
Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: no further information: /127.0.0.1:25853
]
[WARN] [02/24/2014 16:13:40.577] [plesio-system-akka.actor.default-dispatcher-18] [akka://plesio-system/system/remote-watcher] Detected unreachable: [akka.tcp://thinclient@127.0.0.1:25853]
[INFO] [02/24/2014 16:13:40.578] [plesio-system-akka.actor.default-dispatcher-3] [Remoting] Address [akka.tcp://thinclient@127.0.0.1:25853] is now quarantined, all messages to this address will be delivered to dead letters.

--- here the DeathWatch realized the remote actor (player) is dead and is removed from the world ---

[ERROR] [02/24/2014 16:14:41.617] [plesio-system-akka.actor.default-dispatcher-15] [akka://plesio-system/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fthinclient%40127.0.0.1%3A25853-1/endpointWriter] AssociationError [akka.tcp://plesio-system@127.0.0.1:25852] -> [akka.tcp://thinclient@127.0.0.1:25853]: Error [Association failed with [akka.tcp://thinclient@127.0.0.1:25853]] [
akka.remote.EndpointAssociationException: Association failed with [akka.tcp://thinclient@127.0.0.1:25853]
Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: no further information: /127.0.0.1:25853
]
[ERROR] [02/24/2014 16:14:42.621] [plesio-system-akka.actor.default-dispatcher-14] [akka://plesio-system/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fthinclient%40127.0.0.1%3A25853-1/endpointWriter] AssociationError [akka.tcp://plesio-system@127.0.0.1:25852] -> [akka.tcp://thinclient@127.0.0.1:25853]: Error [Association failed with [akka.tcp://thinclient@127.0.0.1:25853]] [
akka.remote.EndpointAssociationException: Association failed with [akka.tcp://thinclient@127.0.0.1:25853]
Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: no further information: /127.0.0.1:25853
]

Akka Team

unread,
Feb 24, 2014, 11:13:11 AM2/24/14
to Akka User List
Hi Eniot,


On Mon, Feb 24, 2014 at 4:50 PM, Eniot Nacram <markus....@gmail.com> wrote:
Hi all

I'm using akka 2.2.3 in a game prototype where a player can remotely connect to an existing world.
I'm having trouble properly managing the sudden deconnection of players.
The following happens:
1) The world (an actor) is started
2) A player (an actor) looks up the world remotely and they start exchanging messages
3) Everything is fine
4) The player actor dies
5) Some actors on the world side keep trying to send messages to the player side, hence some errors happen on the world side.
6) The deathwatch on the world side realizes the player is dead and the world discards the dead player (ie: no more messages sent to him or expected from him)
7) A continous flow of EndpointAssociationException happens in the world logs.

Please see the section here to understand the limits of remoting when used in client-server fashion: http://doc.akka.io/docs/akka/2.3.0-RC4/general/remoting.html#Peer-to-Peer_vs__Client-Server

Unfortunately the above is the 2.3-RC4 docs, but the 2.2.4 docs will include this section as well.
 

I would very much like to know:
1) how / where to handle the errors happening at step 5

What kind of errors you mean?
 
2) correctly handle and stop the continous flow of errors happening at step 7

Use the setting:
akka.remote.quarantine-systems-for = 1 d
(or even higher value)

You might also want to use
akka.remote.retry-gate-closed-for = 5 s

Although it is in the 2.3-RC4 docs where remoting settings changed slightly, you might want to familiarize yourself with this: http://doc.akka.io/docs/akka/2.3.0-RC4/java/remoting.html#Lifecycle_and_Failure_Recovery_Model

In general I recommend using 2.3-RC4 since the final version is quite close to being released (hopefully this is the final RC).

I hope this help
-Endre
 
Thanks in advance for any tips or pointers.

Here's a log of this case (lines starting and ending with "---" are my application's logs edited for clarity)

[INFO] [02/24/2014 16:13:21.414] [Main-akka.actor.default-dispatcher-3] [Remoting] Starting remoting
[INFO] [02/24/2014 16:13:21.554] [Main-akka.actor.default-dispatcher-3] [Remoting] Remoting started; listening on addresses :[akka.tcp://plesio...@127.0.0.1:25852]

--- here the player actor joins the world ---
--- here everything is fine ---
--- here the player actor is killed ---

[INFO] [02/24/2014 16:13:34.924] [plesio-system-akka.actor.default-dispatcher-3] [akka://plesio-system/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fplesio-system%40127.0.0.1%3A56227-1] Message [akka.remote.transport.AssociationHandle$Disassociated] from Actor[akka://plesio-system/deadLetters] to Actor[akka://plesio-system/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fplesio-system%40127.0.0.1%3A56227-1#841387870841387870841387870841387870] was not delivered. [1] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[INFO] [02/24/2014 16:13:34.933] [plesio-system-akka.actor.default-dispatcher-5] [akka://plesio-system/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fplesio-system%40127.0.0.1%3A56227-1] Message [akka.remote.transport.ActorTransportAdapter$DisassociateUnderlying] from Actor[akka://plesio-system/deadLetters] to Actor[akka://plesio-system/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fplesio-system%40127.0.0.1%3A56227-1#841387870841387870841387870841387870] was not delivered. [2] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://akka.io/faq/
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/groups/opt_out.



--
Akka Team
Typesafe - The software stack for applications that scale
Blog: letitcrash.com
Twitter: @akkateam
You'll need Skype CreditFree via Skype
You'll need Skype CreditFree via Skype

√iktor Ҡlang

unread,
Feb 24, 2014, 11:36:37 AM2/24/14
to Akka User List
Is there a problem?


--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://akka.io/faq/
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/groups/opt_out.



--
Cheers,
√

———————
Viktor Klang
Chief Architect - Typesafe

Twitter: @viktorklang

Eniot Nacram

unread,
Feb 25, 2014, 9:10:35 AM2/25/14
to akka...@googlegroups.com
Hi again,

Thanks for your answer. I switched to the 2.3-RC like you recommended.
It seems the remote system is now gated instead of quarantined when it's disrupted.

I also tried the configurations you gave me but "akka.remote.quarantine-systems-for = 1 d" but as failed remote are gated now, I dont think I changes anything
"retry-gate-closed-for = 5 s" changes indeed the retry delay for gated systems but I dont know how to make good use of it.

In short, I still have the following problems:
Case 1: a remote actor is shut down and is not restarted.
In this case, my actors keep expecting forever for the failed remote actors to come back (maybe once a day or once a year, it's still waiting forever).
I'd rather them not wait for a potential come back of the failed remotes, but instead just forget they existed.
Maybe my approach is wrong here, then please correct it, but it feels weird to expect forever a sign of life from a disposable actor.

Case 2: a remote actor is shut down and restarted.
Some actor A looked up actor B (successfully), and they were working together for a time, but A was shut down and restarted.
A looks up B again as it restarts. The problem is the following :
Messages from A to B are sent and processed.
Messages from B to A are not sent (dropped maybe?)
Of course I'd like B to be able to send again messages to A, even after a restart.

Thanks again for any further help,

Hi Eniot,


[INFO] [02/24/2014 16:13:21.554] [Main-akka.actor.default-dispatcher-3] [Remoting] Remoting started; listening on addresses :[akka.tcp://plesio-system@127.0.0.1:25852]

--- here the player actor joins the world ---
--- here everything is fine ---
--- here the player actor is killed ---

[INFO] [02/24/2014 16:13:34.924] [plesio-system-akka.actor.default-dispatcher-3] [akka://plesio-system/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fplesio-system%40127.0.0.1%3A56227-1] Message [akka.remote.transport.AssociationHandle$Disassociated] from Actor[akka://plesio-system/deadLetters] to Actor[akka://plesio-system/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fplesio-system%40127.0.0.1%3A56227-1#841387870841387870841387870841387870] was not delivered. [1] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[INFO] [02/24/2014 16:13:34.933] [plesio-system-akka.actor.default-dispatcher-5] [akka://plesio-system/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fplesio-system%40127.0.0.1%3A56227-1] Message [akka.remote.transport.ActorTransportAdapter$DisassociateUnderlying] from Actor[akka://plesio-system/deadLetters] to Actor[akka://plesio-system/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fplesio-system%40127.0.0.1%3A56227-1#841387870841387870841387870841387870] was not delivered. [2] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[ERROR] [02/24/2014 16:13:35.952] [plesio-system-akka.actor.default-dispatcher-20] [akka://plesio-system/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fthinclient%40127.0.0.1%3A25853-0/endpointWriter] AssociationError [akka.tcp://plesio-system@127.0.0.1:25852] -> [akka.tcp://thinclient@127.0.0.1:25853]: Error [Association failed with [akka.tcp://thinclient@127.0.0.1:25853]] [
akka.remote.EndpointAssociationException: Association failed with [akka.tcp://thinclient@127.0.0.1:25853]
Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: no further information: /127.0.0.1:25853
]
[ERROR] [02/24/2014 16:13:36.958] [plesio-system-akka.actor.default-dispatcher-18] [akka://plesio-system/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fthinclient%40127.0.0.1%3A25853-0/endpointWriter] AssociationError [akka.tcp://plesio-system@127.0.0.1:25852] -> [akka.tcp://thinclient@127.0.0.1:25853]: Error [Association failed with [akka.tcp://thinclient@127.0.0.1:25853]] [
akka.remote.EndpointAssociationException: Association failed with [akka.tcp://thinclient@127.0.0.1:25853]
Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: no further information: /127.0.0.1:25853
]
[ERROR] [02/24/2014 16:13:37.965] [plesio-system-akka.actor.default-dispatcher-20] [akka://plesio-system/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fthinclient%40127.0.0.1%3A25853-0/endpointWriter] AssociationError [akka.tcp://plesio-system@127.0.0.1:25852] -> [akka.tcp://thinclient@127.0.0.1:25853]: Error [Association failed with [akka.tcp://thinclient@127.0.0.1:25853]] [
akka.remote.EndpointAssociationException: Association failed with [akka.tcp://thinclient@127.0.0.1:25853]
Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: no further information: /127.0.0.1:25853
]
[INFO] [02/24/2014 16:13:37.970] [plesio-system-akka.actor.default-dispatcher-20] [akka://plesio-system/deadLetters] Message [engine.Ticker$Tick] from Actor[akka://plesio-system/user/director/ticker#-1791557503] to Actor[akka://plesio-system/deadLetters] was not delivered. [3] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[INFO] [02/24/2014 16:13:37.970] [plesio-system-akka.actor.default-dispatcher-20] [akka://plesio-system/deadLetters] Message [engine.World$SunAngle] from Actor[akka://plesio-system/user/director/world#815835863] to Actor[akka://plesio-system/deadLetters] was not delivered. [4] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[INFO] [02/24/2014 16:13:37.970] [plesio-system-akka.actor.default-dispatcher-20] [akka://plesio-system/deadLetters] Message [engine.entity.Player$LookingAt] from Actor[akka://plesio-system/user/director/world#815835863] to Actor[akka://plesio-system/deadLetters] was not delivered. [5] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[INFO] [02/24/2014 16:13:37.970] [plesio-system-akka.actor.default-dispatcher-20] [akka://plesio-system/deadLetters] Message [engine.Ticker$Tick] from Actor[akka://plesio-system/user/director/ticker#-1791557503] to Actor[akka://plesio-system/deadLetters] was not delivered. [6] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[INFO] [02/24/2014 16:13:37.970] [plesio-system-akka.actor.default-dispatcher-20] [akka://plesio-system/deadLetters] Message [engine.World$SunAngle] from Actor[akka://plesio-system/user/director/world#815835863] to Actor[akka://plesio-system/deadLetters] was not delivered. [7] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[INFO] [02/24/2014 16:13:37.971] [plesio-system-akka.actor.default-dispatcher-20] [akka://plesio-system/deadLetters] Message [engine.entity.Player$LookingAt] from Actor[akka://plesio-system/user/director/world#815835863] to Actor[akka://plesio-system/deadLetters] was not delivered. [8] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[INFO] [02/24/2014 16:13:37.971] [plesio-system-akka.actor.default-dispatcher-20] [akka://plesio-system/deadLetters] Message [engine.Ticker$Tick] from Actor[akka://plesio-system/user/director/ticker#-1791557503] to Actor[akka://plesio-system/deadLetters] was not delivered. [9] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[INFO] [02/24/2014 16:13:37.971] [plesio-system-akka.actor.default-dispatcher-20] [akka://plesio-system/deadLetters] Message [engine.entity.Player$LookingAt] from Actor[akka://plesio-system/user/director/world#815835863] to Actor[akka://plesio-system/deadLetters] was not delivered. [10] dead letters encountered, no more dead letters will be logged. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[ERROR] [02/24/2014 16:13:39.015] [plesio-system-akka.actor.default-dispatcher-19] [akka://plesio-system/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fthinclient%40127.0.0.1%3A25853-0/endpointWriter] AssociationError [akka.tcp://plesio-system@127.0.0.1:25852] -> [akka.tcp://thinclient@127.0.0.1:25853]: Error [Association failed with [akka.tcp://thinclient@127.0.0.1:25853]] [
akka.remote.EndpointAssociationException: Association failed with [akka.tcp://thinclient@127.0.0.1:25853]
Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: no further information: /127.0.0.1:25853
]
[ERROR] [02/24/2014 16:13:40.026] [plesio-system-akka.actor.default-dispatcher-2] [akka://plesio-system/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fthinclient%40127.0.0.1%3A25853-0/endpointWriter] AssociationError [akka.tcp://plesio-system@127.0.0.1:25852] -> [akka.tcp://thinclient@127.0.0.1:25853]: Error [Association failed with [akka.tcp://thinclient@127.0.0.1:25853]] [
akka.remote.EndpointAssociationException: Association failed with [akka.tcp://thinclient@127.0.0.1:25853]
Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: no further information: /127.0.0.1:25853
]
[WARN] [02/24/2014 16:13:40.577] [plesio-system-akka.actor.default-dispatcher-18] [akka://plesio-system/system/remote-watcher] Detected unreachable: [akka.tcp://thinclient@127.0.0.1:25853]
[INFO] [02/24/2014 16:13:40.578] [plesio-system-akka.actor.default-dispatcher-3] [Remoting] Address [akka.tcp://thinclient@127.0.0.1:25853] is now quarantined, all messages to this address will be delivered to dead letters.

--- here the DeathWatch realized the remote actor (player) is dead and is removed from the world ---

[ERROR] [02/24/2014 16:14:41.617] [plesio-system-akka.actor.default-dispatcher-15] [akka://plesio-system/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fthinclient%40127.0.0.1%3A25853-1/endpointWriter] AssociationError [akka.tcp://plesio-system@127.0.0.1:25852] -> [akka.tcp://thinclient@127.0.0.1:25853]: Error [Association failed with [akka.tcp://thinclient@127.0.0.1:25853]] [
akka.remote.EndpointAssociationException: Association failed with [akka.tcp://thinclient@127.0.0.1:25853]
Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: no further information: /127.0.0.1:25853
]
[ERROR] [02/24/2014 16:14:42.621] [plesio-system-akka.actor.default-dispatcher-14] [akka://plesio-system/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fthinclient%40127.0.0.1%3A25853-1/endpointWriter] AssociationError [akka.tcp://plesio-system@127.0.0.1:25852] -> [akka.tcp://thinclient@127.0.0.1:25853]: Error [Association failed with [akka.tcp://thinclient@127.0.0.1:25853]] [
akka.remote.EndpointAssociationException: Association failed with [akka.tcp://thinclient@127.0.0.1:25853]

Eniot Nacram

unread,
Feb 25, 2014, 12:06:30 PM2/25/14
to akka...@googlegroups.com
Ok, problems solved.
I used a simplified version of my system to understand better what was happening, and the only problem was that one of the remote actors was not properly watched.
Thanks for the pointers, and thanks for akka too!

Akka Team

unread,
Feb 27, 2014, 4:36:48 AM2/27/14
to Akka User List
Hi Eniot,

Nice to see that your problem is solved! One thing to remember with gating that an actor system will only try to reconnect after a gate event if there is actually some actor trying to send to that remote system after the gate period elapsed.

-Endre


--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://akka.io/faq/
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/groups/opt_out.
Reply all
Reply to author
Forward
0 new messages