Akka 2.2.3: Remote Systems Losing Contact Despite Both Systems Being Up
743 views
Skip to first unread message
Brian Fowler
Jan 24, 2014, 9:58:16 AM1/24/14
to akka...@googlegroups.com
Hi,
I have two actor systems on different boxes under 2.2.3 ActorSystem1 and ActorSystem2. I spin up ActorSystem1 and then ActorSystem2. ActorSystem1 "asks" an actorSelection on ActorSystem2 and ActorSystem2 replies everything is fine. (I can do this lots of times without issue)
Then after 5 minutes or so of not sending messages from ActorSystem1 to ActorSystem2, ActorSystem1 and ActorSystem2 can't communicate anymore despite both systems are up and there being are no network issues. Under Akka 2.1.4 I don't get this issue. Restarting the ActorSystem1 jvm restores the connectivity. Anyone got any ideas? Currently if this isn't fix I'm going to have to go back to using Akka 2.1.4 as we can't have this happening in production. Logs below of when I attempt to send a message from actorSystem1 to actorSystem2 and the ask fails.
Thanks,
Brian
On ActorSystem1 I get
[2014-01-24 14:26:14,620] [DEBUG] [AkkaXLServer-akka.actor.default-dispatcher-10] akka.remote.transport.ProtocolStateActor - stopped
[2014-01-24 14:26:14,620] [DEBUG] [AkkaXLServer-akka.actor.default-dispatcher-10] akka.remote.transport.ProtocolStateActor - stopped
[2014-01-24 14:26:14,620] [INFO] [AkkaXLServer-akka.actor.default-dispatcher-10] akka.actor.LocalActorRef - Message [akka.remote.transport.AssociationHandle$Disassociated] from Actor[akka://AkkaXLServer/deadLetters] to Actor[akka://AkkaXLServer/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2FDataAugmentation%40server002%3A2552-1#1651503039] was not delivered. [1] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[2014-01-24 14:26:14,620] [INFO] [AkkaXLServer-akka.actor.default-dispatcher-10] akka.actor.LocalActorRef - Message [akka.remote.transport.AssociationHandle$Disassociated] from Actor[akka://AkkaXLServer/deadLetters] to Actor[akka://AkkaXLServer/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2FDataAugmentation%40server002%3A2552-1#1651503039] was not delivered. [1] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[2014-01-24 14:26:14,621] [DEBUG] [AkkaXLServer-akka.actor.default-dispatcher-10] akka.remote.EndpointReader - stopped
[2014-01-24 14:26:14,621] [DEBUG] [AkkaXLServer-akka.actor.default-dispatcher-10] akka.remote.EndpointReader - stopped
[2014-01-24 14:26:14,623] [DEBUG] [AkkaXLServer-akka.actor.default-dispatcher-13] akka.remote.EndpointWriter - Disassociated [akka.tcp://AkkaXL...@127.0.0.1:60789] -> [akka.tcp://DataAugmentation@server002:2552]
[2014-01-24 14:26:14,623] [DEBUG] [AkkaXLServer-akka.actor.default-dispatcher-13] akka.remote.EndpointWriter - Disassociated [akka.tcp://AkkaXL...@127.0.0.1:60789] -> [akka.tcp://DataAugmentation@server002:2552]
[2014-01-24 14:26:14,628] [DEBUG] [AkkaXLServer-akka.actor.default-dispatcher-19] akka.remote.EndpointWriter - restarting
[2014-01-24 14:26:14,628] [DEBUG] [AkkaXLServer-akka.actor.default-dispatcher-19] akka.remote.EndpointWriter - restarting
[2014-01-24 14:26:14,630] [DEBUG] [AkkaXLServer-akka.actor.default-dispatcher-2] akka.remote.EndpointWriter - Disassociated [akka.tcp://AkkaXL...@127.0.0.1:60789] -> [akka.tcp://DataAugmentation@server002:2552]
[2014-01-24 14:26:14,630] [DEBUG] [AkkaXLServer-akka.actor.default-dispatcher-2] akka.remote.EndpointWriter - Disassociated [akka.tcp://AkkaXL...@127.0.0.1:60789] -> [akka.tcp://DataAugmentation@server002:2552]
[2014-01-24 14:26:14,630] [INFO] [AkkaXLServer-akka.actor.default-dispatcher-2] akka.actor.LocalActorRef - Message [akka.remote.transport.ActorTransportAdapter$DisassociateUnderlying] from Actor[akka://AkkaXLServer/deadLetters] to Actor[akka://AkkaXLServer/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2FDataAugmentation%40server002%3A2552-1#1651503039] was not delivered. [2] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[2014-01-24 14:26:14,630] [INFO] [AkkaXLServer-akka.actor.default-dispatcher-2] akka.actor.LocalActorRef - Message [akka.remote.transport.ActorTransportAdapter$DisassociateUnderlying] from Actor[akka://AkkaXLServer/deadLetters] to Actor[akka://AkkaXLServer/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2FDataAugmentation%40server002%3A2552-1#1651503039] was not delivered. [2] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[2014-01-24 14:26:14,632] [DEBUG] [AkkaXLServer-akka.actor.default-dispatcher-11] akka.remote.EndpointWriter - restarted
[2014-01-24 14:26:14,632] [DEBUG] [AkkaXLServer-akka.actor.default-dispatcher-11] akka.remote.EndpointWriter - restarted
[2014-01-24 14:26:14,633] [DEBUG] [AkkaXLServer-akka.actor.default-dispatcher-24] akka.remote.transport.AkkaProtocolManager - now supervising Actor[akka://AkkaXLServer/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2FDataAugmentation%40server002%3A2552-2#1273774939]
[2014-01-24 14:26:14,633] [DEBUG] [AkkaXLServer-akka.actor.default-dispatcher-24] akka.remote.transport.AkkaProtocolManager - now supervising Actor[akka://AkkaXLServer/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2FDataAugmentation%40server002%3A2552-2#1273774939]
[2014-01-24 14:26:14,634] [DEBUG] [AkkaXLServer-akka.actor.default-dispatcher-24] akka.remote.transport.ProtocolStateActor - started (akka.remote.transport.ProtocolStateActor@549ad840)
[2014-01-24 14:26:14,634] [DEBUG] [AkkaXLServer-akka.actor.default-dispatcher-24] akka.remote.transport.ProtocolStateActor - started (akka.remote.transport.ProtocolStateActor@549ad840)
[2014-01-24 14:26:17,126] [DEBUG] [AkkaXLServer-akka.actor.default-dispatcher-20] akka.remote.EndpointWriter - Associated [akka.tcp://AkkaXL...@127.0.0.1:60789] -> [akka.tcp://DataAugmentation@server002:2552]
[2014-01-24 14:26:17,126] [DEBUG] [AkkaXLServer-akka.actor.default-dispatcher-20] akka.remote.EndpointWriter - Associated [akka.tcp://AkkaXL...@127.0.0.1:60789] -> [akka.tcp://DataAugmentation@server002:2552]
[2014-01-24 14:26:17,127] [DEBUG] [AkkaXLServer-akka.actor.default-dispatcher-20] akka.remote.EndpointWriter - now supervising Actor[akka://AkkaXLServer/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FDataAugmentation%40server002%3A2552-0/endpointWriter/endpointReader-akka.tcp%3A%2F%2FDataAugmentation%40server002%3A2552-0#936701101]
[2014-01-24 14:26:17,127] [DEBUG] [AkkaXLServer-akka.actor.default-dispatcher-20] akka.remote.EndpointWriter - now supervising Actor[akka://AkkaXLServer/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FDataAugmentation%40server002%3A2552-0/endpointWriter/endpointReader-akka.tcp%3A%2F%2FDataAugmentation%40server002%3A2552-0#936701101]
[2014-01-24 14:26:17,129] [DEBUG] [AkkaXLServer-akka.actor.default-dispatcher-18] akka.remote.EndpointReader - started (akka.remote.EndpointReader@1bd97d0d)
[2014-01-24 14:26:17,129] [DEBUG] [AkkaXLServer-akka.actor.default-dispatcher-18] akka.remote.EndpointReader - started (akka.remote.EndpointReader@1bd97d0d)
[2014-01-24 14:26:17,129] [DEBUG] [AkkaXLServer-akka.actor.default-dispatcher-18] akka.remote.EndpointReader - now monitoring Actor[akka://AkkaXLServer/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FDataAugmentation%40server002%3A2552-0/endpointWriter#226701486]
[2014-01-24 14:26:17,129] [DEBUG] [AkkaXLServer-akka.actor.default-dispatcher-18] akka.remote.EndpointReader - now monitoring Actor[akka://AkkaXLServer/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FDataAugmentation%40server002%3A2552-0/endpointWriter#226701486]
On ActorSystem2
[2014-01-24 14:26:16,473] [ERROR] [DataAugmentation-akka.actor.default-dispatcher-234] akka.remote.EndpointWriter - AssociationError [akka.tcp://DataAugmentation@server002:2552] -> [akka.tcp://AkkaXL...@127.0.0.1:60789]: Error [Association failed with [akka.tcp://AkkaXL...@127.0.0.1:60789]] [
akka.remote.EndpointAssociationException: Association failed with [akka.tcp://AkkaXL...@127.0.0.1:60789]
Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: /127.0.0.1:60789
Akka Team
Jan 24, 2014, 10:05:12 AM1/24/14
to Akka User List
Hi Brian,
On ActorSystem1 I get
Actually the interesting part is more probably earlier in this log. Btw, is the port 60789 firewalled? It seems like ActorSystem2 is trying to connect back to system1 and it cannot.
-Endre
--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://akka.io/faq/
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/groups/opt_out.
--
Akka Team
Typesafe - The software stack for applications that scale
Brian Fowler
Jan 24, 2014, 10:14:04 AM1/24/14
to akka...@googlegroups.com
Hi Endre,
No the two systems can comunicate initially, the problem just occurs after a quiet period of about 5 minutes. The problem also does not occur with Akka 2.1.4 so I don't think it is a network issue.
Thanks,
Brian
On Friday, 24 January 2014 15:05:12 UTC, Akka Team wrote:
Hi Brian,
On ActorSystem1 I get
Actually the interesting part is more probably earlier in this log. Btw, is the port 60789 firewalled? It seems like ActorSystem2 is trying to connect back to system1 and it cannot.
-Endre
[2014-01-24 14:26:14,620] [DEBUG] [AkkaXLServer-akka.actor.default-dispatcher-10] akka.remote.transport.ProtocolStateActor - stopped
[2014-01-24 14:26:14,620] [DEBUG] [AkkaXLServer-akka.actor.default-dispatcher-10] akka.remote.transport.ProtocolStateActor - stopped
[2014-01-24 14:26:14,620] [INFO] [AkkaXLServer-akka.actor.default-dispatcher-10] akka.actor.LocalActorRef - Message [akka.remote.transport.AssociationHandle$Disassociated] from Actor[akka://AkkaXLServer/deadLetters] to Actor[akka://AkkaXLServer/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2FDataAugmentation%40server002%3A2552-1#1651503039] was not delivered. [1] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[2014-01-24 14:26:14,620] [INFO] [AkkaXLServer-akka.actor.default-dispatcher-10] akka.actor.LocalActorRef - Message [akka.remote.transport.AssociationHandle$Disassociated] from Actor[akka://AkkaXLServer/deadLetters] to Actor[akka://AkkaXLServer/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2FDataAugmentation%40server002%3A2552-1#1651503039] was not delivered. [1] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[2014-01-24 14:26:14,621] [DEBUG] [AkkaXLServer-akka.actor.default-dispatcher-10] akka.remote.EndpointReader - stopped
[2014-01-24 14:26:14,621] [DEBUG] [AkkaXLServer-akka.actor.default-dispatcher-10] akka.remote.EndpointReader - stopped
[2014-01-24 14:26:14,623] [DEBUG] [AkkaXLServer-akka.actor.default-dispatcher-13] akka.remote.EndpointWriter - Disassociated [akka.tcp://AkkaXLServer@127.0.0.1:60789] -> [akka.tcp://DataAugmentation@server002:2552]
[2014-01-24 14:26:14,623] [DEBUG] [AkkaXLServer-akka.actor.default-dispatcher-13] akka.remote.EndpointWriter - Disassociated [akka.tcp://AkkaXLServer@127.0.0.1:60789] -> [akka.tcp://DataAugmentation@server002:2552]
[2014-01-24 14:26:14,628] [DEBUG] [AkkaXLServer-akka.actor.default-dispatcher-19] akka.remote.EndpointWriter - restarting
[2014-01-24 14:26:14,628] [DEBUG] [AkkaXLServer-akka.actor.default-dispatcher-19] akka.remote.EndpointWriter - restarting
[2014-01-24 14:26:14,630] [DEBUG] [AkkaXLServer-akka.actor.default-dispatcher-2] akka.remote.EndpointWriter - Disassociated [akka.tcp://AkkaXLServer@127.0.0.1:60789] -> [akka.tcp://DataAugmentation@server002:2552]
[2014-01-24 14:26:14,630] [DEBUG] [AkkaXLServer-akka.actor.default-dispatcher-2] akka.remote.EndpointWriter - Disassociated [akka.tcp://AkkaXLServer@127.0.0.1:60789] -> [akka.tcp://DataAugmentation@server002:2552]
[2014-01-24 14:26:14,630] [INFO] [AkkaXLServer-akka.actor.default-dispatcher-2] akka.actor.LocalActorRef - Message [akka.remote.transport.ActorTransportAdapter$DisassociateUnderlying] from Actor[akka://AkkaXLServer/deadLetters] to Actor[akka://AkkaXLServer/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2FDataAugmentation%40server002%3A2552-1#1651503039] was not delivered. [2] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[2014-01-24 14:26:14,630] [INFO] [AkkaXLServer-akka.actor.default-dispatcher-2] akka.actor.LocalActorRef - Message [akka.remote.transport.ActorTransportAdapter$DisassociateUnderlying] from Actor[akka://AkkaXLServer/deadLetters] to Actor[akka://AkkaXLServer/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2FDataAugmentation%40server002%3A2552-1#1651503039] was not delivered. [2] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[2014-01-24 14:26:14,632] [DEBUG] [AkkaXLServer-akka.actor.default-dispatcher-11] akka.remote.EndpointWriter - restarted
[2014-01-24 14:26:14,632] [DEBUG] [AkkaXLServer-akka.actor.default-dispatcher-11] akka.remote.EndpointWriter - restarted
[2014-01-24 14:26:14,633] [DEBUG] [AkkaXLServer-akka.actor.default-dispatcher-24] akka.remote.transport.AkkaProtocolManager - now supervising Actor[akka://AkkaXLServer/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2FDataAugmentation%40server002%3A2552-2#1273774939]
[2014-01-24 14:26:14,633] [DEBUG] [AkkaXLServer-akka.actor.default-dispatcher-24] akka.remote.transport.AkkaProtocolManager - now supervising Actor[akka://AkkaXLServer/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2FDataAugmentation%40server002%3A2552-2#1273774939]
[2014-01-24 14:26:14,634] [DEBUG] [AkkaXLServer-akka.actor.default-dispatcher-24] akka.remote.transport.ProtocolStateActor - started (akka.remote.transport.ProtocolStateActor@549ad840)
[2014-01-24 14:26:14,634] [DEBUG] [AkkaXLServer-akka.actor.default-dispatcher-24] akka.remote.transport.ProtocolStateActor - started (akka.remote.transport.ProtocolStateActor@549ad840)
[2014-01-24 14:26:17,126] [DEBUG] [AkkaXLServer-akka.actor.default-dispatcher-20] akka.remote.EndpointWriter - Associated [akka.tcp://AkkaXLServer@127.0.0.1:60789] -> [akka.tcp://DataAugmentation@server002:2552]
[2014-01-24 14:26:17,126] [DEBUG] [AkkaXLServer-akka.actor.default-dispatcher-20] akka.remote.EndpointWriter - Associated [akka.tcp://AkkaXLServer@127.0.0.1:60789] -> [akka.tcp://DataAugmentation@server002:2552]
[2014-01-24 14:26:17,127] [DEBUG] [AkkaXLServer-akka.actor.default-dispatcher-20] akka.remote.EndpointWriter - now supervising Actor[akka://AkkaXLServer/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FDataAugmentation%40server002%3A2552-0/endpointWriter/endpointReader-akka.tcp%3A%2F%2FDataAugmentation%40server002%3A2552-0#936701101]
[2014-01-24 14:26:17,127] [DEBUG] [AkkaXLServer-akka.actor.default-dispatcher-20] akka.remote.EndpointWriter - now supervising Actor[akka://AkkaXLServer/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FDataAugmentation%40server002%3A2552-0/endpointWriter/endpointReader-akka.tcp%3A%2F%2FDataAugmentation%40server002%3A2552-0#936701101]
[2014-01-24 14:26:17,129] [DEBUG] [AkkaXLServer-akka.actor.default-dispatcher-18] akka.remote.EndpointReader - started (akka.remote.EndpointReader@1bd97d0d)
[2014-01-24 14:26:17,129] [DEBUG] [AkkaXLServer-akka.actor.default-dispatcher-18] akka.remote.EndpointReader - started (akka.remote.EndpointReader@1bd97d0d)
[2014-01-24 14:26:17,129] [DEBUG] [AkkaXLServer-akka.actor.default-dispatcher-18] akka.remote.EndpointReader - now monitoring Actor[akka://AkkaXLServer/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FDataAugmentation%40server002%3A2552-0/endpointWriter#226701486]
[2014-01-24 14:26:17,129] [DEBUG] [AkkaXLServer-akka.actor.default-dispatcher-18] akka.remote.EndpointReader - now monitoring Actor[akka://AkkaXLServer/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FDataAugmentation%40server002%3A2552-0/endpointWriter#226701486]
On ActorSystem2
[2014-01-24 14:26:16,473] [ERROR] [DataAugmentation-akka.actor.default-dispatcher-234] akka.remote.EndpointWriter - AssociationError [akka.tcp://DataAugmentation@server002:2552] -> [akka.tcp://AkkaXLServer@127.0.0.1:60789]: Error [Association failed with [akka.tcp://AkkaXLServer@127.0.0.1:60789]] [
akka.remote.EndpointAssociationException: Association failed with [akka.tcp://AkkaXLServer@127.0.0.1:60789]
Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: /127.0.0.1:60789
--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://akka.io/faq/
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/groups/opt_out.
Akka Team
Jan 24, 2014, 10:23:05 AM1/24/14
to Akka User List
Hi Brian,
On Fri, Jan 24, 2014 at 4:14 PM, Brian Fowler <brian.w...@gmail.com> wrote:
Hi Endre,
No the two systems can comunicate initially, the problem just occurs after a quiet period of about 5 minutes. The problem also does not occur with Akka 2.1.4 so I don't think it is a network issue.
Akka 2.1.4 had no real recovery mechanism (and other important features) so it probably never triggers it.
That the systems can communicate initially is not really a proof because actor systems reuse the incoming connection when possible. So it can be like this:
1. system1 connects to system2 on address akka.tcp://DataAugmentation@server002:2552 (this direction is fine)
2. communication is fine
3. for some reason system1 shuts of the connection (most probably heartbeat misses, you should increase the akka.remote.transport-failure-detector.acceptable-heartbeat-pause setting to something like 10s -- but I need more log entries).
4. Now system2 tries to recover by connecting to system1 on address 127.0.0.1:60789
and this direction fails.
What is suspicious to me is this error: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: /127.0.0.1:60789
This is thrown by netty and means the connection is not accepted.
You can try to force this scenario by setting akka.remote.use-passive-connections = off on both nodes (this prevents system2 to reuse the connection from system1). Then we might see what happens.
-Endre
Brian Fowler
Jan 24, 2014, 10:44:46 AM1/24/14
to akka...@googlegroups.com
Thanks Endre, I'll try increasing the akka.remote.transport-failure-
However I'm a little concerned in general that the ActorSystems are not recovering their connectivity without restarting the ActorSystem1 JVM. Do I need to change the config or but in a different Supervisor strategy to make this work?
ActorSystem1 seems to make some attempt at it but it doesn't seem to work..
detector.acceptable-heartbeat-pause
However I'm a little concerned in general that the ActorSystems are not recovering their connectivity without restarting the ActorSystem1 JVM. Do I need to change the config or but in a different Supervisor strategy to make this work?
ActorSystem1 seems to make some attempt at it but it doesn't seem to work..
[2014-01-24 14:26:14,632] [DEBUG] [AkkaXLServer-akka.actor.
default-dispatcher-11] akka.remote.EndpointWriter - restarted
[2014-01-24 14:26:14,632] [DEBUG] [AkkaXLServer-akka.actor.default-dispatcher-11] akka.remote.EndpointWriter - restarted
[2014-01-24 14:26:14,633] [DEBUG] [AkkaXLServer-akka.actor.default-dispatcher-24] akka.remote.transport.AkkaProtocolManager - now supervising Actor[akka://AkkaXLServer/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2FDataAugmentation%40server002%3A2552-2#1273774939]
[2014-01-24 14:26:14,633] [DEBUG] [AkkaXLServer-akka.actor.default-dispatcher-24] akka.remote.transport.AkkaProtocolManager - now supervising Actor[akka://AkkaXLServer/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2FDataAugmentation%40server002%3A2552-2#1273774939]
[2014-01-24 14:26:14,634] [DEBUG] [AkkaXLServer-akka.actor.default-dispatcher-24] akka.remote.transport.ProtocolStateActor - started (akka.remote.transport.ProtocolStateActor@549ad840)
[2014-01-24 14:26:14,634] [DEBUG] [AkkaXLServer-akka.actor.default-dispatcher-24] akka.remote.transport.ProtocolStateActor - started (akka.remote.transport.ProtocolStateActor@549ad840)
[2014-01-24 14:26:17,126] [DEBUG] [AkkaXLServer-akka.actor.default-dispatcher-20] akka.remote.EndpointWriter - Associated [akka.tcp://AkkaXLServer@127.0.0.1:60789] -> [akka.tcp://DataAugmentation@server002:2552]
[2014-01-24 14:26:17,126] [DEBUG] [AkkaXLServer-akka.actor.default-dispatcher-20] akka.remote.EndpointWriter - Associated [akka.tcp://AkkaXLServer@127.0.0.1:60789] -> [akka.tcp://DataAugmentation@server002:2552]
[2014-01-24 14:26:17,127] [DEBUG] [AkkaXLServer-akka.actor.default-dispatcher-20] akka.remote.EndpointWriter - now supervising Actor[akka://AkkaXLServer/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FDataAugmentation%40server002%3A2552-0/endpointWriter/endpointReader-akka.tcp%3A%2F%2FDataAugmentation%40server002%3A2552-0#936701101]
[2014-01-24 14:26:17,127] [DEBUG] [AkkaXLServer-akka.actor.default-dispatcher-20] akka.remote.EndpointWriter - now supervising Actor[akka://AkkaXLServer/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FDataAugmentation%40server002%3A2552-0/endpointWriter/endpointReader-akka.tcp%3A%2F%2FDataAugmentation%40server002%3A2552-0#936701101]
[2014-01-24 14:26:17,129] [DEBUG] [AkkaXLServer-akka.actor.default-dispatcher-18] akka.remote.EndpointReader - started (akka.remote.EndpointReader@1bd97d0d)
[2014-01-24 14:26:17,129] [DEBUG] [AkkaXLServer-akka.actor.default-dispatcher-18] akka.remote.EndpointReader - started (akka.remote.EndpointReader@1bd97d0d)
[2014-01-24 14:26:17,129] [DEBUG] [AkkaXLServer-akka.actor.default-dispatcher-18] akka.remote.EndpointReader - now monitoring Actor[akka://AkkaXLServer/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FDataAugmentation%40server002%3A2552-0/endpointWriter#226701486]
[2014-01-24 14:26:17,129] [DEBUG] [AkkaXLServer-akka.actor.default-dispatcher-18] akka.remote.EndpointReader - now monitoring Actor[akka://AkkaXLServer/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FDataAugmentation%40server002%3A2552-0/endpointWriter#226701486]
[2014-01-24 14:26:14,632] [DEBUG] [AkkaXLServer-akka.actor.default-dispatcher-11] akka.remote.EndpointWriter - restarted
[2014-01-24 14:26:14,633] [DEBUG] [AkkaXLServer-akka.actor.default-dispatcher-24] akka.remote.transport.AkkaProtocolManager - now supervising Actor[akka://AkkaXLServer/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2FDataAugmentation%40server002%3A2552-2#1273774939]
[2014-01-24 14:26:14,633] [DEBUG] [AkkaXLServer-akka.actor.default-dispatcher-24] akka.remote.transport.AkkaProtocolManager - now supervising Actor[akka://AkkaXLServer/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2FDataAugmentation%40server002%3A2552-2#1273774939]
[2014-01-24 14:26:14,634] [DEBUG] [AkkaXLServer-akka.actor.default-dispatcher-24] akka.remote.transport.ProtocolStateActor - started (akka.remote.transport.ProtocolStateActor@549ad840)
[2014-01-24 14:26:14,634] [DEBUG] [AkkaXLServer-akka.actor.default-dispatcher-24] akka.remote.transport.ProtocolStateActor - started (akka.remote.transport.ProtocolStateActor@549ad840)
[2014-01-24 14:26:17,126] [DEBUG] [AkkaXLServer-akka.actor.default-dispatcher-20] akka.remote.EndpointWriter - Associated [akka.tcp://AkkaXLServer@127.0.0.1:60789] -> [akka.tcp://DataAugmentation@server002:2552]
[2014-01-24 14:26:17,126] [DEBUG] [AkkaXLServer-akka.actor.default-dispatcher-20] akka.remote.EndpointWriter - Associated [akka.tcp://AkkaXLServer@127.0.0.1:60789] -> [akka.tcp://DataAugmentation@server002:2552]
[2014-01-24 14:26:17,127] [DEBUG] [AkkaXLServer-akka.actor.default-dispatcher-20] akka.remote.EndpointWriter - now supervising Actor[akka://AkkaXLServer/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FDataAugmentation%40server002%3A2552-0/endpointWriter/endpointReader-akka.tcp%3A%2F%2FDataAugmentation%40server002%3A2552-0#936701101]
[2014-01-24 14:26:17,127] [DEBUG] [AkkaXLServer-akka.actor.default-dispatcher-20] akka.remote.EndpointWriter - now supervising Actor[akka://AkkaXLServer/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FDataAugmentation%40server002%3A2552-0/endpointWriter/endpointReader-akka.tcp%3A%2F%2FDataAugmentation%40server002%3A2552-0#936701101]
[2014-01-24 14:26:17,129] [DEBUG] [AkkaXLServer-akka.actor.default-dispatcher-18] akka.remote.EndpointReader - started (akka.remote.EndpointReader@1bd97d0d)
[2014-01-24 14:26:17,129] [DEBUG] [AkkaXLServer-akka.actor.default-dispatcher-18] akka.remote.EndpointReader - started (akka.remote.EndpointReader@1bd97d0d)
[2014-01-24 14:26:17,129] [DEBUG] [AkkaXLServer-akka.actor.default-dispatcher-18] akka.remote.EndpointReader - now monitoring Actor[akka://AkkaXLServer/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FDataAugmentation%40server002%3A2552-0/endpointWriter#226701486]
[2014-01-24 14:26:17,129] [DEBUG] [AkkaXLServer-akka.actor.default-dispatcher-18] akka.remote.EndpointReader - now monitoring Actor[akka://AkkaXLServer/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FDataAugmentation%40server002%3A2552-0/endpointWriter#226701486]
Akka Team
Jan 24, 2014, 10:59:21 AM1/24/14
to Akka User List
Hi Brian,
On Fri, Jan 24, 2014 at 4:44 PM, Brian Fowler <brian.w...@gmail.com> wrote:
Thanks Endre, I'll try increasing the akka.remote.transport-failure-detector.acceptable-heartbeat-pause
However I'm a little concerned in general that the ActorSystems are not recovering their connectivity without restarting the ActorSystem1 JVM. Do I need to change the config or but in a different Supervisor strategy to make this work?
Restart is not needed, unless one of the nodes quarantined the other (but that would be in the logs). I suspect something else here: from your log it seems that system one connects fine:
akka.remote.EndpointWriter - Associated [akka.tcp://AkkaXL...@127.0.0.1:60789] -> [akka.tcp://DataAugmentation@server002:2552]
akka.remote.EndpointWriter - Associated [akka.tcp://AkkaXL...@127.0.0.1:60789] -> [akka.tcp://DataAugmentation@server002:2552]
As I said, I suspect the direction from system2 to system1 to be problematic. I think the two systems are trying to connect at the same time, and therefore the connection from system1 on system2 to is accepted for reading only, but the writing direction (system2 to system1) is failing for some reason (Connection Refused).
You can debug what messages are actually getting through:
akka.remote.log-received-messages = on
akka.remote.log-sent-messages = on
You can debug what messages are actually getting through:
akka.remote.log-received-messages = on
akka.remote.log-sent-messages = on
Then you will see the actual remote messages that are attempted to be sent and that are actually received in the log.
Btw without a longer log it is hard to see the whole picture.
-Endre
Brian Fowler
Jan 27, 2014, 6:54:36 AM1/27/14
to akka...@googlegroups.com
Hi Endre,
Upping the akka.remote.transport-failure-detector.acceptable-heartbeat-pause timeout, made the systems stay connected with each other for longer however they still failed in the same manner after time. Due to our release schedules we are rolling back to 2.1.4. I'll have to investigate further at another time.
Thanks for you help,
Brian
Upping the akka.remote.transport-failure-detector.acceptable-heartbeat-pause timeout, made the systems stay connected with each other for longer however they still failed in the same manner after time. Due to our release schedules we are rolling back to 2.1.4. I'll have to investigate further at another time.
Thanks for you help,
Brian
On Friday, 24 January 2014 15:59:21 UTC, Akka Team wrote:
Hi Brian,
On Fri, Jan 24, 2014 at 4:44 PM, Brian Fowler <brian.w...@gmail.com> wrote:
Thanks Endre, I'll try increasing the akka.remote.transport-failure-detector.acceptable-heartbeat-pause
However I'm a little concerned in general that the ActorSystems are not recovering their connectivity without restarting the ActorSystem1 JVM. Do I need to change the config or but in a different Supervisor strategy to make this work?
Restart is not needed, unless one of the nodes quarantined the other (but that would be in the logs). I suspect something else here: from your log it seems that system one connects fine:
Endre Varga
Jan 27, 2014, 7:13:58 AM1/27/14
to akka...@googlegroups.com
Hi Brian,
On Mon, Jan 27, 2014 at 12:54 PM, Brian Fowler <brian.w...@gmail.com> wrote:
Hi Endre,
Upping the akka.remote.transport-failure-detector.acceptable-heartbeat-pause timeout, made the systems stay connected with each other for longer however they still failed in the same manner after time.
This indicates load issues though.
Due to our release schedules we are rolling back to 2.1.4. I'll have to investigate further at another time.
Sorry to hear that but I understand the choice. I think you should investigate the issue a bit more even on 2.1.4 because the fact that the transport Failure Detector triggers means that it misses many heartbeats -- which can indicate high latencies (can be CPU, GC pressure or network load). These problems are not apparent in 2.1.x because its remoting does not monitor response times.
Thanks for you help,
Thank you for your patience.
-Endre
Brian Fowler
Jan 27, 2014, 12:05:54 PM1/27/14
to akka...@googlegroups.com
Hi Endre,
Think I can reliably recreate the problem. You need two linux boxes. On one create a actor system with a typed actor bound to an endpoint. I just had a simple one method that returned the string hello when the method was called with no arguments.
On the the the other create an actor system that calls the method. Call the method once then sleep for 30 seconds and call it again. Simulate a slow system by after the first method is called and a response is succesfully received suspend the process on the calling system i.e. Ctrl-Z then after10 seconds then but it back in the foreground.
Logs below
TypedActor Endpoint
[2014-01-27 16:45:43,780] [INFO] [example-akka.actor.default-dispatcher-9] akka.event.slf4j.Slf4jLogger - Slf4jLogger started
[2014-01-27 16:45:43,780] [INFO] [example-akka.actor.default-dispatcher-9] akka.event.slf4j.Slf4jLogger - Slf4jLogger started
[2014-01-27 16:45:43,794] [DEBUG] [example-akka.actor.default-dispatcher-11] akka.event.EventStream - logger log1-Slf4jLogger started
[2014-01-27 16:45:43,794] [DEBUG] [example-akka.actor.default-dispatcher-11] akka.event.EventStream - logger log1-Slf4jLogger started
[2014-01-27 16:45:43,795] [DEBUG] [example-akka.actor.default-dispatcher-14] akka.event.EventStream - Default Loggers started
[2014-01-27 16:45:43,795] [DEBUG] [example-akka.actor.default-dispatcher-14] akka.event.EventStream - Default Loggers started
[2014-01-27 16:45:43,795] [DEBUG] [example-akka.actor.default-dispatcher-16] akka.actor.LocalActorRefProvider$SystemGuardian - now supervising Actor[akka://example/system/UnhandledMessageForwarder#-2027631171]
[2014-01-27 16:45:43,795] [DEBUG] [example-akka.actor.default-dispatcher-16] akka.actor.LocalActorRefProvider$SystemGuardian - now supervising Actor[akka://example/system/UnhandledMessageForwarder#-2027631171]
[2014-01-27 16:45:43,796] [DEBUG] [example-akka.actor.default-dispatcher-17] akka.event.LoggingBus$$anonfun$startDefaultLoggers$2$$anon$1 - started (akka.event.LoggingBus$$anonfun$startDefaultLoggers$2$$anon$1@435db13f)
[2014-01-27 16:45:43,796] [DEBUG] [example-akka.actor.default-dispatcher-17] akka.event.LoggingBus$$anonfun$startDefaultLoggers$2$$anon$1 - started (akka.event.LoggingBus$$anonfun$startDefaultLoggers$2$$anon$1@435db13f)
[2014-01-27 16:45:43,796] [DEBUG] [example-akka.actor.default-dispatcher-18] akka.actor.LocalActorRefProvider$SystemGuardian - now supervising Actor[akka://example/system/remoting-terminator#-658149313]
[2014-01-27 16:45:43,796] [DEBUG] [example-akka.actor.default-dispatcher-18] akka.actor.LocalActorRefProvider$SystemGuardian - now supervising Actor[akka://example/system/remoting-terminator#-658149313]
[2014-01-27 16:45:43,809] [DEBUG] [example-akka.actor.default-dispatcher-19] akka.remote.RemoteActorRefProvider$RemotingTerminator - started (akka.remote.RemoteActorRefProvider$RemotingTerminator@5bab2b55)
[2014-01-27 16:45:43,809] [DEBUG] [example-akka.actor.default-dispatcher-19] akka.remote.RemoteActorRefProvider$RemotingTerminator - started (akka.remote.RemoteActorRefProvider$RemotingTerminator@5bab2b55)
[2014-01-27 16:45:43,821] [DEBUG] [example-akka.actor.default-dispatcher-24] akka.actor.LocalActorRefProvider$SystemGuardian - now supervising Actor[akka://example/system/transports#128266947]
[2014-01-27 16:45:43,821] [DEBUG] [example-akka.actor.default-dispatcher-24] akka.actor.LocalActorRefProvider$SystemGuardian - now supervising Actor[akka://example/system/transports#128266947]
[2014-01-27 16:45:43,822] [DEBUG] [example-akka.actor.default-dispatcher-26] akka.remote.Remoting$TransportSupervisor - started (akka.remote.Remoting$TransportSupervisor@c569c60)
[2014-01-27 16:45:43,822] [DEBUG] [example-akka.actor.default-dispatcher-26] akka.remote.Remoting$TransportSupervisor - started (akka.remote.Remoting$TransportSupervisor@c569c60)
[2014-01-27 16:45:43,824] [DEBUG] [example-akka.actor.default-dispatcher-29] akka.remote.RemoteActorRefProvider$RemotingTerminator - now monitoring Actor[akka://example/system]
[2014-01-27 16:45:43,824] [DEBUG] [example-akka.actor.default-dispatcher-29] akka.remote.RemoteActorRefProvider$RemotingTerminator - now monitoring Actor[akka://example/system]
[2014-01-27 16:45:43,825] [DEBUG] [example-akka.actor.default-dispatcher-30] akka.actor.LocalActorRefProvider$SystemGuardian - now supervising Actor[akka://example/system/endpointManager#39348252]
[2014-01-27 16:45:43,825] [DEBUG] [example-akka.actor.default-dispatcher-30] akka.actor.LocalActorRefProvider$SystemGuardian - now supervising Actor[akka://example/system/endpointManager#39348252]
[2014-01-27 16:45:43,835] [DEBUG] [example-akka.actor.default-dispatcher-31] akka.remote.EndpointManager - started (akka.remote.EndpointManager@32602b6b)
[2014-01-27 16:45:43,835] [DEBUG] [example-akka.actor.default-dispatcher-31] akka.remote.EndpointManager - started (akka.remote.EndpointManager@32602b6b)
[2014-01-27 16:45:44,059] [DEBUG] [example-akka.actor.default-dispatcher-241] akka.remote.Remoting$TransportSupervisor - now supervising Actor[akka://example/system/transports/akkaprotocolmanager.tcp0#-1577819875]
[2014-01-27 16:45:44,059] [DEBUG] [example-akka.actor.default-dispatcher-241] akka.remote.Remoting$TransportSupervisor - now supervising Actor[akka://example/system/transports/akkaprotocolmanager.tcp0#-1577819875]
[2014-01-27 16:45:44,061] [DEBUG] [example-akka.actor.default-dispatcher-243] akka.remote.transport.AkkaProtocolManager - started (akka.remote.transport.AkkaProtocolManager@2d42c778)
[2014-01-27 16:45:44,061] [DEBUG] [example-akka.actor.default-dispatcher-243] akka.remote.transport.AkkaProtocolManager - started (akka.remote.transport.AkkaProtocolManager@2d42c778)
[2014-01-27 16:45:44,075] [DEBUG] [example-akka.actor.default-dispatcher-250] akka.actor.LocalActorRefProvider$SystemGuardian - now supervising Actor[akka://example/system/remote-watcher#-1405704646]
[2014-01-27 16:45:44,075] [DEBUG] [example-akka.actor.default-dispatcher-250] akka.actor.LocalActorRefProvider$SystemGuardian - now supervising Actor[akka://example/system/remote-watcher#-1405704646]
[2014-01-27 16:45:44,076] [DEBUG] [example-akka.actor.default-dispatcher-253] akka.actor.LocalActorRefProvider$SystemGuardian - now supervising Actor[akka://example/system/remote-deployment-watcher#1221709328]
[2014-01-27 16:45:44,076] [DEBUG] [example-akka.actor.default-dispatcher-253] akka.actor.LocalActorRefProvider$SystemGuardian - now supervising Actor[akka://example/system/remote-deployment-watcher#1221709328]
[2014-01-27 16:45:44,077] [DEBUG] [example-akka.actor.default-dispatcher-254] akka.actor.LocalActorRefProvider$SystemGuardian - now supervising Actor[akka://example/system/deadLetterListener#1681690402]
[2014-01-27 16:45:44,077] [DEBUG] [example-akka.actor.default-dispatcher-254] akka.actor.LocalActorRefProvider$SystemGuardian - now supervising Actor[akka://example/system/deadLetterListener#1681690402]
[2014-01-27 16:45:44,078] [DEBUG] [example-akka.actor.default-dispatcher-255] akka.event.DeadLetterListener - started (akka.event.DeadLetterListener@1ff243d1)
[2014-01-27 16:45:44,078] [DEBUG] [example-akka.actor.default-dispatcher-255] akka.event.DeadLetterListener - started (akka.event.DeadLetterListener@1ff243d1)
[2014-01-27 16:45:44,079] [DEBUG] [example-akka.actor.default-dispatcher-256] akka.remote.RemoteDeploymentWatcher - started (akka.remote.RemoteDeploymentWatcher@7458e2a1)
[2014-01-27 16:45:44,079] [DEBUG] [example-akka.actor.default-dispatcher-256] akka.remote.RemoteDeploymentWatcher - started (akka.remote.RemoteDeploymentWatcher@7458e2a1)
[2014-01-27 16:45:44,082] [DEBUG] [example-akka.actor.default-dispatcher-257] akka.remote.RemoteWatcher - started (akka.remote.RemoteWatcher@7b0bbd42)
[2014-01-27 16:45:44,082] [DEBUG] [example-akka.actor.default-dispatcher-257] akka.remote.RemoteWatcher - started (akka.remote.RemoteWatcher@7b0bbd42)
[2014-01-27 16:45:44,084] [DEBUG] [example-akka.actor.default-dispatcher-260] akka.actor.LocalActorRefProvider$Guardian - now supervising Actor[akka://example/user/test-monitor#-1100372654]
[2014-01-27 16:45:44,084] [DEBUG] [example-akka.actor.default-dispatcher-260] akka.actor.LocalActorRefProvider$Guardian - now supervising Actor[akka://example/user/test-monitor#-1100372654]
[2014-01-27 16:45:44,102] [DEBUG] [example-akka.actor.default-dispatcher-269] akka.actor.LocalActorRefProvider$Guardian - now supervising Actor[akka://example/user/test#-1720775600]
[2014-01-27 16:45:44,102] [DEBUG] [example-akka.actor.default-dispatcher-269] akka.actor.LocalActorRefProvider$Guardian - now supervising Actor[akka://example/user/test#-1720775600]
[2014-01-27 16:45:44,106] [DEBUG] [example-akka.actor.default-dispatcher-270] akka.routing.RouterConfig$$anon$1 - started (akka.routing.RouterConfig$$anon$1@29115152)
[2014-01-27 16:45:44,106] [DEBUG] [example-akka.actor.default-dispatcher-270] akka.routing.RouterConfig$$anon$1 - started (akka.routing.RouterConfig$$anon$1@29115152)
[2014-01-27 16:45:44,107] [DEBUG] [example-akka.actor.default-dispatcher-271] akka.routing.RouterConfig$$anon$1 - now supervising Actor[akka://example/user/test/$a#-1283403612]
[2014-01-27 16:45:44,107] [DEBUG] [example-akka.actor.default-dispatcher-271] akka.routing.RouterConfig$$anon$1 - now supervising Actor[akka://example/user/test/$a#-1283403612]
[2014-01-27 16:45:44,108] [DEBUG] [example-akka.actor.default-dispatcher-272] akka.routing.RouterConfig$$anon$1 - now supervising Actor[akka://example/user/test/$b#-1937476943]
[2014-01-27 16:45:44,108] [DEBUG] [example-akka.actor.default-dispatcher-272] akka.routing.RouterConfig$$anon$1 - now supervising Actor[akka://example/user/test/$b#-1937476943]
[2014-01-27 16:45:44,108] [DEBUG] [example-akka.actor.default-dispatcher-273] akka.routing.RouterConfig$$anon$1 - now supervising Actor[akka://example/user/test/$c#-1267864665]
[2014-01-27 16:45:44,108] [DEBUG] [example-akka.actor.default-dispatcher-273] akka.routing.RouterConfig$$anon$1 - now supervising Actor[akka://example/user/test/$c#-1267864665]
[2014-01-27 16:45:44,109] [DEBUG] [example-akka.actor.default-dispatcher-274] akka.routing.RouterConfig$$anon$1 - now supervising Actor[akka://example/user/test/$d#344287740]
[2014-01-27 16:45:44,109] [DEBUG] [example-akka.actor.default-dispatcher-274] akka.routing.RouterConfig$$anon$1 - now supervising Actor[akka://example/user/test/$d#344287740]
[2014-01-27 16:45:44,109] [DEBUG] [example-akka.actor.default-dispatcher-274] akka.routing.RouterConfig$$anon$1 - now supervising Actor[akka://example/user/test/$e#-89560475]
[2014-01-27 16:45:44,109] [DEBUG] [example-akka.actor.default-dispatcher-274] akka.routing.RouterConfig$$anon$1 - now supervising Actor[akka://example/user/test/$e#-89560475]
[2014-01-27 16:45:44,129] [DEBUG] [example-akka.actor.default-dispatcher-4] akka.actor.TypedActor$TypedActor - started (akka.actor.TypedActor$TypedActor@6eecb3f1)
[2014-01-27 16:45:44,129] [DEBUG] [example-akka.actor.default-dispatcher-4] akka.actor.TypedActor$TypedActor - started (akka.actor.TypedActor$TypedActor@6eecb3f1)
[2014-01-27 16:45:44,129] [DEBUG] [example-akka.actor.default-dispatcher-4] akka.actor.TypedActor$TypedActor - started (akka.actor.TypedActor$TypedActor@7619cc1b)
[2014-01-27 16:45:44,129] [DEBUG] [example-akka.actor.default-dispatcher-4] akka.actor.TypedActor$TypedActor - started (akka.actor.TypedActor$TypedActor@7619cc1b)
[2014-01-27 16:45:44,130] [DEBUG] [example-akka.actor.default-dispatcher-4] akka.actor.TypedActor$TypedActor - started (akka.actor.TypedActor$TypedActor@17a73d30)
[2014-01-27 16:45:44,130] [DEBUG] [example-akka.actor.default-dispatcher-4] akka.actor.TypedActor$TypedActor - started (akka.actor.TypedActor$TypedActor@17a73d30)
[2014-01-27 16:45:44,131] [DEBUG] [example-akka.actor.default-dispatcher-4] akka.actor.TypedActor$TypedActor - started (akka.actor.TypedActor$TypedActor@64a22671)
[2014-01-27 16:45:44,131] [DEBUG] [example-akka.actor.default-dispatcher-4] akka.actor.TypedActor$TypedActor - started (akka.actor.TypedActor$TypedActor@64a22671)
[2014-01-27 16:45:44,131] [DEBUG] [example-akka.actor.default-dispatcher-4] akka.actor.TypedActor$TypedActor - started (akka.actor.TypedActor$TypedActor@5189f854)
[2014-01-27 16:45:44,131] [DEBUG] [example-akka.actor.default-dispatcher-4] akka.actor.TypedActor$TypedActor - started (akka.actor.TypedActor$TypedActor@5189f854)
[2014-01-27 16:45:59,413] [DEBUG] [example-akka.actor.default-dispatcher-22] akka.remote.transport.AkkaProtocolManager - now supervising Actor[akka://example/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fexample%4010.43.15.19%3A36084-1#1695231211]
[2014-01-27 16:45:59,413] [DEBUG] [example-akka.actor.default-dispatcher-22] akka.remote.transport.AkkaProtocolManager - now supervising Actor[akka://example/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fexample%4010.43.15.19%3A36084-1#1695231211]
[2014-01-27 16:45:59,419] [DEBUG] [example-akka.actor.default-dispatcher-4] akka.remote.transport.ProtocolStateActor - started (akka.remote.transport.ProtocolStateActor@68fc8e75)
[2014-01-27 16:45:59,419] [DEBUG] [example-akka.actor.default-dispatcher-4] akka.remote.transport.ProtocolStateActor - started (akka.remote.transport.ProtocolStateActor@68fc8e75)
[2014-01-27 16:45:59,458] [DEBUG] [example-akka.actor.default-dispatcher-25] akka.remote.EndpointManager - now supervising Actor[akka://example/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-0#2034240752]
[2014-01-27 16:45:59,458] [DEBUG] [example-akka.actor.default-dispatcher-25] akka.remote.EndpointManager - now supervising Actor[akka://example/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-0#2034240752]
[2014-01-27 16:45:59,470] [DEBUG] [example-akka.actor.default-dispatcher-235] akka.remote.ReliableDeliverySupervisor - started (akka.remote.ReliableDeliverySupervisor@3de6696c)
[2014-01-27 16:45:59,470] [DEBUG] [example-akka.actor.default-dispatcher-235] akka.remote.ReliableDeliverySupervisor - started (akka.remote.ReliableDeliverySupervisor@3de6696c)
[2014-01-27 16:45:59,470] [DEBUG] [example-akka.actor.default-dispatcher-235] akka.remote.ReliableDeliverySupervisor - now monitoring Actor[akka://example/system/endpointManager#39348252]
[2014-01-27 16:45:59,470] [DEBUG] [example-akka.actor.default-dispatcher-235] akka.remote.ReliableDeliverySupervisor - now monitoring Actor[akka://example/system/endpointManager#39348252]
[2014-01-27 16:45:59,471] [DEBUG] [example-akka.actor.default-dispatcher-235] akka.remote.ReliableDeliverySupervisor - now supervising Actor[akka://example/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-0/endpointWriter#-680224632]
[2014-01-27 16:45:59,471] [DEBUG] [example-akka.actor.default-dispatcher-235] akka.remote.ReliableDeliverySupervisor - now supervising Actor[akka://example/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-0/endpointWriter#-680224632]
[2014-01-27 16:45:59,480] [DEBUG] [example-akka.actor.default-dispatcher-31] akka.remote.EndpointWriter - started (akka.remote.EndpointWriter@485b78cf)
[2014-01-27 16:45:59,480] [DEBUG] [example-akka.actor.default-dispatcher-31] akka.remote.EndpointWriter - started (akka.remote.EndpointWriter@485b78cf)
[2014-01-27 16:45:59,480] [DEBUG] [example-akka.actor.default-dispatcher-31] akka.remote.EndpointWriter - now monitoring Actor[akka://example/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-0#2034240752]
[2014-01-27 16:45:59,480] [DEBUG] [example-akka.actor.default-dispatcher-31] akka.remote.EndpointWriter - now monitoring Actor[akka://example/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-0#2034240752]
[2014-01-27 16:45:59,481] [DEBUG] [example-akka.actor.default-dispatcher-31] akka.remote.EndpointWriter - now supervising Actor[akka://example/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-0/endpointWriter/endpointReader-akka.tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-0#1050380478]
[2014-01-27 16:45:59,481] [DEBUG] [example-akka.actor.default-dispatcher-31] akka.remote.EndpointWriter - now supervising Actor[akka://example/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-0/endpointWriter/endpointReader-akka.tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-0#1050380478]
[2014-01-27 16:45:59,489] [DEBUG] [example-akka.actor.default-dispatcher-242] akka.remote.EndpointReader - started (akka.remote.EndpointReader@4545f5e3)
[2014-01-27 16:45:59,489] [DEBUG] [example-akka.actor.default-dispatcher-242] akka.remote.EndpointReader - started (akka.remote.EndpointReader@4545f5e3)
[2014-01-27 16:45:59,489] [DEBUG] [example-akka.actor.default-dispatcher-242] akka.remote.EndpointReader - now monitoring Actor[akka://example/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-0/endpointWriter#-680224632]
[2014-01-27 16:45:59,489] [DEBUG] [example-akka.actor.default-dispatcher-242] akka.remote.EndpointReader - now monitoring Actor[akka://example/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-0/endpointWriter#-680224632]
[2014-01-27 16:45:59,539] [DEBUG] [example-akka.actor.default-dispatcher-247] akka.remote.EndpointWriter - received local message RemoteMessage: [SelectChildName(user,SelectChildName(test,MethodCall(public abstract java.lang.String com.markit.bonds.akka.ReconnectFailExample$ExampleService.hello(),null)))] to [Actor[akka://example/]]<+[akka://example/] from [Actor[akka.tcp://cli...@127.0.0.1:49659/temp/$a]]
[2014-01-27 16:45:59,539] [DEBUG] [example-akka.actor.default-dispatcher-247] akka.remote.EndpointWriter - received local message RemoteMessage: [SelectChildName(user,SelectChildName(test,MethodCall(public abstract java.lang.String com.markit.bonds.akka.ReconnectFailExample$ExampleService.hello(),null)))] to [Actor[akka://example/]]<+[akka://example/] from [Actor[akka.tcp://cli...@127.0.0.1:49659/temp/$a]]
[2014-01-27 16:45:59,540] [DEBUG] [example-akka.actor.default-dispatcher-247] akka.actor.LocalActorRefProvider$Guardian - received AutoReceiveMessage Envelope(SelectChildName(user,SelectChildName(test,MethodCall(public abstract java.lang.String com.markit.bonds.akka.ReconnectFailExample$ExampleService.hello(),null))),Actor[akka.tcp://cli...@127.0.0.1:49659/temp/$a])
[2014-01-27 16:45:59,540] [DEBUG] [example-akka.actor.default-dispatcher-247] akka.actor.LocalActorRefProvider$Guardian - received AutoReceiveMessage Envelope(SelectChildName(user,SelectChildName(test,MethodCall(public abstract java.lang.String com.markit.bonds.akka.ReconnectFailExample$ExampleService.hello(),null))),Actor[akka.tcp://cli...@127.0.0.1:49659/temp/$a])
[2014-01-27 16:45:59,542] [DEBUG] [example-akka.actor.default-dispatcher-254] akka.actor.LocalActorRefProvider$Guardian - received AutoReceiveMessage Envelope(SelectChildName(test,MethodCall(public abstract java.lang.String com.markit.bonds.akka.ReconnectFailExample$ExampleService.hello(),null)),Actor[akka.tcp://cli...@127.0.0.1:49659/temp/$a])
[2014-01-27 16:45:59,542] [DEBUG] [example-akka.actor.default-dispatcher-254] akka.actor.LocalActorRefProvider$Guardian - received AutoReceiveMessage Envelope(SelectChildName(test,MethodCall(public abstract java.lang.String com.markit.bonds.akka.ReconnectFailExample$ExampleService.hello(),null)),Actor[akka.tcp://cli...@127.0.0.1:49659/temp/$a])
[2014-01-27 16:45:59,561] [DEBUG] [example-akka.actor.default-dispatcher-249] akka.remote.EndpointWriter - sending message RemoteMessage: [hello] to [Actor[akka.tcp://cli...@127.0.0.1:49659/temp/$a]]<+[akka.tcp://cli...@127.0.0.1:49659/temp/$a] from [Actor[akka://example/user/test/$a#-1283403612]]
[2014-01-27 16:45:59,561] [DEBUG] [example-akka.actor.default-dispatcher-249] akka.remote.EndpointWriter - sending message RemoteMessage: [hello] to [Actor[akka.tcp://cli...@127.0.0.1:49659/temp/$a]]<+[akka.tcp://cli...@127.0.0.1:49659/temp/$a] from [Actor[akka://example/user/test/$a#-1283403612]]
[2014-01-27 16:45:59,563] [DEBUG] [example-akka.actor.default-dispatcher-258] akka.serialization.Serialization(akka://example) - Using serializer[akka.serialization.JavaSerializer] for message [java.lang.String]
[2014-01-27 16:45:59,563] [DEBUG] [example-akka.actor.default-dispatcher-258] akka.serialization.Serialization(akka://example) - Using serializer[akka.serialization.JavaSerializer] for message [java.lang.String]
[2014-01-27 16:46:17,470] [DEBUG] [example-akka.actor.default-dispatcher-257] akka.remote.transport.ProtocolStateActor - stopped
[2014-01-27 16:46:17,470] [DEBUG] [example-akka.actor.default-dispatcher-257] akka.remote.transport.ProtocolStateActor - stopped
[2014-01-27 16:46:17,471] [DEBUG] [example-akka.actor.default-dispatcher-257] akka.remote.EndpointReader - stopped
[2014-01-27 16:46:17,471] [DEBUG] [example-akka.actor.default-dispatcher-257] akka.remote.EndpointReader - stopped
[2014-01-27 16:46:17,474] [DEBUG] [example-akka.actor.default-dispatcher-261] akka.remote.EndpointWriter - received AutoReceiveMessage Envelope(Terminated(Actor[akka://example/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-0/endpointWriter/endpointReader-akka.tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-0#1050380478]),Actor[akka://example/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-0/endpointWriter/endpointReader-akka.tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-0#1050380478])
[2014-01-27 16:46:17,474] [DEBUG] [example-akka.actor.default-dispatcher-261] akka.remote.EndpointWriter - received AutoReceiveMessage Envelope(Terminated(Actor[akka://example/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-0/endpointWriter/endpointReader-akka.tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-0#1050380478]),Actor[akka://example/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-0/endpointWriter/endpointReader-akka.tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-0#1050380478])
[2014-01-27 16:46:17,474] [INFO] [example-akka.actor.default-dispatcher-261] akka.actor.LocalActorRef - Message [akka.remote.transport.AssociationHandle$Disassociated] from Actor[akka://example/deadLetters] to Actor[akka://example/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fexample%4010.43.15.19%3A36084-1#1695231211] was not delivered. [1] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[2014-01-27 16:46:17,474] [INFO] [example-akka.actor.default-dispatcher-261] akka.actor.LocalActorRef - Message [akka.remote.transport.AssociationHandle$Disassociated] from Actor[akka://example/deadLetters] to Actor[akka://example/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fexample%4010.43.15.19%3A36084-1#1695231211] was not delivered. [1] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[2014-01-27 16:46:17,476] [DEBUG] [example-akka.actor.default-dispatcher-262] akka.remote.EndpointWriter - Disassociated [akka.tcp://example@server002:3552] <- [akka.tcp://cli...@127.0.0.1:49659]
[2014-01-27 16:46:17,476] [DEBUG] [example-akka.actor.default-dispatcher-262] akka.remote.EndpointWriter - Disassociated [akka.tcp://example@server002:3552] <- [akka.tcp://cli...@127.0.0.1:49659]
[2014-01-27 16:46:17,480] [DEBUG] [example-akka.actor.default-dispatcher-270] akka.remote.EndpointWriter - restarting
[2014-01-27 16:46:17,480] [DEBUG] [example-akka.actor.default-dispatcher-270] akka.remote.EndpointWriter - restarting
[2014-01-27 16:46:17,482] [DEBUG] [example-akka.actor.default-dispatcher-273] akka.remote.EndpointWriter - Disassociated [akka.tcp://example@server002:3552] <- [akka.tcp://cli...@127.0.0.1:49659]
[2014-01-27 16:46:17,482] [DEBUG] [example-akka.actor.default-dispatcher-273] akka.remote.EndpointWriter - Disassociated [akka.tcp://example@server002:3552] <- [akka.tcp://cli...@127.0.0.1:49659]
[2014-01-27 16:46:17,482] [INFO] [example-akka.actor.default-dispatcher-273] akka.actor.LocalActorRef - Message [akka.remote.transport.ActorTransportAdapter$DisassociateUnderlying] from Actor[akka://example/deadLetters] to Actor[akka://example/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fexample%4010.43.15.19%3A36084-1#1695231211] was not delivered. [2] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[2014-01-27 16:46:17,482] [INFO] [example-akka.actor.default-dispatcher-273] akka.actor.LocalActorRef - Message [akka.remote.transport.ActorTransportAdapter$DisassociateUnderlying] from Actor[akka://example/deadLetters] to Actor[akka://example/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fexample%4010.43.15.19%3A36084-1#1695231211] was not delivered. [2] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[2014-01-27 16:46:17,484] [DEBUG] [example-akka.actor.default-dispatcher-2] akka.remote.EndpointWriter - restarted
[2014-01-27 16:46:17,484] [DEBUG] [example-akka.actor.default-dispatcher-2] akka.remote.EndpointWriter - restarted
[2014-01-27 16:46:17,486] [DEBUG] [example-akka.actor.default-dispatcher-274] akka.remote.transport.AkkaProtocolManager - now supervising Actor[akka://example/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-2#-1488353550]
[2014-01-27 16:46:17,486] [DEBUG] [example-akka.actor.default-dispatcher-274] akka.remote.transport.AkkaProtocolManager - now supervising Actor[akka://example/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-2#-1488353550]
[2014-01-27 16:46:17,491] [DEBUG] [example-akka.actor.default-dispatcher-7] akka.remote.transport.ProtocolStateActor - started (akka.remote.transport.ProtocolStateActor@1e9af0b1)
[2014-01-27 16:46:17,491] [DEBUG] [example-akka.actor.default-dispatcher-7] akka.remote.transport.ProtocolStateActor - started (akka.remote.transport.ProtocolStateActor@1e9af0b1)
[2014-01-27 16:46:17,504] [DEBUG] [example-akka.actor.default-dispatcher-14] akka.remote.transport.ProtocolStateActor - stopped
[2014-01-27 16:46:17,504] [DEBUG] [example-akka.actor.default-dispatcher-14] akka.remote.transport.ProtocolStateActor - stopped
[2014-01-27 16:46:17,506] [ERROR] [example-akka.actor.default-dispatcher-16] akka.remote.EndpointWriter - AssociationError [akka.tcp://example@server002:3552] -> [akka.tcp://cli...@127.0.0.1:49659]: Error [Association failed with [akka.tcp://cli...@127.0.0.1:49659]] [
akka.remote.EndpointAssociationException: Association failed with [akka.tcp://cli...@127.0.0.1:49659]
Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: /127.0.0.1:49659
]
[2014-01-27 16:46:17,506] [ERROR] [example-akka.actor.default-dispatcher-16] akka.remote.EndpointWriter - AssociationError [akka.tcp://example@server002:3552] -> [akka.tcp://cli...@127.0.0.1:49659]: Error [Association failed with [akka.tcp://cli...@127.0.0.1:49659]] [
akka.remote.EndpointAssociationException: Association failed with [akka.tcp://cli...@127.0.0.1:49659]
Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: /127.0.0.1:49659
]
[2014-01-27 16:46:17,507] [DEBUG] [example-akka.actor.default-dispatcher-18] akka.remote.EndpointWriter - restarting
[2014-01-27 16:46:17,507] [DEBUG] [example-akka.actor.default-dispatcher-18] akka.remote.EndpointWriter - restarting
[2014-01-27 16:46:17,507] [DEBUG] [example-akka.actor.default-dispatcher-18] akka.remote.EndpointWriter - Disassociated [akka.tcp://example@server002:3552] -> [akka.tcp://cli...@127.0.0.1:49659]
[2014-01-27 16:46:17,507] [DEBUG] [example-akka.actor.default-dispatcher-18] akka.remote.EndpointWriter - Disassociated [akka.tcp://example@server002:3552] -> [akka.tcp://cli...@127.0.0.1:49659]
[2014-01-27 16:46:17,508] [DEBUG] [example-akka.actor.default-dispatcher-18] akka.remote.EndpointWriter - restarted
[2014-01-27 16:46:17,508] [DEBUG] [example-akka.actor.default-dispatcher-18] akka.remote.EndpointWriter - restarted
[2014-01-27 16:46:17,510] [DEBUG] [example-akka.actor.default-dispatcher-19] akka.remote.transport.AkkaProtocolManager - now supervising Actor[akka://example/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-3#1690936949]
[2014-01-27 16:46:17,510] [DEBUG] [example-akka.actor.default-dispatcher-19] akka.remote.transport.AkkaProtocolManager - now supervising Actor[akka://example/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-3#1690936949]
[2014-01-27 16:46:17,510] [DEBUG] [example-akka.actor.default-dispatcher-19] akka.remote.transport.ProtocolStateActor - started (akka.remote.transport.ProtocolStateActor@7463e563)
[2014-01-27 16:46:17,510] [DEBUG] [example-akka.actor.default-dispatcher-19] akka.remote.transport.ProtocolStateActor - started (akka.remote.transport.ProtocolStateActor@7463e563)
[2014-01-27 16:46:17,513] [DEBUG] [example-akka.actor.default-dispatcher-27] akka.remote.transport.ProtocolStateActor - stopped
[2014-01-27 16:46:17,513] [DEBUG] [example-akka.actor.default-dispatcher-27] akka.remote.transport.ProtocolStateActor - stopped
[2014-01-27 16:46:17,513] [ERROR] [example-akka.actor.default-dispatcher-241] akka.remote.EndpointWriter - AssociationError [akka.tcp://example@server002:3552] -> [akka.tcp://cli...@127.0.0.1:49659]: Error [Association failed with [akka.tcp://cli...@127.0.0.1:49659]] [
akka.remote.EndpointAssociationException: Association failed with [akka.tcp://cli...@127.0.0.1:49659]
Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: /127.0.0.1:49659
]
[2014-01-27 16:46:17,513] [ERROR] [example-akka.actor.default-dispatcher-241] akka.remote.EndpointWriter - AssociationError [akka.tcp://example@server002:3552] -> [akka.tcp://cli...@127.0.0.1:49659]: Error [Association failed with [akka.tcp://cli...@127.0.0.1:49659]] [
akka.remote.EndpointAssociationException: Association failed with [akka.tcp://cli...@127.0.0.1:49659]
Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: /127.0.0.1:49659
]
[2014-01-27 16:46:17,514] [DEBUG] [example-akka.actor.default-dispatcher-241] akka.remote.EndpointWriter - restarting
[2014-01-27 16:46:17,514] [DEBUG] [example-akka.actor.default-dispatcher-241] akka.remote.EndpointWriter - restarting
[2014-01-27 16:46:17,514] [DEBUG] [example-akka.actor.default-dispatcher-241] akka.remote.EndpointWriter - Disassociated [akka.tcp://example@server002:3552] -> [akka.tcp://cli...@127.0.0.1:49659]
[2014-01-27 16:46:17,514] [DEBUG] [example-akka.actor.default-dispatcher-241] akka.remote.EndpointWriter - Disassociated [akka.tcp://example@server002:3552] -> [akka.tcp://cli...@127.0.0.1:49659]
[2014-01-27 16:46:17,515] [DEBUG] [example-akka.actor.default-dispatcher-31] akka.remote.EndpointWriter - restarted
[2014-01-27 16:46:17,515] [DEBUG] [example-akka.actor.default-dispatcher-31] akka.remote.EndpointWriter - restarted
[2014-01-27 16:46:17,517] [DEBUG] [example-akka.actor.default-dispatcher-244] akka.remote.transport.AkkaProtocolManager - now supervising Actor[akka://example/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-4#1637151416]
[2014-01-27 16:46:17,517] [DEBUG] [example-akka.actor.default-dispatcher-244] akka.remote.transport.AkkaProtocolManager - now supervising Actor[akka://example/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-4#1637151416]
[2014-01-27 16:46:17,518] [DEBUG] [example-akka.actor.default-dispatcher-247] akka.remote.transport.ProtocolStateActor - started (akka.remote.transport.ProtocolStateActor@6070c38c)
[2014-01-27 16:46:17,518] [DEBUG] [example-akka.actor.default-dispatcher-247] akka.remote.transport.ProtocolStateActor - started (akka.remote.transport.ProtocolStateActor@6070c38c)
[2014-01-27 16:46:17,522] [DEBUG] [example-akka.actor.default-dispatcher-246] akka.remote.transport.ProtocolStateActor - stopped
[2014-01-27 16:46:17,522] [DEBUG] [example-akka.actor.default-dispatcher-246] akka.remote.transport.ProtocolStateActor - stopped
[2014-01-27 16:46:17,523] [ERROR] [example-akka.actor.default-dispatcher-246] akka.remote.EndpointWriter - AssociationError [akka.tcp://example@server002:3552] -> [akka.tcp://cli...@127.0.0.1:49659]: Error [Association failed with [akka.tcp://cli...@127.0.0.1:49659]] [
akka.remote.EndpointAssociationException: Association failed with [akka.tcp://cli...@127.0.0.1:49659]
Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: /127.0.0.1:49659
]
[2014-01-27 16:46:17,523] [ERROR] [example-akka.actor.default-dispatcher-246] akka.remote.EndpointWriter - AssociationError [akka.tcp://example@server002:3552] -> [akka.tcp://cli...@127.0.0.1:49659]: Error [Association failed with [akka.tcp://cli...@127.0.0.1:49659]] [
akka.remote.EndpointAssociationException: Association failed with [akka.tcp://cli...@127.0.0.1:49659]
Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: /127.0.0.1:49659
]
[2014-01-27 16:46:17,524] [DEBUG] [example-akka.actor.default-dispatcher-254] akka.remote.EndpointWriter - Disassociated [akka.tcp://example@server002:3552] -> [akka.tcp://cli...@127.0.0.1:49659]
[2014-01-27 16:46:17,524] [DEBUG] [example-akka.actor.default-dispatcher-254] akka.remote.EndpointWriter - Disassociated [akka.tcp://example@server002:3552] -> [akka.tcp://cli...@127.0.0.1:49659]
[2014-01-27 16:46:17,525] [DEBUG] [example-akka.actor.default-dispatcher-257] akka.remote.EndpointWriter - stopped
[2014-01-27 16:46:17,525] [DEBUG] [example-akka.actor.default-dispatcher-257] akka.remote.EndpointWriter - stopped
[2014-01-27 16:46:17,525] [DEBUG] [example-akka.actor.default-dispatcher-261] akka.remote.ReliableDeliverySupervisor - received AutoReceiveMessage Envelope(Terminated(Actor[akka://example/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-0/endpointWriter#-680224632]),Actor[akka://example/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-0/endpointWriter#-680224632])
[2014-01-27 16:46:17,525] [DEBUG] [example-akka.actor.default-dispatcher-261] akka.remote.ReliableDeliverySupervisor - received AutoReceiveMessage Envelope(Terminated(Actor[akka://example/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-0/endpointWriter#-680224632]),Actor[akka://example/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-0/endpointWriter#-680224632])
[2014-01-27 16:46:22,597] [DEBUG] [example-akka.actor.default-dispatcher-20] akka.remote.transport.AkkaProtocolManager - now supervising Actor[akka://example/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fexample%4010.43.15.19%3A36087-5#-1519118151]
[2014-01-27 16:46:22,597] [DEBUG] [example-akka.actor.default-dispatcher-20] akka.remote.transport.AkkaProtocolManager - now supervising Actor[akka://example/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fexample%4010.43.15.19%3A36087-5#-1519118151]
[2014-01-27 16:46:22,598] [DEBUG] [example-akka.actor.default-dispatcher-20] akka.remote.transport.ProtocolStateActor - started (akka.remote.transport.ProtocolStateActor@2945c761)
[2014-01-27 16:46:22,598] [DEBUG] [example-akka.actor.default-dispatcher-20] akka.remote.transport.ProtocolStateActor - started (akka.remote.transport.ProtocolStateActor@2945c761)
[2014-01-27 16:46:22,602] [DEBUG] [example-akka.actor.default-dispatcher-4] akka.remote.EndpointManager - now supervising Actor[akka://example/system/endpointManager/endpointWriter-akka.tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-1#-592860104]
[2014-01-27 16:46:22,602] [DEBUG] [example-akka.actor.default-dispatcher-4] akka.remote.EndpointManager - now supervising Actor[akka://example/system/endpointManager/endpointWriter-akka.tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-1#-592860104]
[2014-01-27 16:46:22,604] [DEBUG] [example-akka.actor.default-dispatcher-236] akka.remote.EndpointWriter - started (akka.remote.EndpointWriter@7b9bd735)
[2014-01-27 16:46:22,604] [DEBUG] [example-akka.actor.default-dispatcher-236] akka.remote.EndpointWriter - started (akka.remote.EndpointWriter@7b9bd735)
[2014-01-27 16:46:22,605] [DEBUG] [example-akka.actor.default-dispatcher-236] akka.remote.EndpointWriter - now monitoring Actor[akka://example/system/endpointManager#39348252]
[2014-01-27 16:46:22,605] [DEBUG] [example-akka.actor.default-dispatcher-236] akka.remote.EndpointWriter - now monitoring Actor[akka://example/system/endpointManager#39348252]
[2014-01-27 16:46:22,605] [DEBUG] [example-akka.actor.default-dispatcher-236] akka.remote.EndpointWriter - now supervising Actor[akka://example/system/endpointManager/endpointWriter-akka.tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-1/endpointReader-akka.tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-0#-1170796026]
[2014-01-27 16:46:22,605] [DEBUG] [example-akka.actor.default-dispatcher-236] akka.remote.EndpointWriter - now supervising Actor[akka://example/system/endpointManager/endpointWriter-akka.tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-1/endpointReader-akka.tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-0#-1170796026]
[2014-01-27 16:46:22,606] [DEBUG] [example-akka.actor.default-dispatcher-241] akka.remote.EndpointReader - started (akka.remote.EndpointReader@3951f910)
[2014-01-27 16:46:22,606] [DEBUG] [example-akka.actor.default-dispatcher-241] akka.remote.EndpointReader - started (akka.remote.EndpointReader@3951f910)
[2014-01-27 16:46:22,607] [DEBUG] [example-akka.actor.default-dispatcher-241] akka.remote.EndpointReader - now monitoring Actor[akka://example/system/endpointManager/endpointWriter-akka.tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-1#-592860104]
[2014-01-27 16:46:22,607] [DEBUG] [example-akka.actor.default-dispatcher-241] akka.remote.EndpointReader - now monitoring Actor[akka://example/system/endpointManager/endpointWriter-akka.tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-1#-592860104]
[2014-01-27 16:46:29,584] [DEBUG] [example-akka.actor.default-dispatcher-254] akka.remote.EndpointWriter - received local message RemoteMessage: [SelectChildName(user,SelectChildName(test,MethodCall(public abstract java.lang.String com.markit.bonds.akka.ReconnectFailExample$ExampleService.hello(),null)))] to [Actor[akka://example/]]<+[akka://example/] from [Actor[akka.tcp://cli...@127.0.0.1:49659/temp/$b]]
[2014-01-27 16:46:29,584] [DEBUG] [example-akka.actor.default-dispatcher-254] akka.remote.EndpointWriter - received local message RemoteMessage: [SelectChildName(user,SelectChildName(test,MethodCall(public abstract java.lang.String com.markit.bonds.akka.ReconnectFailExample$ExampleService.hello(),null)))] to [Actor[akka://example/]]<+[akka://example/] from [Actor[akka.tcp://cli...@127.0.0.1:49659/temp/$b]]
[2014-01-27 16:46:29,584] [DEBUG] [example-akka.actor.default-dispatcher-254] akka.actor.LocalActorRefProvider$Guardian - received AutoReceiveMessage Envelope(SelectChildName(user,SelectChildName(test,MethodCall(public abstract java.lang.String com.markit.bonds.akka.ReconnectFailExample$ExampleService.hello(),null))),Actor[akka.tcp://cli...@127.0.0.1:49659/temp/$b])
[2014-01-27 16:46:29,584] [DEBUG] [example-akka.actor.default-dispatcher-254] akka.actor.LocalActorRefProvider$Guardian - received AutoReceiveMessage Envelope(SelectChildName(user,SelectChildName(test,MethodCall(public abstract java.lang.String com.markit.bonds.akka.ReconnectFailExample$ExampleService.hello(),null))),Actor[akka.tcp://cli...@127.0.0.1:49659/temp/$b])
[2014-01-27 16:46:29,585] [DEBUG] [example-akka.actor.default-dispatcher-254] akka.actor.LocalActorRefProvider$Guardian - received AutoReceiveMessage Envelope(SelectChildName(test,MethodCall(public abstract java.lang.String com.markit.bonds.akka.ReconnectFailExample$ExampleService.hello(),null)),Actor[akka.tcp://cli...@127.0.0.1:49659/temp/$b])
[2014-01-27 16:46:29,585] [DEBUG] [example-akka.actor.default-dispatcher-254] akka.actor.LocalActorRefProvider$Guardian - received AutoReceiveMessage Envelope(SelectChildName(test,MethodCall(public abstract java.lang.String com.markit.bonds.akka.ReconnectFailExample$ExampleService.hello(),null)),Actor[akka.tcp://cli...@127.0.0.1:49659/temp/$b])
[2014-01-27 16:46:29,588] [DEBUG] [example-akka.actor.default-dispatcher-274] akka.remote.ReliableDeliverySupervisor - now supervising Actor[akka://example/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-0/endpointWriter#680874170]
[2014-01-27 16:46:29,588] [DEBUG] [example-akka.actor.default-dispatcher-274] akka.remote.ReliableDeliverySupervisor - now supervising Actor[akka://example/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-0/endpointWriter#680874170]
[2014-01-27 16:46:29,589] [DEBUG] [example-akka.actor.default-dispatcher-6] akka.remote.EndpointWriter - started (akka.remote.EndpointWriter@395fd251)
[2014-01-27 16:46:29,589] [DEBUG] [example-akka.actor.default-dispatcher-6] akka.remote.EndpointWriter - started (akka.remote.EndpointWriter@395fd251)
[2014-01-27 16:46:29,589] [DEBUG] [example-akka.actor.default-dispatcher-6] akka.remote.EndpointWriter - now monitoring Actor[akka://example/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-0#2034240752]
[2014-01-27 16:46:29,589] [DEBUG] [example-akka.actor.default-dispatcher-6] akka.remote.EndpointWriter - now monitoring Actor[akka://example/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-0#2034240752]
[2014-01-27 16:46:29,590] [DEBUG] [example-akka.actor.default-dispatcher-3] akka.remote.transport.AkkaProtocolManager - now supervising Actor[akka://example/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-6#1495903680]
[2014-01-27 16:46:29,590] [DEBUG] [example-akka.actor.default-dispatcher-3] akka.remote.transport.AkkaProtocolManager - now supervising Actor[akka://example/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-6#1495903680]
[2014-01-27 16:46:29,591] [DEBUG] [example-akka.actor.default-dispatcher-3] akka.remote.transport.ProtocolStateActor - started (akka.remote.transport.ProtocolStateActor@41e8e144)
[2014-01-27 16:46:29,591] [DEBUG] [example-akka.actor.default-dispatcher-3] akka.remote.transport.ProtocolStateActor - started (akka.remote.transport.ProtocolStateActor@41e8e144)
[2014-01-27 16:46:29,598] [DEBUG] [example-akka.actor.default-dispatcher-16] akka.remote.transport.ProtocolStateActor - stopped
[2014-01-27 16:46:29,598] [DEBUG] [example-akka.actor.default-dispatcher-16] akka.remote.transport.ProtocolStateActor - stopped
[2014-01-27 16:46:29,598] [ERROR] [example-akka.actor.default-dispatcher-18] akka.remote.EndpointWriter - AssociationError [akka.tcp://example@server002:3552] -> [akka.tcp://cli...@127.0.0.1:49659]: Error [Association failed with [akka.tcp://cli...@127.0.0.1:49659]] [
akka.remote.EndpointAssociationException: Association failed with [akka.tcp://cli...@127.0.0.1:49659]
Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: /127.0.0.1:49659
]
[2014-01-27 16:46:29,598] [ERROR] [example-akka.actor.default-dispatcher-18] akka.remote.EndpointWriter - AssociationError [akka.tcp://example@server002:3552] -> [akka.tcp://cli...@127.0.0.1:49659]: Error [Association failed with [akka.tcp://cli...@127.0.0.1:49659]] [
akka.remote.EndpointAssociationException: Association failed with [akka.tcp://cli...@127.0.0.1:49659]
Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: /127.0.0.1:49659
]
[2014-01-27 16:46:29,599] [DEBUG] [example-akka.actor.default-dispatcher-18] akka.remote.EndpointWriter - restarting
[2014-01-27 16:46:29,599] [DEBUG] [example-akka.actor.default-dispatcher-18] akka.remote.EndpointWriter - restarting
[2014-01-27 16:46:29,599] [DEBUG] [example-akka.actor.default-dispatcher-18] akka.remote.EndpointWriter - Disassociated [akka.tcp://example@server002:3552] -> [akka.tcp://cli...@127.0.0.1:49659]
[2014-01-27 16:46:29,599] [DEBUG] [example-akka.actor.default-dispatcher-18] akka.remote.EndpointWriter - Disassociated [akka.tcp://example@server002:3552] -> [akka.tcp://cli...@127.0.0.1:49659]
[2014-01-27 16:46:29,600] [DEBUG] [example-akka.actor.default-dispatcher-30] akka.remote.EndpointWriter - restarted
[2014-01-27 16:46:29,600] [DEBUG] [example-akka.actor.default-dispatcher-30] akka.remote.EndpointWriter - restarted
[2014-01-27 16:46:29,602] [DEBUG] [example-akka.actor.default-dispatcher-235] akka.remote.transport.AkkaProtocolManager - now supervising Actor[akka://example/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-7#2118989911]
[2014-01-27 16:46:29,602] [DEBUG] [example-akka.actor.default-dispatcher-235] akka.remote.transport.AkkaProtocolManager - now supervising Actor[akka://example/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-7#2118989911]
[2014-01-27 16:46:29,603] [DEBUG] [example-akka.actor.default-dispatcher-235] akka.remote.transport.ProtocolStateActor - started (akka.remote.transport.ProtocolStateActor@3e4ac866)
[2014-01-27 16:46:29,603] [DEBUG] [example-akka.actor.default-dispatcher-235] akka.remote.transport.ProtocolStateActor - started (akka.remote.transport.ProtocolStateActor@3e4ac866)
[2014-01-27 16:46:29,605] [DEBUG] [example-akka.actor.default-dispatcher-240] akka.remote.transport.ProtocolStateActor - stopped
[2014-01-27 16:46:29,605] [DEBUG] [example-akka.actor.default-dispatcher-240] akka.remote.transport.ProtocolStateActor - stopped
[2014-01-27 16:46:29,605] [ERROR] [example-akka.actor.default-dispatcher-240] akka.remote.EndpointWriter - AssociationError [akka.tcp://example@server002:3552] -> [akka.tcp://cli...@127.0.0.1:49659]: Error [Association failed with [akka.tcp://cli...@127.0.0.1:49659]] [
akka.remote.EndpointAssociationException: Association failed with [akka.tcp://cli...@127.0.0.1:49659]
Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: /127.0.0.1:49659
]
[2014-01-27 16:46:29,605] [ERROR] [example-akka.actor.default-dispatcher-240] akka.remote.EndpointWriter - AssociationError [akka.tcp://example@server002:3552] -> [akka.tcp://cli...@127.0.0.1:49659]: Error [Association failed with [akka.tcp://cli...@127.0.0.1:49659]] [
akka.remote.EndpointAssociationException: Association failed with [akka.tcp://cli...@127.0.0.1:49659]
Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: /127.0.0.1:49659
]
[2014-01-27 16:46:29,606] [DEBUG] [example-akka.actor.default-dispatcher-238] akka.remote.EndpointWriter - restarting
[2014-01-27 16:46:29,606] [DEBUG] [example-akka.actor.default-dispatcher-238] akka.remote.EndpointWriter - restarting
[2014-01-27 16:46:29,606] [DEBUG] [example-akka.actor.default-dispatcher-238] akka.remote.EndpointWriter - Disassociated [akka.tcp://example@server002:3552] -> [akka.tcp://cli...@127.0.0.1:49659]
[2014-01-27 16:46:29,606] [DEBUG] [example-akka.actor.default-dispatcher-238] akka.remote.EndpointWriter - Disassociated [akka.tcp://example@server002:3552] -> [akka.tcp://cli...@127.0.0.1:49659]
[2014-01-27 16:46:29,607] [DEBUG] [example-akka.actor.default-dispatcher-244] akka.remote.EndpointWriter - restarted
[2014-01-27 16:46:29,607] [DEBUG] [example-akka.actor.default-dispatcher-244] akka.remote.EndpointWriter - restarted
[2014-01-27 16:46:29,609] [DEBUG] [example-akka.actor.default-dispatcher-238] akka.remote.transport.AkkaProtocolManager - now supervising Actor[akka://example/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-8#-331254032]
[2014-01-27 16:46:29,609] [DEBUG] [example-akka.actor.default-dispatcher-238] akka.remote.transport.AkkaProtocolManager - now supervising Actor[akka://example/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-8#-331254032]
[2014-01-27 16:46:29,610] [DEBUG] [example-akka.actor.default-dispatcher-247] akka.remote.transport.ProtocolStateActor - started (akka.remote.transport.ProtocolStateActor@3a0b2c64)
[2014-01-27 16:46:29,610] [DEBUG] [example-akka.actor.default-dispatcher-247] akka.remote.transport.ProtocolStateActor - started (akka.remote.transport.ProtocolStateActor@3a0b2c64)
[2014-01-27 16:46:29,612] [DEBUG] [example-akka.actor.default-dispatcher-261] akka.remote.transport.ProtocolStateActor - stopped
[2014-01-27 16:46:29,612] [DEBUG] [example-akka.actor.default-dispatcher-261] akka.remote.transport.ProtocolStateActor - stopped
[2014-01-27 16:46:29,613] [ERROR] [example-akka.actor.default-dispatcher-261] akka.remote.EndpointWriter - AssociationError [akka.tcp://example@server002:3552] -> [akka.tcp://cli...@127.0.0.1:49659]: Error [Association failed with [akka.tcp://cli...@127.0.0.1:49659]] [
akka.remote.EndpointAssociationException: Association failed with [akka.tcp://cli...@127.0.0.1:49659]
Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: /127.0.0.1:49659
]
[2014-01-27 16:46:29,613] [ERROR] [example-akka.actor.default-dispatcher-261] akka.remote.EndpointWriter - AssociationError [akka.tcp://example@server002:3552] -> [akka.tcp://cli...@127.0.0.1:49659]: Error [Association failed with [akka.tcp://cli...@127.0.0.1:49659]] [
akka.remote.EndpointAssociationException: Association failed with [akka.tcp://cli...@127.0.0.1:49659]
Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: /127.0.0.1:49659
]
[2014-01-27 16:46:29,613] [DEBUG] [example-akka.actor.default-dispatcher-261] akka.remote.EndpointWriter - restarting
[2014-01-27 16:46:29,613] [DEBUG] [example-akka.actor.default-dispatcher-261] akka.remote.EndpointWriter - restarting
[2014-01-27 16:46:29,614] [DEBUG] [example-akka.actor.default-dispatcher-261] akka.remote.EndpointWriter - Disassociated [akka.tcp://example@server002:3552] -> [akka.tcp://cli...@127.0.0.1:49659]
[2014-01-27 16:46:29,614] [DEBUG] [example-akka.actor.default-dispatcher-261] akka.remote.EndpointWriter - Disassociated [akka.tcp://example@server002:3552] -> [akka.tcp://cli...@127.0.0.1:49659]
[2014-01-27 16:46:29,614] [DEBUG] [example-akka.actor.default-dispatcher-254] akka.remote.EndpointWriter - restarted
[2014-01-27 16:46:29,614] [DEBUG] [example-akka.actor.default-dispatcher-254] akka.remote.EndpointWriter - restarted
[2014-01-27 16:46:29,616] [DEBUG] [example-akka.actor.default-dispatcher-8] akka.remote.transport.AkkaProtocolManager - now supervising Actor[akka://example/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-9#-56509918]
[2014-01-27 16:46:29,616] [DEBUG] [example-akka.actor.default-dispatcher-8] akka.remote.transport.AkkaProtocolManager - now supervising Actor[akka://example/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-9#-56509918]
[2014-01-27 16:46:29,617] [DEBUG] [example-akka.actor.default-dispatcher-8] akka.remote.transport.ProtocolStateActor - started (akka.remote.transport.ProtocolStateActor@23d8417c)
[2014-01-27 16:46:29,617] [DEBUG] [example-akka.actor.default-dispatcher-8] akka.remote.transport.ProtocolStateActor - started (akka.remote.transport.ProtocolStateActor@23d8417c)
[2014-01-27 16:46:29,620] [DEBUG] [example-akka.actor.default-dispatcher-13] akka.remote.transport.ProtocolStateActor - stopped
[2014-01-27 16:46:29,620] [DEBUG] [example-akka.actor.default-dispatcher-13] akka.remote.transport.ProtocolStateActor - stopped
[2014-01-27 16:46:29,620] [ERROR] [example-akka.actor.default-dispatcher-13] akka.remote.EndpointWriter - AssociationError [akka.tcp://example@server002:3552] -> [akka.tcp://cli...@127.0.0.1:49659]: Error [Association failed with [akka.tcp://cli...@127.0.0.1:49659]] [
akka.remote.EndpointAssociationException: Association failed with [akka.tcp://cli...@127.0.0.1:49659]
Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: /127.0.0.1:49659
]
[2014-01-27 16:46:29,620] [ERROR] [example-akka.actor.default-dispatcher-13] akka.remote.EndpointWriter - AssociationError [akka.tcp://example@server002:3552] -> [akka.tcp://cli...@127.0.0.1:49659]: Error [Association failed with [akka.tcp://cli...@127.0.0.1:49659]] [
akka.remote.EndpointAssociationException: Association failed with [akka.tcp://cli...@127.0.0.1:49659]
Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: /127.0.0.1:49659
]
[2014-01-27 16:46:29,621] [DEBUG] [example-akka.actor.default-dispatcher-13] akka.remote.EndpointWriter - Disassociated [akka.tcp://example@server002:3552] -> [akka.tcp://cli...@127.0.0.1:49659]
[2014-01-27 16:46:29,621] [DEBUG] [example-akka.actor.default-dispatcher-13] akka.remote.EndpointWriter - Disassociated [akka.tcp://example@server002:3552] -> [akka.tcp://cli...@127.0.0.1:49659]
[2014-01-27 16:46:29,621] [DEBUG] [example-akka.actor.default-dispatcher-13] akka.remote.EndpointWriter - stopped
[2014-01-27 16:46:29,621] [DEBUG] [example-akka.actor.default-dispatcher-13] akka.remote.EndpointWriter - stopped
[2014-01-27 16:46:29,621] [INFO] [example-akka.actor.default-dispatcher-13] akka.remote.RemoteActorRefProvider$RemoteDeadLetterActorRef - Message [java.lang.String] from Actor[akka://example/user/test/$b#-1937476943] to Actor[akka://example/deadLetters] was not delivered. [3] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[2014-01-27 16:46:29,621] [INFO] [example-akka.actor.default-dispatcher-13] akka.remote.RemoteActorRefProvider$RemoteDeadLetterActorRef - Message [java.lang.String] from Actor[akka://example/user/test/$b#-1937476943] to Actor[akka://example/deadLetters] was not delivered. [3] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[2014-01-27 16:46:29,622] [DEBUG] [example-akka.actor.default-dispatcher-21] akka.remote.ReliableDeliverySupervisor - received AutoReceiveMessage Envelope(Terminated(Actor[akka://example/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-0/endpointWriter#680874170]),Actor[akka://example/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-0/endpointWriter#680874170])
[2014-01-27 16:46:29,622] [DEBUG] [example-akka.actor.default-dispatcher-21] akka.remote.ReliableDeliverySupervisor - received AutoReceiveMessage Envelope(Terminated(Actor[akka://example/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-0/endpointWriter#680874170]),Actor[akka://example/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-0/endpointWriter#680874170])
[2014-01-27 16:46:45,536] [DEBUG] [example-akka.actor.default-dispatcher-19] akka.remote.transport.ProtocolStateActor - stopped
[2014-01-27 16:46:45,536] [DEBUG] [example-akka.actor.default-dispatcher-19] akka.remote.transport.ProtocolStateActor - stopped
[2014-01-27 16:46:45,536] [DEBUG] [example-akka.actor.default-dispatcher-19] akka.remote.EndpointReader - stopped
[2014-01-27 16:46:45,536] [DEBUG] [example-akka.actor.default-dispatcher-19] akka.remote.EndpointReader - stopped
[2014-01-27 16:46:45,537] [DEBUG] [example-akka.actor.default-dispatcher-241] akka.remote.EndpointWriter - received AutoReceiveMessage Envelope(Terminated(Actor[akka://example/system/endpointManager/endpointWriter-akka.tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-1/endpointReader-akka.tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-0#-1170796026]),Actor[akka://example/system/endpointManager/endpointWriter-akka.tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-1/endpointReader-akka.tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-0#-1170796026])
[2014-01-27 16:46:45,537] [DEBUG] [example-akka.actor.default-dispatcher-241] akka.remote.EndpointWriter - received AutoReceiveMessage Envelope(Terminated(Actor[akka://example/system/endpointManager/endpointWriter-akka.tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-1/endpointReader-akka.tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-0#-1170796026]),Actor[akka://example/system/endpointManager/endpointWriter-akka.tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-1/endpointReader-akka.tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-0#-1170796026])
[2014-01-27 16:46:45,537] [DEBUG] [example-akka.actor.default-dispatcher-241] akka.remote.EndpointWriter - Disassociated [akka.tcp://example@server002:3552] <- [akka.tcp://cli...@127.0.0.1:49659]
[2014-01-27 16:46:45,537] [DEBUG] [example-akka.actor.default-dispatcher-241] akka.remote.EndpointWriter - Disassociated [akka.tcp://example@server002:3552] <- [akka.tcp://cli...@127.0.0.1:49659]
[2014-01-27 16:46:45,538] [DEBUG] [example-akka.actor.default-dispatcher-29] akka.remote.EndpointWriter - Disassociated [akka.tcp://example@server002:3552] <- [akka.tcp://cli...@127.0.0.1:49659]
[2014-01-27 16:46:45,538] [DEBUG] [example-akka.actor.default-dispatcher-29] akka.remote.EndpointWriter - Disassociated [akka.tcp://example@server002:3552] <- [akka.tcp://cli...@127.0.0.1:49659]
[2014-01-27 16:46:45,540] [INFO] [example-akka.actor.default-dispatcher-29] akka.actor.LocalActorRef - Message [akka.remote.transport.ActorTransportAdapter$DisassociateUnderlying] from Actor[akka://example/deadLetters] to Actor[akka://example/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fexample%4010.43.15.19%3A36087-5#-1519118151] was not delivered. [4] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[2014-01-27 16:46:45,540] [INFO] [example-akka.actor.default-dispatcher-29] akka.actor.LocalActorRef - Message [akka.remote.transport.ActorTransportAdapter$DisassociateUnderlying] from Actor[akka://example/deadLetters] to Actor[akka://example/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fexample%4010.43.15.19%3A36087-5#-1519118151] was not delivered. [4] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[2014-01-27 16:46:45,540] [DEBUG] [example-akka.actor.default-dispatcher-29] akka.remote.EndpointWriter - stopped
[2014-01-27 16:46:45,540] [DEBUG] [example-akka.actor.default-dispatcher-29] akka.remote.EndpointWriter - stopped
[2014-01-27 16:46:45,541] [DEBUG] [example-akka.actor.default-dispatcher-29] akka.remote.EndpointManager - received AutoReceiveMessage Envelope(Terminated(Actor[akka://example/system/endpointManager/endpointWriter-akka.tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-1#-592860104]),Actor[akka://example/system/endpointManager/endpointWriter-akka.tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-1#-592860104])
[2014-01-27 16:46:45,541] [DEBUG] [example-akka.actor.default-dispatcher-29] akka.remote.EndpointManager - received AutoReceiveMessage Envelope(Terminated(Actor[akka://example/system/endpointManager/endpointWriter-akka.tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-1#-592860104]),Actor[akka://example/system/endpointManager/endpointWriter-akka.tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-1#-592860104])
System sending to the TypedActor
[2014-01-27 16:45:59,065] [INFO] [client-akka.actor.default-dispatcher-9] akka.event.slf4j.Slf4jLogger - Slf4jLogger started
[2014-01-27 16:45:59,065] [INFO] [client-akka.actor.default-dispatcher-9] akka.event.slf4j.Slf4jLogger - Slf4jLogger started
[2014-01-27 16:45:59,076] [DEBUG] [client-akka.actor.default-dispatcher-10] akka.event.EventStream - logger log1-Slf4jLogger started
[2014-01-27 16:45:59,076] [DEBUG] [client-akka.actor.default-dispatcher-10] akka.event.EventStream - logger log1-Slf4jLogger started
[2014-01-27 16:45:59,077] [DEBUG] [client-akka.actor.default-dispatcher-13] akka.event.EventStream - Default Loggers started
[2014-01-27 16:45:59,077] [DEBUG] [client-akka.actor.default-dispatcher-13] akka.event.EventStream - Default Loggers started
[2014-01-27 16:45:59,078] [DEBUG] [client-akka.actor.default-dispatcher-14] akka.actor.LocalActorRefProvider$SystemGuardian - now supervising Actor[akka://client/system/remoting-terminator#358201843]
[2014-01-27 16:45:59,078] [DEBUG] [client-akka.actor.default-dispatcher-14] akka.actor.LocalActorRefProvider$SystemGuardian - now supervising Actor[akka://client/system/remoting-terminator#358201843]
[2014-01-27 16:45:59,091] [DEBUG] [client-akka.actor.default-dispatcher-15] akka.remote.RemoteActorRefProvider$RemotingTerminator - started (akka.remote.RemoteActorRefProvider$RemotingTerminator@319c0bd6)
[2014-01-27 16:45:59,091] [DEBUG] [client-akka.actor.default-dispatcher-15] akka.remote.RemoteActorRefProvider$RemotingTerminator - started (akka.remote.RemoteActorRefProvider$RemotingTerminator@319c0bd6)
[2014-01-27 16:45:59,100] [DEBUG] [client-akka.actor.default-dispatcher-20] akka.actor.LocalActorRefProvider$SystemGuardian - now supervising Actor[akka://client/system/transports#429508472]
[2014-01-27 16:45:59,100] [DEBUG] [client-akka.actor.default-dispatcher-20] akka.actor.LocalActorRefProvider$SystemGuardian - now supervising Actor[akka://client/system/transports#429508472]
[2014-01-27 16:45:59,100] [DEBUG] [client-akka.actor.default-dispatcher-21] akka.remote.Remoting$TransportSupervisor - started (akka.remote.Remoting$TransportSupervisor@1494cb8b)
[2014-01-27 16:45:59,100] [DEBUG] [client-akka.actor.default-dispatcher-21] akka.remote.Remoting$TransportSupervisor - started (akka.remote.Remoting$TransportSupervisor@1494cb8b)
[2014-01-27 16:45:59,105] [DEBUG] [client-akka.actor.default-dispatcher-24] akka.actor.LocalActorRefProvider$SystemGuardian - now supervising Actor[akka://client/system/endpointManager#237680297]
[2014-01-27 16:45:59,105] [DEBUG] [client-akka.actor.default-dispatcher-24] akka.actor.LocalActorRefProvider$SystemGuardian - now supervising Actor[akka://client/system/endpointManager#237680297]
[2014-01-27 16:45:59,106] [DEBUG] [client-akka.actor.default-dispatcher-26] akka.remote.RemoteActorRefProvider$RemotingTerminator - now monitoring Actor[akka://client/system]
[2014-01-27 16:45:59,106] [DEBUG] [client-akka.actor.default-dispatcher-26] akka.remote.RemoteActorRefProvider$RemotingTerminator - now monitoring Actor[akka://client/system]
[2014-01-27 16:45:59,114] [DEBUG] [client-akka.actor.default-dispatcher-27] akka.remote.EndpointManager - started (akka.remote.EndpointManager@c569c60)
[2014-01-27 16:45:59,114] [DEBUG] [client-akka.actor.default-dispatcher-27] akka.remote.EndpointManager - started (akka.remote.EndpointManager@c569c60)
[2014-01-27 16:45:59,278] [DEBUG] [client-akka.actor.default-dispatcher-139] akka.remote.Remoting$TransportSupervisor - now supervising Actor[akka://client/system/transports/akkaprotocolmanager.tcp0#674870860]
[2014-01-27 16:45:59,278] [DEBUG] [client-akka.actor.default-dispatcher-139] akka.remote.Remoting$TransportSupervisor - now supervising Actor[akka://client/system/transports/akkaprotocolmanager.tcp0#674870860]
[2014-01-27 16:45:59,280] [DEBUG] [client-akka.actor.default-dispatcher-140] akka.remote.transport.AkkaProtocolManager - started (akka.remote.transport.AkkaProtocolManager@72813bc1)
[2014-01-27 16:45:59,280] [DEBUG] [client-akka.actor.default-dispatcher-140] akka.remote.transport.AkkaProtocolManager - started (akka.remote.transport.AkkaProtocolManager@72813bc1)
[2014-01-27 16:45:59,292] [DEBUG] [client-akka.actor.default-dispatcher-148] akka.actor.LocalActorRefProvider$SystemGuardian - now supervising Actor[akka://client/system/remote-watcher#1665380928]
[2014-01-27 16:45:59,292] [DEBUG] [client-akka.actor.default-dispatcher-148] akka.actor.LocalActorRefProvider$SystemGuardian - now supervising Actor[akka://client/system/remote-watcher#1665380928]
[2014-01-27 16:45:59,293] [DEBUG] [client-akka.actor.default-dispatcher-151] akka.actor.LocalActorRefProvider$SystemGuardian - now supervising Actor[akka://client/system/remote-deployment-watcher#-59029643]
[2014-01-27 16:45:59,293] [DEBUG] [client-akka.actor.default-dispatcher-151] akka.actor.LocalActorRefProvider$SystemGuardian - now supervising Actor[akka://client/system/remote-deployment-watcher#-59029643]
[2014-01-27 16:45:59,293] [DEBUG] [client-akka.actor.default-dispatcher-152] akka.actor.LocalActorRefProvider$SystemGuardian - now supervising Actor[akka://client/system/deadLetterListener#959546158]
[2014-01-27 16:45:59,293] [DEBUG] [client-akka.actor.default-dispatcher-152] akka.actor.LocalActorRefProvider$SystemGuardian - now supervising Actor[akka://client/system/deadLetterListener#959546158]
[2014-01-27 16:45:59,295] [DEBUG] [client-akka.actor.default-dispatcher-153] akka.remote.RemoteDeploymentWatcher - started (akka.remote.RemoteDeploymentWatcher@66da9ea4)
[2014-01-27 16:45:59,295] [DEBUG] [client-akka.actor.default-dispatcher-153] akka.remote.RemoteDeploymentWatcher - started (akka.remote.RemoteDeploymentWatcher@66da9ea4)
[2014-01-27 16:45:59,295] [DEBUG] [client-akka.actor.default-dispatcher-154] akka.event.DeadLetterListener - started (akka.event.DeadLetterListener@5374a6e2)
[2014-01-27 16:45:59,295] [DEBUG] [client-akka.actor.default-dispatcher-154] akka.event.DeadLetterListener - started (akka.event.DeadLetterListener@5374a6e2)
[2014-01-27 16:45:59,295] [DEBUG] [client-akka.actor.default-dispatcher-155] akka.remote.RemoteWatcher - started (akka.remote.RemoteWatcher@f786a3c)
[2014-01-27 16:45:59,295] [DEBUG] [client-akka.actor.default-dispatcher-155] akka.remote.RemoteWatcher - started (akka.remote.RemoteWatcher@f786a3c)
[2014-01-27 16:45:59,304] [DEBUG] [client-akka.actor.default-dispatcher-158] akka.actor.LocalActorRefProvider$Guardian - now supervising Actor[akka://client/user/$a#-580707033]
[2014-01-27 16:45:59,304] [DEBUG] [client-akka.actor.default-dispatcher-158] akka.actor.LocalActorRefProvider$Guardian - now supervising Actor[akka://client/user/$a#-580707033]
[2014-01-27 16:45:59,327] [DEBUG] [client-akka.actor.default-dispatcher-167] akka.remote.EndpointManager - now supervising Actor[akka://client/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fexample%40server002%3A3552-0#-560920394]
[2014-01-27 16:45:59,327] [DEBUG] [client-akka.actor.default-dispatcher-167] akka.remote.EndpointManager - now supervising Actor[akka://client/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fexample%40server002%3A3552-0#-560920394]
[2014-01-27 16:45:59,345] [DEBUG] [client-akka.actor.default-dispatcher-169] akka.remote.ReliableDeliverySupervisor - started (akka.remote.ReliableDeliverySupervisor@3ebfbbe3)
[2014-01-27 16:45:59,345] [DEBUG] [client-akka.actor.default-dispatcher-169] akka.remote.ReliableDeliverySupervisor - started (akka.remote.ReliableDeliverySupervisor@3ebfbbe3)
[2014-01-27 16:45:59,345] [DEBUG] [client-akka.actor.default-dispatcher-170] akka.remote.ReliableDeliverySupervisor - now monitoring Actor[akka://client/system/endpointManager#237680297]
[2014-01-27 16:45:59,345] [DEBUG] [client-akka.actor.default-dispatcher-170] akka.remote.ReliableDeliverySupervisor - now monitoring Actor[akka://client/system/endpointManager#237680297]
[2014-01-27 16:45:59,346] [DEBUG] [client-akka.actor.default-dispatcher-170] akka.remote.ReliableDeliverySupervisor - now supervising Actor[akka://client/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fexample%40server002%3A3552-0/endpointWriter#1386378256]
[2014-01-27 16:45:59,346] [DEBUG] [client-akka.actor.default-dispatcher-170] akka.remote.ReliableDeliverySupervisor - now supervising Actor[akka://client/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fexample%40server002%3A3552-0/endpointWriter#1386378256]
[2014-01-27 16:45:59,356] [DEBUG] [client-akka.actor.default-dispatcher-168] akka.remote.EndpointWriter - started (akka.remote.EndpointWriter@5b6b9e62)
[2014-01-27 16:45:59,356] [DEBUG] [client-akka.actor.default-dispatcher-168] akka.remote.EndpointWriter - started (akka.remote.EndpointWriter@5b6b9e62)
[2014-01-27 16:45:59,356] [DEBUG] [client-akka.actor.default-dispatcher-168] akka.remote.EndpointWriter - now monitoring Actor[akka://client/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fexample%40server002%3A3552-0#-560920394]
[2014-01-27 16:45:59,356] [DEBUG] [client-akka.actor.default-dispatcher-168] akka.remote.EndpointWriter - now monitoring Actor[akka://client/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fexample%40server002%3A3552-0#-560920394]
[2014-01-27 16:45:59,366] [DEBUG] [client-akka.actor.default-dispatcher-4] akka.remote.transport.AkkaProtocolManager - now supervising Actor[akka://client/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fexample%40server002%3A3552-1#245794225]
[2014-01-27 16:45:59,366] [DEBUG] [client-akka.actor.default-dispatcher-4] akka.remote.transport.AkkaProtocolManager - now supervising Actor[akka://client/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fexample%40server002%3A3552-1#245794225]
[2014-01-27 16:45:59,374] [DEBUG] [client-akka.actor.default-dispatcher-9] akka.remote.transport.ProtocolStateActor - started (akka.remote.transport.ProtocolStateActor@10c789fb)
[2014-01-27 16:45:59,374] [DEBUG] [client-akka.actor.default-dispatcher-9] akka.remote.transport.ProtocolStateActor - started (akka.remote.transport.ProtocolStateActor@10c789fb)
[2014-01-27 16:45:59,485] [DEBUG] [client-akka.actor.default-dispatcher-17] akka.remote.EndpointWriter - Associated [akka.tcp://cli...@127.0.0.1:49659] -> [akka.tcp://example@server002:3552]
[2014-01-27 16:45:59,485] [DEBUG] [client-akka.actor.default-dispatcher-17] akka.remote.EndpointWriter - Associated [akka.tcp://cli...@127.0.0.1:49659] -> [akka.tcp://example@server002:3552]
[2014-01-27 16:45:59,485] [DEBUG] [client-akka.actor.default-dispatcher-17] akka.remote.EndpointWriter - now supervising Actor[akka://client/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fexample%40server002%3A3552-0/endpointWriter/endpointReader-akka.tcp%3A%2F%2Fexample%40server002%3A3552-0#739192055]
[2014-01-27 16:45:59,485] [DEBUG] [client-akka.actor.default-dispatcher-17] akka.remote.EndpointWriter - now supervising Actor[akka://client/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fexample%40server002%3A3552-0/endpointWriter/endpointReader-akka.tcp%3A%2F%2Fexample%40server002%3A3552-0#739192055]
[2014-01-27 16:45:59,490] [DEBUG] [client-akka.actor.default-dispatcher-24] akka.remote.EndpointWriter - sending message RemoteMessage: [SelectChildName(user,SelectChildName(test,MethodCall(public abstract java.lang.String com.markit.bonds.akka.ReconnectFailExample$ExampleService.hello(),null)))] to [Actor[akka.tcp://example@server002:3552/]]<+[akka.tcp://example@server002:3552/] from [Actor[akka://client/temp/$a]]
[2014-01-27 16:45:59,490] [DEBUG] [client-akka.actor.default-dispatcher-24] akka.remote.EndpointWriter - sending message RemoteMessage: [SelectChildName(user,SelectChildName(test,MethodCall(public abstract java.lang.String com.markit.bonds.akka.ReconnectFailExample$ExampleService.hello(),null)))] to [Actor[akka.tcp://example@server002:3552/]]<+[akka.tcp://example@server002:3552/] from [Actor[akka://client/temp/$a]]
[2014-01-27 16:45:59,493] [DEBUG] [client-akka.actor.default-dispatcher-26] akka.remote.EndpointReader - started (akka.remote.EndpointReader@197ebe66)
[2014-01-27 16:45:59,493] [DEBUG] [client-akka.actor.default-dispatcher-26] akka.remote.EndpointReader - started (akka.remote.EndpointReader@197ebe66)
[2014-01-27 16:45:59,493] [DEBUG] [client-akka.actor.default-dispatcher-26] akka.remote.EndpointReader - now monitoring Actor[akka://client/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fexample%40server002%3A3552-0/endpointWriter#1386378256]
[2014-01-27 16:45:59,493] [DEBUG] [client-akka.actor.default-dispatcher-26] akka.remote.EndpointReader - now monitoring Actor[akka://client/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fexample%40server002%3A3552-0/endpointWriter#1386378256]
[2014-01-27 16:45:59,494] [DEBUG] [client-akka.actor.default-dispatcher-26] akka.serialization.Serialization(akka://client) - Using serializer[akka.remote.serialization.MessageContainerSerializer] for message [akka.actor.SelectChildName]
[2014-01-27 16:45:59,494] [DEBUG] [client-akka.actor.default-dispatcher-26] akka.serialization.Serialization(akka://client) - Using serializer[akka.remote.serialization.MessageContainerSerializer] for message [akka.actor.SelectChildName]
[2014-01-27 16:45:59,500] [DEBUG] [client-akka.actor.default-dispatcher-135] akka.serialization.Serialization(akka://client) - Using serializer[akka.serialization.JavaSerializer] for message [akka.actor.TypedActor$MethodCall]
[2014-01-27 16:45:59,500] [DEBUG] [client-akka.actor.default-dispatcher-135] akka.serialization.Serialization(akka://client) - Using serializer[akka.serialization.JavaSerializer] for message [akka.actor.TypedActor$MethodCall]
[2014-01-27 16:45:59,582] [DEBUG] [client-akka.actor.default-dispatcher-136] akka.remote.EndpointWriter - received local message RemoteMessage: [hello] to [Actor[akka://client/temp/$a]]<+[akka://client/temp/$a] from [Actor[akka.tcp://example@server002:3552/user/test/$a#-1283403612]]
[2014-01-27 16:45:59,582] [DEBUG] [client-akka.actor.default-dispatcher-136] akka.remote.EndpointWriter - received local message RemoteMessage: [hello] to [Actor[akka://client/temp/$a]]<+[akka://client/temp/$a] from [Actor[akka.tcp://example@server002:3552/user/test/$a#-1283403612]]
hello
[2014-01-27 16:46:22,591] [DEBUG] [client-akka.actor.default-dispatcher-169] akka.remote.EndpointReader - stopped
[2014-01-27 16:46:22,591] [DEBUG] [client-akka.actor.default-dispatcher-169] akka.remote.EndpointReader - stopped
[2014-01-27 16:46:22,593] [DEBUG] [client-akka.actor.default-dispatcher-6] akka.remote.transport.ProtocolStateActor - stopped
[2014-01-27 16:46:22,593] [DEBUG] [client-akka.actor.default-dispatcher-6] akka.remote.transport.ProtocolStateActor - stopped
[2014-01-27 16:46:22,593] [INFO] [client-akka.actor.default-dispatcher-6] akka.actor.LocalActorRef - Message [akka.remote.transport.AssociationHandle$Disassociated] from Actor[akka://client/deadLetters] to Actor[akka://client/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fexample%40server002%3A3552-1#245794225] was not delivered. [1] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[2014-01-27 16:46:22,593] [INFO] [client-akka.actor.default-dispatcher-6] akka.actor.LocalActorRef - Message [akka.remote.transport.AssociationHandle$Disassociated] from Actor[akka://client/deadLetters] to Actor[akka://client/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fexample%40server002%3A3552-1#245794225] was not delivered. [1] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[2014-01-27 16:46:22,593] [DEBUG] [client-akka.actor.default-dispatcher-6] akka.remote.EndpointWriter - received AutoReceiveMessage Envelope(Terminated(Actor[akka://client/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fexample%40server002%3A3552-0/endpointWriter/endpointReader-akka.tcp%3A%2F%2Fexample%40server002%3A3552-0#739192055]),Actor[akka://client/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fexample%40server002%3A3552-0/endpointWriter/endpointReader-akka.tcp%3A%2F%2Fexample%40server002%3A3552-0#739192055])
[2014-01-27 16:46:22,593] [DEBUG] [client-akka.actor.default-dispatcher-6] akka.remote.EndpointWriter - received AutoReceiveMessage Envelope(Terminated(Actor[akka://client/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fexample%40server002%3A3552-0/endpointWriter/endpointReader-akka.tcp%3A%2F%2Fexample%40server002%3A3552-0#739192055]),Actor[akka://client/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fexample%40server002%3A3552-0/endpointWriter/endpointReader-akka.tcp%3A%2F%2Fexample%40server002%3A3552-0#739192055])
[2014-01-27 16:46:22,594] [INFO] [client-akka.actor.default-dispatcher-6] akka.actor.LocalActorRef - Message [akka.actor.FSM$Timer] from Actor[akka://client/deadLetters] to Actor[akka://client/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fexample%40server002%3A3552-1#245794225] was not delivered. [2] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[2014-01-27 16:46:22,594] [INFO] [client-akka.actor.default-dispatcher-6] akka.actor.LocalActorRef - Message [akka.actor.FSM$Timer] from Actor[akka://client/deadLetters] to Actor[akka://client/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fexample%40server002%3A3552-1#245794225] was not delivered. [2] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[2014-01-27 16:46:22,594] [DEBUG] [client-akka.actor.default-dispatcher-6] akka.remote.EndpointWriter - Disassociated [akka.tcp://cli...@127.0.0.1:49659] -> [akka.tcp://example@server002:3552]
[2014-01-27 16:46:22,594] [DEBUG] [client-akka.actor.default-dispatcher-6] akka.remote.EndpointWriter - Disassociated [akka.tcp://cli...@127.0.0.1:49659] -> [akka.tcp://example@server002:3552]
[2014-01-27 16:46:22,598] [DEBUG] [client-akka.actor.default-dispatcher-22] akka.remote.EndpointWriter - restarting
[2014-01-27 16:46:22,598] [DEBUG] [client-akka.actor.default-dispatcher-22] akka.remote.EndpointWriter - restarting
[2014-01-27 16:46:22,599] [DEBUG] [client-akka.actor.default-dispatcher-17] akka.remote.EndpointWriter - Disassociated [akka.tcp://cli...@127.0.0.1:49659] -> [akka.tcp://example@server002:3552]
[2014-01-27 16:46:22,599] [DEBUG] [client-akka.actor.default-dispatcher-17] akka.remote.EndpointWriter - Disassociated [akka.tcp://cli...@127.0.0.1:49659] -> [akka.tcp://example@server002:3552]
[2014-01-27 16:46:22,599] [INFO] [client-akka.actor.default-dispatcher-17] akka.actor.LocalActorRef - Message [akka.remote.transport.ActorTransportAdapter$DisassociateUnderlying] from Actor[akka://client/deadLetters] to Actor[akka://client/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fexample%40server002%3A3552-1#245794225] was not delivered. [3] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[2014-01-27 16:46:22,599] [INFO] [client-akka.actor.default-dispatcher-17] akka.actor.LocalActorRef - Message [akka.remote.transport.ActorTransportAdapter$DisassociateUnderlying] from Actor[akka://client/deadLetters] to Actor[akka://client/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fexample%40server002%3A3552-1#245794225] was not delivered. [3] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[2014-01-27 16:46:22,600] [DEBUG] [client-akka.actor.default-dispatcher-18] akka.remote.EndpointWriter - restarted
[2014-01-27 16:46:22,600] [DEBUG] [client-akka.actor.default-dispatcher-18] akka.remote.EndpointWriter - restarted
[2014-01-27 16:46:22,603] [DEBUG] [client-akka.actor.default-dispatcher-23] akka.remote.transport.AkkaProtocolManager - now supervising Actor[akka://client/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fexample%40server002%3A3552-2#-226377435]
[2014-01-27 16:46:22,603] [DEBUG] [client-akka.actor.default-dispatcher-23] akka.remote.transport.AkkaProtocolManager - now supervising Actor[akka://client/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fexample%40server002%3A3552-2#-226377435]
[2014-01-27 16:46:22,604] [DEBUG] [client-akka.actor.default-dispatcher-23] akka.remote.transport.ProtocolStateActor - started (akka.remote.transport.ProtocolStateActor@707efa96)
[2014-01-27 16:46:22,604] [DEBUG] [client-akka.actor.default-dispatcher-23] akka.remote.transport.ProtocolStateActor - started (akka.remote.transport.ProtocolStateActor@707efa96)
[2014-01-27 16:46:22,612] [DEBUG] [client-akka.actor.default-dispatcher-147] akka.remote.EndpointWriter - Associated [akka.tcp://cli...@127.0.0.1:49659] -> [akka.tcp://example@server002:3552]
[2014-01-27 16:46:22,612] [DEBUG] [client-akka.actor.default-dispatcher-147] akka.remote.EndpointWriter - Associated [akka.tcp://cli...@127.0.0.1:49659] -> [akka.tcp://example@server002:3552]
[2014-01-27 16:46:22,613] [DEBUG] [client-akka.actor.default-dispatcher-147] akka.remote.EndpointWriter - now supervising Actor[akka://client/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fexample%40server002%3A3552-0/endpointWriter/endpointReader-akka.tcp%3A%2F%2Fexample%40server002%3A3552-0#125954714]
[2014-01-27 16:46:22,613] [DEBUG] [client-akka.actor.default-dispatcher-147] akka.remote.EndpointWriter - now supervising Actor[akka://client/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fexample%40server002%3A3552-0/endpointWriter/endpointReader-akka.tcp%3A%2F%2Fexample%40server002%3A3552-0#125954714]
[2014-01-27 16:46:22,615] [DEBUG] [client-akka.actor.default-dispatcher-152] akka.remote.EndpointReader - started (akka.remote.EndpointReader@7e8905bd)
[2014-01-27 16:46:22,615] [DEBUG] [client-akka.actor.default-dispatcher-152] akka.remote.EndpointReader - started (akka.remote.EndpointReader@7e8905bd)
[2014-01-27 16:46:22,615] [DEBUG] [client-akka.actor.default-dispatcher-152] akka.remote.EndpointReader - now monitoring Actor[akka://client/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fexample%40server002%3A3552-0/endpointWriter#1386378256]
[2014-01-27 16:46:22,615] [DEBUG] [client-akka.actor.default-dispatcher-152] akka.remote.EndpointReader - now monitoring Actor[akka://client/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fexample%40server002%3A3552-0/endpointWriter#1386378256]
[2014-01-27 16:46:29,592] [DEBUG] [client-akka.actor.default-dispatcher-7] akka.remote.EndpointWriter - sending message RemoteMessage: [SelectChildName(user,SelectChildName(test,MethodCall(public abstract java.lang.String com.markit.bonds.akka.ReconnectFailExample$ExampleService.hello(),null)))] to [Actor[akka.tcp://example@server002:3552/]]<+[akka.tcp://example@server002:3552/] from [Actor[akka://client/temp/$b]]
[2014-01-27 16:46:29,592] [DEBUG] [client-akka.actor.default-dispatcher-7] akka.remote.EndpointWriter - sending message RemoteMessage: [SelectChildName(user,SelectChildName(test,MethodCall(public abstract java.lang.String com.markit.bonds.akka.ReconnectFailExample$ExampleService.hello(),null)))] to [Actor[akka.tcp://example@server002:3552/]]<+[akka.tcp://example@server002:3552/] from [Actor[akka://client/temp/$b]]
Think I can reliably recreate the problem. You need two linux boxes. On one create a actor system with a typed actor bound to an endpoint. I just had a simple one method that returned the string hello when the method was called with no arguments.
On the the the other create an actor system that calls the method. Call the method once then sleep for 30 seconds and call it again. Simulate a slow system by after the first method is called and a response is succesfully received suspend the process on the calling system i.e. Ctrl-Z then after10 seconds then but it back in the foreground.
Logs below
TypedActor Endpoint
[2014-01-27 16:45:43,780] [INFO] [example-akka.actor.default-dispatcher-9] akka.event.slf4j.Slf4jLogger - Slf4jLogger started
[2014-01-27 16:45:43,780] [INFO] [example-akka.actor.default-dispatcher-9] akka.event.slf4j.Slf4jLogger - Slf4jLogger started
[2014-01-27 16:45:43,794] [DEBUG] [example-akka.actor.default-dispatcher-11] akka.event.EventStream - logger log1-Slf4jLogger started
[2014-01-27 16:45:43,794] [DEBUG] [example-akka.actor.default-dispatcher-11] akka.event.EventStream - logger log1-Slf4jLogger started
[2014-01-27 16:45:43,795] [DEBUG] [example-akka.actor.default-dispatcher-14] akka.event.EventStream - Default Loggers started
[2014-01-27 16:45:43,795] [DEBUG] [example-akka.actor.default-dispatcher-14] akka.event.EventStream - Default Loggers started
[2014-01-27 16:45:43,795] [DEBUG] [example-akka.actor.default-dispatcher-16] akka.actor.LocalActorRefProvider$SystemGuardian - now supervising Actor[akka://example/system/UnhandledMessageForwarder#-2027631171]
[2014-01-27 16:45:43,795] [DEBUG] [example-akka.actor.default-dispatcher-16] akka.actor.LocalActorRefProvider$SystemGuardian - now supervising Actor[akka://example/system/UnhandledMessageForwarder#-2027631171]
[2014-01-27 16:45:43,796] [DEBUG] [example-akka.actor.default-dispatcher-17] akka.event.LoggingBus$$anonfun$startDefaultLoggers$2$$anon$1 - started (akka.event.LoggingBus$$anonfun$startDefaultLoggers$2$$anon$1@435db13f)
[2014-01-27 16:45:43,796] [DEBUG] [example-akka.actor.default-dispatcher-17] akka.event.LoggingBus$$anonfun$startDefaultLoggers$2$$anon$1 - started (akka.event.LoggingBus$$anonfun$startDefaultLoggers$2$$anon$1@435db13f)
[2014-01-27 16:45:43,796] [DEBUG] [example-akka.actor.default-dispatcher-18] akka.actor.LocalActorRefProvider$SystemGuardian - now supervising Actor[akka://example/system/remoting-terminator#-658149313]
[2014-01-27 16:45:43,796] [DEBUG] [example-akka.actor.default-dispatcher-18] akka.actor.LocalActorRefProvider$SystemGuardian - now supervising Actor[akka://example/system/remoting-terminator#-658149313]
[2014-01-27 16:45:43,809] [DEBUG] [example-akka.actor.default-dispatcher-19] akka.remote.RemoteActorRefProvider$RemotingTerminator - started (akka.remote.RemoteActorRefProvider$RemotingTerminator@5bab2b55)
[2014-01-27 16:45:43,809] [DEBUG] [example-akka.actor.default-dispatcher-19] akka.remote.RemoteActorRefProvider$RemotingTerminator - started (akka.remote.RemoteActorRefProvider$RemotingTerminator@5bab2b55)
[2014-01-27 16:45:43,821] [DEBUG] [example-akka.actor.default-dispatcher-24] akka.actor.LocalActorRefProvider$SystemGuardian - now supervising Actor[akka://example/system/transports#128266947]
[2014-01-27 16:45:43,821] [DEBUG] [example-akka.actor.default-dispatcher-24] akka.actor.LocalActorRefProvider$SystemGuardian - now supervising Actor[akka://example/system/transports#128266947]
[2014-01-27 16:45:43,822] [DEBUG] [example-akka.actor.default-dispatcher-26] akka.remote.Remoting$TransportSupervisor - started (akka.remote.Remoting$TransportSupervisor@c569c60)
[2014-01-27 16:45:43,822] [DEBUG] [example-akka.actor.default-dispatcher-26] akka.remote.Remoting$TransportSupervisor - started (akka.remote.Remoting$TransportSupervisor@c569c60)
[2014-01-27 16:45:43,824] [DEBUG] [example-akka.actor.default-dispatcher-29] akka.remote.RemoteActorRefProvider$RemotingTerminator - now monitoring Actor[akka://example/system]
[2014-01-27 16:45:43,824] [DEBUG] [example-akka.actor.default-dispatcher-29] akka.remote.RemoteActorRefProvider$RemotingTerminator - now monitoring Actor[akka://example/system]
[2014-01-27 16:45:43,825] [DEBUG] [example-akka.actor.default-dispatcher-30] akka.actor.LocalActorRefProvider$SystemGuardian - now supervising Actor[akka://example/system/endpointManager#39348252]
[2014-01-27 16:45:43,825] [DEBUG] [example-akka.actor.default-dispatcher-30] akka.actor.LocalActorRefProvider$SystemGuardian - now supervising Actor[akka://example/system/endpointManager#39348252]
[2014-01-27 16:45:43,835] [DEBUG] [example-akka.actor.default-dispatcher-31] akka.remote.EndpointManager - started (akka.remote.EndpointManager@32602b6b)
[2014-01-27 16:45:43,835] [DEBUG] [example-akka.actor.default-dispatcher-31] akka.remote.EndpointManager - started (akka.remote.EndpointManager@32602b6b)
[2014-01-27 16:45:44,059] [DEBUG] [example-akka.actor.default-dispatcher-241] akka.remote.Remoting$TransportSupervisor - now supervising Actor[akka://example/system/transports/akkaprotocolmanager.tcp0#-1577819875]
[2014-01-27 16:45:44,059] [DEBUG] [example-akka.actor.default-dispatcher-241] akka.remote.Remoting$TransportSupervisor - now supervising Actor[akka://example/system/transports/akkaprotocolmanager.tcp0#-1577819875]
[2014-01-27 16:45:44,061] [DEBUG] [example-akka.actor.default-dispatcher-243] akka.remote.transport.AkkaProtocolManager - started (akka.remote.transport.AkkaProtocolManager@2d42c778)
[2014-01-27 16:45:44,061] [DEBUG] [example-akka.actor.default-dispatcher-243] akka.remote.transport.AkkaProtocolManager - started (akka.remote.transport.AkkaProtocolManager@2d42c778)
[2014-01-27 16:45:44,075] [DEBUG] [example-akka.actor.default-dispatcher-250] akka.actor.LocalActorRefProvider$SystemGuardian - now supervising Actor[akka://example/system/remote-watcher#-1405704646]
[2014-01-27 16:45:44,075] [DEBUG] [example-akka.actor.default-dispatcher-250] akka.actor.LocalActorRefProvider$SystemGuardian - now supervising Actor[akka://example/system/remote-watcher#-1405704646]
[2014-01-27 16:45:44,076] [DEBUG] [example-akka.actor.default-dispatcher-253] akka.actor.LocalActorRefProvider$SystemGuardian - now supervising Actor[akka://example/system/remote-deployment-watcher#1221709328]
[2014-01-27 16:45:44,076] [DEBUG] [example-akka.actor.default-dispatcher-253] akka.actor.LocalActorRefProvider$SystemGuardian - now supervising Actor[akka://example/system/remote-deployment-watcher#1221709328]
[2014-01-27 16:45:44,077] [DEBUG] [example-akka.actor.default-dispatcher-254] akka.actor.LocalActorRefProvider$SystemGuardian - now supervising Actor[akka://example/system/deadLetterListener#1681690402]
[2014-01-27 16:45:44,077] [DEBUG] [example-akka.actor.default-dispatcher-254] akka.actor.LocalActorRefProvider$SystemGuardian - now supervising Actor[akka://example/system/deadLetterListener#1681690402]
[2014-01-27 16:45:44,078] [DEBUG] [example-akka.actor.default-dispatcher-255] akka.event.DeadLetterListener - started (akka.event.DeadLetterListener@1ff243d1)
[2014-01-27 16:45:44,078] [DEBUG] [example-akka.actor.default-dispatcher-255] akka.event.DeadLetterListener - started (akka.event.DeadLetterListener@1ff243d1)
[2014-01-27 16:45:44,079] [DEBUG] [example-akka.actor.default-dispatcher-256] akka.remote.RemoteDeploymentWatcher - started (akka.remote.RemoteDeploymentWatcher@7458e2a1)
[2014-01-27 16:45:44,079] [DEBUG] [example-akka.actor.default-dispatcher-256] akka.remote.RemoteDeploymentWatcher - started (akka.remote.RemoteDeploymentWatcher@7458e2a1)
[2014-01-27 16:45:44,082] [DEBUG] [example-akka.actor.default-dispatcher-257] akka.remote.RemoteWatcher - started (akka.remote.RemoteWatcher@7b0bbd42)
[2014-01-27 16:45:44,082] [DEBUG] [example-akka.actor.default-dispatcher-257] akka.remote.RemoteWatcher - started (akka.remote.RemoteWatcher@7b0bbd42)
[2014-01-27 16:45:44,084] [DEBUG] [example-akka.actor.default-dispatcher-260] akka.actor.LocalActorRefProvider$Guardian - now supervising Actor[akka://example/user/test-monitor#-1100372654]
[2014-01-27 16:45:44,084] [DEBUG] [example-akka.actor.default-dispatcher-260] akka.actor.LocalActorRefProvider$Guardian - now supervising Actor[akka://example/user/test-monitor#-1100372654]
[2014-01-27 16:45:44,102] [DEBUG] [example-akka.actor.default-dispatcher-269] akka.actor.LocalActorRefProvider$Guardian - now supervising Actor[akka://example/user/test#-1720775600]
[2014-01-27 16:45:44,102] [DEBUG] [example-akka.actor.default-dispatcher-269] akka.actor.LocalActorRefProvider$Guardian - now supervising Actor[akka://example/user/test#-1720775600]
[2014-01-27 16:45:44,106] [DEBUG] [example-akka.actor.default-dispatcher-270] akka.routing.RouterConfig$$anon$1 - started (akka.routing.RouterConfig$$anon$1@29115152)
[2014-01-27 16:45:44,106] [DEBUG] [example-akka.actor.default-dispatcher-270] akka.routing.RouterConfig$$anon$1 - started (akka.routing.RouterConfig$$anon$1@29115152)
[2014-01-27 16:45:44,107] [DEBUG] [example-akka.actor.default-dispatcher-271] akka.routing.RouterConfig$$anon$1 - now supervising Actor[akka://example/user/test/$a#-1283403612]
[2014-01-27 16:45:44,107] [DEBUG] [example-akka.actor.default-dispatcher-271] akka.routing.RouterConfig$$anon$1 - now supervising Actor[akka://example/user/test/$a#-1283403612]
[2014-01-27 16:45:44,108] [DEBUG] [example-akka.actor.default-dispatcher-272] akka.routing.RouterConfig$$anon$1 - now supervising Actor[akka://example/user/test/$b#-1937476943]
[2014-01-27 16:45:44,108] [DEBUG] [example-akka.actor.default-dispatcher-272] akka.routing.RouterConfig$$anon$1 - now supervising Actor[akka://example/user/test/$b#-1937476943]
[2014-01-27 16:45:44,108] [DEBUG] [example-akka.actor.default-dispatcher-273] akka.routing.RouterConfig$$anon$1 - now supervising Actor[akka://example/user/test/$c#-1267864665]
[2014-01-27 16:45:44,108] [DEBUG] [example-akka.actor.default-dispatcher-273] akka.routing.RouterConfig$$anon$1 - now supervising Actor[akka://example/user/test/$c#-1267864665]
[2014-01-27 16:45:44,109] [DEBUG] [example-akka.actor.default-dispatcher-274] akka.routing.RouterConfig$$anon$1 - now supervising Actor[akka://example/user/test/$d#344287740]
[2014-01-27 16:45:44,109] [DEBUG] [example-akka.actor.default-dispatcher-274] akka.routing.RouterConfig$$anon$1 - now supervising Actor[akka://example/user/test/$d#344287740]
[2014-01-27 16:45:44,109] [DEBUG] [example-akka.actor.default-dispatcher-274] akka.routing.RouterConfig$$anon$1 - now supervising Actor[akka://example/user/test/$e#-89560475]
[2014-01-27 16:45:44,109] [DEBUG] [example-akka.actor.default-dispatcher-274] akka.routing.RouterConfig$$anon$1 - now supervising Actor[akka://example/user/test/$e#-89560475]
[2014-01-27 16:45:44,129] [DEBUG] [example-akka.actor.default-dispatcher-4] akka.actor.TypedActor$TypedActor - started (akka.actor.TypedActor$TypedActor@6eecb3f1)
[2014-01-27 16:45:44,129] [DEBUG] [example-akka.actor.default-dispatcher-4] akka.actor.TypedActor$TypedActor - started (akka.actor.TypedActor$TypedActor@6eecb3f1)
[2014-01-27 16:45:44,129] [DEBUG] [example-akka.actor.default-dispatcher-4] akka.actor.TypedActor$TypedActor - started (akka.actor.TypedActor$TypedActor@7619cc1b)
[2014-01-27 16:45:44,129] [DEBUG] [example-akka.actor.default-dispatcher-4] akka.actor.TypedActor$TypedActor - started (akka.actor.TypedActor$TypedActor@7619cc1b)
[2014-01-27 16:45:44,130] [DEBUG] [example-akka.actor.default-dispatcher-4] akka.actor.TypedActor$TypedActor - started (akka.actor.TypedActor$TypedActor@17a73d30)
[2014-01-27 16:45:44,130] [DEBUG] [example-akka.actor.default-dispatcher-4] akka.actor.TypedActor$TypedActor - started (akka.actor.TypedActor$TypedActor@17a73d30)
[2014-01-27 16:45:44,131] [DEBUG] [example-akka.actor.default-dispatcher-4] akka.actor.TypedActor$TypedActor - started (akka.actor.TypedActor$TypedActor@64a22671)
[2014-01-27 16:45:44,131] [DEBUG] [example-akka.actor.default-dispatcher-4] akka.actor.TypedActor$TypedActor - started (akka.actor.TypedActor$TypedActor@64a22671)
[2014-01-27 16:45:44,131] [DEBUG] [example-akka.actor.default-dispatcher-4] akka.actor.TypedActor$TypedActor - started (akka.actor.TypedActor$TypedActor@5189f854)
[2014-01-27 16:45:44,131] [DEBUG] [example-akka.actor.default-dispatcher-4] akka.actor.TypedActor$TypedActor - started (akka.actor.TypedActor$TypedActor@5189f854)
[2014-01-27 16:45:59,413] [DEBUG] [example-akka.actor.default-dispatcher-22] akka.remote.transport.AkkaProtocolManager - now supervising Actor[akka://example/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fexample%4010.43.15.19%3A36084-1#1695231211]
[2014-01-27 16:45:59,413] [DEBUG] [example-akka.actor.default-dispatcher-22] akka.remote.transport.AkkaProtocolManager - now supervising Actor[akka://example/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fexample%4010.43.15.19%3A36084-1#1695231211]
[2014-01-27 16:45:59,419] [DEBUG] [example-akka.actor.default-dispatcher-4] akka.remote.transport.ProtocolStateActor - started (akka.remote.transport.ProtocolStateActor@68fc8e75)
[2014-01-27 16:45:59,419] [DEBUG] [example-akka.actor.default-dispatcher-4] akka.remote.transport.ProtocolStateActor - started (akka.remote.transport.ProtocolStateActor@68fc8e75)
[2014-01-27 16:45:59,458] [DEBUG] [example-akka.actor.default-dispatcher-25] akka.remote.EndpointManager - now supervising Actor[akka://example/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-0#2034240752]
[2014-01-27 16:45:59,458] [DEBUG] [example-akka.actor.default-dispatcher-25] akka.remote.EndpointManager - now supervising Actor[akka://example/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-0#2034240752]
[2014-01-27 16:45:59,470] [DEBUG] [example-akka.actor.default-dispatcher-235] akka.remote.ReliableDeliverySupervisor - started (akka.remote.ReliableDeliverySupervisor@3de6696c)
[2014-01-27 16:45:59,470] [DEBUG] [example-akka.actor.default-dispatcher-235] akka.remote.ReliableDeliverySupervisor - started (akka.remote.ReliableDeliverySupervisor@3de6696c)
[2014-01-27 16:45:59,470] [DEBUG] [example-akka.actor.default-dispatcher-235] akka.remote.ReliableDeliverySupervisor - now monitoring Actor[akka://example/system/endpointManager#39348252]
[2014-01-27 16:45:59,470] [DEBUG] [example-akka.actor.default-dispatcher-235] akka.remote.ReliableDeliverySupervisor - now monitoring Actor[akka://example/system/endpointManager#39348252]
[2014-01-27 16:45:59,471] [DEBUG] [example-akka.actor.default-dispatcher-235] akka.remote.ReliableDeliverySupervisor - now supervising Actor[akka://example/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-0/endpointWriter#-680224632]
[2014-01-27 16:45:59,471] [DEBUG] [example-akka.actor.default-dispatcher-235] akka.remote.ReliableDeliverySupervisor - now supervising Actor[akka://example/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-0/endpointWriter#-680224632]
[2014-01-27 16:45:59,480] [DEBUG] [example-akka.actor.default-dispatcher-31] akka.remote.EndpointWriter - started (akka.remote.EndpointWriter@485b78cf)
[2014-01-27 16:45:59,480] [DEBUG] [example-akka.actor.default-dispatcher-31] akka.remote.EndpointWriter - started (akka.remote.EndpointWriter@485b78cf)
[2014-01-27 16:45:59,480] [DEBUG] [example-akka.actor.default-dispatcher-31] akka.remote.EndpointWriter - now monitoring Actor[akka://example/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-0#2034240752]
[2014-01-27 16:45:59,480] [DEBUG] [example-akka.actor.default-dispatcher-31] akka.remote.EndpointWriter - now monitoring Actor[akka://example/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-0#2034240752]
[2014-01-27 16:45:59,481] [DEBUG] [example-akka.actor.default-dispatcher-31] akka.remote.EndpointWriter - now supervising Actor[akka://example/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-0/endpointWriter/endpointReader-akka.tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-0#1050380478]
[2014-01-27 16:45:59,481] [DEBUG] [example-akka.actor.default-dispatcher-31] akka.remote.EndpointWriter - now supervising Actor[akka://example/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-0/endpointWriter/endpointReader-akka.tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-0#1050380478]
[2014-01-27 16:45:59,489] [DEBUG] [example-akka.actor.default-dispatcher-242] akka.remote.EndpointReader - started (akka.remote.EndpointReader@4545f5e3)
[2014-01-27 16:45:59,489] [DEBUG] [example-akka.actor.default-dispatcher-242] akka.remote.EndpointReader - started (akka.remote.EndpointReader@4545f5e3)
[2014-01-27 16:45:59,489] [DEBUG] [example-akka.actor.default-dispatcher-242] akka.remote.EndpointReader - now monitoring Actor[akka://example/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-0/endpointWriter#-680224632]
[2014-01-27 16:45:59,489] [DEBUG] [example-akka.actor.default-dispatcher-242] akka.remote.EndpointReader - now monitoring Actor[akka://example/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-0/endpointWriter#-680224632]
[2014-01-27 16:45:59,539] [DEBUG] [example-akka.actor.default-dispatcher-247] akka.remote.EndpointWriter - received local message RemoteMessage: [SelectChildName(user,SelectChildName(test,MethodCall(public abstract java.lang.String com.markit.bonds.akka.ReconnectFailExample$ExampleService.hello(),null)))] to [Actor[akka://example/]]<+[akka://example/] from [Actor[akka.tcp://cli...@127.0.0.1:49659/temp/$a]]
[2014-01-27 16:45:59,539] [DEBUG] [example-akka.actor.default-dispatcher-247] akka.remote.EndpointWriter - received local message RemoteMessage: [SelectChildName(user,SelectChildName(test,MethodCall(public abstract java.lang.String com.markit.bonds.akka.ReconnectFailExample$ExampleService.hello(),null)))] to [Actor[akka://example/]]<+[akka://example/] from [Actor[akka.tcp://cli...@127.0.0.1:49659/temp/$a]]
[2014-01-27 16:45:59,540] [DEBUG] [example-akka.actor.default-dispatcher-247] akka.actor.LocalActorRefProvider$Guardian - received AutoReceiveMessage Envelope(SelectChildName(user,SelectChildName(test,MethodCall(public abstract java.lang.String com.markit.bonds.akka.ReconnectFailExample$ExampleService.hello(),null))),Actor[akka.tcp://cli...@127.0.0.1:49659/temp/$a])
[2014-01-27 16:45:59,540] [DEBUG] [example-akka.actor.default-dispatcher-247] akka.actor.LocalActorRefProvider$Guardian - received AutoReceiveMessage Envelope(SelectChildName(user,SelectChildName(test,MethodCall(public abstract java.lang.String com.markit.bonds.akka.ReconnectFailExample$ExampleService.hello(),null))),Actor[akka.tcp://cli...@127.0.0.1:49659/temp/$a])
[2014-01-27 16:45:59,542] [DEBUG] [example-akka.actor.default-dispatcher-254] akka.actor.LocalActorRefProvider$Guardian - received AutoReceiveMessage Envelope(SelectChildName(test,MethodCall(public abstract java.lang.String com.markit.bonds.akka.ReconnectFailExample$ExampleService.hello(),null)),Actor[akka.tcp://cli...@127.0.0.1:49659/temp/$a])
[2014-01-27 16:45:59,542] [DEBUG] [example-akka.actor.default-dispatcher-254] akka.actor.LocalActorRefProvider$Guardian - received AutoReceiveMessage Envelope(SelectChildName(test,MethodCall(public abstract java.lang.String com.markit.bonds.akka.ReconnectFailExample$ExampleService.hello(),null)),Actor[akka.tcp://cli...@127.0.0.1:49659/temp/$a])
[2014-01-27 16:45:59,561] [DEBUG] [example-akka.actor.default-dispatcher-249] akka.remote.EndpointWriter - sending message RemoteMessage: [hello] to [Actor[akka.tcp://cli...@127.0.0.1:49659/temp/$a]]<+[akka.tcp://cli...@127.0.0.1:49659/temp/$a] from [Actor[akka://example/user/test/$a#-1283403612]]
[2014-01-27 16:45:59,561] [DEBUG] [example-akka.actor.default-dispatcher-249] akka.remote.EndpointWriter - sending message RemoteMessage: [hello] to [Actor[akka.tcp://cli...@127.0.0.1:49659/temp/$a]]<+[akka.tcp://cli...@127.0.0.1:49659/temp/$a] from [Actor[akka://example/user/test/$a#-1283403612]]
[2014-01-27 16:45:59,563] [DEBUG] [example-akka.actor.default-dispatcher-258] akka.serialization.Serialization(akka://example) - Using serializer[akka.serialization.JavaSerializer] for message [java.lang.String]
[2014-01-27 16:45:59,563] [DEBUG] [example-akka.actor.default-dispatcher-258] akka.serialization.Serialization(akka://example) - Using serializer[akka.serialization.JavaSerializer] for message [java.lang.String]
[2014-01-27 16:46:17,470] [DEBUG] [example-akka.actor.default-dispatcher-257] akka.remote.transport.ProtocolStateActor - stopped
[2014-01-27 16:46:17,470] [DEBUG] [example-akka.actor.default-dispatcher-257] akka.remote.transport.ProtocolStateActor - stopped
[2014-01-27 16:46:17,471] [DEBUG] [example-akka.actor.default-dispatcher-257] akka.remote.EndpointReader - stopped
[2014-01-27 16:46:17,471] [DEBUG] [example-akka.actor.default-dispatcher-257] akka.remote.EndpointReader - stopped
[2014-01-27 16:46:17,474] [DEBUG] [example-akka.actor.default-dispatcher-261] akka.remote.EndpointWriter - received AutoReceiveMessage Envelope(Terminated(Actor[akka://example/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-0/endpointWriter/endpointReader-akka.tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-0#1050380478]),Actor[akka://example/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-0/endpointWriter/endpointReader-akka.tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-0#1050380478])
[2014-01-27 16:46:17,474] [DEBUG] [example-akka.actor.default-dispatcher-261] akka.remote.EndpointWriter - received AutoReceiveMessage Envelope(Terminated(Actor[akka://example/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-0/endpointWriter/endpointReader-akka.tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-0#1050380478]),Actor[akka://example/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-0/endpointWriter/endpointReader-akka.tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-0#1050380478])
[2014-01-27 16:46:17,474] [INFO] [example-akka.actor.default-dispatcher-261] akka.actor.LocalActorRef - Message [akka.remote.transport.AssociationHandle$Disassociated] from Actor[akka://example/deadLetters] to Actor[akka://example/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fexample%4010.43.15.19%3A36084-1#1695231211] was not delivered. [1] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[2014-01-27 16:46:17,474] [INFO] [example-akka.actor.default-dispatcher-261] akka.actor.LocalActorRef - Message [akka.remote.transport.AssociationHandle$Disassociated] from Actor[akka://example/deadLetters] to Actor[akka://example/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fexample%4010.43.15.19%3A36084-1#1695231211] was not delivered. [1] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[2014-01-27 16:46:17,476] [DEBUG] [example-akka.actor.default-dispatcher-262] akka.remote.EndpointWriter - Disassociated [akka.tcp://example@server002:3552] <- [akka.tcp://cli...@127.0.0.1:49659]
[2014-01-27 16:46:17,476] [DEBUG] [example-akka.actor.default-dispatcher-262] akka.remote.EndpointWriter - Disassociated [akka.tcp://example@server002:3552] <- [akka.tcp://cli...@127.0.0.1:49659]
[2014-01-27 16:46:17,480] [DEBUG] [example-akka.actor.default-dispatcher-270] akka.remote.EndpointWriter - restarting
[2014-01-27 16:46:17,480] [DEBUG] [example-akka.actor.default-dispatcher-270] akka.remote.EndpointWriter - restarting
[2014-01-27 16:46:17,482] [DEBUG] [example-akka.actor.default-dispatcher-273] akka.remote.EndpointWriter - Disassociated [akka.tcp://example@server002:3552] <- [akka.tcp://cli...@127.0.0.1:49659]
[2014-01-27 16:46:17,482] [DEBUG] [example-akka.actor.default-dispatcher-273] akka.remote.EndpointWriter - Disassociated [akka.tcp://example@server002:3552] <- [akka.tcp://cli...@127.0.0.1:49659]
[2014-01-27 16:46:17,482] [INFO] [example-akka.actor.default-dispatcher-273] akka.actor.LocalActorRef - Message [akka.remote.transport.ActorTransportAdapter$DisassociateUnderlying] from Actor[akka://example/deadLetters] to Actor[akka://example/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fexample%4010.43.15.19%3A36084-1#1695231211] was not delivered. [2] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[2014-01-27 16:46:17,482] [INFO] [example-akka.actor.default-dispatcher-273] akka.actor.LocalActorRef - Message [akka.remote.transport.ActorTransportAdapter$DisassociateUnderlying] from Actor[akka://example/deadLetters] to Actor[akka://example/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fexample%4010.43.15.19%3A36084-1#1695231211] was not delivered. [2] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[2014-01-27 16:46:17,484] [DEBUG] [example-akka.actor.default-dispatcher-2] akka.remote.EndpointWriter - restarted
[2014-01-27 16:46:17,484] [DEBUG] [example-akka.actor.default-dispatcher-2] akka.remote.EndpointWriter - restarted
[2014-01-27 16:46:17,486] [DEBUG] [example-akka.actor.default-dispatcher-274] akka.remote.transport.AkkaProtocolManager - now supervising Actor[akka://example/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-2#-1488353550]
[2014-01-27 16:46:17,486] [DEBUG] [example-akka.actor.default-dispatcher-274] akka.remote.transport.AkkaProtocolManager - now supervising Actor[akka://example/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-2#-1488353550]
[2014-01-27 16:46:17,491] [DEBUG] [example-akka.actor.default-dispatcher-7] akka.remote.transport.ProtocolStateActor - started (akka.remote.transport.ProtocolStateActor@1e9af0b1)
[2014-01-27 16:46:17,491] [DEBUG] [example-akka.actor.default-dispatcher-7] akka.remote.transport.ProtocolStateActor - started (akka.remote.transport.ProtocolStateActor@1e9af0b1)
[2014-01-27 16:46:17,504] [DEBUG] [example-akka.actor.default-dispatcher-14] akka.remote.transport.ProtocolStateActor - stopped
[2014-01-27 16:46:17,504] [DEBUG] [example-akka.actor.default-dispatcher-14] akka.remote.transport.ProtocolStateActor - stopped
[2014-01-27 16:46:17,506] [ERROR] [example-akka.actor.default-dispatcher-16] akka.remote.EndpointWriter - AssociationError [akka.tcp://example@server002:3552] -> [akka.tcp://cli...@127.0.0.1:49659]: Error [Association failed with [akka.tcp://cli...@127.0.0.1:49659]] [
akka.remote.EndpointAssociationException: Association failed with [akka.tcp://cli...@127.0.0.1:49659]
Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: /127.0.0.1:49659
]
[2014-01-27 16:46:17,506] [ERROR] [example-akka.actor.default-dispatcher-16] akka.remote.EndpointWriter - AssociationError [akka.tcp://example@server002:3552] -> [akka.tcp://cli...@127.0.0.1:49659]: Error [Association failed with [akka.tcp://cli...@127.0.0.1:49659]] [
akka.remote.EndpointAssociationException: Association failed with [akka.tcp://cli...@127.0.0.1:49659]
Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: /127.0.0.1:49659
]
[2014-01-27 16:46:17,507] [DEBUG] [example-akka.actor.default-dispatcher-18] akka.remote.EndpointWriter - restarting
[2014-01-27 16:46:17,507] [DEBUG] [example-akka.actor.default-dispatcher-18] akka.remote.EndpointWriter - restarting
[2014-01-27 16:46:17,507] [DEBUG] [example-akka.actor.default-dispatcher-18] akka.remote.EndpointWriter - Disassociated [akka.tcp://example@server002:3552] -> [akka.tcp://cli...@127.0.0.1:49659]
[2014-01-27 16:46:17,507] [DEBUG] [example-akka.actor.default-dispatcher-18] akka.remote.EndpointWriter - Disassociated [akka.tcp://example@server002:3552] -> [akka.tcp://cli...@127.0.0.1:49659]
[2014-01-27 16:46:17,508] [DEBUG] [example-akka.actor.default-dispatcher-18] akka.remote.EndpointWriter - restarted
[2014-01-27 16:46:17,508] [DEBUG] [example-akka.actor.default-dispatcher-18] akka.remote.EndpointWriter - restarted
[2014-01-27 16:46:17,510] [DEBUG] [example-akka.actor.default-dispatcher-19] akka.remote.transport.AkkaProtocolManager - now supervising Actor[akka://example/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-3#1690936949]
[2014-01-27 16:46:17,510] [DEBUG] [example-akka.actor.default-dispatcher-19] akka.remote.transport.AkkaProtocolManager - now supervising Actor[akka://example/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-3#1690936949]
[2014-01-27 16:46:17,510] [DEBUG] [example-akka.actor.default-dispatcher-19] akka.remote.transport.ProtocolStateActor - started (akka.remote.transport.ProtocolStateActor@7463e563)
[2014-01-27 16:46:17,510] [DEBUG] [example-akka.actor.default-dispatcher-19] akka.remote.transport.ProtocolStateActor - started (akka.remote.transport.ProtocolStateActor@7463e563)
[2014-01-27 16:46:17,513] [DEBUG] [example-akka.actor.default-dispatcher-27] akka.remote.transport.ProtocolStateActor - stopped
[2014-01-27 16:46:17,513] [DEBUG] [example-akka.actor.default-dispatcher-27] akka.remote.transport.ProtocolStateActor - stopped
[2014-01-27 16:46:17,513] [ERROR] [example-akka.actor.default-dispatcher-241] akka.remote.EndpointWriter - AssociationError [akka.tcp://example@server002:3552] -> [akka.tcp://cli...@127.0.0.1:49659]: Error [Association failed with [akka.tcp://cli...@127.0.0.1:49659]] [
akka.remote.EndpointAssociationException: Association failed with [akka.tcp://cli...@127.0.0.1:49659]
Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: /127.0.0.1:49659
]
[2014-01-27 16:46:17,513] [ERROR] [example-akka.actor.default-dispatcher-241] akka.remote.EndpointWriter - AssociationError [akka.tcp://example@server002:3552] -> [akka.tcp://cli...@127.0.0.1:49659]: Error [Association failed with [akka.tcp://cli...@127.0.0.1:49659]] [
akka.remote.EndpointAssociationException: Association failed with [akka.tcp://cli...@127.0.0.1:49659]
Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: /127.0.0.1:49659
]
[2014-01-27 16:46:17,514] [DEBUG] [example-akka.actor.default-dispatcher-241] akka.remote.EndpointWriter - restarting
[2014-01-27 16:46:17,514] [DEBUG] [example-akka.actor.default-dispatcher-241] akka.remote.EndpointWriter - restarting
[2014-01-27 16:46:17,514] [DEBUG] [example-akka.actor.default-dispatcher-241] akka.remote.EndpointWriter - Disassociated [akka.tcp://example@server002:3552] -> [akka.tcp://cli...@127.0.0.1:49659]
[2014-01-27 16:46:17,514] [DEBUG] [example-akka.actor.default-dispatcher-241] akka.remote.EndpointWriter - Disassociated [akka.tcp://example@server002:3552] -> [akka.tcp://cli...@127.0.0.1:49659]
[2014-01-27 16:46:17,515] [DEBUG] [example-akka.actor.default-dispatcher-31] akka.remote.EndpointWriter - restarted
[2014-01-27 16:46:17,515] [DEBUG] [example-akka.actor.default-dispatcher-31] akka.remote.EndpointWriter - restarted
[2014-01-27 16:46:17,517] [DEBUG] [example-akka.actor.default-dispatcher-244] akka.remote.transport.AkkaProtocolManager - now supervising Actor[akka://example/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-4#1637151416]
[2014-01-27 16:46:17,517] [DEBUG] [example-akka.actor.default-dispatcher-244] akka.remote.transport.AkkaProtocolManager - now supervising Actor[akka://example/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-4#1637151416]
[2014-01-27 16:46:17,518] [DEBUG] [example-akka.actor.default-dispatcher-247] akka.remote.transport.ProtocolStateActor - started (akka.remote.transport.ProtocolStateActor@6070c38c)
[2014-01-27 16:46:17,518] [DEBUG] [example-akka.actor.default-dispatcher-247] akka.remote.transport.ProtocolStateActor - started (akka.remote.transport.ProtocolStateActor@6070c38c)
[2014-01-27 16:46:17,522] [DEBUG] [example-akka.actor.default-dispatcher-246] akka.remote.transport.ProtocolStateActor - stopped
[2014-01-27 16:46:17,522] [DEBUG] [example-akka.actor.default-dispatcher-246] akka.remote.transport.ProtocolStateActor - stopped
[2014-01-27 16:46:17,523] [ERROR] [example-akka.actor.default-dispatcher-246] akka.remote.EndpointWriter - AssociationError [akka.tcp://example@server002:3552] -> [akka.tcp://cli...@127.0.0.1:49659]: Error [Association failed with [akka.tcp://cli...@127.0.0.1:49659]] [
akka.remote.EndpointAssociationException: Association failed with [akka.tcp://cli...@127.0.0.1:49659]
Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: /127.0.0.1:49659
]
[2014-01-27 16:46:17,523] [ERROR] [example-akka.actor.default-dispatcher-246] akka.remote.EndpointWriter - AssociationError [akka.tcp://example@server002:3552] -> [akka.tcp://cli...@127.0.0.1:49659]: Error [Association failed with [akka.tcp://cli...@127.0.0.1:49659]] [
akka.remote.EndpointAssociationException: Association failed with [akka.tcp://cli...@127.0.0.1:49659]
Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: /127.0.0.1:49659
]
[2014-01-27 16:46:17,524] [DEBUG] [example-akka.actor.default-dispatcher-254] akka.remote.EndpointWriter - Disassociated [akka.tcp://example@server002:3552] -> [akka.tcp://cli...@127.0.0.1:49659]
[2014-01-27 16:46:17,524] [DEBUG] [example-akka.actor.default-dispatcher-254] akka.remote.EndpointWriter - Disassociated [akka.tcp://example@server002:3552] -> [akka.tcp://cli...@127.0.0.1:49659]
[2014-01-27 16:46:17,525] [DEBUG] [example-akka.actor.default-dispatcher-257] akka.remote.EndpointWriter - stopped
[2014-01-27 16:46:17,525] [DEBUG] [example-akka.actor.default-dispatcher-257] akka.remote.EndpointWriter - stopped
[2014-01-27 16:46:17,525] [DEBUG] [example-akka.actor.default-dispatcher-261] akka.remote.ReliableDeliverySupervisor - received AutoReceiveMessage Envelope(Terminated(Actor[akka://example/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-0/endpointWriter#-680224632]),Actor[akka://example/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-0/endpointWriter#-680224632])
[2014-01-27 16:46:17,525] [DEBUG] [example-akka.actor.default-dispatcher-261] akka.remote.ReliableDeliverySupervisor - received AutoReceiveMessage Envelope(Terminated(Actor[akka://example/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-0/endpointWriter#-680224632]),Actor[akka://example/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-0/endpointWriter#-680224632])
[2014-01-27 16:46:22,597] [DEBUG] [example-akka.actor.default-dispatcher-20] akka.remote.transport.AkkaProtocolManager - now supervising Actor[akka://example/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fexample%4010.43.15.19%3A36087-5#-1519118151]
[2014-01-27 16:46:22,597] [DEBUG] [example-akka.actor.default-dispatcher-20] akka.remote.transport.AkkaProtocolManager - now supervising Actor[akka://example/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fexample%4010.43.15.19%3A36087-5#-1519118151]
[2014-01-27 16:46:22,598] [DEBUG] [example-akka.actor.default-dispatcher-20] akka.remote.transport.ProtocolStateActor - started (akka.remote.transport.ProtocolStateActor@2945c761)
[2014-01-27 16:46:22,598] [DEBUG] [example-akka.actor.default-dispatcher-20] akka.remote.transport.ProtocolStateActor - started (akka.remote.transport.ProtocolStateActor@2945c761)
[2014-01-27 16:46:22,602] [DEBUG] [example-akka.actor.default-dispatcher-4] akka.remote.EndpointManager - now supervising Actor[akka://example/system/endpointManager/endpointWriter-akka.tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-1#-592860104]
[2014-01-27 16:46:22,602] [DEBUG] [example-akka.actor.default-dispatcher-4] akka.remote.EndpointManager - now supervising Actor[akka://example/system/endpointManager/endpointWriter-akka.tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-1#-592860104]
[2014-01-27 16:46:22,604] [DEBUG] [example-akka.actor.default-dispatcher-236] akka.remote.EndpointWriter - started (akka.remote.EndpointWriter@7b9bd735)
[2014-01-27 16:46:22,604] [DEBUG] [example-akka.actor.default-dispatcher-236] akka.remote.EndpointWriter - started (akka.remote.EndpointWriter@7b9bd735)
[2014-01-27 16:46:22,605] [DEBUG] [example-akka.actor.default-dispatcher-236] akka.remote.EndpointWriter - now monitoring Actor[akka://example/system/endpointManager#39348252]
[2014-01-27 16:46:22,605] [DEBUG] [example-akka.actor.default-dispatcher-236] akka.remote.EndpointWriter - now monitoring Actor[akka://example/system/endpointManager#39348252]
[2014-01-27 16:46:22,605] [DEBUG] [example-akka.actor.default-dispatcher-236] akka.remote.EndpointWriter - now supervising Actor[akka://example/system/endpointManager/endpointWriter-akka.tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-1/endpointReader-akka.tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-0#-1170796026]
[2014-01-27 16:46:22,605] [DEBUG] [example-akka.actor.default-dispatcher-236] akka.remote.EndpointWriter - now supervising Actor[akka://example/system/endpointManager/endpointWriter-akka.tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-1/endpointReader-akka.tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-0#-1170796026]
[2014-01-27 16:46:22,606] [DEBUG] [example-akka.actor.default-dispatcher-241] akka.remote.EndpointReader - started (akka.remote.EndpointReader@3951f910)
[2014-01-27 16:46:22,606] [DEBUG] [example-akka.actor.default-dispatcher-241] akka.remote.EndpointReader - started (akka.remote.EndpointReader@3951f910)
[2014-01-27 16:46:22,607] [DEBUG] [example-akka.actor.default-dispatcher-241] akka.remote.EndpointReader - now monitoring Actor[akka://example/system/endpointManager/endpointWriter-akka.tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-1#-592860104]
[2014-01-27 16:46:22,607] [DEBUG] [example-akka.actor.default-dispatcher-241] akka.remote.EndpointReader - now monitoring Actor[akka://example/system/endpointManager/endpointWriter-akka.tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-1#-592860104]
[2014-01-27 16:46:29,584] [DEBUG] [example-akka.actor.default-dispatcher-254] akka.remote.EndpointWriter - received local message RemoteMessage: [SelectChildName(user,SelectChildName(test,MethodCall(public abstract java.lang.String com.markit.bonds.akka.ReconnectFailExample$ExampleService.hello(),null)))] to [Actor[akka://example/]]<+[akka://example/] from [Actor[akka.tcp://cli...@127.0.0.1:49659/temp/$b]]
[2014-01-27 16:46:29,584] [DEBUG] [example-akka.actor.default-dispatcher-254] akka.remote.EndpointWriter - received local message RemoteMessage: [SelectChildName(user,SelectChildName(test,MethodCall(public abstract java.lang.String com.markit.bonds.akka.ReconnectFailExample$ExampleService.hello(),null)))] to [Actor[akka://example/]]<+[akka://example/] from [Actor[akka.tcp://cli...@127.0.0.1:49659/temp/$b]]
[2014-01-27 16:46:29,584] [DEBUG] [example-akka.actor.default-dispatcher-254] akka.actor.LocalActorRefProvider$Guardian - received AutoReceiveMessage Envelope(SelectChildName(user,SelectChildName(test,MethodCall(public abstract java.lang.String com.markit.bonds.akka.ReconnectFailExample$ExampleService.hello(),null))),Actor[akka.tcp://cli...@127.0.0.1:49659/temp/$b])
[2014-01-27 16:46:29,584] [DEBUG] [example-akka.actor.default-dispatcher-254] akka.actor.LocalActorRefProvider$Guardian - received AutoReceiveMessage Envelope(SelectChildName(user,SelectChildName(test,MethodCall(public abstract java.lang.String com.markit.bonds.akka.ReconnectFailExample$ExampleService.hello(),null))),Actor[akka.tcp://cli...@127.0.0.1:49659/temp/$b])
[2014-01-27 16:46:29,585] [DEBUG] [example-akka.actor.default-dispatcher-254] akka.actor.LocalActorRefProvider$Guardian - received AutoReceiveMessage Envelope(SelectChildName(test,MethodCall(public abstract java.lang.String com.markit.bonds.akka.ReconnectFailExample$ExampleService.hello(),null)),Actor[akka.tcp://cli...@127.0.0.1:49659/temp/$b])
[2014-01-27 16:46:29,585] [DEBUG] [example-akka.actor.default-dispatcher-254] akka.actor.LocalActorRefProvider$Guardian - received AutoReceiveMessage Envelope(SelectChildName(test,MethodCall(public abstract java.lang.String com.markit.bonds.akka.ReconnectFailExample$ExampleService.hello(),null)),Actor[akka.tcp://cli...@127.0.0.1:49659/temp/$b])
[2014-01-27 16:46:29,588] [DEBUG] [example-akka.actor.default-dispatcher-274] akka.remote.ReliableDeliverySupervisor - now supervising Actor[akka://example/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-0/endpointWriter#680874170]
[2014-01-27 16:46:29,588] [DEBUG] [example-akka.actor.default-dispatcher-274] akka.remote.ReliableDeliverySupervisor - now supervising Actor[akka://example/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-0/endpointWriter#680874170]
[2014-01-27 16:46:29,589] [DEBUG] [example-akka.actor.default-dispatcher-6] akka.remote.EndpointWriter - started (akka.remote.EndpointWriter@395fd251)
[2014-01-27 16:46:29,589] [DEBUG] [example-akka.actor.default-dispatcher-6] akka.remote.EndpointWriter - started (akka.remote.EndpointWriter@395fd251)
[2014-01-27 16:46:29,589] [DEBUG] [example-akka.actor.default-dispatcher-6] akka.remote.EndpointWriter - now monitoring Actor[akka://example/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-0#2034240752]
[2014-01-27 16:46:29,589] [DEBUG] [example-akka.actor.default-dispatcher-6] akka.remote.EndpointWriter - now monitoring Actor[akka://example/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-0#2034240752]
[2014-01-27 16:46:29,590] [DEBUG] [example-akka.actor.default-dispatcher-3] akka.remote.transport.AkkaProtocolManager - now supervising Actor[akka://example/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-6#1495903680]
[2014-01-27 16:46:29,590] [DEBUG] [example-akka.actor.default-dispatcher-3] akka.remote.transport.AkkaProtocolManager - now supervising Actor[akka://example/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-6#1495903680]
[2014-01-27 16:46:29,591] [DEBUG] [example-akka.actor.default-dispatcher-3] akka.remote.transport.ProtocolStateActor - started (akka.remote.transport.ProtocolStateActor@41e8e144)
[2014-01-27 16:46:29,591] [DEBUG] [example-akka.actor.default-dispatcher-3] akka.remote.transport.ProtocolStateActor - started (akka.remote.transport.ProtocolStateActor@41e8e144)
[2014-01-27 16:46:29,598] [DEBUG] [example-akka.actor.default-dispatcher-16] akka.remote.transport.ProtocolStateActor - stopped
[2014-01-27 16:46:29,598] [DEBUG] [example-akka.actor.default-dispatcher-16] akka.remote.transport.ProtocolStateActor - stopped
[2014-01-27 16:46:29,598] [ERROR] [example-akka.actor.default-dispatcher-18] akka.remote.EndpointWriter - AssociationError [akka.tcp://example@server002:3552] -> [akka.tcp://cli...@127.0.0.1:49659]: Error [Association failed with [akka.tcp://cli...@127.0.0.1:49659]] [
akka.remote.EndpointAssociationException: Association failed with [akka.tcp://cli...@127.0.0.1:49659]
Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: /127.0.0.1:49659
]
[2014-01-27 16:46:29,598] [ERROR] [example-akka.actor.default-dispatcher-18] akka.remote.EndpointWriter - AssociationError [akka.tcp://example@server002:3552] -> [akka.tcp://cli...@127.0.0.1:49659]: Error [Association failed with [akka.tcp://cli...@127.0.0.1:49659]] [
akka.remote.EndpointAssociationException: Association failed with [akka.tcp://cli...@127.0.0.1:49659]
Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: /127.0.0.1:49659
]
[2014-01-27 16:46:29,599] [DEBUG] [example-akka.actor.default-dispatcher-18] akka.remote.EndpointWriter - restarting
[2014-01-27 16:46:29,599] [DEBUG] [example-akka.actor.default-dispatcher-18] akka.remote.EndpointWriter - restarting
[2014-01-27 16:46:29,599] [DEBUG] [example-akka.actor.default-dispatcher-18] akka.remote.EndpointWriter - Disassociated [akka.tcp://example@server002:3552] -> [akka.tcp://cli...@127.0.0.1:49659]
[2014-01-27 16:46:29,599] [DEBUG] [example-akka.actor.default-dispatcher-18] akka.remote.EndpointWriter - Disassociated [akka.tcp://example@server002:3552] -> [akka.tcp://cli...@127.0.0.1:49659]
[2014-01-27 16:46:29,600] [DEBUG] [example-akka.actor.default-dispatcher-30] akka.remote.EndpointWriter - restarted
[2014-01-27 16:46:29,600] [DEBUG] [example-akka.actor.default-dispatcher-30] akka.remote.EndpointWriter - restarted
[2014-01-27 16:46:29,602] [DEBUG] [example-akka.actor.default-dispatcher-235] akka.remote.transport.AkkaProtocolManager - now supervising Actor[akka://example/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-7#2118989911]
[2014-01-27 16:46:29,602] [DEBUG] [example-akka.actor.default-dispatcher-235] akka.remote.transport.AkkaProtocolManager - now supervising Actor[akka://example/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-7#2118989911]
[2014-01-27 16:46:29,603] [DEBUG] [example-akka.actor.default-dispatcher-235] akka.remote.transport.ProtocolStateActor - started (akka.remote.transport.ProtocolStateActor@3e4ac866)
[2014-01-27 16:46:29,603] [DEBUG] [example-akka.actor.default-dispatcher-235] akka.remote.transport.ProtocolStateActor - started (akka.remote.transport.ProtocolStateActor@3e4ac866)
[2014-01-27 16:46:29,605] [DEBUG] [example-akka.actor.default-dispatcher-240] akka.remote.transport.ProtocolStateActor - stopped
[2014-01-27 16:46:29,605] [DEBUG] [example-akka.actor.default-dispatcher-240] akka.remote.transport.ProtocolStateActor - stopped
[2014-01-27 16:46:29,605] [ERROR] [example-akka.actor.default-dispatcher-240] akka.remote.EndpointWriter - AssociationError [akka.tcp://example@server002:3552] -> [akka.tcp://cli...@127.0.0.1:49659]: Error [Association failed with [akka.tcp://cli...@127.0.0.1:49659]] [
akka.remote.EndpointAssociationException: Association failed with [akka.tcp://cli...@127.0.0.1:49659]
Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: /127.0.0.1:49659
]
[2014-01-27 16:46:29,605] [ERROR] [example-akka.actor.default-dispatcher-240] akka.remote.EndpointWriter - AssociationError [akka.tcp://example@server002:3552] -> [akka.tcp://cli...@127.0.0.1:49659]: Error [Association failed with [akka.tcp://cli...@127.0.0.1:49659]] [
akka.remote.EndpointAssociationException: Association failed with [akka.tcp://cli...@127.0.0.1:49659]
Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: /127.0.0.1:49659
]
[2014-01-27 16:46:29,606] [DEBUG] [example-akka.actor.default-dispatcher-238] akka.remote.EndpointWriter - restarting
[2014-01-27 16:46:29,606] [DEBUG] [example-akka.actor.default-dispatcher-238] akka.remote.EndpointWriter - restarting
[2014-01-27 16:46:29,606] [DEBUG] [example-akka.actor.default-dispatcher-238] akka.remote.EndpointWriter - Disassociated [akka.tcp://example@server002:3552] -> [akka.tcp://cli...@127.0.0.1:49659]
[2014-01-27 16:46:29,606] [DEBUG] [example-akka.actor.default-dispatcher-238] akka.remote.EndpointWriter - Disassociated [akka.tcp://example@server002:3552] -> [akka.tcp://cli...@127.0.0.1:49659]
[2014-01-27 16:46:29,607] [DEBUG] [example-akka.actor.default-dispatcher-244] akka.remote.EndpointWriter - restarted
[2014-01-27 16:46:29,607] [DEBUG] [example-akka.actor.default-dispatcher-244] akka.remote.EndpointWriter - restarted
[2014-01-27 16:46:29,609] [DEBUG] [example-akka.actor.default-dispatcher-238] akka.remote.transport.AkkaProtocolManager - now supervising Actor[akka://example/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-8#-331254032]
[2014-01-27 16:46:29,609] [DEBUG] [example-akka.actor.default-dispatcher-238] akka.remote.transport.AkkaProtocolManager - now supervising Actor[akka://example/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-8#-331254032]
[2014-01-27 16:46:29,610] [DEBUG] [example-akka.actor.default-dispatcher-247] akka.remote.transport.ProtocolStateActor - started (akka.remote.transport.ProtocolStateActor@3a0b2c64)
[2014-01-27 16:46:29,610] [DEBUG] [example-akka.actor.default-dispatcher-247] akka.remote.transport.ProtocolStateActor - started (akka.remote.transport.ProtocolStateActor@3a0b2c64)
[2014-01-27 16:46:29,612] [DEBUG] [example-akka.actor.default-dispatcher-261] akka.remote.transport.ProtocolStateActor - stopped
[2014-01-27 16:46:29,612] [DEBUG] [example-akka.actor.default-dispatcher-261] akka.remote.transport.ProtocolStateActor - stopped
[2014-01-27 16:46:29,613] [ERROR] [example-akka.actor.default-dispatcher-261] akka.remote.EndpointWriter - AssociationError [akka.tcp://example@server002:3552] -> [akka.tcp://cli...@127.0.0.1:49659]: Error [Association failed with [akka.tcp://cli...@127.0.0.1:49659]] [
akka.remote.EndpointAssociationException: Association failed with [akka.tcp://cli...@127.0.0.1:49659]
Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: /127.0.0.1:49659
]
[2014-01-27 16:46:29,613] [ERROR] [example-akka.actor.default-dispatcher-261] akka.remote.EndpointWriter - AssociationError [akka.tcp://example@server002:3552] -> [akka.tcp://cli...@127.0.0.1:49659]: Error [Association failed with [akka.tcp://cli...@127.0.0.1:49659]] [
akka.remote.EndpointAssociationException: Association failed with [akka.tcp://cli...@127.0.0.1:49659]
Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: /127.0.0.1:49659
]
[2014-01-27 16:46:29,613] [DEBUG] [example-akka.actor.default-dispatcher-261] akka.remote.EndpointWriter - restarting
[2014-01-27 16:46:29,613] [DEBUG] [example-akka.actor.default-dispatcher-261] akka.remote.EndpointWriter - restarting
[2014-01-27 16:46:29,614] [DEBUG] [example-akka.actor.default-dispatcher-261] akka.remote.EndpointWriter - Disassociated [akka.tcp://example@server002:3552] -> [akka.tcp://cli...@127.0.0.1:49659]
[2014-01-27 16:46:29,614] [DEBUG] [example-akka.actor.default-dispatcher-261] akka.remote.EndpointWriter - Disassociated [akka.tcp://example@server002:3552] -> [akka.tcp://cli...@127.0.0.1:49659]
[2014-01-27 16:46:29,614] [DEBUG] [example-akka.actor.default-dispatcher-254] akka.remote.EndpointWriter - restarted
[2014-01-27 16:46:29,614] [DEBUG] [example-akka.actor.default-dispatcher-254] akka.remote.EndpointWriter - restarted
[2014-01-27 16:46:29,616] [DEBUG] [example-akka.actor.default-dispatcher-8] akka.remote.transport.AkkaProtocolManager - now supervising Actor[akka://example/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-9#-56509918]
[2014-01-27 16:46:29,616] [DEBUG] [example-akka.actor.default-dispatcher-8] akka.remote.transport.AkkaProtocolManager - now supervising Actor[akka://example/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-9#-56509918]
[2014-01-27 16:46:29,617] [DEBUG] [example-akka.actor.default-dispatcher-8] akka.remote.transport.ProtocolStateActor - started (akka.remote.transport.ProtocolStateActor@23d8417c)
[2014-01-27 16:46:29,617] [DEBUG] [example-akka.actor.default-dispatcher-8] akka.remote.transport.ProtocolStateActor - started (akka.remote.transport.ProtocolStateActor@23d8417c)
[2014-01-27 16:46:29,620] [DEBUG] [example-akka.actor.default-dispatcher-13] akka.remote.transport.ProtocolStateActor - stopped
[2014-01-27 16:46:29,620] [DEBUG] [example-akka.actor.default-dispatcher-13] akka.remote.transport.ProtocolStateActor - stopped
[2014-01-27 16:46:29,620] [ERROR] [example-akka.actor.default-dispatcher-13] akka.remote.EndpointWriter - AssociationError [akka.tcp://example@server002:3552] -> [akka.tcp://cli...@127.0.0.1:49659]: Error [Association failed with [akka.tcp://cli...@127.0.0.1:49659]] [
akka.remote.EndpointAssociationException: Association failed with [akka.tcp://cli...@127.0.0.1:49659]
Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: /127.0.0.1:49659
]
[2014-01-27 16:46:29,620] [ERROR] [example-akka.actor.default-dispatcher-13] akka.remote.EndpointWriter - AssociationError [akka.tcp://example@server002:3552] -> [akka.tcp://cli...@127.0.0.1:49659]: Error [Association failed with [akka.tcp://cli...@127.0.0.1:49659]] [
akka.remote.EndpointAssociationException: Association failed with [akka.tcp://cli...@127.0.0.1:49659]
Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: /127.0.0.1:49659
]
[2014-01-27 16:46:29,621] [DEBUG] [example-akka.actor.default-dispatcher-13] akka.remote.EndpointWriter - Disassociated [akka.tcp://example@server002:3552] -> [akka.tcp://cli...@127.0.0.1:49659]
[2014-01-27 16:46:29,621] [DEBUG] [example-akka.actor.default-dispatcher-13] akka.remote.EndpointWriter - Disassociated [akka.tcp://example@server002:3552] -> [akka.tcp://cli...@127.0.0.1:49659]
[2014-01-27 16:46:29,621] [DEBUG] [example-akka.actor.default-dispatcher-13] akka.remote.EndpointWriter - stopped
[2014-01-27 16:46:29,621] [DEBUG] [example-akka.actor.default-dispatcher-13] akka.remote.EndpointWriter - stopped
[2014-01-27 16:46:29,621] [INFO] [example-akka.actor.default-dispatcher-13] akka.remote.RemoteActorRefProvider$RemoteDeadLetterActorRef - Message [java.lang.String] from Actor[akka://example/user/test/$b#-1937476943] to Actor[akka://example/deadLetters] was not delivered. [3] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[2014-01-27 16:46:29,621] [INFO] [example-akka.actor.default-dispatcher-13] akka.remote.RemoteActorRefProvider$RemoteDeadLetterActorRef - Message [java.lang.String] from Actor[akka://example/user/test/$b#-1937476943] to Actor[akka://example/deadLetters] was not delivered. [3] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[2014-01-27 16:46:29,622] [DEBUG] [example-akka.actor.default-dispatcher-21] akka.remote.ReliableDeliverySupervisor - received AutoReceiveMessage Envelope(Terminated(Actor[akka://example/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-0/endpointWriter#680874170]),Actor[akka://example/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-0/endpointWriter#680874170])
[2014-01-27 16:46:29,622] [DEBUG] [example-akka.actor.default-dispatcher-21] akka.remote.ReliableDeliverySupervisor - received AutoReceiveMessage Envelope(Terminated(Actor[akka://example/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-0/endpointWriter#680874170]),Actor[akka://example/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-0/endpointWriter#680874170])
[2014-01-27 16:46:45,536] [DEBUG] [example-akka.actor.default-dispatcher-19] akka.remote.transport.ProtocolStateActor - stopped
[2014-01-27 16:46:45,536] [DEBUG] [example-akka.actor.default-dispatcher-19] akka.remote.transport.ProtocolStateActor - stopped
[2014-01-27 16:46:45,536] [DEBUG] [example-akka.actor.default-dispatcher-19] akka.remote.EndpointReader - stopped
[2014-01-27 16:46:45,536] [DEBUG] [example-akka.actor.default-dispatcher-19] akka.remote.EndpointReader - stopped
[2014-01-27 16:46:45,537] [DEBUG] [example-akka.actor.default-dispatcher-241] akka.remote.EndpointWriter - received AutoReceiveMessage Envelope(Terminated(Actor[akka://example/system/endpointManager/endpointWriter-akka.tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-1/endpointReader-akka.tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-0#-1170796026]),Actor[akka://example/system/endpointManager/endpointWriter-akka.tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-1/endpointReader-akka.tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-0#-1170796026])
[2014-01-27 16:46:45,537] [DEBUG] [example-akka.actor.default-dispatcher-241] akka.remote.EndpointWriter - received AutoReceiveMessage Envelope(Terminated(Actor[akka://example/system/endpointManager/endpointWriter-akka.tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-1/endpointReader-akka.tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-0#-1170796026]),Actor[akka://example/system/endpointManager/endpointWriter-akka.tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-1/endpointReader-akka.tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-0#-1170796026])
[2014-01-27 16:46:45,537] [DEBUG] [example-akka.actor.default-dispatcher-241] akka.remote.EndpointWriter - Disassociated [akka.tcp://example@server002:3552] <- [akka.tcp://cli...@127.0.0.1:49659]
[2014-01-27 16:46:45,537] [DEBUG] [example-akka.actor.default-dispatcher-241] akka.remote.EndpointWriter - Disassociated [akka.tcp://example@server002:3552] <- [akka.tcp://cli...@127.0.0.1:49659]
[2014-01-27 16:46:45,538] [DEBUG] [example-akka.actor.default-dispatcher-29] akka.remote.EndpointWriter - Disassociated [akka.tcp://example@server002:3552] <- [akka.tcp://cli...@127.0.0.1:49659]
[2014-01-27 16:46:45,538] [DEBUG] [example-akka.actor.default-dispatcher-29] akka.remote.EndpointWriter - Disassociated [akka.tcp://example@server002:3552] <- [akka.tcp://cli...@127.0.0.1:49659]
[2014-01-27 16:46:45,540] [INFO] [example-akka.actor.default-dispatcher-29] akka.actor.LocalActorRef - Message [akka.remote.transport.ActorTransportAdapter$DisassociateUnderlying] from Actor[akka://example/deadLetters] to Actor[akka://example/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fexample%4010.43.15.19%3A36087-5#-1519118151] was not delivered. [4] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[2014-01-27 16:46:45,540] [INFO] [example-akka.actor.default-dispatcher-29] akka.actor.LocalActorRef - Message [akka.remote.transport.ActorTransportAdapter$DisassociateUnderlying] from Actor[akka://example/deadLetters] to Actor[akka://example/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fexample%4010.43.15.19%3A36087-5#-1519118151] was not delivered. [4] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[2014-01-27 16:46:45,540] [DEBUG] [example-akka.actor.default-dispatcher-29] akka.remote.EndpointWriter - stopped
[2014-01-27 16:46:45,540] [DEBUG] [example-akka.actor.default-dispatcher-29] akka.remote.EndpointWriter - stopped
[2014-01-27 16:46:45,541] [DEBUG] [example-akka.actor.default-dispatcher-29] akka.remote.EndpointManager - received AutoReceiveMessage Envelope(Terminated(Actor[akka://example/system/endpointManager/endpointWriter-akka.tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-1#-592860104]),Actor[akka://example/system/endpointManager/endpointWriter-akka.tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-1#-592860104])
[2014-01-27 16:46:45,541] [DEBUG] [example-akka.actor.default-dispatcher-29] akka.remote.EndpointManager - received AutoReceiveMessage Envelope(Terminated(Actor[akka://example/system/endpointManager/endpointWriter-akka.tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-1#-592860104]),Actor[akka://example/system/endpointManager/endpointWriter-akka.tcp%3A%2F%2Fclient%40127.0.0.1%3A49659-1#-592860104])
System sending to the TypedActor
[2014-01-27 16:45:59,065] [INFO] [client-akka.actor.default-dispatcher-9] akka.event.slf4j.Slf4jLogger - Slf4jLogger started
[2014-01-27 16:45:59,065] [INFO] [client-akka.actor.default-dispatcher-9] akka.event.slf4j.Slf4jLogger - Slf4jLogger started
[2014-01-27 16:45:59,076] [DEBUG] [client-akka.actor.default-dispatcher-10] akka.event.EventStream - logger log1-Slf4jLogger started
[2014-01-27 16:45:59,076] [DEBUG] [client-akka.actor.default-dispatcher-10] akka.event.EventStream - logger log1-Slf4jLogger started
[2014-01-27 16:45:59,077] [DEBUG] [client-akka.actor.default-dispatcher-13] akka.event.EventStream - Default Loggers started
[2014-01-27 16:45:59,077] [DEBUG] [client-akka.actor.default-dispatcher-13] akka.event.EventStream - Default Loggers started
[2014-01-27 16:45:59,078] [DEBUG] [client-akka.actor.default-dispatcher-14] akka.actor.LocalActorRefProvider$SystemGuardian - now supervising Actor[akka://client/system/remoting-terminator#358201843]
[2014-01-27 16:45:59,078] [DEBUG] [client-akka.actor.default-dispatcher-14] akka.actor.LocalActorRefProvider$SystemGuardian - now supervising Actor[akka://client/system/remoting-terminator#358201843]
[2014-01-27 16:45:59,091] [DEBUG] [client-akka.actor.default-dispatcher-15] akka.remote.RemoteActorRefProvider$RemotingTerminator - started (akka.remote.RemoteActorRefProvider$RemotingTerminator@319c0bd6)
[2014-01-27 16:45:59,091] [DEBUG] [client-akka.actor.default-dispatcher-15] akka.remote.RemoteActorRefProvider$RemotingTerminator - started (akka.remote.RemoteActorRefProvider$RemotingTerminator@319c0bd6)
[2014-01-27 16:45:59,100] [DEBUG] [client-akka.actor.default-dispatcher-20] akka.actor.LocalActorRefProvider$SystemGuardian - now supervising Actor[akka://client/system/transports#429508472]
[2014-01-27 16:45:59,100] [DEBUG] [client-akka.actor.default-dispatcher-20] akka.actor.LocalActorRefProvider$SystemGuardian - now supervising Actor[akka://client/system/transports#429508472]
[2014-01-27 16:45:59,100] [DEBUG] [client-akka.actor.default-dispatcher-21] akka.remote.Remoting$TransportSupervisor - started (akka.remote.Remoting$TransportSupervisor@1494cb8b)
[2014-01-27 16:45:59,100] [DEBUG] [client-akka.actor.default-dispatcher-21] akka.remote.Remoting$TransportSupervisor - started (akka.remote.Remoting$TransportSupervisor@1494cb8b)
[2014-01-27 16:45:59,105] [DEBUG] [client-akka.actor.default-dispatcher-24] akka.actor.LocalActorRefProvider$SystemGuardian - now supervising Actor[akka://client/system/endpointManager#237680297]
[2014-01-27 16:45:59,105] [DEBUG] [client-akka.actor.default-dispatcher-24] akka.actor.LocalActorRefProvider$SystemGuardian - now supervising Actor[akka://client/system/endpointManager#237680297]
[2014-01-27 16:45:59,106] [DEBUG] [client-akka.actor.default-dispatcher-26] akka.remote.RemoteActorRefProvider$RemotingTerminator - now monitoring Actor[akka://client/system]
[2014-01-27 16:45:59,106] [DEBUG] [client-akka.actor.default-dispatcher-26] akka.remote.RemoteActorRefProvider$RemotingTerminator - now monitoring Actor[akka://client/system]
[2014-01-27 16:45:59,114] [DEBUG] [client-akka.actor.default-dispatcher-27] akka.remote.EndpointManager - started (akka.remote.EndpointManager@c569c60)
[2014-01-27 16:45:59,114] [DEBUG] [client-akka.actor.default-dispatcher-27] akka.remote.EndpointManager - started (akka.remote.EndpointManager@c569c60)
[2014-01-27 16:45:59,278] [DEBUG] [client-akka.actor.default-dispatcher-139] akka.remote.Remoting$TransportSupervisor - now supervising Actor[akka://client/system/transports/akkaprotocolmanager.tcp0#674870860]
[2014-01-27 16:45:59,278] [DEBUG] [client-akka.actor.default-dispatcher-139] akka.remote.Remoting$TransportSupervisor - now supervising Actor[akka://client/system/transports/akkaprotocolmanager.tcp0#674870860]
[2014-01-27 16:45:59,280] [DEBUG] [client-akka.actor.default-dispatcher-140] akka.remote.transport.AkkaProtocolManager - started (akka.remote.transport.AkkaProtocolManager@72813bc1)
[2014-01-27 16:45:59,280] [DEBUG] [client-akka.actor.default-dispatcher-140] akka.remote.transport.AkkaProtocolManager - started (akka.remote.transport.AkkaProtocolManager@72813bc1)
[2014-01-27 16:45:59,292] [DEBUG] [client-akka.actor.default-dispatcher-148] akka.actor.LocalActorRefProvider$SystemGuardian - now supervising Actor[akka://client/system/remote-watcher#1665380928]
[2014-01-27 16:45:59,292] [DEBUG] [client-akka.actor.default-dispatcher-148] akka.actor.LocalActorRefProvider$SystemGuardian - now supervising Actor[akka://client/system/remote-watcher#1665380928]
[2014-01-27 16:45:59,293] [DEBUG] [client-akka.actor.default-dispatcher-151] akka.actor.LocalActorRefProvider$SystemGuardian - now supervising Actor[akka://client/system/remote-deployment-watcher#-59029643]
[2014-01-27 16:45:59,293] [DEBUG] [client-akka.actor.default-dispatcher-151] akka.actor.LocalActorRefProvider$SystemGuardian - now supervising Actor[akka://client/system/remote-deployment-watcher#-59029643]
[2014-01-27 16:45:59,293] [DEBUG] [client-akka.actor.default-dispatcher-152] akka.actor.LocalActorRefProvider$SystemGuardian - now supervising Actor[akka://client/system/deadLetterListener#959546158]
[2014-01-27 16:45:59,293] [DEBUG] [client-akka.actor.default-dispatcher-152] akka.actor.LocalActorRefProvider$SystemGuardian - now supervising Actor[akka://client/system/deadLetterListener#959546158]
[2014-01-27 16:45:59,295] [DEBUG] [client-akka.actor.default-dispatcher-153] akka.remote.RemoteDeploymentWatcher - started (akka.remote.RemoteDeploymentWatcher@66da9ea4)
[2014-01-27 16:45:59,295] [DEBUG] [client-akka.actor.default-dispatcher-153] akka.remote.RemoteDeploymentWatcher - started (akka.remote.RemoteDeploymentWatcher@66da9ea4)
[2014-01-27 16:45:59,295] [DEBUG] [client-akka.actor.default-dispatcher-154] akka.event.DeadLetterListener - started (akka.event.DeadLetterListener@5374a6e2)
[2014-01-27 16:45:59,295] [DEBUG] [client-akka.actor.default-dispatcher-154] akka.event.DeadLetterListener - started (akka.event.DeadLetterListener@5374a6e2)
[2014-01-27 16:45:59,295] [DEBUG] [client-akka.actor.default-dispatcher-155] akka.remote.RemoteWatcher - started (akka.remote.RemoteWatcher@f786a3c)
[2014-01-27 16:45:59,295] [DEBUG] [client-akka.actor.default-dispatcher-155] akka.remote.RemoteWatcher - started (akka.remote.RemoteWatcher@f786a3c)
[2014-01-27 16:45:59,304] [DEBUG] [client-akka.actor.default-dispatcher-158] akka.actor.LocalActorRefProvider$Guardian - now supervising Actor[akka://client/user/$a#-580707033]
[2014-01-27 16:45:59,304] [DEBUG] [client-akka.actor.default-dispatcher-158] akka.actor.LocalActorRefProvider$Guardian - now supervising Actor[akka://client/user/$a#-580707033]
[2014-01-27 16:45:59,327] [DEBUG] [client-akka.actor.default-dispatcher-167] akka.remote.EndpointManager - now supervising Actor[akka://client/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fexample%40server002%3A3552-0#-560920394]
[2014-01-27 16:45:59,327] [DEBUG] [client-akka.actor.default-dispatcher-167] akka.remote.EndpointManager - now supervising Actor[akka://client/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fexample%40server002%3A3552-0#-560920394]
[2014-01-27 16:45:59,345] [DEBUG] [client-akka.actor.default-dispatcher-169] akka.remote.ReliableDeliverySupervisor - started (akka.remote.ReliableDeliverySupervisor@3ebfbbe3)
[2014-01-27 16:45:59,345] [DEBUG] [client-akka.actor.default-dispatcher-169] akka.remote.ReliableDeliverySupervisor - started (akka.remote.ReliableDeliverySupervisor@3ebfbbe3)
[2014-01-27 16:45:59,345] [DEBUG] [client-akka.actor.default-dispatcher-170] akka.remote.ReliableDeliverySupervisor - now monitoring Actor[akka://client/system/endpointManager#237680297]
[2014-01-27 16:45:59,345] [DEBUG] [client-akka.actor.default-dispatcher-170] akka.remote.ReliableDeliverySupervisor - now monitoring Actor[akka://client/system/endpointManager#237680297]
[2014-01-27 16:45:59,346] [DEBUG] [client-akka.actor.default-dispatcher-170] akka.remote.ReliableDeliverySupervisor - now supervising Actor[akka://client/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fexample%40server002%3A3552-0/endpointWriter#1386378256]
[2014-01-27 16:45:59,346] [DEBUG] [client-akka.actor.default-dispatcher-170] akka.remote.ReliableDeliverySupervisor - now supervising Actor[akka://client/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fexample%40server002%3A3552-0/endpointWriter#1386378256]
[2014-01-27 16:45:59,356] [DEBUG] [client-akka.actor.default-dispatcher-168] akka.remote.EndpointWriter - started (akka.remote.EndpointWriter@5b6b9e62)
[2014-01-27 16:45:59,356] [DEBUG] [client-akka.actor.default-dispatcher-168] akka.remote.EndpointWriter - started (akka.remote.EndpointWriter@5b6b9e62)
[2014-01-27 16:45:59,356] [DEBUG] [client-akka.actor.default-dispatcher-168] akka.remote.EndpointWriter - now monitoring Actor[akka://client/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fexample%40server002%3A3552-0#-560920394]
[2014-01-27 16:45:59,356] [DEBUG] [client-akka.actor.default-dispatcher-168] akka.remote.EndpointWriter - now monitoring Actor[akka://client/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fexample%40server002%3A3552-0#-560920394]
[2014-01-27 16:45:59,366] [DEBUG] [client-akka.actor.default-dispatcher-4] akka.remote.transport.AkkaProtocolManager - now supervising Actor[akka://client/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fexample%40server002%3A3552-1#245794225]
[2014-01-27 16:45:59,366] [DEBUG] [client-akka.actor.default-dispatcher-4] akka.remote.transport.AkkaProtocolManager - now supervising Actor[akka://client/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fexample%40server002%3A3552-1#245794225]
[2014-01-27 16:45:59,374] [DEBUG] [client-akka.actor.default-dispatcher-9] akka.remote.transport.ProtocolStateActor - started (akka.remote.transport.ProtocolStateActor@10c789fb)
[2014-01-27 16:45:59,374] [DEBUG] [client-akka.actor.default-dispatcher-9] akka.remote.transport.ProtocolStateActor - started (akka.remote.transport.ProtocolStateActor@10c789fb)
[2014-01-27 16:45:59,485] [DEBUG] [client-akka.actor.default-dispatcher-17] akka.remote.EndpointWriter - Associated [akka.tcp://cli...@127.0.0.1:49659] -> [akka.tcp://example@server002:3552]
[2014-01-27 16:45:59,485] [DEBUG] [client-akka.actor.default-dispatcher-17] akka.remote.EndpointWriter - Associated [akka.tcp://cli...@127.0.0.1:49659] -> [akka.tcp://example@server002:3552]
[2014-01-27 16:45:59,485] [DEBUG] [client-akka.actor.default-dispatcher-17] akka.remote.EndpointWriter - now supervising Actor[akka://client/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fexample%40server002%3A3552-0/endpointWriter/endpointReader-akka.tcp%3A%2F%2Fexample%40server002%3A3552-0#739192055]
[2014-01-27 16:45:59,485] [DEBUG] [client-akka.actor.default-dispatcher-17] akka.remote.EndpointWriter - now supervising Actor[akka://client/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fexample%40server002%3A3552-0/endpointWriter/endpointReader-akka.tcp%3A%2F%2Fexample%40server002%3A3552-0#739192055]
[2014-01-27 16:45:59,490] [DEBUG] [client-akka.actor.default-dispatcher-24] akka.remote.EndpointWriter - sending message RemoteMessage: [SelectChildName(user,SelectChildName(test,MethodCall(public abstract java.lang.String com.markit.bonds.akka.ReconnectFailExample$ExampleService.hello(),null)))] to [Actor[akka.tcp://example@server002:3552/]]<+[akka.tcp://example@server002:3552/] from [Actor[akka://client/temp/$a]]
[2014-01-27 16:45:59,490] [DEBUG] [client-akka.actor.default-dispatcher-24] akka.remote.EndpointWriter - sending message RemoteMessage: [SelectChildName(user,SelectChildName(test,MethodCall(public abstract java.lang.String com.markit.bonds.akka.ReconnectFailExample$ExampleService.hello(),null)))] to [Actor[akka.tcp://example@server002:3552/]]<+[akka.tcp://example@server002:3552/] from [Actor[akka://client/temp/$a]]
[2014-01-27 16:45:59,493] [DEBUG] [client-akka.actor.default-dispatcher-26] akka.remote.EndpointReader - started (akka.remote.EndpointReader@197ebe66)
[2014-01-27 16:45:59,493] [DEBUG] [client-akka.actor.default-dispatcher-26] akka.remote.EndpointReader - started (akka.remote.EndpointReader@197ebe66)
[2014-01-27 16:45:59,493] [DEBUG] [client-akka.actor.default-dispatcher-26] akka.remote.EndpointReader - now monitoring Actor[akka://client/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fexample%40server002%3A3552-0/endpointWriter#1386378256]
[2014-01-27 16:45:59,493] [DEBUG] [client-akka.actor.default-dispatcher-26] akka.remote.EndpointReader - now monitoring Actor[akka://client/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fexample%40server002%3A3552-0/endpointWriter#1386378256]
[2014-01-27 16:45:59,494] [DEBUG] [client-akka.actor.default-dispatcher-26] akka.serialization.Serialization(akka://client) - Using serializer[akka.remote.serialization.MessageContainerSerializer] for message [akka.actor.SelectChildName]
[2014-01-27 16:45:59,494] [DEBUG] [client-akka.actor.default-dispatcher-26] akka.serialization.Serialization(akka://client) - Using serializer[akka.remote.serialization.MessageContainerSerializer] for message [akka.actor.SelectChildName]
[2014-01-27 16:45:59,500] [DEBUG] [client-akka.actor.default-dispatcher-135] akka.serialization.Serialization(akka://client) - Using serializer[akka.serialization.JavaSerializer] for message [akka.actor.TypedActor$MethodCall]
[2014-01-27 16:45:59,500] [DEBUG] [client-akka.actor.default-dispatcher-135] akka.serialization.Serialization(akka://client) - Using serializer[akka.serialization.JavaSerializer] for message [akka.actor.TypedActor$MethodCall]
[2014-01-27 16:45:59,582] [DEBUG] [client-akka.actor.default-dispatcher-136] akka.remote.EndpointWriter - received local message RemoteMessage: [hello] to [Actor[akka://client/temp/$a]]<+[akka://client/temp/$a] from [Actor[akka.tcp://example@server002:3552/user/test/$a#-1283403612]]
[2014-01-27 16:45:59,582] [DEBUG] [client-akka.actor.default-dispatcher-136] akka.remote.EndpointWriter - received local message RemoteMessage: [hello] to [Actor[akka://client/temp/$a]]<+[akka://client/temp/$a] from [Actor[akka.tcp://example@server002:3552/user/test/$a#-1283403612]]
hello
[2014-01-27 16:46:22,591] [DEBUG] [client-akka.actor.default-dispatcher-169] akka.remote.EndpointReader - stopped
[2014-01-27 16:46:22,591] [DEBUG] [client-akka.actor.default-dispatcher-169] akka.remote.EndpointReader - stopped
[2014-01-27 16:46:22,593] [DEBUG] [client-akka.actor.default-dispatcher-6] akka.remote.transport.ProtocolStateActor - stopped
[2014-01-27 16:46:22,593] [DEBUG] [client-akka.actor.default-dispatcher-6] akka.remote.transport.ProtocolStateActor - stopped
[2014-01-27 16:46:22,593] [INFO] [client-akka.actor.default-dispatcher-6] akka.actor.LocalActorRef - Message [akka.remote.transport.AssociationHandle$Disassociated] from Actor[akka://client/deadLetters] to Actor[akka://client/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fexample%40server002%3A3552-1#245794225] was not delivered. [1] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[2014-01-27 16:46:22,593] [INFO] [client-akka.actor.default-dispatcher-6] akka.actor.LocalActorRef - Message [akka.remote.transport.AssociationHandle$Disassociated] from Actor[akka://client/deadLetters] to Actor[akka://client/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fexample%40server002%3A3552-1#245794225] was not delivered. [1] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[2014-01-27 16:46:22,593] [DEBUG] [client-akka.actor.default-dispatcher-6] akka.remote.EndpointWriter - received AutoReceiveMessage Envelope(Terminated(Actor[akka://client/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fexample%40server002%3A3552-0/endpointWriter/endpointReader-akka.tcp%3A%2F%2Fexample%40server002%3A3552-0#739192055]),Actor[akka://client/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fexample%40server002%3A3552-0/endpointWriter/endpointReader-akka.tcp%3A%2F%2Fexample%40server002%3A3552-0#739192055])
[2014-01-27 16:46:22,593] [DEBUG] [client-akka.actor.default-dispatcher-6] akka.remote.EndpointWriter - received AutoReceiveMessage Envelope(Terminated(Actor[akka://client/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fexample%40server002%3A3552-0/endpointWriter/endpointReader-akka.tcp%3A%2F%2Fexample%40server002%3A3552-0#739192055]),Actor[akka://client/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fexample%40server002%3A3552-0/endpointWriter/endpointReader-akka.tcp%3A%2F%2Fexample%40server002%3A3552-0#739192055])
[2014-01-27 16:46:22,594] [INFO] [client-akka.actor.default-dispatcher-6] akka.actor.LocalActorRef - Message [akka.actor.FSM$Timer] from Actor[akka://client/deadLetters] to Actor[akka://client/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fexample%40server002%3A3552-1#245794225] was not delivered. [2] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[2014-01-27 16:46:22,594] [INFO] [client-akka.actor.default-dispatcher-6] akka.actor.LocalActorRef - Message [akka.actor.FSM$Timer] from Actor[akka://client/deadLetters] to Actor[akka://client/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fexample%40server002%3A3552-1#245794225] was not delivered. [2] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[2014-01-27 16:46:22,594] [DEBUG] [client-akka.actor.default-dispatcher-6] akka.remote.EndpointWriter - Disassociated [akka.tcp://cli...@127.0.0.1:49659] -> [akka.tcp://example@server002:3552]
[2014-01-27 16:46:22,594] [DEBUG] [client-akka.actor.default-dispatcher-6] akka.remote.EndpointWriter - Disassociated [akka.tcp://cli...@127.0.0.1:49659] -> [akka.tcp://example@server002:3552]
[2014-01-27 16:46:22,598] [DEBUG] [client-akka.actor.default-dispatcher-22] akka.remote.EndpointWriter - restarting
[2014-01-27 16:46:22,598] [DEBUG] [client-akka.actor.default-dispatcher-22] akka.remote.EndpointWriter - restarting
[2014-01-27 16:46:22,599] [DEBUG] [client-akka.actor.default-dispatcher-17] akka.remote.EndpointWriter - Disassociated [akka.tcp://cli...@127.0.0.1:49659] -> [akka.tcp://example@server002:3552]
[2014-01-27 16:46:22,599] [DEBUG] [client-akka.actor.default-dispatcher-17] akka.remote.EndpointWriter - Disassociated [akka.tcp://cli...@127.0.0.1:49659] -> [akka.tcp://example@server002:3552]
[2014-01-27 16:46:22,599] [INFO] [client-akka.actor.default-dispatcher-17] akka.actor.LocalActorRef - Message [akka.remote.transport.ActorTransportAdapter$DisassociateUnderlying] from Actor[akka://client/deadLetters] to Actor[akka://client/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fexample%40server002%3A3552-1#245794225] was not delivered. [3] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[2014-01-27 16:46:22,599] [INFO] [client-akka.actor.default-dispatcher-17] akka.actor.LocalActorRef - Message [akka.remote.transport.ActorTransportAdapter$DisassociateUnderlying] from Actor[akka://client/deadLetters] to Actor[akka://client/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fexample%40server002%3A3552-1#245794225] was not delivered. [3] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[2014-01-27 16:46:22,600] [DEBUG] [client-akka.actor.default-dispatcher-18] akka.remote.EndpointWriter - restarted
[2014-01-27 16:46:22,600] [DEBUG] [client-akka.actor.default-dispatcher-18] akka.remote.EndpointWriter - restarted
[2014-01-27 16:46:22,603] [DEBUG] [client-akka.actor.default-dispatcher-23] akka.remote.transport.AkkaProtocolManager - now supervising Actor[akka://client/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fexample%40server002%3A3552-2#-226377435]
[2014-01-27 16:46:22,603] [DEBUG] [client-akka.actor.default-dispatcher-23] akka.remote.transport.AkkaProtocolManager - now supervising Actor[akka://client/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fexample%40server002%3A3552-2#-226377435]
[2014-01-27 16:46:22,604] [DEBUG] [client-akka.actor.default-dispatcher-23] akka.remote.transport.ProtocolStateActor - started (akka.remote.transport.ProtocolStateActor@707efa96)
[2014-01-27 16:46:22,604] [DEBUG] [client-akka.actor.default-dispatcher-23] akka.remote.transport.ProtocolStateActor - started (akka.remote.transport.ProtocolStateActor@707efa96)
[2014-01-27 16:46:22,612] [DEBUG] [client-akka.actor.default-dispatcher-147] akka.remote.EndpointWriter - Associated [akka.tcp://cli...@127.0.0.1:49659] -> [akka.tcp://example@server002:3552]
[2014-01-27 16:46:22,612] [DEBUG] [client-akka.actor.default-dispatcher-147] akka.remote.EndpointWriter - Associated [akka.tcp://cli...@127.0.0.1:49659] -> [akka.tcp://example@server002:3552]
[2014-01-27 16:46:22,613] [DEBUG] [client-akka.actor.default-dispatcher-147] akka.remote.EndpointWriter - now supervising Actor[akka://client/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fexample%40server002%3A3552-0/endpointWriter/endpointReader-akka.tcp%3A%2F%2Fexample%40server002%3A3552-0#125954714]
[2014-01-27 16:46:22,613] [DEBUG] [client-akka.actor.default-dispatcher-147] akka.remote.EndpointWriter - now supervising Actor[akka://client/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fexample%40server002%3A3552-0/endpointWriter/endpointReader-akka.tcp%3A%2F%2Fexample%40server002%3A3552-0#125954714]
[2014-01-27 16:46:22,615] [DEBUG] [client-akka.actor.default-dispatcher-152] akka.remote.EndpointReader - started (akka.remote.EndpointReader@7e8905bd)
[2014-01-27 16:46:22,615] [DEBUG] [client-akka.actor.default-dispatcher-152] akka.remote.EndpointReader - started (akka.remote.EndpointReader@7e8905bd)
[2014-01-27 16:46:22,615] [DEBUG] [client-akka.actor.default-dispatcher-152] akka.remote.EndpointReader - now monitoring Actor[akka://client/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fexample%40server002%3A3552-0/endpointWriter#1386378256]
[2014-01-27 16:46:22,615] [DEBUG] [client-akka.actor.default-dispatcher-152] akka.remote.EndpointReader - now monitoring Actor[akka://client/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fexample%40server002%3A3552-0/endpointWriter#1386378256]
[2014-01-27 16:46:29,592] [DEBUG] [client-akka.actor.default-dispatcher-7] akka.remote.EndpointWriter - sending message RemoteMessage: [SelectChildName(user,SelectChildName(test,MethodCall(public abstract java.lang.String com.markit.bonds.akka.ReconnectFailExample$ExampleService.hello(),null)))] to [Actor[akka.tcp://example@server002:3552/]]<+[akka.tcp://example@server002:3552/] from [Actor[akka://client/temp/$b]]
[2014-01-27 16:46:29,592] [DEBUG] [client-akka.actor.default-dispatcher-7] akka.remote.EndpointWriter - sending message RemoteMessage: [SelectChildName(user,SelectChildName(test,MethodCall(public abstract java.lang.String com.markit.bonds.akka.ReconnectFailExample$ExampleService.hello(),null)))] to [Actor[akka.tcp://example@server002:3552/]]<+[akka.tcp://example@server002:3552/] from [Actor[akka://client/temp/$b]]
Patrik Nordwall
Feb 4, 2014, 2:36:29 AM2/4/14
to akka...@googlegroups.com
Hi Brian,
Sorry for the delay, we have not found the time to analyze the logs and the reproducer. I have created a ticket for it: https://www.assembla.com/spaces/akka/tickets/3860
It would be interesting to hear if you have the same problem after upgrading to Akka 2.3.0-RC2.
/Patrik
Patrik Nordwall
Typesafe - Reactive apps on the JVM
Twitter: @patriknw
Reply all
Reply to author
Forward
0 new messages