Re: [akka-user] How to gracefully disconnect actor systems

1,013 views
Skip to first unread message

Roland Kuhn

unread,
Mar 2, 2013, 9:21:50 AM3/2/13
to akka...@googlegroups.com
Hi Daniel,

1 mar 2013 kl. 14:32 skrev Daniel Krzywicki:

Hi,

I want to write a simple client-server app using akka:
  • a server actor awaits for requests in a first actor system
  • a new actor system is created, along with a client actor, which submits a task remotedly to the server actor
  • the server actor computes some result and sends it back to the client actor
  • the client actor processess the results, then shuts down its actor system.

The server actor does not retain references to the client. However, when the client system shuts down, the server starts throwing akka.remote.EndpointException.

Thanks for trying out 2.2-M1! The new remoting maps the connection life cycle to an actor’s life cycle, and that actor then “fails” when the connection is closed. We have a ticket in the pipeline (#2824 - Change logging of actor failures) to enable different logging strategies for actors according to the application’s needs, and we will apply this to sanitize the overly verbose logging of the remoting layer.

Until then please just ignore these exceptions, they are expected when the client disconnects.

I’m assuming that afterwards remoting still works, right? If not, then please give more information (confirm the Akka version used and include full stack traces).

Regards,

Roland


How to gracefully disconnect these systems so that the client one can shutdown without the server one bothering?

Best regards

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://akka.io/faq/
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 



Dr. Roland Kuhn
Akka Tech Lead
Typesafe – Empowering professional developers to build amazing apps.
twitter: @rolandkuhn

Siddhu Warrier

unread,
May 1, 2014, 2:44:40 PM5/1/14
to akka...@googlegroups.com
Hi,

Apologies for resurrecting a long-dead thread, but I am facing a similar problem when using remoting with Akka 2.3.0 with Scala 2.10.2.


I have two actor systems:
  • remote-registration-service: Runs on localhost:8245, and "registers" clients
  • registration-client-system-{UUID}: Runs on localhost:2252 and sends registration request to the "remote-registration-service" using Patterns.ask

I start up the remote-registration-service, and then start up the registration-client-system. The registration-client-system then uses an ActorSelection, a resolveOnce, and a Patterns.ask to send a registration request to, and receive a response from, the remote-registration-service. Here's the relevant code snippets (Java):

                    Future<ActorRef> registrarActorRefFuture = system.actorSelection(registrationServicePath).resolveOne(new
                            FiniteDuration
                            (QUEUE_REGISTRATION_RETRY_TIMEOUT_MS,
                                    TimeUnit.MILLISECONDS));
                    final ActorRef registrarActorRef = Await.result(registrarActorRefFuture, DURATION);
                    Future<Object> registrationResultFuture = Patterns.ask(registrationClientActor, new RegisterMessage(clientId,
                priority), DURATION.toMillis());
                    Object regResult = Await.result(registrationResultFuture, DURATION);

The registration process executes successfully and I receive a response back from the remote-registration-service. The registration-client-service then shuts down.

At this point, the remote-registration-service starts to print out dead letter messages as shown below. I have marked the messages from the remote-registration-service in Bold

2014-05-01 19:33:19,965 DEBUG: c.c.c.c.s.QueueRegistrationService | ...Registration complete. Starting shutdown of registration-client-system... 
[INFO] [05/01/2014 19:33:19.974] [registration-client-system0c0be54e-87d8-4920-8725-019b2e8ab0f0-akka.remote.default-remote-dispatcher-6] [akka.tcp://registration-client-system0c0be...@127.0.0.1:2552/system/remoting-terminator] Shutting down remote daemon.
[INFO] [05/01/2014 19:33:19.976] [registration-client-system0c0be54e-87d8-4920-8725-019b2e8ab0f0-akka.remote.default-remote-dispatcher-6] [akka.tcp://registration-client-system0c0be...@127.0.0.1:2552/system/remoting-terminator] Remote daemon shut down; proceeding with flushing remote transports.
[INFO] [05/01/2014 19:33:19.992] [registration-client-system0c0be54e-87d8-4920-8725-019b2e8ab0f0-akka.actor.default-dispatcher-3] [akka://registration-client-system0c0be54e-87d8-4920-8725-019b2e8ab0f0/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fremote-registration-service%40127.0.0.1%3A8245-0/endpointWriter/endpointReader-akka.tcp%3A%2F%2Fremote-registration-service%40127.0.0.1%3A8245-0] Message [akka.remote.transport.AssociationHandle$Disassociated] from Actor[akka://registration-client-system0c0be54e-87d8-4920-8725-019b2e8ab0f0/deadLetters] to Actor[akka://registration-client-system0c0be54e-87d8-4920-8725-019b2e8ab0f0/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fremote-registration-service%40127.0.0.1%3A8245-0/endpointWriter/endpointReader-akka.tcp%3A%2F%2Fremote-registration-service%40127.0.0.1%3A8245-0#-1326837246] was not delivered. [1] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[INFO] [05/01/2014 19:33:19.993] [registration-client-system0c0be54e-87d8-4920-8725-019b2e8ab0f0-akka.actor.default-dispatcher-3] [akka://registration-client-system0c0be54e-87d8-4920-8725-019b2e8ab0f0/system/transports/akkaprotocolmanager.tcp1/akkaProtocol-tcp%3A%2F%2Fremote-registration-service%40127.0.0.1%3A8245-1] Message [akka.remote.transport.AssociationHandle$Disassociated] from Actor[akka://registration-client-system0c0be54e-87d8-4920-8725-019b2e8ab0f0/deadLetters] to Actor[akka://registration-client-system0c0be54e-87d8-4920-8725-019b2e8ab0f0/system/transports/akkaprotocolmanager.tcp1/akkaProtocol-tcp%3A%2F%2Fremote-registration-service%40127.0.0.1%3A8245-1#-606871149] was not delivered. [2] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[INFO] [05/01/2014 19:33:19.993] [remote-registration-service-akka.actor.default-dispatcher-2] [akka://remote-registration-service/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fremote-registration-service%40127.0.0.1%3A57446-1] Message [akka.remote.transport.AssociationHandle$Disassociated] from Actor[akka://remote-registration-service/deadLetters] to Actor[akka://remote-registration-service/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fremote-registration-service%40127.0.0.1%3A57446-1#-707785622] was not delivered. [1] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
akka.remote.ShutDownAssociation: Shut down address: akka.tcp://registration-client-system0c0be...@127.0.0.1:2552
Caused by: akka.remote.transport.Transport$InvalidAssociationException: The remote system terminated the association because it is shutting down.
]
[INFO] [05/01/2014 19:33:20.003] [remote-registration-service-akka.actor.default-dispatcher-3] [akka://remote-registration-service/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fremote-registration-service%40127.0.0.1%3A57446-1] Message [akka.remote.transport.ActorTransportAdapter$DisassociateUnderlying] from Actor[akka://remote-registration-service/deadLetters] to Actor[akka://remote-registration-service/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fremote-registration-service%40127.0.0.1%3A57446-1#-707785622] was not delivered. [2] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[INFO] [05/01/2014 19:33:20.011] [ForkJoinPool-5-worker-15] [Remoting] Remoting shut down
[INFO] [05/01/2014 19:33:20.011] [registration-client-system0c0be54e-87d8-4920-8725-019b2e8ab0f0-akka.remote.default-remote-dispatcher-6] [akka.tcp://registration-client-system0c0be...@127.0.0.1:2552/system/remoting-terminator] Remoting shut down.

Then, I restart the registration-client-system. This time, it comes up with a different actor-system-name because the suffix UUID is different. But this time, even retrieving the actorref from the ActorSelection fails as shown in the error logs below:

2014-05-01 19:33:22,084 ERROR: c.c.c.c.s.QueueRegistrationService | FAIL! 
akka.actor.ActorNotFound: Actor not found for: ActorSelection[Anchor(akka.tcp://remote-regist...@127.0.0.1:8245/), Path(/user/registrar)]
at akka.actor.ActorSelection$$anonfun$resolveOne$1.apply(ActorSelection.scala:65) ~[akka-actor_2.10-2.3.0.jar:na]
at akka.actor.ActorSelection$$anonfun$resolveOne$1.apply(ActorSelection.scala:63) ~[akka-actor_2.10-2.3.0.jar:na]
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:29) ~[scala-library-2.10.2.jar:na]
at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.processBatch$1(BatchingExecutor.scala:67) ~[akka-actor_2.10-2.3.0.jar:na]
at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:82) ~[akka-actor_2.10-2.3.0.jar:na]
at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59) ~[akka-actor_2.10-2.3.0.jar:na]
at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59) ~[akka-actor_2.10-2.3.0.jar:na]
at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72) ~[scala-library-2.10.2.jar:na]
at akka.dispatch.BatchingExecutor$Batch.run(BatchingExecutor.scala:58) ~[akka-actor_2.10-2.3.0.jar:na]
at akka.dispatch.ExecutionContexts$sameThreadExecutionContext$.unbatchedExecute(Future.scala:74) ~[akka-actor_2.10-2.3.0.jar:na]
at akka.dispatch.BatchingExecutor$class.execute(BatchingExecutor.scala:110) ~[akka-actor_2.10-2.3.0.jar:na]
at akka.dispatch.ExecutionContexts$sameThreadExecutionContext$.execute(Future.scala:73) ~[akka-actor_2.10-2.3.0.jar:na]
at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:37) ~[scala-library-2.10.2.jar:na]
at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:133) ~[scala-library-2.10.2.jar:na]
at akka.pattern.PromiseActorRef.$bang(AskSupport.scala:267) ~[akka-actor_2.10-2.3.0.jar:na]
at akka.remote.DefaultMessageDispatcher.dispatch(Endpoint.scala:86) ~[akka-remote_2.10-2.3.0.jar:na]
at akka.remote.EndpointReader$$anonfun$receive$2.applyOrElse(Endpoint.scala:804) ~[akka-remote_2.10-2.3.0.jar:na]
at akka.actor.Actor$class.aroundReceive(Actor.scala:465) ~[akka-actor_2.10-2.3.0.jar:na]
at akka.remote.EndpointActor.aroundReceive(Endpoint.scala:408) ~[akka-remote_2.10-2.3.0.jar:na]
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516) ~[akka-actor_2.10-2.3.0.jar:na]
at akka.actor.ActorCell.invoke(ActorCell.scala:487) ~[akka-actor_2.10-2.3.0.jar:na]
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238) ~[akka-actor_2.10-2.3.0.jar:na]
at akka.dispatch.Mailbox.run(Mailbox.scala:220) ~[akka-actor_2.10-2.3.0.jar:na]
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393) ~[akka-actor_2.10-2.3.0.jar:na]
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) ~[scala-library-2.10.2.jar:na]
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) ~[scala-library-2.10.2.jar:na]
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) ~[scala-library-2.10.2.jar:na]
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) ~[scala-library-2.10.2.jar:na]


At this point, the registration-client-system-{UUID} begins its shutdown process. Then, I see a very interesting error message from the remote-registration-service which seems to indicate that the remote-registration-service is aware of the registration-client-system, but is for some reason not sending any messages back to it!

akka.remote.ShutDownAssociation: Shut down address: akka.tcp://registration-client-systemfd1fa...@127.0.0.1:2552
Caused by: akka.remote.transport.Transport$InvalidAssociationException: The remote system terminated the association because it is shutting down.

Is there something wrong with our remoting configuration? Could you please help?

Thanks,

Siddhu

Akka Team

unread,
May 2, 2014, 5:15:41 AM5/2/14
to Akka User List
Hi,




The registration process executes successfully and I receive a response back from the remote-registration-service. The registration-client-service then shuts down.

At this point, the remote-registration-service starts to print out dead letter messages as shown below. I have marked the messages from the remote-registration-service in Bold

These are internal messages, you should not be concerned about them. Dead letters are not harmful per se, they usually happen when an actor is stopped but someone still sends them a few messages -- they get delivered to dead letters.
 

akka.remote.ShutDownAssociation: Shut down address: akka.tcp://registration-client-system0c0be...@127.0.0.1:2552
Caused by: akka.remote.transport.Transport$InvalidAssociationException: The remote system terminated the association because it is shutting down.
]
 
The above line means that the remote system has been shut down, and we cannot connect back ever to it.

As a sidenote, be aware that akka-remoting is not a client-server architecture (see http://doc.akka.io/docs/akka/2.3.2/general/remoting.html#Peer-to-Peer_vs__Client-Server) but a peer-to-peer architecture, so any side might open a connection.


At this point, the registration-client-system-{UUID} begins its shutdown process. Then, I see a very interesting error message from the remote-registration-service which seems to indicate that the remote-registration-service is aware of the registration-client-system, but is for some reason not sending any messages back to it!

Yes, it looks like it was connected before. You can get more information by elevating the logging levels in remoting (http://doc.akka.io/docs/akka/2.3.2/general/configuration.html#akka-remote), you can use akka.remote.log-sent-messages, akka.remote.log-received-messages to see what messages are sent and received.

It can be that your timeout is just too small, I recommend trying the resolution several times before giving up. Also, instead of using a Future and Await you should use ordinary message sends for identification: http://doc.akka.io/docs/akka/2.3.2/scala/actors.html#Identifying_Actors_via_Actor_Selection

And you should upgrade to Akka 2.3.2 :)

-Endre
 

>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.



--
Akka Team
Typesafe - The software stack for applications that scale
Blog: letitcrash.com
Twitter: @akkateam
Reply all
Reply to author
Forward
0 new messages