how to avoid or catch an akka.remote.transport.Transport$InvalidAssociationException

[David]

Hello,

I am using Akka remote 2.4.17 with Scala 2.12.1

I have two actor systems: system1 and system2 

An actor1 is created on system1 and an actor2 is created on system2

actor2 looks up actor1 and they exchange messages.

At some point, I shut down system2 and system1 crashes with the following error message:

[error] a.r.EndpointWriter - AssociationError [akka.tcp://sys...@127.0.0.1:2552] <- [akka.tcp://sys...@127.0.0.1:2553]: Error [Shut down address: akka.tcp://sys...@127.0.0.1:2553] [

akka.remote.ShutDownAssociation: Shut down address: akka.tcp://sys...@127.0.0.1:2553

Caused by: akka.remote.transport.Transport$InvalidAssociationException: The remote system terminated the association because it is shutting down.

]

What should I to ensure that system1 stays up and running when system2 goes down?

Thanks,

David

 

[Patrik Nordwall]

system1 doesn't crash, that is just a harmless log message, or do you see anything that actually doesn't work in system1 after that?

/Patrik

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

[David]

Thanks Patrick,

You are right.

System1 runs in Play and indeed the application only crashes when it is launched with sbt.  The problem disappears when the application is properly built and launched.

However, I am still interested in knowing if there is a way for the system1 application to be notified of the akka.remote.ShutDownAssociation event so that it can trigger some cleanup after losing the remote actor.  Looking at the log, all the information needed is there but how can I access it?

Best,

David 

[Arnout Engelen]

On Fri, Apr 14, 2017 at 10:39 AM, David <massar...@gmail.com> wrote:

I am still interested in knowing if there is a way for the system1 application to be notified of the akka.remote.ShutDownAssociation event so that it can trigger some cleanup after losing the remote actor.  Looking at the log, all the information needed is there but how can I access it?

I think you can subscribe to association errors using http://doc.akka.io/docs/akka/current/scala/remoting.html#Remote_Events

Kind regards,

Arnout

 

To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscribe@googlegroups.com.

To post to this group, send email to akka...@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

--

Arnout Engelen

Senior Software Engineer

E: arnout....@lightbend.com

T: https://twitter.com/raboofje

[David]

Thanks Arnout.  It totally works.

I solved my problem by adding the following to actor1:

    override def preStart: Unit =

    {

        super.preStart()

        context.system.eventStream.subscribe(self, classOf[akka.remote.DisassociatedEvent])

    }

Then I just had to add and entry for DisassociatedEvent in receive to trigger the behavior I wanted

    override def receive =

    { 

        case event: akka.remote.DisassociatedEvent => {

            ???

Best,

David

[Patrik Nordwall]

but I'm still confused by that you say that something crash, what is it that stops working?

[David]

Hi Patrick,

Sorry for not being very clear.

I am using Akka together with Play.  I have two Play applications that both create actors.  These actors use Akka remote to communicate with each other.  During my initial tests, I was launching the play applications using "sbt run" and when I was closing one Play application, the other crashed with the [error] a.r.EndpointWriter - AssociationError that I described in my initial post.  This problem disappears when I compile the apps instead of launching them from sbt.

Once this initial "problem" solved, I still needed to detect when the connection to the remote actor is lost, this problem was solved by subscribing to the right event stream.

Best,

David 

[Jakub Liska]

Hey, I think that the : 

ERROR: akka.remote.transport.Transport$InvalidAssociationException: The remote system terminated the association because it is shutting down.

should not exist in case of graceful remote system shutdown. Imagine that you have a system that is forking JVMs with remote actor system

for each task or job (the remote actor system serves only as a way of passing input and result) and you're getting thousands of these ERRORs,

then somebody uses your software and he gets thousands of these errors.

Btw this is nice David, unfortunately it doesn't solve the ERROR logging problem.

  override def preStart(): Unit = context.system.eventStream.subscribe(self, classOf[DisassociatedEvent])

  override def unhandled(message: Any): Unit = message match {
    case DisassociatedEvent(local, remote, _) =>
      log.info(s"Forked executor system $remote disassociated from $local ...")
    case x => super.unhandled(x)
  }

Anybody has an idea how to avoid these ERROR log messages ?

[Zachary Marois]

If these are harmless, would ignoring any ERRORs from the akka.remote.EndpointWriter class be safe? I definitely worry about hiding real errors, but I also don't know how to deal with all these false-negatives.

[Patrik Nordwall]

It should probably be changed to Warning level. PR fixing this would be welcome.

/Patrik

--
*****************************************************************************************************
** New discussion forum: https://discuss.akka.io/ replacing akka-user google-group soon.
** This group will soon be put into read-only mode, and replaced by discuss.akka.io
** More details: https://akka.io/blog/news/2018/03/13/discuss.akka.io-announced
*****************************************************************************************************