[2.1 RC1] After remote connect, system waits on execution-pool-keepalive before full shutdown

[cessationoftime]

1. If I use akka-remote to communicate 2-way with a remote system like in this gist. I end up having to wait on the execution-pool-keepalive timeout at shutdown so that I dont get back to the sbt prompt until the timeout has passed. Is this expected? How can I prevent having to wait on this timeout?

Note that in the gist shutdownLog file the timestamp difference between the last two lines is precisely 30s. It used to take 60s before I modified my timeout (so it definitely is this timeout).

1. # (I) Length in akka.time-unit how long core threads will be kept alive if
   
2. # idling
3. akka.remote.netty.execution-pool-keepalive = 30s

Could this be related? I got the following error once while shutting down, but the system didnt ever fully shutdown after that, this error is also in the gist.

[ERROR] [11/07/2012 21:19:27.657] [node-scheduler-1] [ActorSystem(node)] Uncaught error from thread [node-scheduler-1]

java.lang.IllegalStateException: cannot enqueue after shutdown

        at akka.util.internal.HashedWheelTimer.scheduleTimeout(HashedWheelTimer.java:273)

        at akka.util.internal.HashedWheelTimer$Worker.fetchExpiredTimeouts(HashedWheelTimer.java:355)

        at akka.util.internal.HashedWheelTimer$Worker.fetchExpiredTimeouts(HashedWheelTimer.java:321)

        at akka.util.internal.HashedWheelTimer$Worker.run(HashedWheelTimer.java:308)

        at java.lang.Thread.run(Unknown Source)

Thanks,

Chris

[Björn Antonsson]

Hi Chris,

Are you running multiple actor systems inside the sbt test, and if so, are you shutting down them all?

Normal remoting shouldn't hang in this way.

B/

-- 

Björn Antonsson

Typesafe - The software stack for applications that scale

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://akka.io/faq/
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To post to this group, send email to akka...@googlegroups.com.
To unsubscribe from this group, send email to akka-user+...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user?hl=en.
 
 

[cessationoftime]

I probabIy should have been more thorough in my initial description, my Netty configuration is a little non-standard. I am giving the Netty server an IP address of "0.0.0.0", so the server will accept incoming connections directed to an IP other than the local one. I then give the Akka system my router IP. With this setup I can accept external NAT connections through my router.   Works great except for the shutdown timeout.  Does the Netty shutdown sequence require an IP address as an input somewhere? Perhaps I am supplying it with the wrong value when I shutdown.

This isnt running in a test, this is remoting from one PC to another.

At the end of NettyRemoteTransport.shutdown, If I shutdown the executionHandler like this:

       clientChannelFactory.releaseExternalResources()

       PipelineFactory.executionHandler foreach { _.releaseExternalResources() }

Then the timeout problem goes away.  However, I found a commit by Victor with a comment that explicitly says not to do this. It is pretty old and the comment is gone, is that warning still valid?

I will do a Pull Request shortly so you can view my change, and hopefully Pull it if this gets solved.

Chris

[Björn Antonsson]

Hi Chris,

On Saturday, 10 November 2012 at 02:27, cessationoftime wrote:

I probabIy should have been more thorough in my initial description, my Netty configuration is a little non-standard. I am giving the Netty server an IP address of "0.0.0.0", so the server will accept incoming connections directed to an IP other than the local one. I then give the Akka system my router IP. With this setup I can accept external NAT connections through my router.   Works great except for the shutdown timeout.  Does the Netty shutdown sequence require an IP address as an input somewhere? Perhaps I am supplying it with the wrong value when I shutdown.

I haven't looked into that. But that's definitely something to look at. Could you open a ticket for this time-out issue? https://www.assembla.com/spaces/akka/

 

This isnt running in a test, this is remoting from one PC to another.

At the end of NettyRemoteTransport.shutdown, If I shutdown the executionHandler like this:

       clientChannelFactory.releaseExternalResources()

       PipelineFactory.executionHandler foreach { _.releaseExternalResources() }

Then the timeout problem goes away.  However, I found a commit by Victor with a comment that explicitly says not to do this. It is pretty old and the comment is gone, is that warning still valid?

The PipelineFactory is local to the NettyRemoteTransport, and it has its own ThreadPool, so I think that it could be shut down when the NettyRemoteTransport is shut down. The question is why you have to do this in your case.

 

I will do a Pull Request shortly so you can view my change, and hopefully Pull it if this gets solved.

That would be great.

B/

[cessationoftime]

I added a ticket for this: it is #2697.

Also, I separated this from my modifications for NAT and now I think it may be an issue in the vanilla version of akka-2.1-RC1. It occurs if one sends a message to a remote actor system or if a lone system tries to send a message to a non-existant ip/port.

[cessationoftime]

Since this timeout problem seems to be unrelated to the changes I made.  I created a second ticket and PR for those changes #2698. I am hoping they are simple enough you guys will be able to merge them in for the 2.1 release.

https://github.com/akka/akka/pull/864

[Björn Antonsson]

On Saturday, 10 November 2012 at 23:30, cessationoftime wrote:

Since this timeout problem seems to be unrelated to the changes I made.  I created a second ticket and PR for those changes #2698. I am hoping they are simple enough you guys will be able to merge them in for the 2.1 release.

https://github.com/akka/akka/pull/864

Thank you very much. It is great that you separated the feature from the bug-report.

B/