[2.1.1 Scala] akka system shutdown in production

[Frederic Masion]

Hi,

I have the following surprising log output in production :

2013-04-26 17:27:37,805 - [INFO] - from play in main 
Application started (Prod)
2013-04-26 17:27:37,907 - [INFO] - from play in main 
Listening for HTTP on /0:0:0:0:0:0:0:0:9000
2013-04-26 19:39:13,393 - [INFO] - from play in Thread-7 
Shutdown application default Akka system.

Play stops the default akka system at some point after the server start without any other error message

As I start some actors in the Global.scala (dispatchers and listeners) the next call leads to an application crash

Moreover one of my actors is an zmq Actor that listens on a zmq channel....

Does play stop and restart the actor system when it wants ? 

Does this stop come from an error that isn't logged ?

I'm very surprised of what's happening

Should I create an other actor sytem at server start in order not to depend on play's defaut akka system's lifecycle

Please Help

Thanx

Fred 

 

[Guillaume Bort]

On Tue, Apr 30, 2013 at 10:27 AM, Frederic Masion <f.ma...@kreactive.com> wrote:

Shutdown application default Akka system.

It means that your application is stopped by something. The message comes from the Akka plugin that log this on application.stop(). So I guess, that it is not the cause of your problem but a consequence. 

--
Guillaume Bort, http://guillaume.bort.fr

[Frederic Masion]

Hi Guillaume,

Thanx for the awnser but :

In the application Global.scala I've overriden the onStop methode, this onStop logs the application stop     log.info("Application shutdown...") but there's no trace of that in the application.log file

if the application would stop or restart I would have the Global.onStop method being called right ?

So I don't think the app is really stopped 

And the http server always respond : There are just some actors started in the Global.scala that don't respond any more after some time

#Strange

[Guillaume Bort]

On Tue, Apr 30, 2013 at 2:16 PM, Frederic Masion <f.ma...@kreactive.com> wrote:

In the application Global.scala I've overriden the onStop methode, this onStop logs the application stop     log.info("Application shutdown...") but there's no trace of that in the application.log file

if the application would stop or restart I would have the Global.onStop method being called right ?

Yes I think so. Still this log message comes from the AkkaPlugin.onStop https://github.com/playframework/Play20/blob/master/framework/src/play/src/main/scala/play/api/libs/concurrent/Akka.scala#L67

So I don't see how it could be logged outside of an application stop.

[Frederic Masion]

Hi Guillaume,

You were right : the message really comes from were you expected. I found some reasons about the strange dysfunction :

! @6e6bodhef - Internal server error, for (GET) [/lifepage] ->
play.api.Application$$anon$1: Execution exception[[IllegalStateException: Shutdown in progress]]
at play.api.Application$class.handleError(Application.scala:289) ~[play_2.10.jar:2.1.1]
at play.api.DefaultApplication.handleError(Application.scala:383) [play_2.10.jar:2.1.1]
at play.core.server.netty.PlayDefaultUpstreamHandler$$anonfun$17$$anonfun$apply$24.apply(PlayDefaultUpstreamHandler.scala:326) [play_2.10.jar:2.1.1]
at play.core.server.netty.PlayDefaultUpstreamHandler$$anonfun$17$$anonfun$apply$24.apply(PlayDefaultUpstreamHandler.scala:324) [play_2.10.jar:2.1.1]
at play.api.libs.concurrent.PlayPromise$$anonfun$extend1$1.apply(Promise.scala:113) [play_2.10.jar:2.1.1]
at play.api.libs.concurrent.PlayPromise$$anonfun$extend1$1.apply(Promise.scala:113) [play_2.10.jar:2.1.1]
at play.api.libs.concurrent.PlayPromise$$anonfun$extend$1$$anonfun$apply$1.apply(Promise.scala:104) [play_2.10.jar:2.1.1]
at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24) [scala-library.jar:na]
at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24) [scala-library.jar:na]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [na:1.7.0]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [na:1.7.0]
at java.lang.Thread.run(Thread.java:722) [na:1.7.0]
java.lang.IllegalStateException: Shutdown in progress

The server is trying to stop but something prevents.

That's the reason I have an incomplete shutdown logs and that I still can submit HTTP request while the akka system is allready down

That gives us an understanding of the current situation but don't explain the root cause of the play shutdown... :(

What could be the cause of play shutting down after a period of inactivity ? while the servers receives request every thing seams to work fine. When there's no more request for a variable period (sometime 2h or 15mn) I have this strange message in the log of akka system stop

Second problem what could prevent play to stop properly ? in the akka plugin the log message is writen before the akka system is really stoped. could the applicationSystem.awaitTermination() never end or other plugins onStop() freezing the process ? the only plugins I use are :

400:play.modules.reactivemongo.ReactiveMongoPlugin
100:net.fwbrasil.activate.play.ActivatePlayPlugin

I will do some refactoring in my code because I don't need Activate any more. Maybe it will help...

If you have an idea your welcome

Thanx

Fred

[Manuel Bernhardt]

Hi,

this just hit us in production. Play shutting down (i.e. the
Global#onStop handler was called), without much of an apparent cause.
But apparently it doesn't manage to shut down entirely, and the
requests still get somewhat handled, but given that the application
ActorSystem is down, the logs get flooded with unhappy controller
actions not being able to do their job.

Any idea why the Application would decide to shut down like this?

Manuel

> --
> You received this message because you are subscribed to the Google Groups
> "play-framework" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to play-framewor...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[Tagir Magomedov]

Seeing the same behavior in Play 2.4

[virtualeyes]

You have to find the root cause. Had similar symptom on 2.4; in my case depending on snapshot builds of play and play mailer plugin caused a NoSuchMethodError to be thrown (and swallowed by Akka), real PITA sorting that out.

Anyway, mine was an edge case and somewhat deserved for depending on the bleeding edge ;-)

Good luck!

[Tagir Magomedov]

Is there a way to see the log of the root cause?

I have akka loglevel set to debug, but nothing shows up.

--
You received this message because you are subscribed to a topic in the Google Groups "play-framework" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/play-framework/aBKWMSxVhAs/unsubscribe.
To unsubscribe from this group and all its topics, send an email to play-framewor...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/play-framework/c406c6ad-9135-45bb-97d3-407c170cb69d%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Simon Fakir]

Did you find a solution?