How to properly shutdown actor system (release all resources)

[Jozo Vilcek]

Hi everyone,

I am implementing a demo application using Akka actors (2.0-M2). The
goal is to have a service (Scala 2.9.1), which is bind to a lifecycle
of a Spring application context (3.1.0) (initialize, dispose) and
hosted in Tomcat web server (7.0.23).

So, I deploy my application, string context is loaded, it starts up my
service which in turn will start off some actors and performs stuff on
background. Then, when stopping the application, spring context get
disposed, propagates event to my service which in turn should stop and
cleanup all actor related resources.

When doing so, I get following complain from the Tomcat:
org.apache.catalina.loader.WebappClassLoader
checkThreadLocalMapForLeaks SEVERE: The web application [/DemoService]
created a ThreadLocal with key of type [akka.dispatch.Future$$anon$2]
(value [akka.dispatc
h.Future$$anon$2@51bfa303]) and a value of type [scala.None$] (value
[None]) but failed to remove it when the web application was stopped.
Threads are going to be renewed over time to try and avoid a probable
memory leak.

My shutdown code is like:
1. I send a PoisonPill to a top level actor and wait for him to stop
via mechanism I found in akka.pattern - gracefullStop
2. I call shutdown on ActorSystem instance and thread sleep for 10
seconds

Questions:
1. What what else should be done (or how to do it correctly) to avoid
messages from Tomcat about thread leftovers
2. How to properly wait for ActorSystem shutdown without explicit
thread sleep

Many thanks in advance!

Best,
Jozef

[Patrik Nordwall]

On Wed, Jan 4, 2012 at 11:53 AM, Jozo Vilcek <jozo....@gmail.com> wrote:

Hi everyone,

I am implementing a demo application using Akka actors (2.0-M2). The
goal is to have a service (Scala 2.9.1), which is bind to a lifecycle
of a Spring application context (3.1.0) (initialize, dispose) and
hosted in Tomcat web server (7.0.23).

So, I deploy my application, string context is loaded, it starts up my
service which in turn will start off some actors and performs stuff on
background. Then, when stopping the application, spring context get
disposed, propagates event to my service which in turn should stop and
cleanup all actor related resources.

When doing so, I get following complain from the Tomcat:
org.apache.catalina.loader.WebappClassLoader
checkThreadLocalMapForLeaks SEVERE: The web application [/DemoService]
created a ThreadLocal with key of type [akka.dispatch.Future$$anon$2]
(value [akka.dispatc
h.Future$$anon$2@51bfa303]) and a value of type [scala.None$] (value
[None]) but failed to remove it when the web application was stopped.
Threads are going to be renewed over time to try and avoid a probable
memory leak.

Thanks for reporting. We will look into it. Created ticket: http://www.assembla.com/spaces/akka/tickets/1604-future-threadlocal-leftover

I have an idea that it is due to the _taskStack in object Future. Should probably be scoped by the system, instead of global object, i.e. using an Extension.

 

My shutdown code is like:
1. I send a PoisonPill  to a top level actor and wait for him to stop
via mechanism I found in akka.pattern - gracefullStop

The gracefulStop sends the PoisonPill.

Maybe you don't need to do this at all if you instead shutdown the system, which will stop all actors.

 

2. I call shutdown on ActorSystem instance and thread sleep for 10
seconds

Questions:
1. What what else should be done (or how to do it correctly) to avoid
messages from Tomcat about thread leftovers
2. How to properly wait for ActorSystem shutdown without explicit
thread sleep

system.registerOnTermination might be useful. To block you can await your own CountDownLatch, which you countDown in the registerOnTermination callback.

 

Many thanks in advance!

Best,
Jozef

--
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To post to this group, send email to akka...@googlegroups.com.
To unsubscribe from this group, send email to akka-user+...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/akka-user?hl=en.

--

Patrik Nordwall
Typesafe - Enterprise-Grade Scala from the Experts
Twitter: @patriknw

[Jozo Vilcek]

Hi,

> Thanks for reporting. We will look into it. Created ticket:http://www.assembla.com/spaces/akka/tickets/1604-future-threadlocal-l...

> I have an idea that it is due to the _taskStack in object Future. Should
> probably be scoped by the system, instead of global object, i.e. using an
> Extension.

That is what I was afraid of :)

Does anyone know by heart an older version of Akka which can handle
proper cleanup for my scenario?


Best,
Jozef

On Jan 4, 12:14 pm, Patrik Nordwall <patrik.nordw...@gmail.com> wrote:

> On Wed, Jan 4, 2012 at 11:53 AM, Jozo Vilcek <jozo.vil...@gmail.com> wrote:
> > Hi everyone,
>
> > I am implementing a demo application using Akka actors (2.0-M2). The
> > goal is to have a service (Scala 2.9.1), which is bind to a lifecycle
> > of a Spring application context (3.1.0) (initialize, dispose) and
> > hosted in Tomcat web server (7.0.23).
>
> > So, I deploy my application, string context is loaded, it starts up my
> > service which in turn will start off some actors and performs stuff on
> > background. Then, when stopping the application, spring context get
> > disposed, propagates event to my service which in turn should stop and
> > cleanup all actor related resources.
>
> > When doing so, I get following complain from the Tomcat:
> > org.apache.catalina.loader.WebappClassLoader
> > checkThreadLocalMapForLeaks SEVERE: The web application [/DemoService]
> > created a ThreadLocal with key of type [akka.dispatch.Future$$anon$2]
> > (value [akka.dispatc
> > h.Future$$anon$2@51bfa303]) and a value of type [scala.None$] (value
> > [None]) but failed to remove it when the web application was stopped.
> > Threads are going to be renewed over time to try and avoid a probable
> > memory leak.
>

> Thanks for reporting. We will look into it. Created ticket:http://www.assembla.com/spaces/akka/tickets/1604-future-threadlocal-l...

> Typesafe <http://typesafe.com/> - Enterprise-Grade Scala from the Experts
> Twitter: @patriknw

[Patrik Nordwall]

On Wed, Jan 4, 2012 at 12:50 PM, Jozo Vilcek <jozo....@gmail.com> wrote:

Hi,

> Thanks for reporting. We will look into it. Created ticket:http://www.assembla.com/spaces/akka/tickets/1604-future-threadlocal-l...

> I have an idea that it is due to the _taskStack in object Future. Should
> probably be scoped by the system, instead of global object, i.e. using an
> Extension.

That is what I was afraid of :)

Does anyone know by heart an older version of Akka which can handle
proper cleanup for my scenario?

We will fix it soon, absolutely before 2.0 final.

--

Patrik Nordwall
Typesafe - Enterprise-Grade Scala from the Experts
Twitter: @patriknw

[Jozef Vilcek]

Thanks for the hint. I have tried to implement the shutdown as you
suggested, but at the end I receive even more complain messages from
tomcat about several threads not being stopped [default-dispatcher1],
[default-dispatcher2] ... My code is like :

val latch = new CountDownLatch(1)
system.registerOnTermination(latch.countDown)
system.shutdown
system = null
latch.await(5, java.util.concurrent.TimeUnit.SECONDS)

[Patrik Nordwall]

I thought some more on this, and did some experiments. I'm not sure it is a real problem. Do you see any real memory or thread leaks, or is it just false alarm by Tomcat?

You use a thread from Tomcat to start/stop the ActorSystem. A workaround for avoiding the warning might be to use your own thread, like this:

  val runner = new Thread {

    override def run() {

      val latch = new CountDownLatch(1)

      val system = ActorSystem()

      system.registerOnTermination(latch.countDown())

      val a = system.actorOf(Props[MyActor])

      a ! 1

      Thread.sleep(10000)

      a ! 2

      system.shutdown()

      latch.await()

    }

  }

  runner.start()

[Jozef Vilcek]

On Wed, Jan 4, 2012 at 5:06 PM, Patrik Nordwall <patrik....@gmail.com> wrote:

I thought some more on this, and did some experiments. I'm not sure it is a real problem. Do you see any real memory or thread leaks, or is it just false alarm by Tomcat?

Well, you are right, when I do multiple start / stop exercise, number of threads is pretty still ... so indeed false alarm or some kind of "late collection"

You use a thread from Tomcat to start/stop the ActorSystem. A workaround for avoiding the warning might be to use your own thread, like this:

  val runner = new Thread {

    override def run() {

      val latch = new CountDownLatch(1)

      val system = ActorSystem()

      system.registerOnTermination(latch.countDown())

      val a = system.actorOf(Props[MyActor])

      a ! 1

      Thread.sleep(10000)

      a ! 2

      system.shutdown()

      latch.await()

    }

  }

  runner.start()

I have tried this but it does not work. However, I have tried also tried to wrap up startum of the actor system into separate thread and more importantly setContextClassLoader(null) on those threads, to Tomcat can not "see" them when looking for not stopped threads. Indeed the complain about the threads not being stopped dissapeared but I got some more warnings about uncleared ThreadLocal variables which however can be false alarms based on information here: http://wiki.apache.org/tomcat/MemoryLeakProtection

Anyway, I am not sure if setContextClassLoader(null) is good to do ... probably not in this case.

Because of my knowledge of concurrency and class loaders in Java I am walking blind here.

What really confuses me is following behavior (start / stop routines run directly on Tomcat provided thread and latched shutdown of ActorSystem) :

1. Normally, I receive complain from Tomcat about threads not being stopped (default-dispatcher threads) + one ThreadLocal in Futures which is known defect

2. If I add thread sleep about 500ms after latch.await in stop (shutdown) hook, I get only complain about the ThreadLocal in Futures

3. If I do stop routine to contain "warm up" exercise (start system -> stop system ->start system), then when I call stop routine Tomcat complain only about the ThreadLocal in Futures

 

This is really strange for me and I would love to know what and why is happening here ... It seems not to be entirely Akka issue unless there is some delayed clean-up or something. As I said, I am walking blind here.

If someone experienced can explain or give a clue where to look I would love to hear it.

Many thanks in advance!

[Patrik Nordwall]

On Thu, Jan 5, 2012 at 10:00 AM, Jozef Vilcek <jozo....@gmail.com> wrote:

On Wed, Jan 4, 2012 at 5:06 PM, Patrik Nordwall <patrik....@gmail.com> wrote:

I thought some more on this, and did some experiments. I'm not sure it is a real problem. Do you see any real memory or thread leaks, or is it just false alarm by Tomcat?

Well, you are right, when I do multiple start / stop exercise, number of threads is pretty still ... so indeed false alarm or some kind of "late collection"

You use a thread from Tomcat to start/stop the ActorSystem. A workaround for avoiding the warning might be to use your own thread, like this:

  val runner = new Thread {

    override def run() {

      val latch = new CountDownLatch(1)

      val system = ActorSystem()

      system.registerOnTermination(latch.countDown())

      val a = system.actorOf(Props[MyActor])

      a ! 1

      Thread.sleep(10000)

      a ! 2

      system.shutdown()

      latch.await()

    }

  }

  runner.start()

I have tried this but it does not work. However, I have tried also tried to wrap up startum of the actor system into separate thread and more importantly setContextClassLoader(null) on those threads, to Tomcat can not "see" them when looking for not stopped threads. Indeed the complain about the threads not being stopped dissapeared but I got some more warnings about uncleared ThreadLocal variables which however can be false alarms based on information here: http://wiki.apache.org/tomcat/MemoryLeakProtection

Anyway, I am not sure if setContextClassLoader(null) is good to do ... probably not in this case.

Because of my knowledge of concurrency and class loaders in Java I am walking blind here.

What really confuses me is following behavior (start / stop routines run directly on Tomcat provided thread and latched shutdown of ActorSystem) :

1. Normally, I receive complain from Tomcat about threads not being stopped (default-dispatcher threads) + one ThreadLocal in Futures which is known defect

2. If I add thread sleep about 500ms after latch.await in stop (shutdown) hook, I get only complain about the ThreadLocal in Futures

Akka's shutdown of the scheduler and dispatchers are also done by a shudown hook, and I don't think we guarantee any order of these, so if your hook runs before this will happen. I'll talk to Viktor and Roland about this next week.

[Patrik Nordwall]

... and your hook will run in a default dispatcher thread, so there is no guarantee anyway that everything is shutdown when your hook is run.

[Jozef Vilcek]

Just to double check that I get this right. So you say that when I call :

[ system.registerOnTermination(latch.countDown) ] just before [ system.shutdown ] there is no guarantee that my OnTermination hook is executed as last one. So any internal shutdown code can be executed after my hook. Correct? Then there is no way to observe fully disposed / stopped AgentSystem?

[Patrik Nordwall]

That is correct, as it is right now.

The onTermination callback signals that all actors has been stopped, but not all dispatchers and scheduler.

[Jozo Vilcek]

Hi Patrik,

On Jan 5, 10:27 am, Patrik Nordwall <patrik.nordw...@gmail.com> wrote:

> On Thu, Jan 5, 2012 at 10:23 AM, Jozef Vilcek <jozo.vil...@gmail.com> wrote:
>
> > On Thu, Jan 5, 2012 at 10:12 AM, Patrik Nordwall <
> > patrik.nordw...@gmail.com> wrote:
>
> >> On Thu, Jan 5, 2012 at 10:09 AM, Patrik Nordwall <

> >> patrik.nordw...@gmail.com> wrote:

I just following up on the actor system shutdown as you mentioned you
planed to talk about this with Viktor and Roland last week.
Any updates or plans to implement a hook or something to notify when
whole actor systems is down and standing still?

Thanks.
Jozef

>
>
>
>
>
>
>
>
> >>>> 3. If I do stop routine to contain "warm up" exercise (start system ->
> >>>> stop system ->start system), then when I call stop routine Tomcat complain
> >>>> only about the ThreadLocal in Futures
>
> >>>> This is really strange for me and I would love to know what and why is
> >>>> happening here ... It seems not to be entirely Akka issue unless there is
> >>>> some delayed clean-up or something. As I said, I am walking blind here.
> >>>> If someone experienced can explain or give a clue where to look I would
> >>>> love to hear it.
>
> >>>> Many thanks in advance!
>

> >>>>> On Wed, Jan 4, 2012 at 2:02 PM, Jozef Vilcek <jozo.vil...@gmail.com>wrote:
>
> >>>>>> On Wed, Jan 4, 2012 at 12:14 PM, Patrik Nordwall

> >>>>>> <patrik.nordw...@gmail.com> wrote:
>
> >>>>>> > On Wed, Jan 4, 2012 at 11:53 AM, Jozo Vilcek <jozo.vil...@gmail.com>

> >>>>>>http://www.assembla.com/spaces/akka/tickets/1604-future-threadlocal-l...

> >>>>> Typesafe <http://typesafe.com/> - Enterprise-Grade Scala from the

> >>>>> Experts
> >>>>> Twitter: @patriknw
>
> >>>>>  --
> >>>>> You received this message because you are subscribed to the Google
> >>>>> Groups "Akka User List" group.
> >>>>> To post to this group, send email to akka...@googlegroups.com.
> >>>>> To unsubscribe from this group, send email to
> >>>>> akka-user+...@googlegroups.com.
> >>>>> For more options, visit this group at
> >>>>>http://groups.google.com/group/akka-user?hl=en.
>
> >>>>  --
> >>>> You received this message because you are subscribed to the
>

> ...
>
> read more »

[√iktor Ҡlang]

Already exists:

http://akka.io/api/akka/2.0-M2/#akka.actor.ActorSystem

See "registerOnTermination"

Cheers,
√

 

--
Viktor Klang

Akka Tech Lead

Typesafe - The software stack for applications that scale

Twitter: @viktorklang

[Patrik Nordwall]

When your onTermination callback is run you are pretty close to total shutdown, but there is no guarantee that all dispatchers have been shutdown yet, and the callback run in the default dispatcher. Easiest is probably to just wait while after that, in another thread, but otherwise you can write your own dispatcher, and replace the default dispatcher, and do some other notification on the shutdown of the ExecutorService.

2012/1/16 √iktor Ҡlang <viktor...@gmail.com>

--

Patrik Nordwall
Typesafe -  The software stack for applications that scale
Twitter: @patriknw