Problem reloading webapp with embedded AKKA

[Eckart Hertzler]

Hi,

I am running AKKA embedded in a web application in jetty.

I noticed that the RemoteServer does not shutdown properly when reloading the webapp.
I get this error message on reload:

INF [20100303-16:02:56.403] akka: ==============================
INF [20100303-16:02:56.403] akka: __ __
INF [20100303-16:02:56.403] akka: _____ | | _| | _______
INF [20100303-16:02:56.403] akka: \__ \ | |/ / |/ /\__ \
INF [20100303-16:02:56.403] akka: / __ \| <| < / __ \_
INF [20100303-16:02:56.403] akka: (____ /__|_ \__|_ \(____ /
INF [20100303-16:02:56.403] akka: \/ \/ \/ \/
INF [20100303-16:02:56.403] akka: Running version 0.7-SNAPSHOT
INF [20100303-16:02:56.404] akka: ==============================
INF [20100303-16:02:56.404] akka: Starting Akka...
INF [20100303-16:02:56.405] akka: Starting up Cluster Service
INF [20100303-16:02:56.418] akka: Loading boot class [de.quaggatech.touchpoint.domain.Boot]
INF [20100303-16:02:56.659] akka: Initializing Remote Actors Service...
INF [20100303-16:02:56.667] akka: Remote Actors Service initialized!
INF [20100303-16:02:56.759] akka: Akka started successfully
INF [20100303-16:02:56.765] remote: Starting remote server at [localhost:9999]
ERR [20100303-16:02:56.841] remote: Could not start up remote server
ERR [20100303-16:02:56.841] remote: org.jboss.netty.channel.ChannelException: Failed to bind to: localhost/127.0.0.1:9999
ERR [20100303-16:02:56.841] remote: at org.jboss.netty.bootstrap.ServerBootstrap.bind(ServerBootstrap.java:300)
ERR [20100303-16:02:56.841] remote: at se.scalablesolutions.akka.remote.RemoteServer.liftedTree1$1(RemoteServer.scala:176)
ERR [20100303-16:02:56.841] remote: at se.scalablesolutions.akka.remote.RemoteServer.start(RemoteServer.scala:163)
ERR [20100303-16:02:56.841] remote: at se.scalablesolutions.akka.remote.RemoteServer.start(RemoteServer.scala:158)
ERR [20100303-16:02:56.841] remote: at se.scalablesolutions.akka.remote.BootableRemoteActorService$$anon$1.run(BootableRemoteActorService.scala:20)
ERR [20100303-16:02:56.841] remote: at java.lang.Thread.run(Thread.java:619)
ERR [20100303-16:02:56.841] remote: Caused by java.net.BindException: Address already in use
ERR [20100303-16:02:56.841] remote: at sun.nio.ch.Net.bind(Native Method)
ERR [20100303-16:02:56.841] remote: at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:119)
ERR [20100303-16:02:56.841] remote: at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)
ERR [20100303-16:02:56.841] remote: at org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.bind(NioServerSocketPipelineSink.java:148)
ERR [20100303-16:02:56.841] remote: at org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.handleServerSocket(NioServerSocketPipelineSink.java:100)
ERR [20100303-16:02:56.841] remote: at org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.eventSunk(NioServerSocketPipelineSink.java:74)
ERR [20100303-16:02:56.841] remote: at org.jboss.netty.channel.Channels.bind(Channels.java:469)
ERR [20100303-16:02:56.841] remote: at org.jboss.netty.channel.AbstractChannel.bind(AbstractChannel.java:170)
ERR [20100303-16:02:56.841] remote: at org.jboss.netty.bootstrap.ServerBootstrap$Binder.channelOpen(ServerBootstrap.java:346)
ERR [20100303-16:02:56.841] remote: at org.jboss.netty.channel.Channels.fireChannelOpen(Channels.java:176)
ERR [20100303-16:02:56.841] remote: at org.jboss.netty.channel.socket.nio.NioServerSocketChannel.<init>(NioServerSocketChannel.java:85)
ERR [20100303-16:02:56.841] remote: at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.newChannel(NioServerSocketChannelFactory.java:142)
ERR [20100303-16:02:56.841] remote: at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.newChannel(NioServerSocketChannelFactory.java:90)
ERR [20100303-16:02:56.841] remote: at org.jboss.netty.bootstrap.ServerBootstrap.bind(ServerBootstrap.java:279)
ERR [20100303-16:02:56.841] remote: at se.scalablesolutions.akka.remote.RemoteServer.liftedTree1$1(RemoteServer.scala:176)
ERR [20100303-16:02:56.841] remote: at se.scalablesolutions.akka.remote.RemoteServer.start(RemoteServer.scala:163)
ERR [20100303-16:02:56.841] remote: at se.scalablesolutions.akka.remote.RemoteServer.start(RemoteServer.scala:158)
ERR [20100303-16:02:56.841] remote: at se.scalablesolutions.akka.remote.BootableRemoteActorService$$anon$1.run(BootableRemoteActorService.scala:20)
ERR [20100303-16:02:56.841] remote: at java.lang.Thread.run(Thread.java:619)
l

additionally there is a problem with the STM trying to register itself to JMX again:

ERR [20100303-16:02:58.661] actor: at se.scalablesolutions.akka.dispatch.ThreadPoolBuilder$MonitorableThread.run(ThreadPoolBuilder.scala:244)
ERR [20100303-16:02:58.661] actor: Caused by java.lang.IllegalArgumentException: Unable to register MBean 'org.multiverse.utils.monitoring.ProfilePublisher@1ce508a' under name 'org.multiverse.utils.monitoring:type=ProfilePublisher' due to: org.multiverse.utils.monitoring:type=ProfilePublisher
ERR [20100303-16:02:58.661] actor: at org.multiverse.utils.monitoring.jmx.JmxUtils.registerMBean(JmxUtils.java:30)
ERR [20100303-16:02:58.661] actor: at org.multiverse.api.GlobalStmInstance.<clinit>(GlobalStmInstance.java:69)
ERR [20100303-16:02:58.661] actor: at se.scalablesolutions.akka.stm.Transaction$$anon$2.<init>(Transaction.scala:130)
ERR [20100303-16:02:58.661] actor: at se.scalablesolutions.akka.stm.Transaction$.atomic(Transaction.scala:129)
ERR [20100303-16:02:58.661] actor: at se.scalablesolutions.akka.actor.Actor$class.transactionalDispatch(Actor.scala:921)
ERR [20100303-16:02:58.661] actor: at se.scalablesolutions.akka.actor.Actor$class.liftedTree1$1(Actor.scala:865)
ERR [20100303-16:02:58.661] actor: at se.scalablesolutions.akka.actor.Actor$class.invoke(Actor.scala:864)
ERR [20100303-16:02:58.661] actor: at de.quaggatech.touchpoint.domain.AccountActor.invoke(Actors.scala:34)
ERR [20100303-16:02:58.661] actor: at se.scalablesolutions.akka.dispatch.MessageInvocation.invoke(Reactor.scala:22)
ERR [20100303-16:02:58.661] actor: at se.scalablesolutions.akka.dispatch.ExecutorBasedEventDrivenDispatcher$$anon$1.run(ExecutorBasedEventDrivenDispatcher.scala:68)
ERR [20100303-16:02:58.661] actor: at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
ERR [20100303-16:02:58.661] actor: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
ERR [20100303-16:02:58.661] actor: at java.lang.Thread.run(Thread.java:619)
ERR [20100303-16:02:58.661] actor: at se.scalablesolutions.akka.dispatch.ThreadPoolBuilder$MonitorableThread.run(ThreadPoolBuilder.scala:244)

Anything I can do about this?

Eckart

[Jonas Bonér]

How would the RemoteServer know that it should shut down?

> --
> You received this message because you are subscribed to the Google Groups "Akka User List" group.
> To post to this group, send email to akka...@googlegroups.com.
> To unsubscribe from this group, send email to akka-user+...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/akka-user?hl=en.
>
>

--
Jonas Bonér

twitter: @jboner
blog: http://jonasboner.com
work: http://scalablesolutions.se
code: http://github.com/jboner
code: http://akkasource.org
also: http://letitcrash.com

[Viktor Klang]

Hello Eckart,

Are you using the

<web-app>
...
  <listener>

    <listener-class>se.scalablesolutions.akka.Kernel</listener-class>

  </listener>
...
</web-app>


?

If so it should invoke remoteNode.shutdown, but you're perhaps starting other RemoteServers?

Is this of any help?

Cheers,

Viktor Klang
| "A complex system that works is invariably
| found to have evolved from a simple system
| that worked." - John Gall

Akka - the Actor Kernel: Akkasource.org
Twttr: twitter.com/viktorklang

[Jonas Bonér]

I think he wants it to be done automatically when redeploy.

[Eckart Hertzler]

Am Mittwoch, 3. März 2010 17:01:30 schrieb Viktor Klang:
> Hello Eckart,
>
> Are you using the
>
> <web-app>
> ...
> <listener>
> <listener-class>se.scalablesolutions.akka.Kernel</listener-class>
> </listener>
> ...
> </web-app>
>
> ?

yes, exactly like this.

>
> If so it should invoke remoteNode.shutdown, but you're perhaps

I can see this on shutdown:

INF [20100303-17:17:45.667] akka: Shutting down Akka...
INF [20100303-17:17:45.673] actor: Shutting down all actors in the system...
INF [20100303-17:17:45.741] actor: Shutting actor down: Actor[de.quaggatech.touchpoint.domain.AccountActor:1268991565268]
INF [20100303-17:17:45.742] actor: Stopping supervisor: Actor[se.scalablesolutions.akka.actor.Supervisor:1268991565344]
INF [20100303-17:17:45.743] actor: All actors have been shut down and unregistered from ActorRegistry
INF [20100303-17:17:45.745] akka: Akka succesfully shut down

So Akka does seem to know it should shut down but it does not unbind the server...

> starting other RemoteServers?

definitly not.

> > akka-user+...@googlegroups.com<akka-user%2Bunsu...@googlegroups.com>

> > .
> > > For more options, visit this group at
> > http://groups.google.com/group/akka-user?hl=en.
> > >
> > >
> >
> >
> >
> > --
> > Jonas Bonér
> >
> > twitter: @jboner
> > blog: http://jonasboner.com
> > work: http://scalablesolutions.se
> > code: http://github.com/jboner
> > code: http://akkasource.org
> > also: http://letitcrash.com
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "Akka User List" group.
> > To post to this group, send email to akka...@googlegroups.com.
> > To unsubscribe from this group, send email to

> > akka-user+...@googlegroups.com<akka-user%2Bunsu...@googlegroups.com>

[Eckart Hertzler]

Am Mittwoch, 3. März 2010 17:17:11 schrieb Jonas Bonér:
> I think he wants it to be done automatically when redeploy.

Yes.

and I can see the the Kernel tries to do this, when the context is destroyed:

------------
def contextDestroyed(e : ServletContextEvent) : Unit = Kernel.shutdown
------------

but as I said, the shutdown is not complete...

[dustin....@gmail.com]

There was a bug not too long ago related to the remote client not shutting down properly: http://www.assembla.com/spaces/akka/tickets/104-automatic-shutdown-of-remoteclient

Perhaps this is the same thing. The bug would continue to exist in 0.6, if I'm not mistaken.

Dustin

On Mar 3, 2010 11:22am, Eckart Hertzler <eck...@hertzler.de> wrote:
> Am Mittwoch, 3. März 2010 17:17:11 schrieb Jonas Bonér:
>
> > I think he wants it to be done automatically when redeploy.
>
>
>
> Yes.
>
>
>
> and I can see the the Kernel tries to do this, when the context is destroyed:
>
>
>
> ------------
>
>   def contextDestroyed(e : ServletContextEvent) : Unit = Kernel.shutdown
>
> ------------
>
>
>
> but as I said, the shutdown is not complete...
>
>
>
>
>
> >
>
> > On 3 March 2010 17:01, Viktor Klang viktor...@gmail.com> wrote:
>
> > > Hello Eckart,
>
> > >
>
> > > Are you using the
>
> > >
>
> > >
>

> > > ...
>
> > >  
>
> > >
>
> > >     se.scalablesolutions.akka.Kernel
>
> > >
>
> > >  
>
> > > ...

>
> > >
>
> > >
>
> > > ?
>
> > >
>
> > > If so it should invoke remoteNode.shutdown, but you're perhaps starting
>
> > > other RemoteServers?
>
> > >
>
> > > Is this of any help?
>
> > >
>
> > > Cheers,
>
> > >
>
> > > On Wed, Mar 3, 2010 at 4:54 PM, Jonas Bonér jbo...@gmail.com> wrote:
>
> > >>
>
> > >> How would the RemoteServer know that it should shut down?
>
> > >>
>
> > >> On 3 March 2010 16:12, Eckart Hertzler eck...@hertzler.de> wrote:
>
> > >> > Hi,
>
> > >> >
>
> > >> > I am running AKKA embedded in a web application in jetty.
>
> > >> >
>
> > >> > I noticed that the RemoteServer does not shutdown properly when
>
> > >> > reloading the webapp.
>
> > >> > I get this error message on reload:
>
> > >> >
>
> > >> > INF [20100303-16:02:56.403] akka: ==============================
>
> > >> > INF [20100303-16:02:56.403] akka:           __    __
>
> > >> > INF [20100303-16:02:56.403] akka:  _____  |  | _|  | _______
>
> > >> > INF [20100303-16:02:56.403] akka:  \__  \ |  |/ /  |/ /\__  \
>
> > >> > INF [20100303-16:02:56.403] akka:   / __ \|    

> > >> > org.jboss.netty.channel.socket.nio.NioServerSocketChannel.(NioServerSocketChannel.java:85)

> > >> > org.multiverse.api.GlobalStmInstance.(GlobalStmInstance.java:69)

>
> > >> > ERR [20100303-16:02:58.661] actor:     at
>

> > >> > se.scalablesolutions.akka.stm.Transaction$$anon$2.(Transaction.scala:130)

[Viktor Klang]

On Wed, Mar 3, 2010 at 5:22 PM, Eckart Hertzler <eck...@hertzler.de> wrote:

Am Mittwoch, 3. März 2010 17:17:11 schrieb Jonas Bonér:

> I think he wants it to be done automatically when redeploy.

Yes.

and I can see the the Kernel tries to do this, when the context is destroyed:

------------
  def contextDestroyed(e : ServletContextEvent) : Unit = Kernel.shutdown
------------

but as I said, the shutdown is not complete...

Eckart, I unfortunately don't have any possibility to go hands on right now, but could you please try to add in RemoteServer.shutdown:

log.debug("RemoteServer has been shut down") at the bottom of the method?

And see if it gets called properly?

Also, do you see:

"Shutting down Remote Actors Service"

in your log?

Cheers,

[Eckart Hertzler]

Am Mittwoch, 3. März 2010 17:32:07 schrieb Viktor Klang:
> Eckart, I unfortunately don't have any possibility to go hands on right now,
> but could you please try to add in RemoteServer.shutdown:
>
> log.debug("RemoteServer has been shut down") at the bottom of the method?
>
> And see if it gets called properly?
>
> Also, do you see:
>
> "Shutting down Remote Actors Service"

No, all I see is this

INF [20100303-17:17:45.667] akka: Shutting down Akka...
INF [20100303-17:17:45.673] actor: Shutting down all actors in the system...
INF [20100303-17:17:45.741] actor: Shutting actor down: Actor[de.quaggatech.touchpoint.domain.AccountActor:1268991565268]
INF [20100303-17:17:45.742] actor: Stopping supervisor: Actor[se.scalablesolutions.akka.actor.Supervisor:1268991565344]
INF [20100303-17:17:45.743] actor: All actors have been shut down and unregistered from ActorRegistry
INF [20100303-17:17:45.745] akka: Akka succesfully shut down

>
> in your log?
>
> Cheers,

[Eckart Hertzler]

Am Mittwoch, 3. März 2010 17:32:07 schrieb Viktor Klang:

> Eckart, I unfortunately don't have any possibility to go hands on right now,
> but could you please try to add in RemoteServer.shutdown:
>
> log.debug("RemoteServer has been shut down") at the bottom of the method?
>

ok, I added th

> And see if it gets called properly?
>
> Also, do you see:
>
> "Shutting down Remote Actors Service"

It seems, the RemoteServer is not shutdown at all.

At startup:

INF [20100303-19:08:52.359] akka: Starting Akka...
INF [20100303-19:08:52.360] akka: Starting up Cluster Service
INF [20100303-19:08:52.377] akka: Loading boot class [de.quaggatech.touchpoint.domain.Boot]
INF [20100303-19:08:52.582] akka: Initializing Remote Actors Service...
INF [20100303-19:08:52.585] akka: Remote Actors Service initialized!
INF [20100303-19:08:52.586] akka: Akka started successfully
INF [20100303-19:08:52.805] remote: Starting remote server at [localhost:9999]

at shutdown

INF [20100303-19:10:34.262] akka: Shutting down Akka...
INF [20100303-19:10:34.269] actor: Shutting down all actors in the system...
INF [20100303-19:10:34.283] actor: Shutting actor down: Actor[de.quaggatech.touchpoint.domain.AccountActor:1268981909760]
INF [20100303-19:10:34.284] actor: Stopping supervisor: Actor[se.scalablesolutions.akka.actor.Supervisor:1268981909822]
INF [20100303-19:10:34.285] actor: All actors have been shut down and unregistered from ActorRegistry
INF [20100303-19:10:34.288] akka: Akka succesfully shut down

And the reason for the missing shutdown seems to that the removeServerThread is not alive

------------------------------------------------------------------------------------
trait BootableRemoteActorService extends Bootable with Logging {
self : BootableActorLoaderService =>

[ ... skipped ..]

abstract override def onUnload = {
super.onUnload

log.info("remote server thread is alive = "+remoteServerThread.isAlive) // a new debug statement

if (remoteServerThread.isAlive) {
log.info("Shutting down Remote Actors Service")
RemoteNode.shutdown
remoteServerThread.join(1000)
}

Cluster.shutdown
}

}
-------------------------------------------------------------------------------------

INF [20100303-19:27:49.907] akka: Shutting down Akka...
INF [20100303-19:27:49.914] actor: Shutting down all actors in the system...
INF [20100303-19:27:49.924] actor: Shutting actor down: Actor[de.quaggatech.touchpoint.domain.AccountActor:1268983035294]
INF [20100303-19:27:49.925] actor: Stopping supervisor: Actor[se.scalablesolutions.akka.actor.Supervisor:1268983035549]
INF [20100303-19:27:49.926] actor: All actors have been shut down and unregistered from ActorRegistry

INF [20100303-19:27:49.926] akka: remote server thread is alive = false <------------

INF [20100303-19:27:49.929] akka: Akka succesfully shut down

>
> in your log?
>
> Cheers,
>

[Viktor Klang]

*laugh* Looks like we're on the same track! :-)

This is what I just compiled before reading your e-mail!

 abstract override def onUnload = {
    super.onUnload
       

    log.info("Shutting down Remote Actors Service")
    RemoteNode.shutdown
       

    if (remoteServerThread.isAlive)
      remoteServerThread.join(1000)
    log.info("Shutting down Cluster")
    Cluster.shutdown
    log.info("Remote Actors Service has been shut down")
  }

Can you try that out and see if it works as you expect?
 

------------------------------------------------------------------------------------
trait BootableRemoteActorService extends Bootable with Logging {
 self : BootableActorLoaderService =>

 [ ... skipped ..]

 abstract override def onUnload = {
       super.onUnload

       log.info("remote server thread is alive = "+remoteServerThread.isAlive)   // a new debug statement

       if (remoteServerThread.isAlive) {
         log.info("Shutting down Remote Actors Service")
         RemoteNode.shutdown
         remoteServerThread.join(1000)
       }

       Cluster.shutdown
 }

}
-------------------------------------------------------------------------------------

INF [20100303-19:27:49.907] akka: Shutting down Akka...
INF [20100303-19:27:49.914] actor: Shutting down all actors in the system...
INF [20100303-19:27:49.924] actor: Shutting actor down: Actor[de.quaggatech.touchpoint.domain.AccountActor:1268983035294]
INF [20100303-19:27:49.925] actor: Stopping supervisor: Actor[se.scalablesolutions.akka.actor.Supervisor:1268983035549]
INF [20100303-19:27:49.926] actor: All actors have been shut down and unregistered from ActorRegistry

INF [20100303-19:27:49.926] akka: remote server thread is alive = false  <------------

INF [20100303-19:27:49.929] akka: Akka succesfully shut down




>
> in your log?
>

> Cheers,
>

--
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To post to this group, send email to akka...@googlegroups.com.
To unsubscribe from this group, send email to akka-user+...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/akka-user?hl=en.

[Viktor Klang]

*laugh* Looks like we're on the same track! :-)

This is what I just compiled before reading your e-mail!

 abstract override def onUnload = {
    super.onUnload
       

    log.info("Shutting down Remote Actors Service")
    RemoteNode.shutdown
       

    if (remoteServerThread.isAlive)
      remoteServerThread.join(1000)
    log.info("Shutting down Cluster")
    Cluster.shutdown
    log.info("Remote Actors Service has been shut down")
  }

Can you try that out and see if it works as you expect?
 

------------------------------------------------------------------------------------
trait BootableRemoteActorService extends Bootable with Logging {
 self : BootableActorLoaderService =>

 [ ... skipped ..]

 abstract override def onUnload = {
       super.onUnload

       log.info("remote server thread is alive = "+remoteServerThread.isAlive)   // a new debug statement

       if (remoteServerThread.isAlive) {
         log.info("Shutting down Remote Actors Service")
         RemoteNode.shutdown
         remoteServerThread.join(1000)
       }

       Cluster.shutdown
 }

}
-------------------------------------------------------------------------------------

INF [20100303-19:27:49.907] akka: Shutting down Akka...
INF [20100303-19:27:49.914] actor: Shutting down all actors in the system...
INF [20100303-19:27:49.924] actor: Shutting actor down: Actor[de.quaggatech.touchpoint.domain.AccountActor:1268983035294]
INF [20100303-19:27:49.925] actor: Stopping supervisor: Actor[se.scalablesolutions.akka.actor.Supervisor:1268983035549]
INF [20100303-19:27:49.926] actor: All actors have been shut down and unregistered from ActorRegistry

INF [20100303-19:27:49.926] akka: remote server thread is alive = false  <------------

INF [20100303-19:27:49.929] akka: Akka succesfully shut down




>
> in your log?
>

> Cheers,
>

--
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To post to this group, send email to akka...@googlegroups.com.
To unsubscribe from this group, send email to akka-user+...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/akka-user?hl=en.

[Eckart Hertzler]

yes it does.

I tried that myself an verified it just a few seconds before your mail arrived :-)

> > akka-user+...@googlegroups.com<akka-user%2Bunsu...@googlegroups.com>

[Viktor Klang]

Awesome! Will you do the honors and commit it or shall I?
 

To unsubscribe from this group, send email to akka-user+...@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/akka-user?hl=en.

[Eckart Hertzler]

I just did.

thanks for your help (awesome!)

> > <akka-user%2Bunsu...@googlegroups.com<akka-user%252Buns...@googlegroups.com>