Dev reload (possibly evolutions-specific) not restarting Akka properly

[Nick Howes]

Hello,

I've a Play app with a clustered actor ref provider, binding to port 2552. It's all great in production, and mostly fine on dev. The problem starts when certain app reloads need to happen (use when an evolution needs to be applied) it doesn't seem to shut down the original ActorSystem, so I get 

Caused by: org.jboss.netty.channel.ChannelException: Failed to bind to: /127.0.1.1:2552

at org.jboss.netty.bootstrap.ServerBootstrap.bind(ServerBootstrap.java:272)

at akka.remote.transport.netty.NettyTransport$$anonfun$listen$1.apply(NettyTransport.scala:410)

at akka.remote.transport.netty.NettyTransport$$anonfun$listen$1.apply(NettyTransport.scala:406)

at scala.util.Success$$anonfun$map$1.apply(Try.scala:237)

Caused by: java.net.BindException: Address already in use

at sun.nio.ch.Net.bind0(Native Method)

at sun.nio.ch.Net.bind(Net.java:437)

I tried using ApplicationLifecycle to set a shutdown hook to terminate() the ActorSystem and wait for terminateWhen, with some logging to check for the hook being called, and sometimes (generally after the "apply evolutions" button has been pressed) it just doesn't do the hook at all - no clean shutdown seems to occur. I can see that ApplicationEvolutions just calls buildLink.forceReload() to do the reload, so don't know why it would be doing an unclean restart.

Has anyone else experienced this?

[Jorge Mangeruga]

Hi Nick!!

I've just found the same problem after enabling evolutions on my project, so far, everything has been working perfectly (I'm only using dev mode). I've surfed a little bit the web, but I've only found your question about the matter.

In my case, the problem appears when I use bindActor in a Guice module and it's the first time the application starts, so evolutions must be applied. After that, when I start the application again, nothing wrong happens (in this case evolutions were already applied the time before). If I comment the line (it isn't viable) in the Guice module or I disable evolutions, all works OK.

I believe this looks like a bug, what do you think?.

Regards,

[Nick Howes]

Hi Jorge - We bind an actor in a Guice module too so our situations are pretty similar. I do think it's a bug, since it isn't letting the usual Application shutdown hooks run before reloading. If I do something else to trigger a reload, like changing some code, then I see my shutdown log lines before it reloads.

[Christian Schmitt]

How do you bind your Cluster?
Actually I never occured this (we migrated away from evolutions) but even when we used them we had another Actor Cluster running.
This is how we did it:

@Singleton
class ClusterActorSystemProvider @Inject()(configuration: Configuration, applicationLifecycle: ApplicationLifecycle) extends Provider[ActorSystem] {

  private val logger = Logger(this.getClass)

  override lazy val get: ActorSystem = {
    val akkaSystem = ActorSystem("cluster", configuration.getConfig("cluster").get.underlying)
    logger.info(s"Starting Akka Cluster System")
    applicationLifecycle.addStopHook(() => akkaSystem.terminate())
    akkaSystem
  }

}
class ClusterActorRefProvider[T <: Actor: ClassTag](name: String, props: Props => Props) extends Provider[ActorRef] {

  //noinspection VarCouldBeVal
  @Inject
  @Named("cluster")
  private var actorSystem: ActorSystem = _
  //noinspection VarCouldBeVal
  @Inject
  private var injector: Injector = _

  lazy val get = {
    val creation = Props(injector.instanceOf[T])
    actorSystem.actorOf(props(creation), name)
  }

}

Guice Module:
def bindGuiceClusterActor[T <: Actor : ClassTag](name: String, props: Props => Props = identity): Unit = {
  bind(classOf[ActorRef])
      .annotatedWith(Names.named(name))
      .toProvider(Providers.guicify(new ClusterActorRefProvider(name, identity)))
      .asEagerSingleton()
}

override def configure(): Unit = {
  bind(classOf[ActorSystem]).annotatedWith(Names.named("cluster")).toProvider(classOf[ClusterActorSystemProvider])
  bindGuiceClusterActor[ClusterActor]("cluster-actor")
}

[Jorge Mangeruga]

Sorry for answer so late. Seeing your code I've remembered that actually I'm using Java and Play 2.5.4 (I didn't mention it before). Related to Akka systems and actors, I simply have the next line in the Guice module of the node with "front-end" role (the Play Application).

@Override
protected void configure() {
  ...
  bindActor(ActorA.class, "actorAName");
  ...
}

Internally, this ActorA creates other actor:

@Override
public void preStart() {
  self().tell(INIT, ActorRef.noSender());
}
 
@Override
public void onReceive(Object message) throws Exception {
  if (INIT.equals(message)) {
    context().system().actorOf(ActorB.props(gpsService, jpaService), configuration.getString(CONF_KEY));
  } else
    unhandled(message);
}

On the other hand, we have other node (a naive Java/Akka application) that when it realizes a node with front end role has joined the cluster, sends the message that ActorB is waiting for.

Let me know if this answers your question.

[Daniel Sanchez]

Hello, I think it is not evolution only problem.

We are using compile time DI with MacWire in Scala Play 2.5.4, and starting some akka schedules (We are using it for start Reactive-kafka consumers), they start fine the fist time, but does not reinitializing when DevApp is reloading.

Just a little example code... for reference:

class KafkaExample(actorSystem: ActorSystem) {
  implicit val materializer = ActorMaterializer()(actorSystem)

  val kafkaSettings = akka.kafka.ConsumerSettings[String, String](
    actorSystem,
    new StringDeserializer,
    new StringDeserializer,
    Set("topic-test")
  ).withBootstrapServers("localhost:9092")
  
  //This Scheluder does not restart at Dev reload!
  actorSystem.scheduler.scheduleOnce(3.seconds) {
    play.Logger.info("Starting reactive-kafka example.")
    
    //Consumer 1
    Consumer.plainSource(
      kafkaSettings
      .withGroupId("consumer1")
      .withClientId("consumer1")
    ).runForeach { record =>
      println(s"Reader 1: «${record.toString}»")
    }

    //Consumer 2
    Consumer.plainSource(
      kafkaSettings
        .withGroupId("consumer2")
        .withClientId("consumer2")
    ).runForeach { record =>
      println(s"Reader 2: «${record.toString}»")
    }
  }
}

If this is happening with Scala and Compile time DI, then this could be caused by some "bug" in framework?

[Igmar Palsenberg]

Has anyone else experienced this?

You need to cleanup afterwards. We use : 

public class FooBarProvider implements Provider<FooBar> {

       @Inject

        FooBarProvider(Environment environment, Configuration configuration ApplicationLifecycle applicationLifecycle) {

            // Some stuff

            applicationLifecycle.addStopHook(() {

                  // Kill of subsystem

            }

        }

        // Rest of the code 

}

That works fine when a reload occurs.

Igmar

 

[Nick Howes]

On Wednesday, 29 June 2016 15:26:37 UTC+1, Igmar Palsenberg wrote:

Has anyone else experienced this?

You need to cleanup afterwards.

Igmar, 

I mentioned in my original post that I already have a stop hook. The problem is that sometimes a clean reload doesn't happen, and the application is restarted without the proper shutdown. No shutdown hooks are called, and it tries to start a new actor system without stopping the original one. This is Play's own Actor System so it should be already managing it anyway.

[Nick Howes]

I've create a demo app here that exhibits the problem. 

https://github.com/UniversityofWarwick/play-migration-reload

[Christian Schmitt]

This:
 life.addStopHook { () =>
   println("Stopping ClusterService..")
   cluster.terminate()
 }
instead of:

life.addStopHook { () =>

println("Stopping ClusterService..")

Future.successful(Unit)

[Will Sargent]

Since this is an Akka cluster, it's generally cleaner to register the termination for after the member has been officially removed using a short term actor:

cluster.registerOnMemberRemoved {

   self ! "okay-to-shutdown"

}

And then tell the cluster you're leaving:

cluster.leave(cluster.selfAddress)

Once you've got the ok (or an appropriate timeout has elapsed), then you can go ahead and map to a terminate.  I'm pretty sure that clluster.terminate() will attempt leaving the cluster internally, but you may want different timeouts for a network failure vs an internal actorsystem terminaton.  See http://doc.akka.io/docs/akka/current/scala/cluster-usage.html#Leaving for more details.

Will.

--
You received this message because you are subscribed to the Google Groups "play-framework" group.
To unsubscribe from this group and stop receiving emails from it, send an email to play-framewor...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/play-framework/e2543bcc-2eb6-4464-a2f4-de3dbc117b61%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Nick Howes]

There is no terminate method on Cluster - that's on ActorSystem, which Play will be managing already. We've tried all manner of waiting for members to leave the cluster but really it doesn't matter what we put in there because the whole problem is that it doesn't call these shutdown hooks at all before trying to reload a new intance binding to the same port.

--
You received this message because you are subscribed to a topic in the Google Groups "play-framework" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/play-framework/1hHsCr_3y2U/unsubscribe.
To unsubscribe from this group and all its topics, send an email to play-framewor...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/play-framework/e2543bcc-2eb6-4464-a2f4-de3dbc117b61%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Nick Howes
Send me a file: https://files.warwick.ac.uk/nickhowes/sendto

[Nick Howes]

Perhaps I've confused things by including Cluster in my example - it has exactly the same problem if I do not initialise any kind of cluster and just use RemoteActorRefProvider instead. So it's nothing to do with cluster membership or shutdown - it's entirely due to the fact that the dev server is not going through shutdown (including shutdown hooks) before starting a new Application + ActorSystem.

--
You received this message because you are subscribed to a topic in the Google Groups "play-framework" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/play-framework/1hHsCr_3y2U/unsubscribe.
To unsubscribe from this group and all its topics, send an email to play-framewor...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/play-framework/CAJmgB627eckwUmjoEo1RnrqFS4xg5rpZv9qN%2BgXs9PH_VtUYMg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

[Nick Howes]

It's not even particularly specific to Akka - any eagerly injected service that tried to bind to a specific port would probably fail in the same way as it found the old version was still holding on to the port.

[Igmar Palsenberg]

 

It's not even particularly specific to Akka - any eagerly injected service that tried to bind to a specific port would probably fail in the same way as it found the old version was still holding on to the port.

We also use shutdown hooks extensively, and it works as expected (mainly to kill off the actors, and things like Jedis connection pools).

Igmar