InvalidActorNameException when recreating actor with context.watch

[Iain Hull]

I am using context.watch in a simple DeathReportActor, Clients send the DeathReportActor a Watch message, then when the fated actor dies a specified message is sent to the mourner. I want to use this for restarting actors with different constructor parameters.  However when I test this actor I keep getting intermittent InvalidActorNameExceptions. 

It seems that the actors name is still in use after I receive the Terminated message, so sometimes creating an actor straight away results in an error.  I thought I was guaranteed to be able to recreate the actor upon recipt of the Terminated message, is this correct or do I have bug in my code?

Here is a simple gist that reproduces the error.  I have tested with akka 2.0.5, 2.2.3 and 2.3.1 and get the same results when I retry this operation enough times is a very tight loop.

https://gist.github.com/IainHull/10129911

Thanks,
Iain.

[Iain Hull]

I have been looking into this in more detail and wonder if this could be an Akka bug.

The docs say: http://doc.akka.io/docs/akka/snapshot/java/untyped-actors.html

Since stopping an actor is asynchronous, you cannot immediately reuse the name of the child you just stopped; this will result in an InvalidActorNameException. Instead, watch the terminating actor and create its replacement in response to the Terminated message which will eventually arrive.

I am trying to recreate an actor in response to the Terminated message.  When I send the Recreate message to the GuardianActor it works most of the time but fails nearly one time in 10,000.

class GuardianActor extends Actor {
  val deathReporter = context.actorOf(DeathReportActor.props)
  var fatedOpt: Option[ActorRef] = None
 
  def receive = {
    case Create(value) =>
      fatedOpt = Some(context.actorOf(pheonixProps(value), "someName"))
    case Recreate(value) =>
      fatedOpt foreach { fated =>
        deathReporter ! DeathReportActor.Watch(fated, self, Create(value))
        context.system.stop(fated)
      }
  }
}

 
class DeathReportActor extends Actor {
  import DeathReportActor._
 
  var watches = Map[ActorRef, Watch]()
 
  def receive = {
    case w @ Watch(fated, _, _) =>
      watches += fated -> w
      context.watch(fated)
    case Terminated(fated) =>
      watches.get(fated) foreach {
        case Watch(_, mourner, message) =>
          mourner ! message
      }
  }
}



[INFO] [04/08/2014 16:08:23.670] [DeathReportActorTest-akka.actor.default-dispatcher-6] [akka://DeathReportActorTest/user/$a/someName] Message [akka.dispatch.sysmsg.Suspend] from Actor[akka://DeathReportActorTest/user/$a/someName#697148684] to Actor[akka://DeathReportActorTest/user/$a/someName#697148684] was not delivered. [1] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[ERROR] [04/08/2014 16:08:23.671] [DeathReportActorTest-akka.actor.default-dispatcher-8] [akka://DeathReportActorTest/user/$a] actor name [someName] is not unique!
akka.actor.InvalidActorNameException: actor name [someName] is not unique!
	at akka.actor.dungeon.ChildrenContainer$NormalChildrenContainer.reserve(ChildrenContainer.scala:130)
	at akka.actor.dungeon.Children$class.reserveChild(Children.scala:77)
	at akka.actor.ActorCell.reserveChild(ActorCell.scala:369)
	at akka.actor.dungeon.Children$class.makeChild(Children.scala:202)
	at akka.actor.dungeon.Children$class.actorOf(Children.scala:38)
	at akka.actor.ActorCell.actorOf(ActorCell.scala:369)
	at DeathReportActorTest$GuardianActor$$anonfun$receive$2.applyOrElse(TestFoo.scala:45)
	at akka.actor.Actor$class.aroundReceive(Actor.scala:465)
	at DeathReportActorTest$GuardianActor.aroundReceive(TestFoo.scala:39)
	at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
	at akka.actor.ActorCell.invoke(ActorCell.scala:487)
	at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)
	at akka.dispatch.Mailbox.run(Mailbox.scala:220)
	at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393)
	at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
	at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
	
Could there be a race condition in this code? I found this ticket, but it could not be reproduced.
https://www.assembla.com/spaces/akka/tickets/3007-invalidactornameexception-even-after-terminated-received-for-that-nam#/activity/ticket:

Iain.

[Martynas Mickevičius]

Hi Iain,

I think the problem could be in your code here:

  deathReporter ! DeathReportActor.Watch(fated, self, Create(value))

  context.system.stop(fated)

Here I can see a race condition. In some cases fated actor can be stopped before Watch message in handled. So DeathReportActor never gets Terminated message. If I move

  context.system.stop(fated)

to DeathReportActor after watch call

  context.watch(fated)

  context.system.stop(fated)

then your test case passes.

-- 

Martynas Mickevičius

Typesafe – Reactive Apps on the JVM

[Iain Hull]

Hi Martynas,

Thank you for you fix.  This does indeed fix my case and I am very happy change me code accordingly.

However can you expand on the semantics of watch?  Is it only safe to call watch from the same actor that calls stop?  Or is it only safe to recreate a dead actor if the watch was called from the same actor as stop?

The documentation says

It should be noted that the Terminated message is generated independent of the order in which registration and termination occur. In particular, the watching actor will receive a Terminated message even if the watched actor has already been terminated at the time of registration.

In my case when the watch and the stop are performed by different actors the Terminated message is sent as expected, however it is not safe to recreated the terminated actor in response to this message.  Am I missing something regarding when these affects are visible or should be this behavior be documented?

Iain.

[√iktor Ҡlang]

This sounds like a bug to me, but I'll leave it to the Akka team to weigh in on :-)

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

[Patrik Nordwall]

On Wed, Apr 9, 2014 at 9:04 PM, Iain Hull <iain...@gmail.com> wrote:

Hi Martynas,

Thank you for you fix.  This does indeed fix my case and I am very happy change me code accordingly.

However can you expand on the semantics of watch?  Is it only safe to call watch from the same actor that calls stop?  Or is it only safe to recreate a dead actor if the watch was called from the same actor as stop?

The documentation says

It should be noted that the Terminated message is generated independent of the order in which registration and termination occur. In particular, the watching actor will receive a Terminated message even if the watched actor has already been terminated at the time of registration.

In my case when the watch and the stop are performed by different actors the Terminated message is sent as expected, however it is not safe to recreated the terminated actor in response to this message. 

I think the only thing that is guaranteed is that if the parent watch a child it should be able to create a new child with the same name after it has received Terminated. In your code it is another actor that is watching, and it might see Terminated before the parent knows about it.

/Patrik

 

Am I missing something regarding when these affects are visible or should be this behavior be documented?

Iain.

On Wednesday, 9 April 2014 15:04:32 UTC+1, Martynas Mickevičius wrote:

Hi Iain,

I think the problem could be in your code here:

  deathReporter ! DeathReportActor.Watch(fated, self, Create(value))

  context.system.stop(fated)

Here I can see a race condition. In some cases fated actor can be stopped before Watch message in handled. So DeathReportActor never gets Terminated message. If I move

  context.system.stop(fated)

to DeathReportActor after watch call

  context.watch(fated)

  context.system.stop(fated)

then your test case passes.

-- 

Martynas Mickevičius

Typesafe – Reactive Apps on the JVM

--

>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

--

Patrik Nordwall
Typesafe -  Reactive apps on the JVM
Twitter: @patriknw

JOIN US. REGISTER TODAY!

Scala

Days

June 16th-18th,

Berlin

[Roland Kuhn]

9 apr 2014 kl. 15:04 skrev Martynas Mickevičius <martynas.m...@typesafe.com>:

Hi Iain,

I think the problem could be in your code here:

  deathReporter ! DeathReportActor.Watch(fated, self, Create(value))

  context.system.stop(fated)

Here I can see a race condition. In some cases fated actor can be stopped before Watch message in handled. So DeathReportActor never gets Terminated message.

This is not true: DeathWatch is idempotent in the sense that you will always get the Terminated message at your earliest convenience when both of the following are true:

• context.watch(target) has been called
• the `target` actor does not exist

Regards,

Roland

If I move

  context.system.stop(fated)

to DeathReportActor after watch call

  context.watch(fated)

  context.system.stop(fated)

then your test case passes.

-- 

Martynas Mickevičius

Typesafe – Reactive Apps on the JVM

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

Dr. Roland Kuhn
Akka Tech Lead
Typesafe – Reactive apps on the JVM.
twitter: @rolandkuhn

[Roland Kuhn]

10 apr 2014 kl. 07:08 skrev Patrik Nordwall <patrik....@gmail.com>:

On Wed, Apr 9, 2014 at 9:04 PM, Iain Hull <iain...@gmail.com> wrote:

Hi Martynas,

Thank you for you fix.  This does indeed fix my case and I am very happy change me code accordingly.

However can you expand on the semantics of watch?  Is it only safe to call watch from the same actor that calls stop?  Or is it only safe to recreate a dead actor if the watch was called from the same actor as stop?

The documentation says

It should be noted that the Terminated message is generated independent of the order in which registration and termination occur. In particular, the watching actor will receive a Terminated message even if the watched actor has already been terminated at the time of registration.

In my case when the watch and the stop are performed by different actors the Terminated message is sent as expected, however it is not safe to recreated the terminated actor in response to this message. 

I think the only thing that is guaranteed is that if the parent watch a child it should be able to create a new child with the same name after it has received Terminated. In your code it is another actor that is watching, and it might see Terminated before the parent knows about it.

Yes, this is exactly the reason. This demonstrates quite directly what it means that actors are only consistent on the inside; between actors there is at most eventual consistency.

Regards,

Roland

[√iktor Ҡlang]

On Thu, Apr 10, 2014 at 2:00 PM, Roland Kuhn <goo...@rkuhn.info> wrote:

10 apr 2014 kl. 07:08 skrev Patrik Nordwall <patrik....@gmail.com>:

On Wed, Apr 9, 2014 at 9:04 PM, Iain Hull <iain...@gmail.com> wrote:

Hi Martynas,

Thank you for you fix.  This does indeed fix my case and I am very happy change me code accordingly.

However can you expand on the semantics of watch?  Is it only safe to call watch from the same actor that calls stop?  Or is it only safe to recreate a dead actor if the watch was called from the same actor as stop?

The documentation says

It should be noted that the Terminated message is generated independent of the order in which registration and termination occur. In particular, the watching actor will receive a Terminated message even if the watched actor has already been terminated at the time of registration.

In my case when the watch and the stop are performed by different actors the Terminated message is sent as expected, however it is not safe to recreated the terminated actor in response to this message. 

I think the only thing that is guaranteed is that if the parent watch a child it should be able to create a new child with the same name after it has received Terminated. In your code it is another actor that is watching, and it might see Terminated before the parent knows about it.

Yes, this is exactly the reason. This demonstrates quite directly what it means that actors are only consistent on the inside; between actors there is at most eventual consistency.

Very good point. It clearly demonstrate that news does not travel faster than the speed of light.

--

Cheers,

√

[Iain Hull]

Thanks for your answers, it makes perfect sense now.  Although I am a little disappointed as this means my goal of abstracting a common restart pattern into into a separate actor is fruitless.  However I will look for another approach.

[Iain Hull]

On Friday, 11 April 2014 10:35:46 UTC+1, Iain Hull wrote:

However I will look for another approach.

I think I have found a clean way to inject the DeathReportActor code into a parent actor.  I have made this actor a trait and renamed it to ChildMonitor.  I would be very grateful if someone could advise if this is a valid use of inheritance with actors?  I put the traits logic into the unhandled method as this allows subclasses to use the normal receive method as well as context.become and context.unbecome.  However I am not sure if this is a valid approach to stacking actor behavior or does it break the Actor's contract.  Can I safely modify state and access the other Actor methods in unhandled?

trait ChildMonitor extends Actor {
  private var watches = Map[ActorRef, Any]()
 
  def monitorChild(child: ActorRef, message: Any): Unit = {
      watches += child -> message
      context.watch(child)    
  }
  
  override def unhandled(message: Any): Unit = message match {
    case Terminated(child) =>
      watches.get(child) foreach (m => self ! m)
      watches -= child
    case _ =>
      super.unhandled(message)
  } 
}

class GuardianActor extends Actor with ChildMonitor {

  var fatedOpt: Option[ActorRef] = None
 
  def receive = {
    case Create(value) =>

      fatedOpt = Some(context.actorOf(pheonixProps(value), "pheonix"))
    case Recreate(value) =>
      fatedOpt match { 
        case Some(fated) =>
          monitorChild(fated, Create(value))
          context.system.stop(fated)
          // fated ! PoisonPill
        case None =>
          fatedOpt = Some(context.actorOf(pheonixProps(value), "pheonix"))
      }
  }
}

Thanks for your help,
Iain.

[Roland Kuhn]

Hi Iain,

11 apr 2014 kl. 12:37 skrev Iain Hull <iain...@gmail.com>:

On Friday, 11 April 2014 10:35:46 UTC+1, Iain Hull wrote:

However I will look for another approach.

I think I have found a clean way to inject the DeathReportActor code into a parent actor.  I have made this actor a trait and renamed it to ChildMonitor.  I would be very grateful if someone could advise if this is a valid use of inheritance with actors?  I put the traits logic into the unhandled method as this allows subclasses to use the normal receive method as well as context.become and context.unbecome.  However I am not sure if this is a valid approach to stacking actor behavior or does it break the Actor's contract.  Can I safely modify state and access the other Actor methods in unhandled?

Yes, that is safe.

trait ChildMonitor extends Actor {
  private var watches = Map[ActorRef, Any]()
 
  def monitorChild(child: ActorRef, message: Any): Unit = {
      watches += child -> message
      context.watch(child)    
  }
  
  override def unhandled(message: Any): Unit = message match {
    case Terminated(child) =>

One problem is that this grabs the handling of all Terminated messages which get this far; you could add a guard `if watches contains child`, but if different layers of your composed actor all watch a target then only one of them gets the Terminated message.

      watches.get(child) foreach (m => self ! m)
      watches -= child
    case _ =>
      super.unhandled(message)
  } 
}

class GuardianActor extends Actor with ChildMonitor {
  var fatedOpt: Option[ActorRef] = None
 
  def receive = {
    case Create(value) =>
      fatedOpt = Some(context.actorOf(pheonixProps(value), "pheonix"))

This means that this actor can be remote-crashed by sending Create twice (using InvalidActorNameException), but that might be your intention.

    case Recreate(value) =>
      fatedOpt match { 
        case Some(fated) =>
          monitorChild(fated, Create(value))
          context.system.stop(fated)

I would just use `context.stop(fated)`.

          // fated ! PoisonPill
        case None =>
          fatedOpt = Some(context.actorOf(pheonixProps(value), "pheonix"))
      }
  }
}

Thanks for your help,

You’re welcome!

Roland

Iain.

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.