DeadLetters when sending to self

[Frederic]

Hello,

I found a quite specific case involving Actors, Future callbacks and exceptions where self becomes deadLetters.

I have the feeling I'm using self the way it is meant to be used, as described here: http://doc.akka.io/docs/akka/2.3.11/general/jmm.html#jmm-shared-state

Am I doing something wrong or is it a bug?

Please see code and output below to reproduce/illustrate the issue.

Thanks, Fred

import akka.actor._

import scala.concurrent._

case object Done

case object Greet

case class Complete(p: Promise[Unit])

class Greeter extends Actor {

  import context.dispatcher

  def receive = {

    case Done => println("Done (system still running)!")

    case Complete(p) => p.success(())

    case Greet =>

      val p = Promise[Unit]()

      val saveMyself = self

      p.future onComplete {

        case _ =>

          println(s"self is $self")

          println(s"self is expexted to be $saveMyself")

          // cannot use self to send a message, bug?

          self ! Done

          saveMyself ! Done

      }

      self ! Complete(p)

      throw new Exception("Messing with onComplete callback, self will be dead letters")

  }

}

object HelloAkkaScala extends App {

  val system = ActorSystem("helloakka")

  val greeter = system.actorOf(Props[Greeter], "greeter")

  greeter ! Greet

}

Program output :

Running HelloAkkaScala
self is Actor[akka://helloakka/deadLetters]
[ERROR] [06/18/2015 14:25:41.510] [helloakka-akka.actor.default-dispatcher-4] [akka://helloakka/user/greeter] Messing with onComplete callback, self will be dead letters
java.lang.Exception: Messing with onComplete callback, self will be dead letters
at Greeter$anonfun$receive$1.applyOrElse(HelloAkkaScala.scala:27)
at akka.actor.Actor$class.aroundReceive(Actor.scala:467)
at Greeter.aroundReceive(HelloAkkaScala.scala:8)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
at akka.actor.ActorCell.invoke(ActorCell.scala:487)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)
at akka.dispatch.Mailbox.run(Mailbox.scala:220)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

self is expexted to be Actor[akka://helloakka/user/greeter#990239115]
Done (system still running)!
[INFO] [06/18/2015 14:25:41.516] [helloakka-akka.actor.default-dispatcher-3] [akka://helloakka/deadLetters] Message [Done$] from Actor[akka://helloakka/deadLetters] to Actor[akka://helloakka/deadLetters] was not delivered. [1] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.

[Frederic]

There's more... as context is also null in the Future's callback, anything using context won't work, see below for an updated example using the Akka's scheduler and dispatcher:

import akka.actor._

import scala.concurrent._

import scala.concurrent.duration._

case class Done(s: String)

case object Greet

case class Complete(p: Promise[Unit])

class Greeter extends Actor {

  import context.dispatcher

  def receive = {

    case Done(s) => println(s"Done($s) (system still running)!")

    case Complete(p) => p.success(())

    case Greet =>

      val p = Promise[Unit]()

      val saveMyself = self

      val sched = context.system.scheduler

      val disp = context.dispatcher

      p.future onComplete {

        case _ =>

          println(s"self is $self")

          println(s"self is expexted to be $saveMyself")

          // cannot use self to send a message, need to close on self

          self ! Done("Send using self")

          saveMyself ! Done("Send using saveMyself")

          // are we supposed to avoid using context here?

          println(s"context is $context") // outputs: context is null

          // is this meant to work?

          // context.system.scheduler.scheduleOnce(1.millisecond, saveMyself, Done)

          // will fail because we can't get the scheduler as context is null

          try { context.system.scheduler.scheduleOnce(1.millisecond, saveMyself, Done("scheduler 1")) } catch { case e => println(e) }

          // The following will still fail because implicit dispatcher is context.dispatcher and context is null

          try { sched.scheduleOnce(1.millisecond, saveMyself, Done("scheduler 2")) } catch { case e => println(e) }

          // that will work...

          sched.scheduleOnce(1.millisecond, saveMyself, Done("scheduler 3"))(disp, saveMyself)

      }

      self ! Complete(p)

      throw new Exception("Messing with onComplete callback, self will be dead letters")

  }

}

object HelloAkkaScala extends App {

  val system = ActorSystem("helloakka")

  val greeter = system.actorOf(Props[Greeter], "greeter")

  greeter ! Greet

}

Program output :

Running HelloAkkaScala
self is Actor[akka://helloakka/deadLetters]

[ERROR] [06/18/2015 16:47:50.464] [helloakka-akka.actor.default-dispatcher-3] [akka://helloakka/user/greeter] Messing with onComplete callback, self will be dead letters

java.lang.Exception: Messing with onComplete callback, self will be dead letters

at Greeter$anonfun$receive$1.applyOrElse(HelloAkkaScala.scala:40)
at akka.actor.Actor$class.aroundReceive(Actor.scala:467)
at Greeter.aroundReceive(HelloAkkaScala.scala:9)

at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
at akka.actor.ActorCell.invoke(ActorCell.scala:487)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)
at akka.dispatch.Mailbox.run(Mailbox.scala:220)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

self is expexted to be Actor[akka://helloakka/user/greeter#-1327535750]
context is null
java.lang.NullPointerException
java.lang.NullPointerException
Done(Send using saveMyself) (system still running)!
[INFO] [06/18/2015 16:47:50.480] [helloakka-akka.actor.default-dispatcher-4] [akka://helloakka/deadLetters] Message [Done] from Actor[akka://helloakka/deadLetters] to Actor[akka://helloakka/deadLetters] was not delivered. [1] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
Done(scheduler 3) (system still running)!

[Roland Kuhn]

Hi Frederic,

you violate the first rule of Actors: never call any of their methods from outside of their context of execution. This is true for all of them, including context() and self(). While the result of calling self() can then safely be shared afterwards (an ActorRef is thread-safe), the result of context() cannot (because ActorContext is not thread-safe).

What happens in your sample is that the Actor is destroyed due to the Exception (that triggers a restart), so anything referencing the old Actor instance will see a dead entity with nulls in it (in order to minimize memory leaks in case someone does something wrong).

Regards,

Roland

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

Dr. Roland Kuhn
Akka Tech Lead
Typesafe – Reactive apps on the JVM.
twitter: @rolandkuhn

[Michael Frank]

the current documentation appears to be incorrect, which i believe was the source of confusion.  on line 20 of the code block of the section "Actors and shared mutable state" on page http://doc.akka.io/docs/akka/2.3.11/general/jmm.html#jmm-shared-state it specifically states:

 18   // Completely safe, "self" is OK to close over
 19   // and it's an ActorRef, which is thread-safe
 20   Future { expensiveCalculation() } onComplete { f => self ! f.value.get }

i opened an issue for it: https://github.com/akka/akka/issues/17805

-Michael

[Patrik Nordwall]

Thanks for reporting!

--

Patrik Nordwall
Typesafe -  Reactive apps on the JVM
Twitter: @patriknw

[Frederic]

Yes Michael, you are right. I based my example code on the documentation you referenced, assuming it was correct.

What would be the correct way to do, the same as when sender is involved or is there something special here?
// is this the safe way to proceed?
val s = self
Future { expensiveCalculation() } onComplete { f => s ! f.value.get }

// or is this better?

Future { expensiveCalculation() } pipeTo self()

Could someone help me understand the following statement from the newly opened issue https://github.com/akka/akka/issues/17805#issuecomment-114389213: "(after all, even if you share the result of self() the message will end up in DeadLetters anyway)". This doesn't match my experience, as actors are restarted, message sent to a copy of self() end up in the actor's mailbox and not in deadLetters.

Thanks, Fred

[Endre Varga]

On Wed, Jun 24, 2015 at 8:20 PM, Frederic <freder...@gmail.com> wrote:

Yes Michael, you are right. I based my example code on the documentation you referenced, assuming it was correct.

What would be the correct way to do, the same as when sender is involved or is there something special here?
// is this the safe way to proceed?
val s = self
Future { expensiveCalculation() } onComplete { f => s ! f.value.get }

// or is this better?

Future { expensiveCalculation() } pipeTo self()

Could someone help me understand the following statement from the newly opened issue https://github.com/akka/akka/issues/17805#issuecomment-114389213: "(after all, even if you share the result of self() the message will end up in DeadLetters anyway)". This doesn't match my experience, as actors are restarted, message sent to a copy of self() end up in the actor's mailbox and not in deadLetters.

What I wanted to say that if you stop the actor then:

 - sending to the the ActorRef of that actor (independently if it was returned by self() or aquired from actorOf) will end up delivering the message to dead letters

 - closing over self() and then calling it after the actor has been stopped will return the ActorRef of DeadLetters, so again, sending messages to it will end up in dead letters

-Endre

[Patrik Nordwall]

We have investigated and fixed the issue. The conclusion was that the change of self during restarts was not needed and incorrect. We have changed the implementation to match the documentation "// Completely safe, "self" is OK to close over"

It will be fixed in upcoming 2.3.12.

Using pipeTo is still the recommended bridge between futures and actors.

You must still not close over and use anything in the context from other threads than the actor's message receive.

/Patrik

[Frederic]

Thank you!

Fred