Akka - Fault tolerance scenario - test fails

[Vincenzo D'Amore]

I'm simulating in a test the conversation between three actors (A, B, C)

    A --->  MessageA2B  ---> B --->  MessageB2C ---> C 

  

When `MessageB2C` is successfully arrived to C then the acknowledgement is sent back to the origin.

    C --> MessageB2C_Ack --> B --> MessageA2B_Ack --> A

The only peculiarity of this conversation is the message `MessageB2C`.

`MessageB2C` is sent at least every 50 ms until C does not answer with its acknowledgement.

I've implemented this simple conversation with scala testkit framework, but the test fail in a particular situation. Here is the github repo https://github.com/freedev/reactive-akka-testkit

When ActorB retries to send MessageB2C more then once time, then is unable to receive the answers from ActorC. And the reply from ActorC to ActorB goes to deadLetters.   

test("expectNoMessage-case: actorB retries MessageB2C every 50 milliseconds") {

   val actorA = TestProbe()

   val actorC = TestProbe()

   val actorB = system.actorOf(ActorB.props(Props(classOf[TestRefWrappingActor], actorC)), "step1-case2-primary")

    

   actorA.send(actorB, MessageA2B())

    

   actorA.expectNoMessage(100.milliseconds)

    

   actorC.expectMsg(MessageB2C())

    

        // Retries form above

   actorC.expectMsg(200.milliseconds, MessageB2C())

        // Never reach this point with 100 ms frequency

   actorC.expectMsg(200.milliseconds, MessageB2C())

    

   actorA.expectNoMessage(100.milliseconds)

    

   actorC.reply(MessageB2C_Ack())

    

        // Never reach this point with MessageB2C 50 ms frequency

   actorA.expectMsg(MessageA2B_Ack())

}

This is the `ActorB` code:

class ActorB(actorCProps: Props) extends Actor {

  import ActorB._

  import context.dispatcher

    

  val log = Logging(context.system, this)

    

  val actorC = context.actorOf(actorCProps)

    

  def receive = {

    case r:MessageA2B => {

      val client = context.sender()

      implicit val timeout = Timeout(100.milliseconds)

      implicit val scheduler=context.system.scheduler

      val p = MessageB2C()

    

      RetrySupport.retry(() => {

         Patterns.ask(actorC, p, 50.millisecond)

      }, 10, 50.millisecond)

      .onSuccess({

         case p: MessageB2C_Ack => {

              client ! MessageA2B_Ack()

         }

      })

    }

  }

}

My eterne gratitude to anyone is able to understand why the reply from ActorC to ActorB goes to deadLetters.  

[Brian Maso]

I would assume first that a message to ActorB going to dead letters means actor B has suffered an unexpected exception and terminated. 

Brian Maso

--
You received this message because you are subscribed to the Google
Groups "scalatest-users" group.
To post to this group, send email to scalate...@googlegroups.com
To unsubscribe from this group, send email to
scalatest-use...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/scalatest-users?hl=en
ScalaTest itself, and documentation, is available here:
http://www.artima.com/scalatest
---
You received this message because you are subscribed to the Google Groups "scalatest-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scalatest-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/scalatest-users/85582647-8422-4b0a-aa35-e0ea19fe7c6f%40googlegroups.com.

[Vincenzo D'Amore]

Hi Brian, 

Yes, Patterns.ask rises an AskTimeoutException for each tentative, but AFAIK they are all caught within the RetrySupport.retry and the actor is not terminated. 

Do you think I should try to avoid to rise any exception at all?

Just to be sure if the code rises any exception, I've tried another approach removing the RetrySupport.retry and adding a custom function retry:

  def retry(actorRef: ActorRef, message: Any, maxAttempts: Int, attempt: Int): Future[Any] = {
    log.info("ActorB - sent message MessageB2C to ActorC " + actorC)
    val future = Patterns.ask(actorRef, message, 50.millisecond) recover {
      case e: AskTimeoutException =>
        if (attempt <= maxAttempts) retry(actorRef, message, maxAttempts, attempt + 1)
        else None // Return default result according to your requirement, if actor is non-reachable.
    }
    future
  }

/// omissis ////

      retry(actorC, p, 10) onSuccess({
        case p: MessageB2C_Ack => {
          log.info("ActorB - Received MessageB2C_Ack so now sending an MessageA2B_Ack to client " + client)
          client ! MessageA2B_Ack()
        }
      })

Running the new version I have the same behaviour:

[INFO] [09/16/2019 23:11:43.621] [KVStoreSuite-akka.actor.default-dispatcher-3] [akka://KVStoreSuite/user/actorb-step1] ActorB received message MessageA2B from client Actor[akka://KVStoreSuite/system/testProbe-1#-1982393505]
[INFO] [09/16/2019 23:11:43.622] [KVStoreSuite-akka.actor.default-dispatcher-3] [akka://KVStoreSuite/user/actorb-step1] ActorB - sent message MessageB2C to ActorC Actor[akka://KVStoreSuite/user/actorb-step1/$a#-815876914]
[INFO] [09/16/2019 23:11:43.695] [KVStoreSuite-akka.actor.default-dispatcher-4] [akka://KVStoreSuite/user/actorb-step1] ActorB - sent message MessageB2C to ActorC Actor[akka://KVStoreSuite/user/actorb-step1/$a#-815876914]
[INFO] [09/16/2019 23:11:43.762] [KVStoreSuite-akka.actor.default-dispatcher-2] [akka://KVStoreSuite/user/actorb-step1] ActorB - sent message MessageB2C to ActorC Actor[akka://KVStoreSuite/user/actorb-step1/$a#-815876914]
[INFO] [09/16/2019 23:11:43.834] [KVStoreSuite-akka.actor.default-dispatcher-3] [akka://KVStoreSuite/user/actorb-step1] ActorB - sent message MessageB2C to ActorC Actor[akka://KVStoreSuite/user/actorb-step1/$a#-815876914]
[INFO] [09/16/2019 23:11:43.865] [KVStoreSuite-akka.actor.default-dispatcher-2] [akka://KVStoreSuite/deadLetters] Message [damore.ActorB$MessageB2C_Ack] from Actor[akka://KVStoreSuite/system/testProbe-2#43200690] to Actor[akka://KVStoreSuite/deadLetters] was not delivered. [1] dead letters encountered. If this is not an expected behavior, then [Actor[akka://KVStoreSuite/deadLetters]] may have terminated unexpectedly, This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[INFO] [09/16/2019 23:11:43.904] [KVStoreSuite-akka.actor.default-dispatcher-4] [akka://KVStoreSuite/user/actorb-step1] ActorB - sent message MessageB2C to ActorC Actor[akka://KVStoreSuite/user/actorb-step1/$a#-815876914]
[INFO] [09/16/2019 23:11:43.972] [KVStoreSuite-akka.actor.default-dispatcher-2] [akka://KVStoreSuite/user/actorb-step1] ActorB - sent message MessageB2C to ActorC Actor[akka://KVStoreSuite/user/actorb-step1/$a#-815876914]
[INFO] [09/16/2019 23:11:44.042] [KVStoreSuite-akka.actor.default-dispatcher-3] [akka://KVStoreSuite/user/actorb-step1] ActorB - sent message MessageB2C to ActorC Actor[akka://KVStoreSuite/user/actorb-step1/$a#-815876914]
[INFO] [09/16/2019 23:11:44.112] [KVStoreSuite-akka.actor.default-dispatcher-2] [akka://KVStoreSuite/user/actorb-step1] ActorB - sent message MessageB2C to ActorC Actor[akka://KVStoreSuite/user/actorb-step1/$a#-815876914]
[INFO] [09/16/2019 23:11:44.182] [KVStoreSuite-akka.actor.default-dispatcher-4] [akka://KVStoreSuite/user/actorb-step1] ActorB - sent message MessageB2C to ActorC Actor[akka://KVStoreSuite/user/actorb-step1/$a#-815876914]
[INFO] [09/16/2019 23:11:44.253] [KVStoreSuite-akka.actor.default-dispatcher-3] [akka://KVStoreSuite/user/actorb-step1] ActorB - sent message MessageB2C to ActorC Actor[akka://KVStoreSuite/user/actorb-step1/$a#-815876914]
[INFO] [09/16/2019 23:11:44.323] [KVStoreSuite-akka.actor.default-dispatcher-4] [akka://KVStoreSuite/user/actorb-step1] ActorB - sent message MessageB2C to ActorC Actor[akka://KVStoreSuite/user/actorb-step1/$a#-815876914]


assertion failed: timeout (3 seconds) during expectMsg while waiting for MessageA2B_Ack()
java.lang.AssertionError: assertion failed: timeout (3 seconds) during expectMsg while waiting for MessageA2B_Ack()
at scala.Predef$.assert(Predef.scala:170)
at akka.testkit.TestKitBase$class.expectMsg_internal(TestKit.scala:402)
at akka.testkit.TestKitBase$class.expectMsg(TestKit.scala:379)
at akka.testkit.TestKit.expectMsg(TestKit.scala:896)
at damore.Step1$$anonfun$1.apply(Step1.scala:42)
at damore.Step1$$anonfun$1.apply(Step1.scala:23)

To view this discussion on the web visit https://groups.google.com/d/msgid/scalatest-users/CADq3_Yd%3DvfShTryh%3DkOi02NVdm9%2BSCw46msfF_fghamjggnuKw%40mail.gmail.com.

--

Vincenzo D'Amore

[Vincenzo D'Amore]

Hi,

I'm still trying to understand what's wrong in this test. As far as I see the exception is contained. What I did wrong? 

I also noticed that the test complete successfully removing the line actorA.expectNoMessage(100.milliseconds)

--

Vincenzo D'Amore

[Tim Moore]

Hi Vincenzo,

Something that might not be obvious about using Patterns.ask, is that it spawns a temporary actor that only exists to receive the reply and complete the Future returned from ask. That temporary actor is used as the sender of the MessageB2C message send to actor C. That means that when C replies, it is not to B, but rather to the temporary actor created by ask. In each retry, that will be a new actor, and when the ask times out, the temporary actor will be stopped.

Then, when you use expectNoMessage, the test will have to wait for the entire duration that you pass to it (100ms) before proceeding. That means that by the time you call "actorC.reply(MessageB2C_Ack())", the temporary sender of the previous message will have timed out and stopped, and there should be a new retried message in actor C's mailbox. That's why the reply goes to dead letters. I believe you will be able to solve this by adding another "actorC.expectMsg(200.milliseconds, MessageB2C())" in between "actorA.expectNoMessage(100.milliseconds)" and "actorC.reply(MessageB2C_Ack())".

In general, using an ask from inside an actor like this is often discouraged in favor of directly modelling the request and response flow directly in the sending actor. This is often simpler and less "magical", with fewer surprises like this one. This article explains some of the tradeoffs pretty well https://medium.com/@yuriigorbylov/akka-ask-antipattern-8361e9698b20

I also want to point out, if you aren't already aware, that there is an Akka discussion forum at https://discuss.akka.io. This kind of question is more specific to Akka than to ScalaTest, so it might be a better place for related questions in the future.

Best,

Tim

To view this discussion on the web visit https://groups.google.com/d/msgid/scalatest-users/CA%2B8z68k6%2BGWU%2B_pykYw8vDg-AADGYE-YqHvMHuhMwRpb_vn6bw%40mail.gmail.com.

--

Tim Moore

Sr. Manager Engineering and Product Management, Lightbend, Inc.

tim....@lightbend.com
+61 420 981 589

[Vincenzo D'Amore]

Great Tim, thanks for explaining how Patterns.ask works, and for sharing the article, I'll read it immediately. 

Regarding discussion.akka.io, I already wrote there few days ago https://discuss.lightbend.com/t/retry-send-message-every-50-milliseconds-fails/5065

To be honest, I've written also on stackoverflow https://stackoverflow.com/questions/57945303/akka-fault-tolerance-scenario-test-fails

and yet in the another forum, but basically I haven't received any clue at all. I was struggling to understand what's wrong, so I thought that in a Scala forum I would find someone with the helpfulness to help me. Thanks again for your time, indeed very precious, can I take advantage for another little question, could you please suggest me also a book or a course to follow in order to getting more fluent in the reactive programming with Akka? 

To view this discussion on the web visit https://groups.google.com/d/msgid/scalatest-users/CAApsmORK4zTsEdoqpC_9aeYccpbksuzTVKS0fsve8HnK-hYXwg%40mail.gmail.com.

--

Vincenzo D'Amore

[Tim Moore]

I apologise for not seeing your thread on discuss.lightbend.com. I've copied my answer there to make it easier for more people to find it in the future.

As for your other question, there is an existing topic on the forum about that: https://discuss.lightbend.com/t/book-other-resource-for-akka/298

Best,

Tim

To view this discussion on the web visit https://groups.google.com/d/msgid/scalatest-users/CA%2B8z68n1nbyxk5yEcwka7Tu--vsQx%3DwCALtSTATrkmpHx%2Bq8vg%40mail.gmail.com.