Can Akka actors deadlock?

[kraythe]

Lets say that we create a pool of 5 actors using round robin logic and then we send 5 messages at the same time and those messages in the course of being processed send another 5 messages each to the actor and wait for the responses to send back to the caller. If this was a java executor it would eventually deadlock waiting for the second set of calls to complete. Is the same deadlock possible with Akka? If not then what would prevent it? 

Thanks in advance. 

[Viktor Klang]

define "wait for the responses to send back to the caller"

On Tue, Aug 25, 2015 at 12:04 PM, kraythe <kra...@gmail.com> wrote:

Lets say that we create a pool of 5 actors using round robin logic and then we send 5 messages at the same time and those messages in the course of being processed send another 5 messages each to the actor and wait for the responses to send back to the caller. If this was a java executor it would eventually deadlock waiting for the second set of calls to complete. Is the same deadlock possible with Akka? If not then what would prevent it? 

Thanks in advance. 

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

--

Cheers,

√

[kraythe]

Call another actor using an ask pattern.

[Viktor Klang]

how does that wait?

--
Cheers,
√

[Roland Kuhn]

The normal way to implement this would not use the ask pattern: if A wants to know something and sends GetSomething to B, but B does not have it and needs to defer to C, then in the Actor world it is possible and completely idiomatic to have C send the result back to A directly. This is the big difference between ActorRefs (which can be passed around for arbitrary messaging patterns) and RPC (which is strictly request–response).

Regards,

Roland

Dr. Roland Kuhn
Akka Tech Lead
Typesafe – Reactive apps on the JVM.
twitter: @rolandkuhn

[kraythe]

If you need the results of the generated futures then it waits for them all to be completed. Process A needs the results of Process B in order to complete. This is a pretty common paradigm in our software. 

-- Robert

[kraythe]

However, this has to be combined with some synchronous calls. Many parts of the system are Asynch but not all. We have users hitting web pages expecting to get certain data. We cant say "thanks for your submission and we will mail you your account information. " We need to go fetch that information which might involve nested calls to several actors and that might recurse to some of the same actors. So we have a situation where user hits a web page and it invokes ask() on Actor A which has a collection of Data to be gotten from Actor B which in turn needs some info from Actor C. Finally Actor C requires a peice of information from actor A (a different piece of information) and then it is all returned. Now I cant complete process A until all of this is done and in an executor concurrent world that means DEADLOCK if each invocation doesn't have its own executor. 

Now one can say, "Don't design a system like this" and I can reply "ok, Ill just go to my boss and say we have to throw away 20 million in capital investment and redesign it with another 20 million to get the right system that looks good to the development world." Im afraid that would not end well. Furthermore, I don't know I could ever get around the problem of needing synchronous calls at times. Users interacting on a web page need things synchronously. Of course we could just get rid of the pesky users and the system would look great. Of course we wouldn't have a business. 

I am sorry but I am more than a bit frustrated hearing stuff like that from the Akka community. Im not talking about use cases that are way out of the norm. This is the type of thing that powers millions of web sites. Now I guess it could be you come back and say "Well, Akka is not the system to handle this." In which case I wonder what it is for at all. But if I cant handle this simple use case in Akka then it is of no use to me and I might as well leave the system using completable futures and Executors and strip out all Akka references. Thats not that hard since I haven't wormed it that far into the system yet. The documentation and promotional materials advertise Akka as solving the problems of concurrent, highly scalable systems. However, if this simple easy use case is not answerable then I question the literature.

-- Robert

[Roland Kuhn]

Hi Robert!

26 aug 2015 kl. 17:40 skrev kraythe <kra...@gmail.com>:

However, this has to be combined with some synchronous calls. Many parts of the system are Asynch but not all. We have users hitting web pages expecting to get certain data. We cant say "thanks for your submission and we will mail you your account information. "

From this I assume that you assume (yes, I know, double-bad, please correct if wrong!) that the web server is implemented as some sort of synchronous routine that needs to present the client response as its return value—but this is not at all the case. If you look at Play (which has been around for quite a while) and the upcoming Akka HTTP then you’ll notice that responses can well be presented in an asynchronous fashion, and that is even the most common mode of operation.

We need to go fetch that information which might involve nested calls to several actors and that might recurse to some of the same actors. So we have a situation where user hits a web page and it invokes ask() on Actor A which has a collection of Data to be gotten from Actor B which in turn needs some info from Actor C. Finally Actor C requires a peice of information from actor A (a different piece of information) and then it is all returned. Now I cant complete process A until all of this is done and in an executor concurrent world that means DEADLOCK if each invocation doesn't have its own executor. 

Again, I must infer from your wording that you assume that using the ask pattern somehow blocks actor A—but that is not at all the case. The ask pattern is a slightly more efficient short-hand for creating a new single-use actor that is used as the sender for the message that is sent to the target ActorRef, so that the eventual reply will be received by that mini-actor which only completes the associated ask-Future and then shuts itself down. There is no blocking in any of this. If you need to transform the results (e.g. by aggregating replies from multiple back-end services) then you do that by using the non-blocking compositional features of Scala’s Futures; non-blocking composition is also used to complete the Future that finally transmits the HTTP response back to the client. Because all of this is non-blocking the class of thread-and-lock deadlock that you are referring to is simply impossible—of course you can still implement logical deadlocks where two actors A and B each wait for the other to make the first step, but that is not a problem in the case we are discussing.

To your rant below I can only say that thousands of production users prove the opposite—and integrating “legacy” components definitely is the norm and not exceptional. I’m currently writing the chapter of Reactive Design Patterns that describes how this is done, so here the hint must suffice that whatever you want to bring in you just wrap inside an Actor and expose a message-driven interface to the Akka-based part of your system.

Regards,

Roland

[kraythe]

Im not attempting to rant, sorry. Im just frustrated at being told what I percieve is, "hey your entire software structure is wrong, rewrite it all." But lets leave that as it is. 

So you are saying if I invoke an ask() inside an actor which invokes an ask() inside another actor that there is no blocking? What if I need ALL of the results of process be for a collection of data to finish A? I.e. A has a collection of IDs and to finish process A, he has to invoke process B on every one of those ids and get the data back. Now I can have process b send it back to A but then they would all arrive as separate distinct messages, not the collection we need to complete process A. This is where the confusion sets in. I CANT finish A without all of the Bs comming back and at some point if a B calls an A actor and the pool of A actors are all busy it would seem to be a problem. You are saying this is not the case? 

class A {
 override  def receive : Receive = {
    case msg: Msg1 =>
     // get domain object.
     // For each id in a collection invoke ask() on actor B
     // when all ask futures complete ... collect results and add to info
     sender ! info
  }
}
class B {
 override  def receive : Receive = {
    case msg: Msg1 =>
     // get domain object.
     // Add some information to result.
     // send ask() to actor A to get some piece of info.
     sender ! info
  }
}

So what am I missing here ? 

[Richard Rodseth]

ask() returns a future. You can do things like use Future.sequence, or for-comprehensions.

That said, in my project we worked pretty hard to avoid using the ask() pattern.

http://techblog.net-a-porter.com/2013/12/ask-tell-and-per-request-actors/

[Adam]

What kind of web container (if any are you using)?

I'd say using ask and blocking on it is still an improvement with pre 3.0 servlets.

With asynchronous servlets you can push the AsyncContext down in a message or use as in the ask future's onComplete callback.

With other frameworks (Netty, Jetty, Akka HTTP, non blocking 3.1 servlets, etc.) things will be even better.

There's just no need to use ask in other places, That's all. In other words it makes perfect sense to use it for the integration point between your existing blocking code and your new Akka based code and slowly change code to or add new asynchronous code.

[kraythe]

Perhaps I am being abnormally stupid but how can you avoid an ask pattern or some other kind of future completion when a user hits a REST endpoint and expects data back in the response? 

-- Robert

[Viktor Klang]

I still have no idea what you mean by "wait" here. Do you mean "block"?
Please either show some code to illustrate or be more specific, it's impossible to answer otherwise and it just ends up being a lot of replying for nothing. :(

--
Cheers,
√

[Richard Rodseth]

Robert, in my case the REST endoint is using Spray. The per-request actor has a reference to the RequestContext, and calls complete() on it, before stopping itself.

I don't have time to check, but it might be modelled on this Activator Template (which I think is referenced in the net-a-porter blog post).

http://www.typesafe.com/activator/template/spray-actor-per-request

[Richard Rodseth]

You could also look at Requester (I haven't)

https://github.com/jducoeur/Requester

[kraythe]

I am using play framework with a mix of scala and java. I can return a promise to the play framework and that promise can call the actor to get the information in a future. No Problem there. The problem is when an actor (A) needs something from another actor (B) and B needs something from A. A sends to a list of Bs and B sends a different message to A and then composes it. Think of generating JSON from a Customer -> Order Situation where Orders can have several different customers. The order JSON needs to have info from the customer in detail but for each of the orders we have to return all customer ids involved in the order. At some point I need to collect the results from all the calls to B and shove them into the result map. I can pass the map in the message to B and have it add the data but then if I have a pool of 5 actors and 5 customers are processing then when it calls back into A to get the customer ids of other customers involved in the order, it needs to use an instance of actor A in the pool to accomplish this. If they are waiting on the completion of a future then that cant happen. 

An example of what I am talking about is below. Please excuse that I coded this cold without compiling but hopefully you get the idea. 

class CustomerActor extends Actor {

    override def receive: Receive = {

      case msg: OrderHistoryJsonMsg =>
        val customer = Customer.findInCacheById(msg.customerId)
        val jsonMap = scala.collection.mutable.Map()
        // init JSON with customer data.
        val selection = Akka.system().actorSelection("/user/orderActors")
        val futures = customer.orderIds.map(orderId => {
          val future = selection ? new OrderJsonMsg(orderId)
          future.onComplete(r => {
            // add data to JSON.
          })
          future
        })
        futures.foreach(f => Await.result(f, 1 seconds))
        sender ! jsonMap
      case msg =>
        unhandled(msg)
    }
}

class OrderActor extends Actor {

  override def receive: Receive = {

    case msg: OrderJsonMsg =>
      val jsonMap = scala.collection.mutable.Map()
      // init JSON with order data.
      val order = Order.findInCacheById(msg.orderId)
      // NOTE HERE WE SEND MESSAGE BACK TO CUSTOMER ACTOR
      val selection = Akka.system().actorSelection("/user/orderActors")
      val futures =  order.customers.foreach(customerId => {
        val future = selection ? new CustomerNameMsg(customerId)
        future.onComplete(r => {
          // add data to JSON.
          sender ! jsonMap
        })
      })
    case msg =>
      unhandled(msg)
  }
}

[Richard Rodseth]

Have you looked at Future.sequence? It turns a List[Future[A]] into a Future[List[A]].

Then you can pipe that result to the same actor or another, or add a complete handler as you have done.

[kraythe]

Either way I have to return just the map to the sender that called with OrderHistory. In fact the sender could be on another node in the cluster so sending the futures is not a viable solution. This is a multiple node environment where the actors MUST run local to where Hazelcast stores the object to avoid heavy serialization overhead. In fact each of the orders themselves might be on another node. 

However the principle question remains, "Will what I posted deadlock?"

-- Robert

[אדם חונן‎]

Compiler or not, the code above has a race condition and I'd bet that it would return an empty map in most cases. The actor is sending the reply without waiting fir the future to end. (It should not deadlock BTW). If anything, I'd pipe the result to the sender.

But a future is not needed at all in this case. Once an actor exits the receive method it's free to process the next messages in it's mailbox. As you haven't written any code that waits, a deadlock is impossible. That said, it is possible to create a deadlock in akka (or any other multithreaded framework). It's just harder, because everything is more explicit and "waiting" is not even one if the primitives supported by actors - you have to implement them.

For this use case you need to define a protocol of messages between the two actors (if they can be remote I'd include handling of dropped messages). Such a protocol can distinguish between the initial request to the customer actor and subsequent requests if that turns out to be needed.

BTW, you should always send immutable data between actors and there's no need for the default case where you call unhandled, if your using the Scala API.

You received this message because you are subscribed to a topic in the Google Groups "Akka User List" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/akka-user/WmK8EXGWKK0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to akka-user+...@googlegroups.com.

[kraythe]

Hmm I don't know why you would think there is a race condition. The code calls an Await() on all the futures from the ASK to Order. Furthermore, the method NEEDS the result to the call to Order to finish the JSON map. How would you do this without a list of futures? Remember all of the data from the Order needs to be in the map before its returned. 

[אדם חונן‎]

Ah... there it is - I missed that little Await there...
Sorry :-)

Anyway, I'd rewrite it differently.

In my eyes you have a few options.

If in order to fulfill a request, you don't need to maintain any state (in your program, not in a DB, or something like Hazelcast), and the request is local, then you can just use a future to begin with.

Otherwise, use an actor.

Even if you do need to maintain state, the question is for how long.

In most cases, at least for parts of the request handling, you can extract logic that only maintains state for the duration of the request.

In such cases an actor per request pattern is very useful (we use it a lot) and what you'll discover is that you often end up with a small (and very simple) FSM.

In your example it's possible to use this pattern:

Actor A receives a request, for order history. It gets a customer from cache and then starts an "actor per request" (Actor B) for the purpose of accumalating all of the customer's orders.

Actor B creates as many actor per request of type C and accumulates their responses.

Each Actor C gets 1 order from cache according to the id it got and asks for the name of the customer of that order from Actor A (which isn't blocked).

Once Actor B is finished it can either reply back to the original requester, or reply to A which will route the reply to the requester - it's up to your messages protocol.

And if that seems too complex, then you're correct.

In the case above, there isn't really any state for any of the above actors, so it's also possible to utilize the new Akka streams and write the above logic much more succinctly (and probably with less bugs...). So this is very similar to using futures, only it's implemented with actors.

There are probably also other "prettier" options.

The point is, though, that it's entirely possible to implement an asynchronous chain of events that can never deadlock.

[kraythe]

Interesting ideas. Actor per request is something i didn't know about. I will research that. That would of course prevent a deadlock because the pool of actors would not be fixed. The question then becomes, in the domain management model I have (basically the actors are managing and extracting info from the domain model) id there a reason to NOT do an actor per request? Also I have to add one complication to the issue. Basically that the Actor has to run on the node that the data is resident on. Hazelcast splits up the data among the nodes and serializing the data (especially for large volumes) is an unscalable solution. I wrote a PartitionAwareRouter that sends the message to the node that holds that part of the map. This has to go to a fixed actor (specifically the actor for the router) on another node. I suppose one solution would be to have a variable size pool that can expand should there be need to do so. I wouldnt know how to go about that though. Essentially the pool would have a collection of x actors pre-started and only send a message to an actor if it is not processing a message and then expand the size of the actor pool if all actors are busy with a temporary actor. I wonder if I would be better off writing that as an actor with routing characteristics rather than a router. Mainly because I don't think I have a lot of control over the pool in a router. 

Any thoughts on that idea ? 

Thanks man. 

-- RObert

[Ryan Tanner]

I'm not quite sure but it sounds like work pulling pattern + cluster sharding is what you're looking for.

[אדם חונן‎]

In our case, we're indeed using work pulling, but each worker encapsulates a single "per request" chain.

But we do this because of our specific requirements of prioritization, back-pressure and gradual service degradation when overloaded.

With simpler requirements I'd likely prefer to use a router which is supposed to be more efficient at, well, routing.

In your case I'd stick with the router for as long as it still felt like a router.

I guess the answer is the ever so annoying "it depends"...

[kraythe]

A variable sized pool could work. Because of the nature of the way our memory cache (Hazelcast) distributes map data and the essential need to run the operations local to that data, its essential we route the message to an actor on the node co-resident with the data, hence the router. Im not usign Akka clusterign but rather Akka remoting where the remote reference is stored in a map in hazelcast. Essentially the router says "is this key on my node? If not figure out what node its on and send it to that node, if so then use whatever local routing logic (round robin, etc) the user wanted in the router. I have never used the akka clustering features. Can you think of an easier way to do this ? 

Also I think I might have to implement a variable sized pool but if so thats fine so long as it is possible. 

Very helpful adam, thanks a bunch. 

-- Robert

[אדם חונן‎]

Does a variable sized pool really have any advantage?

Or maybe you meant the size varies according to number of nodes.

Within the same JVM I would assume that a fixed size pool is good enough as messages can be buffered and back pressure can be applied when the load increases beyond what this pool of workers can handle.

Regarding routing - I'm not familiar enough with Akka's clustering support to give a reliable opinion - I've only played with it for a bit out of interest. We use an internal implementation for that purpose. As a rule of thumb, though, I'd assume it's best to keep the number of components that have to handle clustering logic to a minimum. Each such component has to be managed, so it increases the operational cost. So I'd say that if what you described can handle the various issues that can (or actually will) arise in a distributed system, I'd go for it. It at least sounds simple enough.

[Robert Simmons]

Adam, 

We have some operations that run very quickly and on a limited circumstance and a fixed pool would be no problem. However, we have other operations that take a domain object, find the ids of related objects and do an operation on all of them. Throughout this thread I have been talking Customer and Order as common example, but we are really not processing that kind of data. I am just under some disclosure restraints. But imagine a customer has 200,000 orders and you get the picture. The domain object A is related to hundreds of thousands of Bs. Now I can push them all through a fixed pool but then I risk running out of bandwidth in the pool. You talked about an an actor per request but that becomes problematic when the data is processed on several different nodes. 

The 200,000 ids are located and split up according to where they live in the cluster, in a 4 node system roughly 1/4 will go to each node. Then on that node they will respond to the message. This is important because of the balancing of the memory cache and to avoid unnecessary serialization in high volume situations. Finally, they will all send back responses 

to the caller. To send the messages to the node, I use a router. Its possible there is a way to have a router create actor per request, but I don't know how that could be done. If so that would be fine in a way but not in another way. There is a limited resource also there, the database, which can run only so many transactions at once so when we update all 200,000 objects we can't do it all at once with only 1000 db connections. What we have to do is funnel them through a pool so only so many are running concurrently. The fear I have is a deadlock from something Calling actor A from actor B and thats why I posted this thread. A variable pool, I was thinking, could realize all the actors are busy somehow and create a new set of actors that handle the recursive throughput. This is the problem I am trying to handle and I want to build it in such a way that its kid proof, that a dev with little to no akka experience can't deadlock the system waiting on a pool of a certain number of actors (currently a certain executor) while executing an operation. 

-- Robert

[אדם חונן‎]

So, what we have that resolves this issue, is a fixed pool of worker actors that each start an actor per request and then wait for it to reply or otherwise die (and up to a certain amount of time).

This pool effectively limits the total amount of work that can be done within a single JVM.

If I had to rebuild it today, I'd probably use Akka streams and cut down on lots of written code.

It simply wasn't available when I wrote the code.

As for "Kid proof" - I'm a bit skeptic regarding this.

Asynchronous and distributed execution is more complex than good old synchronous execution and it cannot be completely abstracted out. Not even with Akka.

Moreover, whatever library you use (Akka in this case), other developers will have to understand the underlying concepts.