Cluster router significantly outperforms remote router

[Jeff]

I am building an actor system that basically has a topology of spray -> business logic actor -> slick. I initially built these using akka remote, but have recently switch over to akka cluster. In testing these systems, I have noticed that round trip calls through the system take half the time using akka cluster as they did using akka remote. Literally, the only thing changed between benchmarks is the actor provider. Why would akka cluster provider perform so much better on identical workloads? I was under the impression that akka cluster built on top of akka remote.

[Endre Varga]

Hi Jeff,

On Thu, Apr 9, 2015 at 7:59 AM, Jeff <jknigh...@gmail.com> wrote:

I am building an actor system that basically has a topology of spray -> business logic actor -> slick. I initially built these using akka remote, but have recently switch over to akka cluster. In testing these systems, I have noticed that round trip calls through the system take half the time using akka cluster as they did using akka remote. Literally, the only thing changed between benchmarks is the actor provider. Why would akka cluster provider perform so much better on identical workloads? I was under the impression that akka cluster built on top of akka remote.

This is indeed strange. ClusterActorRefProvider actually extends RemoteActorRefProvider: https://github.com/akka/akka/blob/8485cd2ebb46d2fba851c41c03e34436e498c005/akka-cluster/src/main/scala/akka/cluster/ClusterActorRefProvider.scala#L26

Do you have a small example that demonstrates this? It would be interesting to investigate what is the source of difference. Maybe deathwatch, I don't know.

-Endre

 

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

[Jeff]

I have been able to isolate the code that is running differently between the two different actor providers. Below is the receive of the offending actor:

    var now: Long = _

  def receive = {

    case UserMessage.LookupOrgsResponse(_, uo) =>

      now = new Date().getTime

      db ! OrgMessage.BatchLookupByIds(uo.map( _.orgId ))

      db ! PersonMessage.BatchLookupById(uo.map( _.personId ))

    case OrgMessage.BatchLookupResponse(os) =>

      println(s"orgs took ${new Date().getTime - now} ms")

      orgs = Some(os); process

    case PersonMessage.BatchLookupByIdResponse(ps) =>

      println(s"persons took ${new Date().getTime - now} ms")

      persons = Some(ps); process

  }

When running with provider = "akka.remote.RemoteActorRefProvider", I get the following times: 

persons took 15 ms

orgs took 57 ms

orgs took 13 ms

persons took 55 ms

orgs took 15 ms

persons took 57 ms

orgs took 19 ms

persons took 61 ms

orgs took 12 ms

persons took 52 ms

persons took 27 ms

orgs took 68 ms

orgs took 14 ms

persons took 55 ms

And when I run it with provider = "akka.cluster.ClusterActorRefProvider" I get: 

orgs took 12 ms

persons took 14 ms

orgs took 15 ms

persons took 20 ms

persons took 13 ms

orgs took 15 ms

orgs took 12 ms

persons took 16 ms

orgs took 10 ms

persons took 12 ms

Again, nothing in the business logic is changing. I am simply changing the provider in the application.conf and making the routers cluster aware. These results are consistent on a warmed jvm. 

[Jeff]

I have created a sample project, but I am not able to reproduce the results when not under load. I'll keep digging

On Thursday, April 9, 2015 at 12:10:38 AM UTC-7, drewhk wrote:

[Endre Varga]

Wow! This is an interesting finding, we need to investigate. 

-Endre

[Akka Team]

I added a ticket: https://github.com/akka/akka/issues/17171

Akka Team

Typesafe - Reactive apps on the JVM

Blog: letitcrash.com
Twitter: @akkateam

[Jeff]

I have updated the ticket with the config I am using.