Fwd: Dispatcher Configuration For Better Performance:Akka Persistence/DDD/CQRS

[Prakhyat Mallikarjun]

Team,

I working on an app using akka persistence,akka remoting,akka sharding. We are taking DDD/CQRS approach.

Below is the app configuration,

akka {

    loglevel = INFO

  

  actor {

    provider = "akka.cluster.ClusterActorRefProvider"

  }

  

  remote {

    log-remote-lifecycle-events = off

    netty.tcp {

      hostname = "127.0.0.1"

      port = 9088

         }

  }

  

  cluster {

    seed-nodes = [

      "akka.tcp://akka...@127.0.0.1:9088"

         ]

    auto-down-unreachable-after = 20s

  }

  

  persistence {

    journal.plugin = "cassandra-journal"

    

  }

}

We are creating one actor system for entire app. Correct me if I am wrong, the default dispatcher by default uses a fork-join thread pool with at least 8 threads.

We need to support millions of requests from client. How to tune dispatcher of actor system when using akka persistence/sharding/clustering,

1. What is Recommended Type of Dispatcher
   
2. How to calculate threads per thread pool
3. Do we need to consider cores?
4. Do we need to consider type of OS
5. Will JVM configuration matter

[Note:] Suggest any missed parameter in above list. Also how to specify custom dispatcher configuration in app conf.

Basically want to understand criteria's to decide performing dispatcher per actor system. 

Is there any other configuration required to tune akka persistence/cluster/shard kind of application for good performance.

Also recommendations for tuning akka persistence/cluster/shard kind of application would be great help.

-Prakhyat M M

[Akka Team]

Hi Prakhyat,

We are creating one actor system for entire app. Correct me if I am wrong, the default dispatcher by default uses a fork-join thread pool with at least 8 threads.

We need to support millions of requests from client. How to tune dispatcher of actor system when using akka persistence/sharding/clustering,

1. What is Recommended Type of Dispatcher
   

Usually the ForkJoin pool is the best option for most of the workloads, but this is something you need to measure. It is also a very good practice to use separate dispatchers for various parts of the system. Then they might be tuned individually.

1. How to calculate threads per thread pool

This is documented in the configuration file:

        # The parallelism factor is used to determine thread pool size using the
        # following formula: ceil(available processors * factor). Resulting size

        # is then bounded by the parallelism-min and parallelism-max values.

 

1. Do we need to consider cores?

Yes, for example if you have only 8 cores then configuring 100 threads in the pool makes usually no sense. If you have 8 cores then you might also want to have at least 8 threads.

 

1. Do we need to consider type of OS

I don't have any benchmarks of Akka that compares different OSes on the same machine. Also, this is probably workload dependent.

 

1. Will JVM configuration matter

Yes of course. Again, it is not easy to give any particular configuration, these are all use case depedent.

 

[Note:] Suggest any missed parameter in above list. Also how to specify custom dispatcher configuration in app conf.

Basically want to understand criteria's to decide performing dispatcher per actor system. 

Is there any other configuration required to tune akka persistence/cluster/shard kind of application for good performance.

Also recommendations for tuning akka persistence/cluster/shard kind of application would be great help.

Apart from very generic advice we cannot really do much more without detailed information about your application. If you want in-depth expert advice tailored to your needs you might want to look into our consulting offerings: http://typesafe.com/how/consulting

-Endre

 

-Prakhyat M M

--

>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.

For more options, visit https://groups.google.com/d/optout.

--

Akka Team

Typesafe - The software stack for applications that scale

Blog: letitcrash.com
Twitter: @akkateam

[Prakhyat Mallikarjun]

Hi Andre,

Thanks.

For point [1] What do you mean by "It is also a very good practice to use separate dispatchers for various parts of the system."

I have not seen any app with separate dispatchers. Kindly share any article which describes this and app over github would be help.

I have my app using akka persistence/remoting/sharding. 

-Prakhyat M M

[Akka Team]

Hi Prakhyat,

On Thu, Aug 28, 2014 at 11:24 AM, Prakhyat Mallikarjun <prakh...@gmail.com> wrote:

Hi Andre,

Thanks.

For point [1] What do you mean by "It is also a very good practice to use separate dispatchers for various parts of the system."

I have not seen any app with separate dispatchers. Kindly share any article which describes this and app over github would be help.

Akka itself is using separate dispatchers. For example, of configured so, clustering and remoting can run on their own dispatchers, separately from the default dispatcher (so these subsystems do not need to compete with user actors for the same set of threads). Also Akka IO uses dedicated dispathers to handle IO (again, configurable).

This is a somewhat old article about dispatchers: http://letitcrash.com/post/40755146949/tuning-dispatchers-in-akka-applications

Also I believe most of the Akka books discuss dispatchers and their importance.

-Endre

[Prakhyat Mallikarjun]

Hi Andre,

Thanks. I looked into the post.

If we use multiple dispatchers and corresponding thread pools. How these dispatchers will share cores in one machine.

Consider 3 dispatchers with 8 threads each on a dual core machine.

-Prakhyat M M 

[Akka Team]

Hi Prakhyat,

Assuming the OS scheduler is fair, if all threads are busy then all the computational resources will be distributed roughly equally.

Consider 3 dispatchers with 8 threads each on a dual core machine.

I am not sure what you mean by each on a dual core machine. Each of the actor systems?

Anyway, taking a more interesting example:

There are:

 - 8 cores

 - 2 dispatchers

 - one dispatcher having 12 threads hosting 100 actors

 - second dispatcher having 4 threads hosting 1000 actors

If one core is 100% then we have 800% available on the machine. There are 16 threads in total, each will get roughly 50% (800/16). This means that the first dispatcher will get roughly 12*50% = 600% while the second will get 50%*4 = 200%. Of course this is only true if both dispatchers have active actors, for example if the second one is idle, then the first one will get all the 800%.

Also, the first dispatcher hosts 100 actors, so you will get roughly 6% for each actor (if all actors are busy) and for the second dispatcher you will get 0.2% for each actor (if all actors are busy).

Of course if not every actor is busy (very common) then the active ones will get a much larger share.

-Endre