Akka using 99% of CPU, netty 1%

[Franz Bettag]

Good Night everyone,

we are using akka heavily in our newest product which spawns an actor for every comet/long-polling user (currently 4k users/actors, peak 10k).

Using netty we accept the requests and send them to a ManagementActor which sends the message to the appropriate User-Actor.

Everything works fine, but after upgrading to Netty 4alpha5 and Akka (from 2.0.1 to 2.0.3) we noticed some heavy differences. Initially we thought that it would be netty, but we ran some fairly simple YourKit analysis today and found that netty actually uses 1% of all time and akka 99%.

Since we couldn't use YourKit analysis on the production box with 4k clients, i am not sure if these results are correct. Once we start the process and 1k users connect in one second, it jumps to 400% cpu eating everything - so yourkit would just kill it. It worked like a charm before and we are currently trying to track down the issue as hard as anyone can imagine. :)

I also ready about ForkJoinPool which destroys a thread if it has gone for 5 seconds without interaction, but i have no idea if this is an issue here.

The worst part is that it's closed source (for now), so i cannot share the source freely.

Seriously *any* help is appreciated.

[√iktor Ҡlang]

What happens if you configure to use a thread-pool-executor instead of FJ?

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://akka.io/faq/
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To post to this group, send email to akka...@googlegroups.com.
To unsubscribe from this group, send email to akka-user+...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user?hl=en.
 
 

--
Viktor Klang

Akka Tech Lead

Typesafe - The software stack for applications that scale

Twitter: @viktorklang

[Jeremy Pierre]

We did something similar for our comet stuff over here alongside
websockets - outside of peak time we average at least 6-7k comet
connections, peak time is easily double to triple. A few questions:

1. Lots of future/promise stuff(ask, etc)? If so, maybe you're just
not giving the Akka dispatcher enough leeway (assuming you're using
the default one only)
2. Why send them through a management actor? We simply look up
remote actor's addresses in memcache and use remoting to get the
newest updates for a client.
3. Executors for Netty vs Akka?

I see Viktor's weighed in as well along the lines of #3 ;)

Jeremy

[√iktor Ҡlang]

Could also be a JVM issue, are you running on a 1.7 JRE? If so either switch to the last 1.6 JRE OR the last 1.7

[Franz Bettag]

java version "1.7.0_07"

Java(TM) SE Runtime Environment (build 1.7.0_07-b10)

Java HotSpot(TM) 64-Bit Server VM (build 23.3-b01, mixed mode)

I'll try to update to a newer oracle if possible. Is IceTea an option?

[Franz Bettag]

1. Yes, lots of futures and ask :)

2. Well we currently only need a single quad core for our 10k users (with the old version it used ~60% cpu @ 2G heap)

3. Netty uses Thread and Akka FJ

I already have some new results which indicate an issue with netty, but in any case i'll keep you posted.

thanks for the suggestions, we'll have a look into them first since akka does prolly 80% of the work. :)

[√iktor Ҡlang]

You on OSX? If so, I definitely recommend staying with Java 6, there are some nasty concurrency and NIO issues with JDK7

[√iktor Ҡlang]

On Thu, Nov 8, 2012 at 12:34 AM, Franz Bettag <fr...@bett.ag> wrote:

1. Yes, lots of futures and ask :)

2. Well we currently only need a single quad core for our 10k users (with the old version it used ~60% cpu @ 2G heap)

3. Netty uses Thread and Akka FJ

I'd try switching Akka over to use thread-pool-executor to see if it could be some interaction between the JVM and FJ.

[Alec Zorab]

Hopefully a silly question, but your actors aren't all top level are they?

[√iktor Ҡlang]

Of course not, that would go against all recommendations ;-)

[Johannes Rudolph]

Hi,

we see a similar issue in spray-io. The basic setup is a common work-load where request arriving from the outside are handled first by per-connection handler actors which send the request to a singleton service and then for a response to send back to the outside.

I extracted the basic pattern into a simple ping-pong example [1] where 1000 actors send a "ping" message to a singleton service actor which just answers with "pong". This is done in a loop thousands of times. Alternatively, I tried scheduling the external stimulus but results didn't change much.

I tested the setup with three different dispatcher configurations:

fork-join-default {

}

fork-join-1 {
    fork-join-executor {
        parallelism-min = 1
        parallelism-max = 1
    }
}

thread-pool-dispatcher {
    executor = "thread-pool-executor"
}

The results [2] are surprising:

fork-join-default (akka default config):
Total time elapsed: 14s
avg RTT for ping/pong: ~ 600µs
CPUs utilized: 3.3
branch-misses: 685 millions

fork-join-1 (fork-join pool with just 1 thread):
Total time elapsed: 12.7s
avg RTT for ping/pong: ~ 950µs
CPUs utilized: 1.185
branch-misses: 135 millions

(thread-pool-dispatcher results omitted, something is wrong here because many threads are spawned but only one thread is used all the time)

So, using a fork-join-pool with just one thread is not only faster but uses only 1/3 of the total CPU-time compared to the default setting. The RTT, in contrast, is better with the default settings.

This case is somewhat similar to a recent case discussed on the scala-user mailing list. In the recent case a completely serial process was implemented using akka and showed a similar behavior as here. Roland then analyzed that using just one thread speeds up the processing much. Our case here is similar because if you just look at only one actor making requests it follows a completely serial work path as well. However, there is an important difference: there are 1000 actors doing work in parallel so there should be never too little work to do.

Maybe Roland's explanation holds for this case equally but I tried to analyze the problem further.

I profiled where most of the time is spent and it looks roughly like that [2]:

Events: 48K cycles                                                                            
 31.70%  perf-24003.map                 [.] Lakka/jsr166y/ForkJoinPool;.scan
 11.28%  perf-24003.map                 [.] Lakka/dispatch/Dispatcher;.dispatch
  9.60%  perf-24003.map                 [.] 0x7f4325006187
  4.79%  perf-24003.map                 [.] Lakka/actor/ActorCell;.invoke
  4.40%  perf-24003.map                 [.] Lakka/jsr166y/ForkJoinPool;.execute

As the default settings produces a huge number of branch misses I checked where the branch-misses come from:

Events: 49K branch-misses                                                                      
 73.19%  perf-24756.map                 [.] Lakka/jsr166y/ForkJoinPool;.scan
  5.23%  perf-24756.map                 [.] 0x7fc6e3d89fa7
  4.04%  perf-24756.map                 [.] Lakka/dispatch/ForkJoinExecutorConfigurator$Mailbox
 
It seems to be coming from a particular `if` statement inside the `scan` method [4] which seems to have very bad branch prediction. I can only speculate if you could speed up things at this point by rewriting the conditions of the `if`-statement in some way or if the bad predictability comes out of the algorithm used (e.g. randomization seems to be part of the algorithm).

To reproduce use [1] and run `sbt assembly` and then use the command-lines as seen in [2]. I let it run on my machine a core i5 M560, 64bit linux ubuntu 12.04, kernel 3.2.0-32-generic

java -version
java version "1.7.0_09"
OpenJDK Runtime Environment (IcedTea7 2.3.3) (7u9-2.3.3-0ubuntu1~12.04.1)
OpenJDK 64-Bit Server VM (build 23.2-b09, mixed mode)

I checked on another debian machine and with an oracle java 6 version as well with similar results.

To get Java method information in `perf` you need a java agent which isn't currently published anywhere but which I can provide if someone is interested.

[1] https://gist.github.com/4119067
[2] https://gist.github.com/4125537
[3] https://groups.google.com/d/msg/scala-user/S_blkFo7KNs/9r4hThn2j_UJ
[4] https://github.com/akka/akka/blob/v2.0.4/akka-actor/src/main/java/akka/jsr166y/ForkJoinPool.java#L1533


--
Johannes

-----------------------------------------------
Johannes Rudolph
http://virtual-void.net