perfomance issue with spray-can server

[yossi l]

Hi 

we have developed an application that receives about 20k requests per second ( with keep alive )

on top we see that java process uses 2000+ % cpu

we see instansly with visualvm cpu sampling scala.concurrent.forkjoin.ForkJoinPool.scan() uses 70-80% cpu time 

when we profile only java packages it shows sun.misc.Unsafe.park[native]

this is 24 cores and 400G server

We are using spray-can 1.3.1, and akka 2.3.6. and didn't changed the default configuration of spray

We would to know how to find the reason for such CPU hogging and recommendation for how to increase the throughput of spray

Thank you

[Mathias Doenitz]

Yossi,

since you don’t tell us nothing about what you application is doing or what its architecture looks like it’s very hard to give you any optimisation advice.

Cheers,
Mathias


---
mat...@spray.io
http://spray.io

> --
> You received this message because you are subscribed to the Google Groups "spray.io User List" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to spray-user+...@googlegroups.com.
> Visit this group at http://groups.google.com/group/spray-user.
> To view this discussion on the web visit https://groups.google.com/d/msgid/spray-user/6d198e83-c69a-4cca-972d-95db4011b74d%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[yossi l]

Thanks for you response Mathias

Our application should be able to receive many json POST requests, parse them and answer quickly (under 50 ms) with a processed response (using keep-alive).
In order to process the request we are using external services such memcached and redis server.

We are seeing low response time (around 20 ms) but are facing this CPU problem...... 


object Server extends scala.App {

  implicit val system = ActorSystem()

  val myListener = system.actorOf(Props(classOf[RequestHandlerActor], cfgActor))

  val port = system.settings.config.getInt("8080")

  IO(Http) ! Http.Bind(myListener, interface = "0.0.0.0", port = port, backlog = 2048)
}

class RequestHandlerActor(cfgActor: ActorRef) extends Actor{

  override def receive: Receive = {
    case _: Http.Connected =>
      val workerActor = context.actorOf(Props(classOf[GeneralWorkerActor], cfgActor))
      sender ! Http.Register(workerActor)
    case a: Any =>
      println("default " + a)

[Mathias Doenitz]

Yossi,

50ms per request is quite a “long time” given your performance expectations.
With a throughput of 20K requests per sec you will have about 1000 concurrent requests in flight at any given time.
Depending on how many total actor messages need to be sent and received during the processing path of a single request this might already present some serious load on your system.
Especially, if, additionally, you want/need to do inter-process communication for every request (talking to redis or memcached).

Without knowing anything about your backend it seems to me that you are experiencing a scalability bottleneck introduced by your application architecture, not by anything that’s directly spray- or akka related.

Cheers,
Mathias

---
mat...@spray.io
http://spray.io

> To view this discussion on the web visit https://groups.google.com/d/msgid/spray-user/5a1237f5-334e-42af-958f-d07d93231be0%40googlegroups.com.

[Mariot Chauvin]

Hi Yossi,

May you try to upgrade to akka 2.3.9 (binary compatible) and run against your benchmark to see if you see any improvements?

Cheers,

Mariot

To view this discussion on the web visit https://groups.google.com/d/msgid/spray-user/D7FB4DFB-F238-45A8-B7D3-986CE25B3664%40spray.io.

For more options, visit https://groups.google.com/d/optout.

---

Visit theguardian.com. On your mobile and tablet, download the Guardian iPhone and Android apps theguardian.com/guardianapp and our tablet editions theguardian.com/editions.  Save up to 57% by subscribing to the Guardian and Observer - choose the papers you want and get full digital access.  Visit subscribe.theguardian.com

This e-mail and all attachments are confidential and may also be privileged. If you are not the named recipient, please notify the sender and delete the e-mail and all attachments immediately. Do not disclose the contents to another person. You may not use the information for any purpose, or store, or copy, it in any way.  Guardian News & Media Limited is not liable for any computer viruses or other material transmitted with or as part of this e-mail. You should employ virus checking software.

 

Guardian News & Media Limited is a member of Guardian Media Group plc. Registered Office: PO Box 68164, Kings Place, 90 York Way, London, N1P 2AP.  Registered in England Number 908396

[Johannes Rudolph]

Hi,

On Mon, Mar 9, 2015 at 10:34 AM, yossi l <ma...@pay-per-leads.com> wrote:
> scala.concurrent.forkjoin.ForkJoinPool.scan() uses 70-80% cpu time

This usually means that the ForkJoinPool is over-provisioned with
threads that then start battling for work. For some reason Akka's
default is to allocate 3 times the numbers of cores (hyper-threads?)
the machine reports as the default size of the thread pool (see the
`akka.actor.default-dispatcher.fork-join-executor.parallelism-factor`
setting). Usually, setting this number to 1 gives better performance
if your code doesn't contain any blocking calls and the JVM can rely
on an accurate number of cores. This and the `parallelism-max` would
be the first settings I would experiment with.

> when we profile only java packages it shows sun.misc.Unsafe.park[native]

`sun.misc.Unsafe.park` means the threads are sleeping so it shouldn't
turn up in the profiles but may be if the profiler isn't advanced
enough.

Btw. searching for "parallelism-factor" in spray-user or akka-user
gives other previous discussion on the topic.

HTH
Johannes

--
Johannes

-----------------------------------------------
Johannes Rudolph
http://virtual-void.net

[yossi l]

thanks alot for all responses 

i will check these settings and will see if it helps