Low throughput of spray application

[Olga Gorun]

Hi all,

I am working on spray application that is expected to work as a part of real time bidder so both latency and throughput are very important (~80 ms - maximal latency, N*10000 rps are required). But now I experience a very low throughput (less than 1000 rps) on AWS machine of c4.2xlarge type (8 cores CPU, 16GB memory,  ulimit - 500000). As I can see from jvm and system monitoring, it is far from being limited by CPU, memory, disk IO or networking. 

The system contains mainly two pools of actors on request critical path and I don't see that throughput is increased when I increase pools (when pools are increased, I mostly see increased latency). So I think the problem can be in context switching. So I'd be glad to see advises regarding actor dispatchers configuration.

Some more details about the application architecture.  

The system contains two sets of actors on request critical path: pool of ServerActor (extends spray HttpService) and pool of BiddingAgentActor. ServerActor just sends message with ask pattern (route is handled with its future) to BidderAgentActor. BidderAgentActor handles request mostly synchronously.  It uses Futures for work concurrently when possible, but waits for result of final future. No external calls are made in this process (no DB queries, no HTTP calls...). There are several more actors in the application and BiddingAgentActor sends to them messages for offline processing.

Spray and akka configuration:

spray.can {

  server {

    idle-timeout = 5 s

    request-timeout = 85 ms

    timeout-timeout = 5 ms

    pipelining-limit = 5

  }

}

akka {

  loggers = [akka.event.slf4j.Slf4jLogger]

  loglevel = debug

  actor {

    deployment {

      user/bidder-agent-service {

        router = round-robin-pool

        nr-of-instances = 5

      }

      user/bidder-agent-actor {

        router = round-robin-pool

        nr-of-instances = 5

      }

      user/bid-notification-service {

        router = round-robin-pool

        nr-of-instances = 2

      }

      user/wh-events-bidRequest-service {

        router = round-robin-pool

        nr-of-instances = 3

      }

      user/wh-events-adBudgets-service {

        router = round-robin-pool

        nr-of-instances = 3

      }

      user/wh-events-bidDecision-service {

        router = round-robin-pool

        nr-of-instances = 3

      }

      user/wh-events-bidResponse-service {

        router = round-robin-pool

        nr-of-instances = 3

      }

    }

  }

}

Thank you in advance,

Olga Gorun