Performance Comparisons with Scalatra

[Douglas Rapp]

Hi,

I've been doing load tests on various servers, including scalatra. My spray test app is based off of this spray routing example

https://github.com/spray/spray/tree/e54d6708c90dbb5ee4b8d7f94b1bad8c42aeb1a2/examples/spray-routing/on-spray-can

For the "Hello World" endpoints that return immediately the performance is virtually identical, but when I add a DB lookup, the equivalent app using scalatra seems to be 2x faster. I'm wondering if the slowness is inherent to spray, or if we're just doing something wrong, being new to spray and actors.

What's the right way to do something like a DB lookup? I've tried just doing it, executing it in a promise with a threadpool, and also in a separate actor spawned like this

          val actor = context actorOf Props(new DBLookupActor)

I haven't seen any noticeable performance difference with any of those attempts. There's some subtle stuff hidden behind the scenes, just in terms of which actors get created every request, and how much latency that entails that I don't think is documented beyond the source code. Are the main actors for each request in any sort of pool where they can be reused?

The tests were done via ab -n 5000 -c 100. This was on a linux server, so the mac ab problems I was running into earlier shouldn't apply

Spray:
Concurrency Level:      100
Time taken for tests:   11.077 seconds
Complete requests:      5000
Failed requests:        0
Write errors:           0
Non-2xx responses:      5000
Total transferred:      540000 bytes
HTML transferred:       0 bytes
Requests per second:    451.40 [#/sec] (mean)
Time per request:       221.532 [ms] (mean)
Time per request:       2.215 [ms] (mean, across all concurrent requests)
Transfer rate:          47.61 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.6      0       6
Processing:    24  219  34.9    220     434
Waiting:       24  219  34.9    220     434
Total:         29  220  34.8    221     435

Percentage of the requests served within a certain time (ms)
  50%    221
  66%    230
  75%    237
  80%    241
  90%    253
  95%    269
  98%    291
  99%    302
 100%    435 (longest request)

Scalatra:
Concurrency Level:      100
Time taken for tests:   5.625 seconds
Complete requests:      5000
Failed requests:        0
Write errors:           0
Non-2xx responses:      5000
Total transferred:      7300000 bytes
HTML transferred:       6430000 bytes
Requests per second:    888.87 [#/sec] (mean)
Time per request:       112.502 [ms] (mean)
Time per request:       1.125 [ms] (mean, across all concurrent requests)
Transfer rate:          1267.34 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0   18 227.6      0    2998
Processing:    10   94  33.4     88     305
Waiting:       10   93  32.7     88     305
Total:         13  111 230.5     89    3104

Percentage of the requests served within a certain time (ms)
  50%     89
  66%    103
  75%    110
  80%    114
  90%    127
  95%    140
  98%    187
  99%    298
 100%   3104 (longest request)

[Douglas Rapp]

Disregard those test results, I think they were bugged. Here's another example using a longer running db op. Scalatra is still faster, but not by quite so much. The spray example is creating a new actor for every request to do the db lookup. That's the part I'm uncertain about

Spray:
Concurrency Level:      100
Time taken for tests:   58.140 seconds

Complete requests:      5000
Failed requests:        0
Write errors:           0

Total transferred:      780000 bytes
HTML transferred:       110000 bytes
Requests per second:    86.00 [#/sec] (mean)
Time per request:       1162.808 [ms] (mean)
Time per request:       11.628 [ms] (mean, across all concurrent requests)
Transfer rate:          13.10 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max

Connect:        0    1   1.1      0      11
Processing:    49 1153 303.8   1082    2821
Waiting:       49 1153 303.8   1082    2821
Total:         54 1154 303.6   1083    2821

Percentage of the requests served within a certain time (ms)

  50%   1083
  66%   1139
  75%   1188
  80%   1222
  90%   1335
  95%   1446
  98%   2481
  99%   2587
 100%   2821 (longest request)

Scalatra:
Concurrency Level:      100
Time taken for tests:   50.956 seconds

Complete requests:      5000
Failed requests:        0
Write errors:           0

Total transferred:      665000 bytes
HTML transferred:       110000 bytes
Requests per second:    98.12 [#/sec] (mean)
Time per request:       1019.119 [ms] (mean)
Time per request:       10.191 [ms] (mean, across all concurrent requests)
Transfer rate:          12.74 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max

Connect:        0   11 179.4      0    2997
Processing:    21 1000 690.8    763    5908
Waiting:       21 1000 690.8    763    5908
Total:         21 1011 712.3    772    5909

Percentage of the requests served within a certain time (ms)

  50%    772
  66%   1027
  75%   1182
  80%   1473
  90%   1984
  95%   2483
  98%   3201
  99%   3636
 100%   5909 (longest request)

[Mathias]

Doug,

thanks for posting your analysis.
Even though not exactly slow spray-routing on spray-can M7 left quite a bit of performance optimization potential on the table, especially with regard to HTTP header parsing and caching. We are in the middle of getting this cleaned up for M8.
However, at a rate of less than 100 requests/sec HTTP parsing is not anywhere close to becoming a bottleneck.

> Percentage of the requests served within a certain time (ms)

> 50% 772
> 66% 1027
> 75% 1182
> 80% 1473
> 90% 1984
> 95% 2483
> 98% 3201
> 99% 3636

These numbers show quite an unstable system performance which makes your benchmark appear somewhat fishy.
Why would 50% of requests be served in less than half the time of the 80% percentile?

Cheers,
Mathias

---
mat...@spray.io
http://spray.io

> --
> You received this message because you are subscribed to the Google Groups "spray-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to spray-user+...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[Douglas Rapp]

That's why I'm wondering if there's something wrong we're doing with concurrency that could cause the unstability, like creating a new actor for each db operation. The server and db seem fine in the scalatra test, which is why I'm narrowing in on our implementation with spray

[Douglas Rapp]

To answer my own question, the problems seem to have stemmed from trying to use the core ActorSystem to spawn actors to handle long running db requests. Using a regular old threadpool with promises everything worked itself out. Should it be possible to use the ActorSystem built into spray for other things, or should we just leave it alone?

On Tuesday, April 23, 2013 7:05:59 AM UTC-7, pat...@denkselbst.de wrote:

Why would 50% of requests be served in less than half the time of the 80% percentile?

This seems to be the case in the Scalatra data set and not in the Spray data set, though.

Patrick

[Mathias]

Douglas,

spray doesn't have "its own ActorSystem".
It simply puts its own actors inside the one that you create.
As such it is perfectly ok to use the ActorSystem which you have spray attach to also for all kinds of other tasks…

Cheers,
Mathias

---
mat...@spray.io
http://spray.io

[Johannes Rudolph]

To expand a bit more on Mathias' message:

One ActorSystem should be enough in most cases. However, you must be
always cautious when you have actors run blocking tasks like db
requests.

Here's the topic from the akka documentation about this:

http://doc.akka.io/docs/akka/snapshot/general/actor-systems.html#Blocking_Needs_Careful_Management

Johannes

> --
> You received this message because you are subscribed to the Google Groups
> "spray-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to spray-user+...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

--
Johannes

-----------------------------------------------
Johannes Rudolph
http://virtual-void.net