Clojure web server benchmarks

[Peter Taoussanis]

Hi all,

Quick post to mention that I've put up some rough benchmarks for a number of Clojure web servers: https://github.com/ptaoussanis/clojure-web-server-benchmarks.

Getting semi-reliable numbers was a real PITA, and I'm sure there's still plenty of room left for improvement (suggestions welcome!). Usual disclaimers apply: the numbers vary by hardware, by OS, by concurrency, and by the response size (among other things). The servers also vary considerably in their feature sets. Not to mention that your web server is seldom going to be the bottleneck when generating dynamic content. Tl;dr: this is all very difficult to generalize: you'll need to bench in your own environment with your own workloads to get an accurate picture.

Anyway I think the numbers are interesting (and pretty impressive all-round). Nginx can almost certainly be tweaked faster (ideas?), but I wouldn't have expected something like the standard Ring Jetty adapter even to be playing in the same ballpark - so that's nice.

I'll try keep this updated as servers are updated. And as I've mentioned on the GitHub page, pull-requests welcome for anything I may have missed, including other servers or mis-configurations, etc. Hopefully with some tweaking we can converge on some reasonably accurate/useful common-case numbers.

Cheers!

[Peter Taoussanis]

A couple folks asked for higher concurrency numbers, so I've bumped the max to 92. Many of the servers can comfortably hit 500+ on my hardware, but others begin acting erratically.

Want to first eliminate the possibility that the trouble is on my end (e.g. OS TCP tuning), then I'll bump the numbers again.

[James Reeves]

This is very interesting. Have you tried running the Ring Jetty adapter with a larger thread pool? It's set lower than the default so as not to overload cloud hosts like Heroku.

- James

--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

[Peter Taoussanis]

This is very interesting. Have you tried running the Ring Jetty adapter with a larger thread pool? It's set lower than the default so as not to overload cloud hosts like Heroku.

You mean the :max-threads? No, I left it at the default (50, if I recall correctly?). I'll try bump it next time I run the benchmarks (probably next week). I'm just on a dual core though, so I wouldn't expect any major changes. As it is, I think the standard Jetty adapter performs remarkably well.

[Peter Taoussanis]

This is very interesting. Have you tried running the Ring Jetty adapter with a larger thread pool? It's set lower than the default so as not to overload cloud hosts like Heroku.

Okay, bumped the :max-threads from 50 to 100 without seeing much change to the results at these relatively low concurrency levels. May well make a difference at higher concurrency &/or with better hardware (both on the cards for later).

Anecdotally, I've seen http-kit happily running dozens of thousands of concurrent connections in a high-load testing environment. Would like to see how far I can push it on a 16 logical core box.

[Dmitry Groshev]

It would be *really* nice to see latencies as well. Some webservers trade latency to throughput, and there is no point in throughput if your service is hardly usable due to 100ms+ responses.

[Peter Taoussanis]

Hi Dmitry,

Raw bench results are provided in the `results` folder: https://github.com/ptaoussanis/clojure-web-server-benchmarks/blob/master/results/20130116-01-25

There you'll find a breakdown of the times to connect, process, and wait. Also the cumulative % of requests served relative to max response time.

I'd consider adding a graph if folks think these numbers are sufficiently interesting?

[Bob Hutchison]

On 2013-01-25, at 6:24 AM, Peter Taoussanis <ptaou...@gmail.com> wrote:

I'd consider adding a graph if folks think these numbers are sufficiently interesting?

That would be great! This is very important information for me. And, unless things have improved markedly over that last year or so (I hope so), I think you'll be 'amused' by what you see.

Cheers,

Bob

[Shantanu Kumar]

On Jan 25, 4:24 pm, Peter Taoussanis <ptaoussa...@gmail.com> wrote:
> Hi Dmitry,
>
> Raw bench results are provided in the `results`

> folder:https://github.com/ptaoussanis/clojure-web-server-benchmarks/blob/mas...

>
> There you'll find a breakdown of the times to connect, process, and wait.
> Also the cumulative % of requests served relative to max response time.
>
> I'd consider adding a graph if folks think these numbers are sufficiently
> interesting?

This is important info and would be great to see plotted. More like a
3D graph of (a) concurrency distribution, (b) throughput distribution,
(c) latency distribution, where you vary the concurrency to get the
other numbers.

Shantanu

[Feng Shen]

Hey, thanks for point out latency, I haven't look at them deeply.  Here is how http-kit compare to Nginx when concurrency is 96:  

Server Software:        http-kit
Server Hostname:        127.0.0.1
Server Port:            8087

Document Path:          /
Document Length:        1163 bytes

Concurrency Level:      96
Time taken for tests:   2.858 seconds
Complete requests:      120000
Failed requests:        0
Write errors:           0
Keep-Alive requests:    120000
Total transferred:      156960000 bytes
HTML transferred:       139560000 bytes
Requests per second:    41981.59 [#/sec] (mean)
Time per request:       2.287 [ms] (mean)
Time per request:       0.024 [ms] (mean, across all concurrent requests)
Transfer rate:          53624.92 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.1      0       4
Processing:     1    2   0.7      2       7
Waiting:        1    2   0.7      2       7
Total:          1    2   0.7      2       7

Percentage of the requests served within a certain time (ms)
  50%      2
  66%      2
  75%      2
  80%      2
  90%      3
  95%      4
  98%      5
  99%      5

 100% 7 (longest request) 

Server Software:        nginx
Server Hostname:        127.0.0.1
Server Port:            8081

Document Path:          /
Document Length:        1163 bytes

Concurrency Level:      96
Time taken for tests:   3.492 seconds
Complete requests:      120000
Failed requests:        0
Write errors:           0
Keep-Alive requests:    118844
Total transferred:      164754220 bytes
HTML transferred:       139560000 bytes
Requests per second:    34365.55 [#/sec] (mean)
Time per request:       2.793 [ms] (mean)
Time per request:       0.029 [ms] (mean, across all concurrent requests)
Transfer rate:          46076.41 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.1      0       4
Processing:     0    3   2.2      2      23
Waiting:        0    3   2.2      2      23
Total:          0    3   2.2      2      23

Percentage of the requests served within a certain time (ms)
  50%      2
  66%      3
  75%      4
  80%      5
  90%      6
  95%      7
  98%      9
  99%      9
 100%     23 (longest request)

btw, I am the author of http-kit:  https://github.com/shenfeng/http-kit . http-kit was written from scratch for performance.  Clojure web application can be very fast, as fast as Nginx can do. Clojure web application can handle high concurrency as Nginx can do too.

[Xfeep Zhang]

If reduce worker_connections from 20000 to 1024 or 128, I think nignx will be the fastest one in this test (the max concurrent level is only 96).

[Peter Taoussanis]

Hi Xfeep,

Thank you, I could never understand what configuration (setting) was wrong. I do not have time to update the project now, but if you give me your GitHub user name - I can add you to the repo?

You can update the tests and/or results if you want to.

Thank you also for your work on nginx-Clojure!

Cheers :-)

--

Peter Taoussanis

[Xfeep Zhang]

In this test there are other problems which maybe unfriendly to Nginx.

(1) Nginx will read the file from DISK  for EVERY request. Again the sendfile is disabled.  Worse！ Although OS will cahed this file but KERNEL -> Nginx Process Space will still cost a lot.

(2) Both Ring-Jetty and Http-kit just write the memory string to every request because the file has been load into Clojure var "response".


See the code : "

(def response {:status 200 :headers {"content-type" "text/html"}
                :body (slurp "resources/index.html")
})

;;;; Handlers

(defn handler             [request] response)

[Xfeep Zhang]

You are welcome.
My Github user name is xfeep.
I'm glad to join the repo. Thanks for your invitation!

[Peter Taoussanis]

You are welcome.
My Github user name is xfeep.
I'm glad to join the repo. Thanks for your invitation!

Okay, great - I have added you. You can make any changes you like. I would be happy if you or someone else wants to maintain (update) the repo.

Cheers :-)

--

Peter Taoussanis

[Xfeep Zhang]

I have committed some update.

• org.clojure/clojure 1.4.0 --> 1.5.1
• compojure 1.1.4 --> 1.1.6
• ring 1.1.6 --> 1.2.1
• aleph 0.3.0-beta13 --> 0.3.0
• http-kit 1.3.0-alpha2 --> 2.1.16
• ring-netty-adapter 0.0.3 --> netty-ring-adapter 0.4.6
• remove testing about pure nginx which generally dosen't service dymanic contents without other modules.
• add nginx with php5-fpm 5.5 testing
• add nginx-clojure 0.1.0 testing
• add 128 clients testing

Cheers :-)

[Peter Taoussanis]

Oh wow, that's fantastic!

For those watching: Xfeep's fixed a number of config issues, updated all the servers, and completely updated the chart.

Absolutely well worth a look if you're interested in Clojure web server performance.

* Results are here: https://github.com/ptaoussanis/clojure-web-server-benchmarks

* His GitHub page: https://github.com/xfeep

* His nginx/clojure lib: https://github.com/xfeep/nginx-clojure

Cheers! :-)

--

Peter Taoussanis

[Shen, Feng]

Hi,  nginx-clojure looks great!

A small tip: ab may not be the best tool, since it's single threaded.   A better tool is wrk:  https://github.com/wg/wrk

You can use wrk to better test nginx-clojure,  It should perform even better than others.

沈锋

http://shenfeng.me

--

--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

---
You received this message because you are subscribed to a topic in the Google Groups "Clojure" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/clojure/UrRLCdex7d4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to clojure+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Xfeep Zhang]

You 're right!

ab has many limitations.

Maybe httpload , wrk, Weighttp or ABs  ( several instances of AB)  are better when test client and server do not run on the same computer.

[Jim Crossley]

I just submitted a PR to include Immutant in the benchmarks, per a request from Chas on Twitter, and then I noticed Shantanu filed an issue to include Undertow, so I created a PR for it, too, since Immutant 2.x will be based on it.

The tl;dr is that Immutant performs a little worse than the :gen-class servlets included in the benchmarks, and a little better than the Jetty Ring Adapter, which is about what I'd expect. Undertow's numbers are just a skosh below http-kit, and can probably even be improved since the adapter's dep is about 20 betas behind the current release.

Jim

[Peter Taoussanis]

Thanks Jim, that's terrific!

For those following, updated results now available: https://github.com/ptaoussanis/clojure-web-server-benchmarks