benchmarking against node, java, jruby

[Wilson MacGyver]

I saw this article. http://maxpert.tumblr.com/post/9677133069/node-on-nails

I was curious how gretty measured up against it.

since he's getting 4k-5k/sec for node.js and jruby. 8-8.5k/sec for pure java.

so I wrote

import org.mbte.gretty.httpserver.*

@GrabResolver(name='gretty',
root='http://groovypp.artifactoryonline.com/groovypp/libs-releases-local')
@Grab('org.mbte.groovypp:gretty:0.4.301')

GrettyServer server = []
server.groovy = [
localAddress: new InetSocketAddress("0.0.0.0", 8080),
defaultHandler: {
response.redirect "/"
},
"/": {
get { response.text = 'Hello World\n' }
}
]
server.start()

ran it on amazon 7.5gb instance. made sure jvm is properly warmed up
using the same
flag he did '-server -XX:+PrintCompilation -XX:+UseConcMarkSweepGC ',
and keep running till
JVM doesn't output anything else.

ab -c 1000 -n 100000 http://localhost:8080/

This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 10000 requests
Completed 20000 requests
Completed 30000 requests
Completed 40000 requests
Completed 50000 requests
Completed 60000 requests
Completed 70000 requests
Completed 80000 requests
Completed 90000 requests
Completed 100000 requests
Finished 100000 requests

Server Software:
Server Hostname: localhost
Server Port: 8080

Document Path: /
Document Length: 12 bytes

Concurrency Level: 1000
Time taken for tests: 25.575 seconds
Complete requests: 100000
Failed requests: 0
Write errors: 0
Total transferred: 9204784 bytes
HTML transferred: 1200624 bytes
Requests per second: 3910.14 [#/sec] (mean)
Time per request: 255.745 [ms] (mean)
Time per request: 0.256 [ms] (mean, across all concurrent requests)
Transfer rate: 351.48 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 205 768.1 24 9027
Processing: 0 33 71.4 28 4806
Waiting: 0 25 70.5 19 4792
Total: 0 238 782.7 55 9072

Percentage of the requests served within a certain time (ms)
50% 55
66% 74
75% 87
80% 94
90% 122
95% 3036
98% 3077
99% 3102
100% 9072 (longest request)

gretty is peaking up at around 4k/sec no matter how many times I run
it, which surprised me a bit.

--
Omnem crede diem tibi diluxisse supremum.

[Alex Tkachman]

I think comparison make sense only on the same environment. So without knowing how pure java app performs on the same amazon machine I don't see a point in deeper investigation. Do I miss something?

[Wilson MacGyver]

sorry, my bad, here is the result from pure java

ab -c 1000 -n 100000 http://localhost:8080/
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 10000 requests
Completed 20000 requests
Completed 30000 requests
Completed 40000 requests
Completed 50000 requests
Completed 60000 requests
Completed 70000 requests
Completed 80000 requests
Completed 90000 requests
Completed 100000 requests
Finished 100000 requests

Server Software:
Server Hostname: localhost
Server Port: 8080

Document Path: /
Document Length: 21 bytes

Concurrency Level: 1000
Time taken for tests: 12.309 seconds

Complete requests: 100000
Failed requests: 0
Write errors: 0

Total transferred: 11800000 bytes
HTML transferred: 2100000 bytes
Requests per second: 8124.41 [#/sec] (mean)
Time per request: 123.086 [ms] (mean)
Time per request: 0.123 [ms] (mean, across all concurrent requests)
Transfer rate: 936.21 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median max

Connect: 0 83 509.8 3 9004
Processing: 0 14 178.4 4 8698
Waiting: 0 13 178.3 3 8696
Total: 4 97 575.6 7 11591

Percentage of the requests served within a certain time (ms)

50% 7
66% 12
75% 16
80% 18
90% 29
95% 39
98% 3015
99% 3036
100% 11591 (longest request)

I didn't mention it because I'm getting 8k+.

Thanks

[Alex Tkachman]

Interesting that in java code he close connection after sending request back despite the fact of reporting keep-alive. I am wondering if that effect performance. Do you have chance to do the same test with higher number of concurrent requests? Does replacing groovy with groovy++ has any performance effect?

[Alex Tkachman]

BTW, I very curious what will happen if to add 5ms delay in processing for Gretty version and his Java version.

[Dave cramer]

So I tried this today and get 

Benchmarking 127.0.0.1 (be patient)

apr_socket_recv: Connection reset by peer (104)

Total of 8618 requests completed

I presume this is due to not having enough sockets or some other system config problem?

Dave

[Wilson MacGyver]

or it could be you are triggering an anti-DoS counter measure in the OS.

[Dave Cramer]

my test on node.js doesn't have this problem ?

Dave Cramer

[Wilson MacGyver]

Hmm, not sure. Since Alex is working on kotlin these days, groovy++ and greety isn't really moving.

I've been playing around with vert.x

It has a good groovy API

http://vertx.io/

[Dave Cramer]

Actually you were exactly correct it was tcp synflood. Have you
benchmarked vert.x ? Gretty didn't do well against node. About the
same for a simple request.

I'll have a look at vert.x

Thanks,

Dave Cramer

[Dave Cramer]

So why would you use this instead of jetty or tomcat ? What is
compelling about it ?

Dave Cramer