impressive concurrency benchmark for redis

[Jak Sprats]

i wanted to test how the number of GETs varied as concurrency went up.

I wrote a script that varied the concurrency of GETs from 1-50000, and
as concurrency goes up, it waits until there are only 2000 open tcp
sockets on the machine before going forward w/ the next concurrency
level.

Here is the script: http://allinram.info/redis/concurrency/Concurrency_test_redis.sh
NOTE 1: i became root and did a "ulimit -n 90000" to up the amount of
open file descriptors on both cilent and server
NOTE 2: A.) I commented out all operations except SET in redis-
benchmark and renamed the file redis-benchmark-populate
B.) I commented out all operations except GET in redis-
benchmark and renamed the file redis-benchmark-get

Here is a graph of the concurrency from 1-10 requests
http://allinram.info/redis/concurrency/1-10.png
Concurrency of one is bad, but by 10 concurrent connections we have
already peaked at 107K GET/s

Here is the graph of concurrency from 1-27000 requests
http://allinram.info/redis/concurrency/1-27000.png
This blew my mind, there is minimal performance degradation starting
at 4000 concurrent requests and at 26000 concurrent requests the
performance is 87.6K/s .. unbelievably good

In a 3 tier architecture, if the webserver is something like nginx
(also event driven) and the backend is redis, the system can support
unheard of concurrent connections. I dont think this angle is
mentioned anywhere, but its worth benchmarking and publicizing. This
setup could be benchmarked w/ ruby or php (both work w/ nginx w/
FstCGI) against the standard LAMP or Ruby+LAM and nginx + redis should
destroy the competition.

Question for the community: I messed around w/ a bunch of /proc/sys/
tcp params and tried to fiddle w/ anything that I could think of (open
file descs,etc...), but I could not get the concurrency higher than
27000. Anyone know any tricks here, I was thinking, since I was using
separate machines for client and server I should be able to approach
64K, but no, just 27K ... anyone got any advice?

[Jak Sprats]

retrying w/ hint "ae.h:#define AE_SETSIZE (1024*10) ... redefining it
as 1024*64 let me have our much higher number of clients."
from http://code.google.com/p/redis/issues/detail?id=165

I got this same bug in my tests too http://allinram.info/redis/concurrency/SEGV

On Sep 5, 3:40 pm, Jak Sprats <jakspr...@gmail.com> wrote:
> i wanted to test how the number of GETs varied as concurrency went up.
>
> I wrote a script that varied the concurrency of GETs from 1-50000, and
> as concurrency goes up, it waits until there are only 2000 open tcp
> sockets on the machine before going forward w/ the next concurrency
> level.
>
> Here is the script:http://allinram.info/redis/concurrency/Concurrency_test_redis.sh
> NOTE 1: i became root and did a "ulimit -n 90000" to up the amount of
> open file descriptors on both cilent and server
> NOTE 2: A.) I commented out all operations except SET in redis-
> benchmark and renamed the file redis-benchmark-populate
>               B.) I commented out all operations except GET in redis-
> benchmark and renamed the file redis-benchmark-get
>

> Here is a graph of the concurrency from 1-10 requestshttp://allinram.info/redis/concurrency/1-10.png

> Concurrency of one is bad, but by 10 concurrent connections we have
> already peaked at 107K GET/s
>

> Here is the graph of concurrency from 1-27000 requestshttp://allinram.info/redis/concurrency/1-27000.png

[Jak Sprats]

made it up to 28K concurrent requests with this fix ... no higher ..

On Sep 5, 3:53 pm, Jak Sprats <jakspr...@gmail.com> wrote:
> retrying w/ hint "ae.h:#define AE_SETSIZE (1024*10) ... redefining it
> as 1024*64 let me have our much higher number of clients."

> fromhttp://code.google.com/p/redis/issues/detail?id=165
>
> I got this same bug in my tests toohttp://allinram.info/redis/concurrency/SEGV

[teleo]

What was the hardware used as your clients and servers?

[Pieter Noordhuis]

Hi Jak,

Nice results! I've done some tests using your concurrency script and
found that (with patched AE_SETSIZE) I also hit a wall at 28K
concurrent clients. This is caused by the Linux limit on ephemeral
ports that is set to 32768-61000 (or 28233 ports). So, instead of
hitting the maximum number of open file descriptors, you're hitting
the maximum number of open ports. This can be changed in
/proc/sys/net/ipv4/ip_local_port_range . Changing this range
effectively changes your maximum number of concurrent requests, thus
got me to 50K+ concurrent requests. The latency, however, is huge when
there are this many concurrent clients.

I wasn't able to reproduce the assertion error you got when hitting
28K. I'd like to explore this a little more, because the assertion
error would imply there is something weird going on in this scenario.
My tests were done on a vanilla Ubuntu 10.04 (running in a VM), with
ulimit -n 90000. Can you elaborate a little on the environment, and
check out if the same assertion error also happens inside a VM?

Thanks!
Pieter

> --
> You received this message because you are subscribed to the Google Groups "Redis DB" group.
> To post to this group, send email to redi...@googlegroups.com.
> To unsubscribe from this group, send email to redis-db+u...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/redis-db?hl=en.
>
>

[Salvatore Sanfilippo]

Amazing work Jak, thank you for your hacking explorations :)

Cheers,
Salvatore

> --
> You received this message because you are subscribed to the Google Groups "Redis DB" group.
> To post to this group, send email to redi...@googlegroups.com.
> To unsubscribe from this group, send email to redis-db+u...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/redis-db?hl=en.
>
>

--
Salvatore 'antirez' Sanfilippo
http://invece.org

"We are what we repeatedly do. Excellence, therefore, is not an act,
but a habit." -- Aristotele

[Jak Sprats]

The hardware I have is standard: Standard NIC (A780GM-A motherboard
integrated), Phenom X4 CPU @3.0GHz, 8GB RAM PC3200. Two machines
connected via a standard 1GigE Netgear Switch.
OS: Ubuntu 9.10 64bit.

Pieter, to reproduce the assert() I think you need to turn off the
throttling in my script, so the last two "conc_reqs 2000", change to
"conc_reqs", and i think there are just too many open file descriptors
server side and some strange code path is taken.

Tommorow, I will try my tests using Pieters /proc/sys/next/ipv4
change, and the next day test w/ 2 NICs per box ... would be nice to
have a mean and 80/20 and 99% latency graph on concurrent connections,
also to identify if the bottleneck is OS or NIC related (cuz NICs are
cheap).

Pedro, are you using PHP for your redis + nginx solution. I have read
Hiphopping your PHP and using nginx can be VERY fast.

I thought about the server implementation to best exploit redis' high
concurrency, and the simplest answer was NODE.js. and the next one was
hiphop+php+nginx ... dont know the most concurrent ruby server setup
(ruby event machine, rubinius??). this requires a lot of exploration/
experimentation as well.

If both the webserver and redis can run triggered as network-events on
ALL sides (client<->webserver<->redis) many major bottlenecks (speed,
latency, concurrency, memory usage) should be fundamentally changed.

And its just great news that webserver concurrency is an issue,
because datastore concurrency always bottlenecks first ...

[teleo]

Pieter, would it be possible to document the recommended Linux
configuration for Redis in the Wiki?

Also, it would be useful if AE_SETSIZE were configurable (i.e. not a C
define).

Te.

> > > You received this message because you are subscribed to the Google Groups "Redis DB" group.> > To post to this group, send email tored...@googlegroups.com.> > To unsubscribe from this group, send email toredis-db+...@googlegroups.com.

[Peter Mescalchin]

Jak,

thanks for your work here - those are some mighty impressive
results. I'm thinking if you have a app that requires more than 28K
connections, then sharding would be a wise idea! But it's warm and
fuzzy to know that out of the box Redis has an extremely high
connection ceiling!


Pete

[Pieter Noordhuis]

Hi Jak,

I found the cause of the assertion error and this is now fixed on both
2.0 and master. This means we can go forward with pushing Redis
concurrency to its limits. With the fix applied (and proper ulimit -n,
enlarged ephemeral port range, echo 1 >
/proc/sys/net/ipv4/tcp_tw_recycle), the benchmark tool gets "Cannot
assign requested address" on connect when the kernel has run out of
free ephemeral ports (tcp_tw_recycle helps a bit, but not much). This
leaves the open issue of manually editing AE_SETSIZE, but my guess is
we can leave it this way for a while as 99% of the users don't need
this much concurrent connections.

Eager to see the throughput you get with 50K clients!

Cheers,
Pieter

[teleo]

Hi Pieter,

It is quite possible in production scenarios to reach an unanticipated
traffic peak. In fact, this happens often in successful systems. In
such situations there is often no choice but to try to buy some time
by tweaking the datastore into handling the increased load.

I hope that when Redis has cluster support, there will be other ways
to deal with unanticipated load. But even then, I believe that a
parameter that can help to scale up the system should not be hard-
coded.

Te.

> > You received this message because you are subscribed to the Google Groups "Redis DB" group.> To post to this group, send email tored...@googlegroups.com.> To unsubscribe from this group, send email toredis-db+...@googlegroups.com.

[Jak Sprats]

here is an excel sheet representing latency (2-62000) that has columns
milliseconds, percentage, concurrency
http://allinram.info/redis/concurrency/Conc_Test%20(version%202).xlsb

I dont have any good 3-D graphing, so i used excel's bubble charts,
which are not at all how I want to represent this data, but oh well
Graphs:
http://allinram.info/redis/concurrency/concurrency.html
(remember bubble thickness is request duration in millisecond AND each
graph is relative, check out the bubble size definition at the right).

These look pretty much like how you would expect them to be (below 26K
should be acceptable for everybody), but these graphs hide important
data w/ these overly fat bubbles. It also seems that the 99th
percentile contains the vital info, and this is where SLAs are
defined ... but still this software is at the SLA level of stability
for stuff like this, the graphs look very consistent, no spikes
whatsoever.

What is the best software to do a real 3-D plot of this data ...
Mathematica?

I can do it on win, osx, or ubuntu ... just need a good suggestion ...
or if someone is a whiz at graphing, the link to the excel sheet is up
above, and here is one to text: http://allinram.info/redis/concurrency/3-D.txt
if you are a confirmed linux biggot :)

[Jak Sprats]

and reqs/sec from 2-62K concurrent connections
http://allinram.info/redis/concurrency/req_sec_2-62K.png

On Sep 8, 6:26 am, Jak Sprats <jakspr...@gmail.com> wrote:
> here is an excel sheet representing latency (2-62000) that has columns

> milliseconds, percentage, concurrencyhttp://allinram.info/redis/concurrency/Conc_Test%20(version%202).xlsb

[Dustin S.]

@Jak Sprats:

Regarding extreme performance in conjunction with Nginx: I use Nginx
with the NginxHttpRedis Plugin (http://wiki.nginx.org/NginxHttpRedis)
for caching and it's blazingly fast! If you'd want to get things a
little more dynamic and actually process something as fast as possible
I don't think PHP or Ruby (my personal weapon of choice) will cut it.
I recently discovered the LuaJIT project, which aims to add a just-in-
time compiler to the Lua language. Speed is almost as fast as Java and
blows "standard" Web-App languages out of the water (http://
shootout.alioth.debian.org/u32/benchmark.php?
test=all&lang=luajit&lang2=php).. the cool thing is: There is a Nginx
plugin for Lua (http://github.com/chaoslawful/lua-nginx-module)! I
didn't try it out, as I cannot program in Lua (yet), but I figure that
Nginx + Redis + LuaJIT would yield incredible performance :)

Btw: Redis is really, really awesome and becoming better everyday.. a
big thank you for this great piece of software!

[Jak Sprats]

so i went and made real 3-D graphs, and wrote short analysis...
http://allinram.info/redis/concurrency/test2/graphs.html

it should be noted that this as meant more as a burst test ... bursts
of 100K requests, I do not think that these numbers are sustainable
for very long periods of time.

The numbers are very very good. Most of the graphs look like L's ...
meaning: the only high latency happens right at the very end in the
99.9th percentile.

The graphs tell the story better and yes the webpage is done in 1992
style :)