understand the redis benchmark

[S Ahmed]

I understand benchmarks are usually not the idea and  you should test your actual use case but I want to understand something about the benchmarks.

Say my server gets 50K get's per second.

Does this mean that if I use lua scripting, and my script preforms 5 get operations that my effective throughput is about 10K operations per second?

Or is time saved b/c there is no round trip?

Also, how could I get a benchmark of my lua script with the same client the built-in benchmarks use, or would that mean I would need to use C?

[Josiah Carlson]

You are getting 50k get requests/second because of network round-trip and processing overhead. Performing Redis get calls within a script will let you perform far more queries.

As an example:

josiah@linuxdev ~:$ redis-benchmark -n 1000000 -t get

====== GET ======

  1000000 requests completed in 4.04 seconds

  50 parallel clients

  3 bytes payload

  keep alive: 1

100.00% <= 0 milliseconds

247279.91 requests per second

josiah@linuxdev ~:$ cat test.lua

redis.call('get', 'foo')

redis.call('get', 'foo')

redis.call('get', 'foo')

redis.call('get', 'foo')

redis.call('get', 'foo')

# for the side-effect of loading the script

josiah@linuxdev ~:$ redis-cli --eval test.lua ,

(nil)

josiah@linuxdev ~:$ sha1sum test.lua

deccf55d13cf3eea10cf42169492977f5f5e4679  test.lua

josiah@linuxdev ~:$ redis-benchmark -n 1000000 evalsha deccf55d13cf3eea10cf42169492977f5f5e4679 0

====== evalsha deccf55d13cf3eea10cf42169492977f5f5e4679 0 ======

  1000000 requests completed in 7.00 seconds

  50 parallel clients

  3 bytes payload

  keep alive: 1

99.98% <= 1 milliseconds

99.99% <= 2 milliseconds

100.00% <= 3 milliseconds

100.00% <= 3 milliseconds

142795.95 requests per second

So, at the end I was able to perform roughly 140k lua calls/second, each of which did 5 get calls, for a total of ~700k total get requests performed/second. Compare that with the earlier ~247k get requests/second. Now in practice the performance difference will vary significantly, depending on what you are actually doing. But generally speaking, using Lua scripting will be faster than pipelined or non-pipelined requests.

And to answer your last question about executing the script as part of a redis-benchmark call, please see the different parts (the redis-cli --eval to load the script, the sha1sum to get the hash, and the redis-benchmark call to actually execute the script).

 - Josiah

--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+u...@googlegroups.com.
To post to this group, send email to redi...@googlegroups.com.
Visit this group at http://groups.google.com/group/redis-db.
For more options, visit https://groups.google.com/d/optout.

[Thomas Love]

I was doubtful about your claim that Lua would be faster in general than pipelined requests, so I tried it myself (also with a 5*GET script): 

[herro]$ redis-benchmark -s /tmp/redis.sock -n 1000000 -t GET

====== GET ======

  1000000 requests completed in 3.37 seconds

  50 parallel clients

  3 bytes payload

  keep alive: 1

100.00% <= 0 milliseconds

297176.81 requests per second

[herro]$ redis-benchmark -s /tmp/redis.sock -n 1000000 evalsha 99b8f75657e436246769678af02a4832e93f9753 0

====== evalsha 99b8f75657e436246769678af02a4832e93f9753 0 ======

  1000000 requests completed in 8.59 seconds

  50 parallel clients

  3 bytes payload

  keep alive: 1

100.00% <= 1 milliseconds

100.00% <= 1 milliseconds

116414.43 requests per second

[herro]$ redis-benchmark -s /tmp/redis.sock -n 1000000 -t GET -P 5

====== GET ======

  1000000 requests completed in 0.95 seconds

  50 parallel clients

  3 bytes payload

  keep alive: 1

100.00% <= 0 milliseconds

1054852.25 requests per second

That looks like a significant performance advantage to the pipeline, correct? 

I found (unsurprisingly) that the advantage deteriorated and eventually reversed with increasing script/pipeline length, hitting a ceiling at 1.4 million/sec at -P 40 while evalsha went on to 1.9 million effective commands/sec on a script of 100 GETs. But in practice my "queries" are 2-5 Redis commands in length, pipelined. Also, they're generally a bit more computationally difficult than 5xGET, which would seem to diminish further any advantage in scripting them up. 

Thomas

[Josiah Carlson]

I'm going to tell you something that will surprise and perhaps frustrate you.

Lua being slower than pipelines was a bug, which has been fixed in Redis the most recent 2.8, 3.0, and unstable branches. It was caused by some unnecessary memory churn. You can look through the collection of "Scripting:" related changes on May 7, 2014.

This is what I get when running against a freshly compiled Redis 2.8.9:

josiah@linuxdev ~:$ redis-cli info | grep redis_version

redis_version:2.8.9

josiah@linuxdev ~:$ redis-benchmark -n 1000000 -t get

====== GET ======

  1000000 requests completed in 4.10 seconds

  50 parallel clients

  3 bytes payload

  keep alive: 1

100.00% <= 0 milliseconds

243902.44 requests per second

josiah@linuxdev ~:$ redis-benchmark -n 1000000 -P 5 -t get

====== GET ======

  1000000 requests completed in 0.92 seconds

  50 parallel clients

  3 bytes payload

  keep alive: 1

100.00% <= 0 milliseconds

1089324.62 requests per second

josiah@linuxdev ~:$ redis-benchmark -n 1000000 evalsha deccf55d13cf3eea10cf42169492977f5f5e4679 0

====== evalsha deccf55d13cf3eea10cf42169492977f5f5e4679 0 ======

  1000000 requests completed in 4.89 seconds

  50 parallel clients

  3 bytes payload

  keep alive: 1

99.95% <= 1 milliseconds

100.00% <= 2 milliseconds

100.00% <= 2 milliseconds

204540.81 requests per second

Pulling out my calculator: 5 * 204540.81 = 1022704.05, which is greater than the pipelined request rate, if only by a small amount. ;)

Regards,

 - Josiah

--

[Josiah Carlson]

Or, actually I apparently can't compare numbers. :P

But the rate is *pretty close*. That said, the bandwidth reduction and performance difference should continually get better on the scripting side of things, as long as you aren't only ever performing eval calls.

 - Josiah

[Thomas Love]

I must admit I've been on a bit of an emotional roller coaster here. The ~80% advantage to pipelining seemed a little extreme. But I did willfully ignore the clue -- lower performance on evalsha compared to you after normalizing against -t GET. After a first glance at your follow-up I was a little frustrated, but not surprised. Now I'm just curious to find out what the curves look like for pipelines vs scripts for varying "command queue" lengths and complexities. Quite a ride on the OT side. 

But back to the OP's questions, I would say: 

1. The benchmark measures the command you send, which if it's EVALSHA implies the full (and atomic!) script run, so if you see 50k/sec on a 5-command script, you're seeing Redis do 250k ops per sec. 

2. Scripts *might* save time on round trips, dramatically so when they are long on fast commands, and indeed that is the point. However, between 2 and x commands of this "kind", pipelining *might* save even more time.

Thomas