Jedis not as performant as Redis Benchmark

[Abhishek K]

Hi 

I just started using Jedis Client and I have found a large disparity in the latencies observed by set function of Jedis client compared to the latencies reported by the redis-benchmark tool included in Redis distribution.

Ran the test with 128 threads and 

Jedis Client

Average : 9.57 ms

50 % <= 8.32 ms

90 % <= 17.72 ms

99 % <= 58.17 ms

100 % <= 258.81 ms

Redis Benchmark

100000 requests completed in 0.87 seconds

128 parallel clients

500 bytes payload

keep alive: 1

99.98% <= 1 milliseconds

100.00% <= 1 milliseconds

Does any one has observed such disparity before or are these the numbers every expected from Jedis.

Redis Server and client run on the same 4 core 2.4 GHZ machine with 8GB Ram. 

-Abhishek Kona

-Abhishek Kona

[Jonathan Leibiusky]

Hi,
how did you run the test?

some comments:

Jedis is in general a very fast redis client comparing with other clients out there (ruby, php, etc.). I doubt Jedis will ever be as good as redis-benchmark because it is Java and it is in general slower than c (not always, but usually it is) and also something that I noticed in redis-benchmark is that it is not waiting for the response in the socket (at least what I could understand from the code). That changes a lot numbers, if you do the same in Jedis you'll see a huge improvement, of course it doesn't make any sense for a client.

redis-benchmark is more oriented to benchmark Redis server, probably that is why they are not waiting and reading response from socket.

Also I am not sure if when they run these tests, they use pipelines, which will speed up everything a lot.

I am very interested in digging more into this. Maybe there is something wrong in Jedis somewhere, but such a big difference usually tells that the benchmark is not exactly the same on both.

[Abhishek Kona]

On 07/06/11 9:41 PM, Jonathan Leibiusky wrote:

Hi,
how did you run the test?

• I have a common JedisPool initialized and set of threads performing SET operation by borrowing a Jedis instance from the pool and returning it after each operation.
• A Java threadPool executor filled with jobs is started with required no. of core and max threads (128) and time taken by each operation is noted.
• Config : 128 threads in Executor. 10000 opearations performed.
  

If you feel I could be doing something wrong, I can try to share some outline code.

some comments:

Jedis is in general a very fast redis client comparing with other clients out there (ruby, php, etc.). I doubt Jedis will ever be as good as redis-benchmark because it is Java and it is in general slower than c (not always, but usually it is) and also something that I noticed in redis-benchmark is that it is not waiting for the response in the socket (at least what I could understand from the code). That changes a lot numbers, if you do the same in Jedis you'll see a huge improvement, of course it doesn't make any sense for a client.

Also in the IRC of redis, I read that redis-benchmark is used to measure the no. of operations that a Redis instance can handle. It is not used to measure client latencies. But the documentation at http://redis.io/topics/benchmarks is ambiguous.

redis-benchmark is more oriented to benchmark Redis server, probably that is why they are not waiting and reading response from socket.

Also I am not sure if when they run these tests, they use pipelines, which will speed up everything a lot.

I am very interested in digging more into this. Maybe there is something wrong in Jedis somewhere, but such a big difference usually tells that the benchmark is not exactly the same on both.

Most probably the case.
Are there any Jedis benchmarks available (or any tool similar to redis-benchmark) which can be used to compare the expected benchmarks and my code.

Thanks for the quick reply.
-Abhishek Kona

[Jonathan Leibiusky]

and how do you measure redis benchmark with redis-benchmark? with which parameters?

[Abhishek K]

# redis-benchmark -n 100000 - c 128 -d 500
-Abhishek kona

On 08-Jun-2011 3:31 AM, "Jonathan Leibiusky" <iona...@gmail.com> wrote:
> and how do you measure redis benchmark with redis-benchmark? with which
> parameters?
>
> On Tue, Jun 7, 2011 at 2:45 PM, Abhishek Kona <abhish...@gmail.com>wrote:
>
>> On 07/06/11 9:41 PM, Jonathan Leibiusky wrote:
>>
>> Hi,
>> how did you run the test?
>>
>>

>> - I have a common JedisPool initialized and set of threads performing

>> SET operation by borrowing a Jedis instance from the pool and returning it
>> after each operation.

>> - A Java threadPool executor filled with jobs is started with required

>> no. of core and max threads (128) and time taken by each operation is noted.

>> - Config : 128 threads in Executor. 10000 opearations performed.

>>> *
>>> *
>>> *Jedis Client*

>>> Average : 9.57 ms
>>> 50 % <= 8.32 ms
>>> 90 % <= 17.72 ms
>>> 99 % <= 58.17 ms
>>> 100 % <= 258.81 ms
>>>

>>> *Redis Benchmark*

[Jonathan Leibiusky]

Ok, after digging a while...

redis-benchmark won't measure time for the actual parsing of responses, just receiving the response but not parsing it. So there will be difference between redis-benchmark and Jedis.

Also redis-benchmark is not using threads but FD's, where your test with Jedis uses A LOT of threads (128), that means A LOT of thread-context-switches which is probably killing the benchmark.
Also redis-benchmark won't use CPU assigned to Redis, while JVM probably will do.

So I would say that benchmarking this way is somehow unfair.

What happens if you use less threads? With 4 cores, one for Redis, you should probably use something between 10-20 threads? Not sure about that actually.

What kind of operations are you performing?

[Abhishek Kona]

Hi Jonathan,
Thanks for digging a bit deep.
As of now I am testing only the Redis SET operation. I ran the test with different thread count.
Have a look at https://gist.github.com/1016035
( Both the redis and the test were running on My MacBook , so might not be very accurate, will post results on a linux Box).

You can have a  look at https://github.com/sheki/sheki/tree/master/jedis-benchmark for the code.

On 09/06/11 4:37 AM, Jonathan Leibiusky wrote:

Ok, after digging a while...

redis-benchmark won't measure time for the actual parsing of responses, just receiving the response but not parsing it. So there will be difference between redis-benchmark and Jedis.

Also redis-benchmark is not using threads but FD's,

Each Client is a separate process then?

where your test with Jedis uses A LOT of threads (128), that means A LOT of thread-context-switches which is probably killing the benchmark.
Also redis-benchmark won't use CPU assigned to Redis, while JVM probably will do.

So I would say that benchmarking this way is somehow unfair.

What happens if you use less threads? With 4 cores, one for Redis, you should probably use something between 10-20 threads? Not sure about that actually.

I can see the performance degrading with 32 and beyond threads.  I think it was the high thread  count which was skewing my numbers.

What kind of operations are you performing?

Only Set with 500 bytes of Data.

[Jonathan Leibiusky]

Thanks a lot for sending the code. Comments inline.

On Thu, Jun 9, 2011 at 1:14 AM, Abhishek Kona <abhish...@gmail.com> wrote:

Hi Jonathan,
Thanks for digging a bit deep.
As of now I am testing only the Redis SET operation. I ran the test with different thread count.
Have a look at https://gist.github.com/1016035
( Both the redis and the test were running on My MacBook , so might not be very accurate, will post results on a linux Box).

You can have a  look at https://github.com/sheki/sheki/tree/master/jedis-benchmark for the code.

Nice code.
Some considerations. You are setting setOnBorrow=true, that means 1 extra PING command per SET.
https://github.com/sheki/sheki/blob/master/jedis-benchmark/src/main/java/in/sheki/jedis/benchmark/Benchmark.java#L37
I would set testOnBorrow=false and testOnReturn=false for a bechmark, to be as close to redis-benchmark as possible.
Also you are including time spent to get a resource from the pool (which should be very little), but still, for more realistic result it might be better to leave it outside.
https://github.com/sheki/sheki/blob/master/jedis-benchmark/src/main/java/in/sheki/jedis/benchmark/Benchmark.java#L58

You can change this and check the benchmark again. I guess you should see different numbers! :)
 

On 09/06/11 4:37 AM, Jonathan Leibiusky wrote:

Ok, after digging a while...

redis-benchmark won't measure time for the actual parsing of responses, just receiving the response but not parsing it. So there will be difference between redis-benchmark and Jedis.

Also redis-benchmark is not using threads but FD's,

Each Client is a separate process then?

No. It means that redis-benchmark is just one process but opens several connections and use OS poll to manage those connections with FDs instead of threads. That means that context switches are very very fast comparing with thread context switches. But in java we can't actually do a lot regard this.

A way to speed up this would be to use a pipeline. But I would try first changing the testOnBorrow thing first
 

where your test with Jedis uses A LOT of threads (128), that means A LOT of thread-context-switches which is probably killing the benchmark.
Also redis-benchmark won't use CPU assigned to Redis, while JVM probably will do.

So I would say that benchmarking this way is somehow unfair.

What happens if you use less threads? With 4 cores, one for Redis, you should probably use something between 10-20 threads? Not sure about that actually.

I can see the performance degrading with 32 and beyond threads.  I think it was the high thread  count which was skewing my numbers.

Yes, with 4 cores there is no really big gain with lots of threads. It is actually worse. I wouldn't go beyond 10-15 threads probably.
 

[ib84]

Hi Jonathan,

there are lots of posts about performance lately.  You said once, the String <-> Byte Array  interconversion draws lot of performance. I still have some love left for the cast-based String <-> Byte Array converter, it's > 2x as fast as Java's String conversions, and it's one-way compatible (it handles existing byte[] encoded Strings). 

If you say it's feasible without too much brain split, to decouple the "workload strings" from Protocoll-relevant strings ( those that have to be understood by Redis), then I'd give it a try.  I.e. have to 2 different encoders: SafeEncoder and FastEncoder.

ingvar

[Naveen Raj]

When You run the code using pausable executor service you are not guaranteed that 128 threads?clients are active at the same time. I ran your code and checked the number of clients simultaneously. It showed less number of clients running parallely. redis-cli --stat