speeding up IO operations

[Raghava Mutharaju]

Apologies for cross-posting..... (wanted to get general advice)

Hi,

In case of IO intensive applications, what are the general practices to speed it up?

Now for some info on my redis setup and usage.

I only use set operations -- sadd, smembers for reads & writes. Redis is setup over a small cluster and I use ShardedJedis for communicating with all redis instances. I use pipelining wherever possible (i.e. wherever I know that couple of keys go to the same shard). On a 5-node cluster, there are about 400k keys on each node.

I have done a bit of testing. Writes take a lot of time compared to reads and when I use pipelining for writes, it is almost always 4 times faster than without pipelining (this is in the case of 2 nodes with 1 node is reading & writing to the other node).

After that experiment, I thought of having a global pipeline across all writes in the application -- maintain a queue for each shard and put each write operation as an entry in the queue. Now, instead of calling sadd, I would call my own method (say psadd) which would put this call in appropriate queue. After the queue has reasonable number of entries (how many is reasonable?), I would do a pipeline for all the entries.

I found that reads are pretty quick, but would it help to do something similar for reads as well?

I am hoping that this would speed up the application quite a bit. Are there any other commonly used techniques for speeding up IO operations? 

Thank you in advance.

Regards,

Raghava.

[Josiah Carlson]

On Tue, Jun 7, 2011 at 7:51 PM, Raghava Mutharaju
<m.vijay...@gmail.com> wrote:
> Apologies for cross-posting..... (wanted to get general advice)
> Hi,
> In case of IO intensive applications, what are the general practices to
> speed it up?

What you describe doesn't seem terribly IO intensive to me, but you
also haven't given the number of sadd calls you are performing per
second, the number of smembers calls per second, the average size of
your sets, the memory use on each of your shards, etc.

For me, the first thing I ask is "are my operations limited by Redis,
or by something else". If under a pure write scenario you are able to
push 50k-150k operations per second to a single node (and 250k-750k
per second to your cluster of 5 nodes), then the problem is not with
your code, it is because Redis has probably at it's current limit. If
you are sustaining that rate, then there may be small incremental
improvements (depending on what the hardware could sustain). On the
other hand, if you are only able to perform 10k operations per second,
then there are a lot of efficiencies to be gained.

> Now for some info on my redis setup and usage.
> I only use set operations -- sadd, smembers for reads & writes. Redis is
> setup over a small cluster and I use ShardedJedis for communicating with all
> redis instances. I use pipelining wherever possible (i.e. wherever I know
> that couple of keys go to the same shard). On a 5-node cluster, there are
> about 400k keys on each node.
> I have done a bit of testing. Writes take a lot of time compared to reads
> and when I use pipelining for writes, it is almost always 4 times faster
> than without pipelining (this is in the case of 2 nodes with 1 node is
> reading & writing to the other node).

What kind of write throughput are you getting with your pipelined
writes? 10k/second? 20k/second? 50k/second? 100k/second?

> After that experiment, I thought of having a global pipeline across all
> writes in the application -- maintain a queue for each shard and put each
> write operation as an entry in the queue. Now, instead of calling sadd, I
> would call my own method (say psadd) which would put this call in
> appropriate queue. After the queue has reasonable number of entries (how
> many is reasonable?), I would do a pipeline for all the entries.
> I found that reads are pretty quick, but would it help to do something
> similar for reads as well?

What does "pretty quick" mean? Faster than your smaller pipelines that
you had been creating and executing (probably one at a time, waiting
for up to 5 round trips)?

Generally when I am doing automatic pipelining and I don't care about
the results (I just want the commands to be sent), I will typically
batch them up in chunks of 1000-5000. Even with an older non-hiredis
accelerated version of redis-py, I can usually sustain 50k
writes/second without much difficulty with that setup.

In terms of reads, you may be able to do the same and get some
performance, but if your sets are even moderately sized (say 100 or
1000 elements or larger), you may not realize as much of a gain as you
would expect, though it really all depends on the network behavior
between your two machines. Obviously, you should test.

> I am hoping that this would speed up the application quite a bit. Are there
> any other commonly used techniques for speeding up IO operations?

Pipelining, using clients with a good protocol parser/driver (if Jedis
uses hiredis, this is probably the case), making sure to locally cache
stuff when reasonable, ... that's it off the top of my head for
general "this is how you make it better".

Regards,
- Josiah

[Raghava Mutharaju]

Hello Josiah,

Thank you for the reply and for asking about the ops/sec. Previously, I did not check it in terms of reads & writes per sec.

I used a sample program to test the read & write speed. They do not even touch the 10k/second mark that you specified. I generate 500k random keys and do sadd and smembers operations. Each set has 20 elements. Here are the results

For localhost (i.e. redis is on Node1 and the application which reads & writes is also on the same node)

Writes:

without pipeline: 330 secs (throughput = 1515/s

        pipelined: 54 secs (in batches of 1000) (throughput = 500k/54 = 9259/s)

Reads:

without pipeline: 30 secs (throughput = 16k/s)

        pipelined: 32 secs (ok, this is strange) (throughput = 15.6k/s)

For non-local (i.e. redis is on Node1 and the application which reads & writes is on Node2)

Writes:

without pipeline: 22 mins

        pipelined: 60 secs

Reads:

without pipeline: 100 secs

        pipelined: 56 secs

I guess these results are well below the expected throughput. How can I improve it?

I am using Redis 2.0.2 and Jedis 1.5.2.

Regards,

Raghava.

--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To post to this group, send email to redi...@googlegroups.com.
To unsubscribe from this group, send email to redis-db+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/redis-db?hl=en.

[Didier Spezia]

Hi,

I would start by migrating to the last stable Redis (2.2.8) and
Jedis (2-0-0) versions.

Then, consider using redis-benchmark to qualify the platform: this
will
give you an upper bound of the performance level you can expect with
Redis on this platform.

If your pipelined implementation is very far from the result of Redis
benchmark launched with only one connection, then have a look at the
CPU consumption of the Redis process and the client process in order
to identify the bottleneck.

Regards,
Didier.

[Raghava Mutharaju]

Hi,

I am using Redis 2.2.8 now and also ran the redis-benchmark with the previous version of Redis that I had (2.0.8) and with 2.2.8. It still doesn't come anywhere close to the 110k/s mark.

with Redis 2.2.8, default settings of redis-benchmark

====== SET ======

  10000 requests completed in 0.15 seconds

  50 parallel clients

  3 bytes payload

  keep alive: 1

100.00% <= 0 milliseconds

68493.15 requests per second

====== SADD ======

  10000 requests completed in 0.15 seconds

  50 parallel clients

  3 bytes payload

  keep alive: 1

99.90% <= 1 milliseconds

99.95% <= 2 milliseconds

99.99% <= 3 milliseconds

100.00% <= 3 milliseconds

67114.09 requests per second

====== SPOP ======

  10000 requests completed in 0.15 seconds

  50 parallel clients

  3 bytes payload

  keep alive: 1

100.00% <= 0 milliseconds

67114.09 requests per second

With Redis 2.28, 1 client and 10 bytes size

====== SET ======

  10000 requests completed in 0.82 seconds

  1 parallel clients

  10 bytes payload

  keep alive: 1

100.00% <= 0 milliseconds

12121.21 requests per second

====== SADD ======

  10000 requests completed in 0.82 seconds

  1 parallel clients

  10 bytes payload

  keep alive: 1

100.00% <= 1 milliseconds

12224.94 requests per second

====== SPOP ======

  10000 requests completed in 0.81 seconds

  1 parallel clients

  10 bytes payload

  keep alive: 1

100.00% <= 0 milliseconds

12391.57 requests per second

The CPU utilization for redis-server process does go upto 90% in case of 50 parallel clients run of redis-benchmark. I am using the default redis.conf file.

I am running the redis-server on a Linux 8 core processor with 16GB RAM. But there are other processes (like Hadoop) running on it.

Is there any way to improve the throughput further?

In the Readme file, it was mentioned that using tcmalloc is better. Also, I remember reading somewhere in the mailing list that compiling for 32bit (if the underlying platform is 64bit) helps. Would these two changes help in improving the performance?

Thank you.

Regards,

Raghava.

[Didier Spezia]

One way to improve the throughput further if your clients are on the
same node than the server is to use a unix domain socket.
You need to activate it in the redis.conf file, and then you can
run the benchmark with the -s option.

There is something surprising in your results. On my system, the
throughput with 1 connection is half the throughput with 50
connections. On your system, this ratio is different. I cannot
really explain why ...

But IMO, on this system, it is not possible to achieve 110 q/s
with a unique connection targeting a unique Redis instance.

What is the result of grep "model name" /proc/cpuinfo ?
Is it a VM or a physical box?

Regards,
Didier.

[Raghava Mutharaju]

Hello Didier,

Thank you for your guidance :).

The shift to unix domain socket did it -- now the speeds are hovering at the range of 105k/s to 144k/s.

With default settings of redis-benchmark,

====== GET ======

  10000 requests completed in 0.10 seconds

  50 parallel clients

  3 bytes payload

  keep alive: 1

100.00% <= 0 milliseconds

99009.90 requests per second

====== SADD ======

  10000 requests completed in 0.07 seconds

  50 parallel clients

  3 bytes payload

  keep alive: 1

99.42% <= 1 milliseconds

100.00% <= 1 milliseconds

140845.06 requests per second

====== SPOP ======

  10000 requests completed in 0.07 seconds

  50 parallel clients

  3 bytes payload

  keep alive: 1

100.00% <= 0 milliseconds

138888.89 requests per second

With 1 client and 10 bytes, redis-benchmark,

====== GET ======

  10000 requests completed in 0.57 seconds

  1 parallel clients

  10 bytes payload

  keep alive: 1

100.00% <= 0 milliseconds

17636.69 requests per second

====== SADD ======

  10000 requests completed in 0.58 seconds

  1 parallel clients

  10 bytes payload

  keep alive: 1

100.00% <= 1 milliseconds

17361.11 requests per second

====== SPOP ======

  10000 requests completed in 0.57 seconds

  1 parallel clients

  10 bytes payload

  keep alive: 1

100.00% <= 3 milliseconds

17605.63 requests per second

Machine's model name: Quad-Core AMD Opteron(tm) Processor 2376

It is not a VM, it is a physical box.

Couple of questions

1) How can a client (Jedis in my case) connect to the server using unix domain sockets? As far as I remember, Jedis only has an option to specify host & port.

2) Can I use sockets even in a cluster setting? 

3) Are there any optimum number of parallel connections? With a single connection, the performance is pretty low compared to 50 connections?

4) Would use of tcmalloc & compiling redis with 32bit option increase the performance?

Thank you again.

Regards,

Raghava.

[Josiah Carlson]

There may be a secondary redis connection that takes a path to a unix
domain socket (that's what some other libraries do). If one is not
available, then it would seem that Jedis hasn't implemented unix
domain socket support yet.

> 2) Can I use sockets even in a cluster setting?

You mean unix domain sockets? No. You need to use standard TCP/IP
sockets for remote connections, unix domain sockets are local machine
only.

> 3) Are there any optimum number of parallel connections? With a single
> connection, the performance is pretty low compared to 50 connections?

The benchmark application doesn't use pipelining, it uses multiple
connections. To optimize for number of connections, how many to use
depends on your operations, your processor, your network behavior,
etc. You would need to benchmark to find your sweet spot.

> 4) Would use of tcmalloc & compiling redis with 32bit option increase the
> performance?

No.

If you have done as much as you can to pipeline your calls to Redis,
and you are still getting poor performance, then it sounds like Jedis
just isn't all that fast. Could you paste a bit of your code so that
those with experience using Jedis might be able to point out any
potential slowdowns?

Regards,
- Josiah

[Jonathan Leibiusky]

Hi!
Right now Jedis is not using unix sockets, the reason for that is Java. It doesn't support it natively which makes things a little hard, as a we need to add a dependency and that dependency is platform dependent, etc. But I have a branch where I implemented unix sockets and I am just thinking on the best way to include this before actually doing it.

Regard pipelines and Jedis and speed. You should probably try to share that part of your code to see what and how are you using pipelines, etc.

In my machine (a very bad one) I can run around 200k get/sets in a pipeline. And I know of users who could get even more than that.
But maybe there are improvements to make. So it will be awesome to play with your use case and check where is the bottleneck.

Jonathan

[Raghava Mutharaju]

Thank you for the replies.

Until I ran the redis benchmark and couple of my own tests to check the speed of Redis/Jedis, I did not understand the significance of pipelining. After doing that, I got a fair bit of idea how much speed improvement pipelining can offer. Previously, I did use pipelining but it was limited to local pipelining. Did not use it for sharded calls. So, I plan on redesigning my application read/write calls to take advantage of pipelining (I mentioned how I intend to do it in my original post of this thread).

My concern is more towards not being able to replicate the sort of numbers that are mentioned for Redis/Jedis. The numbers I get are not even close.

I only do sadd & smembers calls. For 100k keys, with each key (set) having 20 elements, following are the statistics (with pipeline).

On local node (i.e. test code & server on same machine)

 

write: 5 secs

read: 3 secs

On a different node (i.e. test code & server on different machine)

write: 12 secs

read: 7 secs

The code I used is here -- http://pastebin.com/tdsxtuZ4. You need to give the host name and the total number of keys as input.

@Josaih:

In your last post, you mentioned about caching. I didn't do that as well. For eg, if there is a sharded jedis call to key X, by caching, do you mean that I should store it locally (local redis), so that the next sharded jedis call to the same key can be saved? Do you use any particular library/strategies for caching?

Regards,

Raghava.

[Josiah Carlson]

In terms of caching, I was mostly referring to a local cache *inside*
your application. That would save any/all round-trips for entries if
you are okay with potentially stale data.

- Josiah