Using Redis as a disk-backed cache

[VC]

Hi all --

First off, Redis is an amazing tool! We've used it fairly extensively
for many different things in our company, and have very happy with the
performance and results. So thanks for all of the effort and
development!

We recently switched our caching store from Memcached to Redis. The
primary reason we did this was to take advantage of Redis' VM feature
to allow some of the lesser-accessed values to be swapped to disk,
boosting the overall size of cache beyond what could be stored in
memory. We have found this to be true; indeed, we are able to store
many more values in our Redis-backed cache then we have been able to
store in a Memcache-only cluster, where everything must be stored in
memory. So on this front we are quite happy.

However, one thing we have noticed is that read performance out of the
cache can be quite erratic and unpredictable. Most of the read
requests finish very quickly (~1ms) but a large number of them finish
orders of magnitude more slowly (10ms - 1s). Even worse, there seem to
be times when many such requests will be slow, leading to degradation
in our site performance (see graph):

https://skitch.com/vincentchu/ri12f/gnuplot

As background, we have a Redis cluster of two boxes, each with 16 GB
of RAM. Each box runs one redis-instance (v2.2.2) with the following
configuration file:

https://gist.github.com/877761

We do an average of around 500 reads / second out of the cluster, with
peaks of around 800-900 reads / sec. Thus each Redis instance will see
around 400-500 reads / sec MAX. We are using these redis instances as
a pure key => value stores, with simple string values of around 1-4k
in size. Each box has around 250 connected clients. One thing I
noticed is that once the redis boxes began to fill with keys is that
both boxes became very I/O bound. Disk utilization shot up to near
100% and has not gone down since. Moreover, memory usage has shot up
to the full 16GB. I've even seen the box start to swap a bit, though
this has remained pretty stable. Our read patterns are pretty long-
tail, so a subset of keys are accessed very heavily.

Using redis-tools, as recommended by http://redis.io/topics/virtual-memory,
to view how many keys are going in / out of swap. Most of the time,
there is no "swap-out" activity. But every once in awhile, I see the
following:

https://gist.github.com/877771

In any case, would be great if I could get some hints as to how best
to configure our Redis instances to maximize read rate and minimize
variability in read throughput:

* Basic question: Am I even using the right redis configuration? Data
durability / recovery would be nice, but it's not hugely important for
us since we just want to use Redis as a cache that can use disk to
boost the number of keys that it is able to store.

* Should we increase the size of the swap that redis is allowed to
use? In our current setup, vm-max-memory is set to 3221225472 (3GB).
If we do increase this, will this force Redis' aggregate memory usage
to exceed the 16 GB of RAM we have on each box, causing it to swap?
Redis is currently using all of the RAM on the box.

* Am I bumping into the maximum performance that I could expect? If
so, purchasing a bigger box (either more RAM, or just adding more
boxes to the cluster) is prudent?

* Compressing string values on the fly? One thing we've thought about
is compressing the strings on their way in/out of the cluster. This
would of course mean more work for our app servers, but this might
also lower the amount of I/O coming in and out of the disk, and allow
Redis to cache more in memory.

Thanks again, and hats off to the Redis team for all of the hard work.
Any help or hints would be much appreciated!

Cheers,

Vince

[Didier Spezia]

Hi Vince,

I do not like much the VM feature in Redis, so the following
advices may not be pertinent. I tend to think it has
its usage on dev/test systems, but not in production.

Regarding your configuration, you have a 64 GB swap file for
a Redis swap memory of only 3 GB. Your boxes have 16 GB of
physical RAM.

>> Should we increase the size of the swap that redis is
allowed to use?

I would try this yes, but you need to keep in mind that the
keys are never swapped out (only the values are). What is the
average size of your keys? How many items do you have in Redis?
I would put at least 8 GB for Redis. Anyway with such a ratio
between memory and swap, do not expect miracles ...

Also think about the double buffering. Redis is using normal
filesystem operations to deal with its swap file. So the swap
file is cached by the filesystem. Supposing you have only Redis
running on these boxes, the filesystem cache should be about:
16 GB - size of Redis in memory - 512 MB

Is the Redis process really using all the RAM of the box?
What is the resident set size of Redis in memory?
It could be a good idea to post a top display and also the
result of INFO command.

What is the size of your filesystem cache? (free command on Linux),
and which OS do you use btw?

>> Am I bumping into the maximum performance that I could expect?

Hard to tell. I would try to optimize it a bit further though.
Here are a few ideas (assuming you are running a Linux 64 bits
system).

- set the swappiness kernel parameter to 0
(to reduce system swap activity)

- replace the unique 64 bits instance of Redis by a number of
32 bits instances. You will save some memory, and will better
use your CPUs.

- use the AOF for persistency rather than snapshotting.

- try other values for the page size.

- rather than adding more RAM or more machines, you also have
the possibility to use SSD hardware to store the Redis swap file.
Consumer-grade SSD hardware is getting affordable these days ...
You need to pay attention to endurance problems though.

>> Compressing string values on the fly?

On-the-fly compression using LZO, LZF or Quicklz is usually cheap,
and generally a win. Quicklz is not widely known, but it is one
killer library - see http://www.quicklz.com

You loose the capability to use redis-client to perform
quick investigations on your data though.

If you use compression, the page size will have to be assessed
again.

Hope this helps ...

Regards,
Didier.

> Using redis-tools, as recommended byhttp://redis.io/topics/virtual-memory,

[Pieter Noordhuis]

Hi Vince,

The performance issues you have are concerned with the VM. Consider
the scenario where you have 100 clients accessing swapped out keys.
This means there are 100 swap-ins being done concurrently, causing 100
random read ops against your disk. If you're not using SSD's, this can
be pretty slow. Reads against keys in memory will be served as quickly
as if the entire set was in memory, so I'm pretty confident that
something like this is causing the increased latency from time to
time.

Another thing: you have a 64G swap file, with a page size of 512 bytes
and the redis-stat output show 10M used pages. This amounts to about
5G on-disk swap. Did you try disabling the VM to see how far these
boxes could take you? I noticed you configured Redis to use the
allkeys-lru eviction policy (which does not make sense without a
"maxmemory" directive), so you can remove values as needed. If you
configure maxmemory to be something like 12G (or 80% of the RAM in a
box, so there is some slack for fragmentation without going into OS
swap) and disable the VM, you could check out how many keys it can
store without going to disk.

With values of 1k-4k a little compression on the app-level couldn't
hurt as well (if the values can be compressed by a significant factor,
>50% or so).

Cheers,
Pieter

> --
> You received this message because you are subscribed to the Google Groups "Redis DB" group.
> To post to this group, send email to redi...@googlegroups.com.
> To unsubscribe from this group, send email to redis-db+u...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/redis-db?hl=en.
>
>