Redis using too much memory

[David Celis]

Hoping somebody here can help me with this. Firstly I've read the
pages on redis.io about memory optimization and virtual memory but I
still seem to be having some issues.

I am running a small, lightweight Rails application that is currently
leaning heavily on Redis (though it uses MySQL as its primary data
store). The data types I use in Redis are mostly sets and sorted sets
with a few hashes. The only values I have stored in these lists are
integers and floats, which the memory-optimization article said was
good. The Rails app, MySQL and Redis were all running on a basic
Linode (512 MB of RAM) and things were going okay until I reached
about 130 users. That's a very small number of users, so I didn't
think I was going to have any problems. However, there were maybe
between 5 and 10 people accessing the site at once. Redis started to
crash and I assume it's because it was running out of RAM. I can only
assume this, however, because although I set logging to verbose, the
only problem showing up in my logfile was the following one:

WARNING overcommit_memory is set to 0! Background save may fail under
low memory condition. To fix this issue add 'vm.overcommit_memory = 1'
to /etc/sysctl.conf and then reboot or run the command 'sysctl
vm.overcommit_memory=1' for this to take effect.

I upgraded the Linode to have 1024 MB of RAM and Redis was still
sucking up as much RAM as it could, which I assumed was normal.
However, I'm thinking Redis shouldn't have been crashing with that
small number of users. I've known people to run websites with
thousands of users on a small server using Redis without any issues. I
decided to give virtual memory a shot (note that my redis server also
runs in appendonly mode). The following is what my redis.conf file
looks like as of now:

https://gist.github.com/1047923

I've restricted Redis to using 256MB of RAM max and use a swapfile for
the rest. However, running `top` reports that Redis is using about 350
MB of RAM at any given time; 100 MB more than I told it to in my conf
file. Additionally, running `redis-cli info` tells me that Redis is
storing about 250 MB worth of data. Is it normal for Redis to be using
100 MB more RAM than seems necessary?

Sorry if the problem I'm having is a basic one, but I'm not exactly a
sysadmin. I'm not too sure of what I'm doing, but I feel like I really
shouldn't be having these performance issues with such a small rails
app that has so few users. Thanks for reading!

-David

[Didier Spezia]

Hi,

it depends on your data, but 350/256 = 1.36 which is the kind
of ratio you can experience when memory is fragmented, or when
you store a lot of very small objects. This is the overhead
of the memory allocator.

Compiling Redis against tcmalloc or jemalloc may help.

Anyway, virtual box + Redis VM + Rails + MySQL, running on
a memory starved machine, looks optimistic to me. Do not
expect stellar performance.

Regards,
Didier.