redis in production environment

[Oleg Ruchovets]

Hello.

   We are planning to go production in a couple of months.

Can community share best practices for configuration of Redis for production environment.

On what we should especially pay attention?

Thanks in advance.

Oleg.  

[Matt Palmer]

On Mon, Nov 04, 2013 at 09:33:03PM -0800, Oleg Ruchovets wrote:
> We are planning to go production in a couple of months.
> Can community share best practices for configuration of Redis for
> production environment.
> On what we should especially pay attention?

AOF configuration. Backups. Maxmemory. Load testing. Replication.
Scalability. Monitoring. Documentation. Disaster recovery. Load
shedding.

Anyone else got anything more?

- Matt

--
Generally the folk who love the environment in vague, frilly ways are at
odds with folk who love the environment next to the mashed potatoes.
-- Anthony de Boer, in a place that does not exist

[Oleg Ruchovets]

Great , Thank you.

   Is there any links or blogs which describes the procedure.

Thanks

Oleg.

[Dvir Volk]

One other thing - for an EC2 environment beware AOF, consider avoiding EBS and beware anything other than M3 instances.

--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+u...@googlegroups.com.
To post to this group, send email to redi...@googlegroups.com.
Visit this group at http://groups.google.com/group/redis-db.
For more options, visit https://groups.google.com/groups/opt_out.

--

Dvir Volk

Chief Architect, Everything.me

http://everything.me 

[Greg Andrews]

Monitor, monitor, monitor.  Put all aspects of the Redis machine's operation into graphs - cpu consumption, ram, disk space, disk performance, network traffic levels.  Put important Redis operations into graphs (commands processed per minute, connections accepted per minute, average and peak ram usage levels).  When something goes wrong, you'll need these graphs to understand what happened and prevent it from happening again.

Also, create fault monitoring that will tell you if your Redis processes are running, accepting connections, responding to commands, and that slaves are properly replicating from their masters.

As for the operating environment, one rule of thumb is to give a Redis instance 2-3x the amount of ram needed to hold its full data set, so saving its data to disk will go smoothly, even when the data set is changing rapidly (lots of writes).  Another rule of thumb is to give Redis 2-4 cpu cores (1 for the main command processing, 1-2 for network i/o with clients, 1 for the child process that saves data to disk).

Redis runs at the speed of ram, which is substantially faster than SSD or hard drives (see this previous post of mine), so Redis can saturate your storage subsystem's i/o as it saves a large data set to disk.  If the other processes on the server are also trying to write to the saturated disk, their performance can suffer.  Plan where, when, and how Redis saves to disk accordingly.  (or monitor the disk performance so you can move Redis to its own disk when its data set gets large enough to hurt the performance of other processes)

Monitor, monitor, monitor.

  -Greg

--

[Oleg Ruchovets]

Thanks Greg for useful information.

I also added couple of questions inline.

Thanks

Oleg.

     

On Tuesday, November 5, 2013 11:52:34 PM UTC+8, GregA wrote:

Monitor, monitor, monitor.  Put all aspects of the Redis machine's operation into graphs - cpu consumption, ram, disk space, disk performance, network traffic levels.  Put important Redis operations into graphs (commands processed per minute, connections accepted per minute, average and peak ram usage levels).  When something goes wrong, you'll need these graphs to understand what happened and prevent it from happening again.

   Can you advice which monitoring tool is better to use?

 

Also, create fault monitoring that will tell you if your Redis processes are running, accepting connections, responding to commands, and that slaves are properly replicating from their masters.

As for the operating environment, one rule of thumb is to give a Redis instance 2-3x the amount of ram needed to hold its full data set, so saving its data to disk will go smoothly, even when the data set is changing rapidly (lots of writes).

 

  Another rule of thumb is to give Redis 2-4 cpu cores (1 for the main command processing, 1-2 for network i/o with clients, 1 for the child process that saves data to disk).

     How this can be configured? And where can I get more details information how to separate processes to be responsible for command , networking and child processes?

[Greg Andrews]

Oled asked:

Can you advice which monitoring tool is better to use?

I have used Ganglia and Cacti at my present job, and we're starting to use Graphite.

Ganglia (http://ganglia.sourceforge.net/) is a "push" design.  Daemons run on each server being monitored, and they send (push) data to the server that holds the RRD files and makes the graphs available via the web interface.  For Linux (and some Unix) machines, this is very easy to install and configure.  There's also a command-line utility available on each server to submit custom graph data into the RRD files from scripts.

Cacti (http://www.cacti.net/) is usually implemented as a "pull" design.  Processes running on the Cacti machine send querys to the SNMP daemon on each server being monitored and "pull" the data from them.  I don't remember if there's a way to push data from the servers into Cacti.  Cacti has a large community of people writing modules to pull/graph data for relational databases, network equipment, and lots of other things.

Graphite (http://graphite.wikidot.com/) is also usually implemented as a "pull" design using SNMP, though it has a listener allowing data to be pushed from servers.  Graphite is newer than a lot of other monitoring/graphing tools, and its FAQ says it's oriented toward larger scale.

I briefly tried Zenoss (http://www.zenoss.com/) several years ago, but I haven't kept myself informed on its development.  It offered both fault monitoring and performance graphing, using a "pull" design via SNMP.

There are probably others also available that I'm not aware of. I don't have a recommendation for a particular tool.  The change from non-graphing to graphing is a far larger advancement than the differences between the tools.  Have a look at their FAQs and pick one that looks like it's a good match for you.

Oleg wrote (regarding giving Redis 2-4 cpu cores):

How this can be configured? And where can I get more details information how to separate processes to be responsible for command , networking and child processes?

If you're running Redis in certain kinds of virtual machines, the virtualization layer can give you control over how cpu resources are allocated.  In other environments it's more a situation of choosing which applications you will run on the same server as Redis, and which you'll run on another server because they'll try to use all the cpu cores.

  -Greg

[Colin Vipurs]

I can give a +1 to graphite, which we also use in combination with statsd.  It can definitely handle large scale - we collect around 250,000 stats continuously.  The interface is a bit "meh" but you soon get used to it and there are other front ends you can bolt on if you really want.

Maybe she awoke to see the roommate's boyfriend swinging from the chandelier wearing a boar's head.

Something which you, I, and everyone else would call "Tuesday", of course.