Rolling BGSAVE instead of (pre)-configured save points

[Andy G.]

Hey,

this post is more about a small discussion about advantages and disadvantages of a Rolling BGSAVE instead of (pre)-configured save points.

Imagine you got a big amount of web server who access a small amount of Redis server without pipelining or persistent connections.

You are running Redis in a non cluster mode with Master - Slave replica.

You persist your data via rdb file and you are not using AOF.

Redis iselfs comes with a preconfiguration of save points. See https://github.com/antirez/redis/blob/unstable/redis.conf#L120

If such a save point is triggered a BGSAVE-Command will be called.

During a BGSAVE-Command a new process will be forked (because of the BG (Background) in BGSAVE).

The fork process is blocking, no other command will be executed until the fork of the process was finished.

Depending on the size of the memory table this fork can took a while.

We saw fork times up to ~400ms on bare metal (no AWS). See https://groups.google.com/forum/#!searchin/redis-db/Scaling$20of$20a$20%22busy%22$20redis$20server$20in$20a$20web$20environment/redis-db/uTKXtI_HMow/hS8JJdSgy40J for some information.

In our environment we saw a lot of requests running into timeouts due to a triggered BGSAVE command by a pre configured save point.

So we was looking for a solution and found a case study posted on pivotal: "Case Study: How Hulu Scaled Serving 4 Billion Videos Using Redis" (http://blog.pivotal.io/pivotal/case-studies-2/case-study-how-hulu-scaled-serving-4-billion-videos-using-redis).

Hulu is using a kind of "Rolling BGSAVE":

Since Redis usually can only use one CPU core, and our boxes have 24  CPUs, we are running 16 shards per box trying to achieve a one to one CPU ratio to maximize our hardware utilization. However, when bgsave is enabled, Redis will fork a new process and may even use double the ram. To keep performance consistent on the shards, we disabled the writing to disk across all the shards, and we have a cron job that runs at 4am everyday doing a rolling “BGSAVE” command on each individual instance.  We also use this rdb file for analyzing the Redis data without affecting the performance of the application.

So we tried the same. Disable all save points and triggered BGSAVE in times where we know that the amount of traffic is not so high (during night for example).

And voila, we reduced the connections running into a timeout a lot.

This is the situation (in short).

Now i want to discuss with you if this is a good or bad solution.

I`d collected some reasons. I like to know your reasons + opinion as well :)

Advantage of a preconfigured save point:

* Save is triggered by the application itselfs (redis)

* Persistence is triggered only if this is necessary (depends on the amount of changed + configured keys)

Disadvantage of a preconfigured save point:

* BGSAVE is triggered independent by time, is triggered by seconds + data changes

* If only a small amount of keys changed, there can be a long timeframe without persistence (depend on your configuration)

* In replication there can be the case that all servers triggered a BGSAVE in the same timeframe

Advantage of a rolling BGSAVE:

* You can time it to times where traffic is not high

* You explicity know when you got a dump / when your last dump was triggered

* You can configure when which server fires a BGSAVE (Rolling)

Disadvantage of a rolling BGSAVE:

* Maybe there is a bigger timeframe between the save points (depends on the amount of command changes)

* You have to take care that the BGSAVE is triggeres

Thanks,

Andy

[Josiah Carlson]

The mechanism that Redis uses to take periodic snapshots is meant to be "good enough for most", not "good enough for everyone". If you find that the automatic background saving via the "save" configuration option is not sufficient, then whatever you pick as your new save mechanism is your choice. You've decided to do your own rolling BGSAVE operations, like Hulu, and like tens/hundreds of other organizations that didn't blog about it after.

Does it work for you? Yes? Great! You're done! Is this the right answer for others? That depends on their requirements.

As an example, a few years ago I had a bunch of data in a single Redis server, somewhere around 40 gigabytes. It took Redis 10+ seconds to fork (the VM was the partial cuase), 20+ minutes to flush the rdb to disk, and if the server was under load, we were altering enough data that the child would use over 20 gigabytes of memory during the snapshot process. For a 68 gig server (the AWS m2.4xlarge), this was running a little close. But we really only needed to keep a snapshot of this data once/day, because we could recover 24 hours of progress in 6 hours, or 1 week of progress in 24 hours. So what we did instead was to make our entire platform aware of the snapshot process to prevent them from using that Redis server at around 3AM, performed a SAVE operation (which completed in 3 minutes instead of 20+ minutes), backed up the snapshot, then told the system that it could continue. This worked for us because it fit our requirements.

What works or doesn't work for others will also depend on their requirements. Rolling BGSAVEs are periodically recommended here for some situations, just as the "pause the world and use SAVE" method I described is also recommended on occasion. Requirements differ, as do recommendations.

For you, the rolling BGSAVE looks like a good answer. I'm not sure you'll do much better with a different but similarly simple solution.

 - Josiah

P.S. And if you are getting pressure from your organization to get validation for having decided to use this method, you can tell them that the guy that wrote this book: http://manning.com/carlson/ says your solution is fine, but he really thinks you should be using twemproxy instances to minimize new connections to your Redis server ;)

--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+u...@googlegroups.com.
To post to this group, send email to redi...@googlegroups.com.
Visit this group at http://groups.google.com/group/redis-db.
For more options, visit https://groups.google.com/d/optout.