Incremental persistence
speedy
incremental persistence models, but it seems to always get pulled to a
different topic, and some of the discussion may be outdated since
redis has evolved amazingly in the last year... so I was wondering if
someone could shed light on the current best practices for this.
What do I mean by incremental persistence? Here's the scenario. Say
we're OK with losing a few minutes of data, but not more. And we're ok
with having enough client-side sharding such that on any redis
instance we'll have less than half the RAM full of data, so let's not
deal with what happens if we have more data than fits in memory. We do
have a high-write scenario with relatively small keys, and very very
high data rates, so a significant subset of keys gets changed very
very often (think counters) To be specific, let's say we have 10 GB of
data per instance (so > 20GB of RAM per instance).
It seems, unless I'm misunderstanding something, that AOF would be too
slow: it writes every change to disk, and even though it only fsyncs
occasionally, eventually when it does fsync it's writing way too
much.
We could alternatively snapshot every minute, but that seems very
wasteful to write the entire thing every minute when only a fraction
changes every minute: our data per instance would be limited by how
much could be snapshotted every minute.
It seems what we want is to persist the final state of the keys at the
end of every minute. That's what the BGREWRITEAOF does when you call
it, but that's *in addition to* the AOF write, so it can only slow
things down even further (though it will speed up restarts). That's
what I mean by "incremental persistence": keep track of the keys that
have changed during the minute, and persist just those, then do it
again the next minute, and occasionally also take a full backup, i.e.
a snapshot. It seems this should be what you'd get if you could have
the BGREWRITEAOF happen without the AOF itself.
Is this the right way to think about the best way to persist redis in
our scenario, and is it possible with the current redis?
Thanks in advance!
Hampus Wessman
Hi speedy,
interesting thoughts. Keep in mind that the AOF (and BGSAVE too) writes
sequentially to the disk, so you should get pretty good disk performance
with both HDDs and SSDs. Updating a counter (or similar) doesn't
generate that much data in the AOF, so it may not be a practical problem
that every single update is logged. I don't think you will be able to
max out your hard drive by running lots of INCR for example. It looks
like my computer is limited by the CPU long before that happens (and
it's running a lot of queries per second by then!).
Another way to achieve something like what you're talking about would be
for Redis to keep a queue of things to write to the AOF, do the writes
asynchronously and compact/rewrite the queue when new updates were
added. That way, if the disk couldn't keep up completely we would
probably be able to merge some changes in the queue before they were
written to disk and thereby improving performance (e.g. several updates
of a single counter would be merged into one). If we don't want it to
use the disk more than necessarily, we could even delay the writing a
bit so that we got more chances to merge updates...
I think that would be a very interesting improvement. I've been thinking
a bit about something like that before. Do note, though, that writes to
the AOF are done synchronously right now so the idea above would require
quite a few changes. I don't think it would make much difference in many
cases either. It should be useful for frequently updated counters at least.
Also see earlier discussions about asynchronous AOF I/O if you're
interested in that aspect.
Cheers,
Hampus
speedy
only have a few hotspots for a few counters we might be able to keep
up via AOF, though it still seems like the persistence model is
qualitatively suboptimal: especially for things like counters, there
is little/no value in the individual changes written to disk, just to
the last value at any given time, so if we're willing to tolerate a
bit of loss it should be possible to do a lot better by an incremental
"backup" rather than the detailed redo log that AOF is writing. We're
also doing this on EC2 so all sorts of talk about SSDs and optimized
systems leaves us just jealous ;-)
I agree that your asynchronous ideas are very interesting. In some
sense, we're looking to do the ultimate version of that: don't write
an AOF at all, just keep track of what's changed in a certain period
of time, and write that asynchronously. Is that a huge change to redis
-- or alternatively is it something one can get by tweaking some
parameters and maybe just a bit of code? We're very eager to get this
somehow ASAP and do what's needed to get there, but we don't have the
people to actually hack at it if that's what it'll take.
Felix Gallo
The current best practice is to separate concerns; keep the production system fast and deterministic by turning off all persistence, but keep a dedicated slave on another machine that has whichever persistence semantics you choose. The master isn't affected by the slave doing replication; the slave has to catch back up once it's done, but unless your load is so high that you are pegging out your production system, the slave falls back into lockstep fairly quickly.
Further, if you're pegging out your production system, consider sharding into multiple redii across several servers. Management of many redii is not that bad (certainly better than managing multiple mysqls), and if you do consistent hashing on the front end then you can end up with a nice, predictable set of systems with sensible failure and recovery mechanics.
EC2's disk options are either horrible or unbelievably horrible; localstore is about the only trustable option in my experience/opinion, and EBS needs a radical overhaul before I'd use it in a production environment. You might consider slaving to somewhere that has far more sensible characteristics. For example, Rackspace has Raid-10 persistent local store and durable instances; although their offering is significantly less mature than EC2, the disk story is far more credible, and AOF/dump seem like actual options there.
F.
--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To post to this group, send email to redi...@googlegroups.com.
To unsubscribe from this group, send email to redis-db+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/redis-db?hl=en.
Josiah Carlson
something like 10 gigs of counters per instance. If you used an AOF,
writing once per second, that's roughly on the order of 2 megs/second
to write (which isn't bad). Obviously rewriting is going to be painful
if you do it in the background (various people have found forking to
be slow on huge processes, which induce unpredictable lagging, and can
cause some nasty memory churn), but if you can "pause the world" for a
bit, doing a SAVE works great. I have a 60 gig Redis instance that
does a foreground save in about 4 minutes (we cycle through an
"offline mode" at 3AM which induces a SAVE when all of our workers
have finished), resulting in a 3.5 gig dump.
That said, it's also entirely possible that Redis isn't the right
solution. Twitter has had a need for a huge number of counters in a
distributed system, and developed a set of patches on top of Cassandra
to make them work in a reliable and fast way:
http://www.slideshare.net/mubarakss/cassandra-at-twitter-distributed-counters
. I've not looked into it as my need for counters is a couple orders
of magnitude smaller than yours, but it may be the right solution for
you.
Regards,
- Josiah
P.S. If you are really looking to work the Redis angle, here is a way
of offering rolling data syncs:
1. Whenever you increment a counter, also perform "SADD known_keys
<key>" inside a MULTI/EXEC call.
2. Have a daemon perform the following sequence of operations as fast
as possible:
v = SRANDMEMBER known_keys
if v is not null:
MULTI
SREM known_keys <v>
data = GET <v>
EXEC
# store v and data in something like Amazon SimpleDB, RDS, etc.
else:
sleep(1)
If you want to make counters save faster (and smaller), store your
counters in sub-keys of hashes, with a target size of roughly 128-1024
entries (updating the hash-max-zipmap-entries and
hash-max-zipmap-value config entries appropriately), which should
reduce round-trips between Redis, and whatever data store you are
using. You can also perform something up to 1024 SRANDMEMBER calls,
dedupe, and perform fetch on those keys to get data for a large chunk
of keys, which should allow for pretty reasonable streaming to
SimpleDB, RDS, etc.
You don't get point in time persistence, you do rely on your own code
being proper (never mind another Amazon service not going down), but
you can run more than one backup process against each Redis if one is
not able to keep up (with a little work, Redis scripting makes it a
lot easier), never mind various opportunities to further scale a
single backup process per Redis (I offered a couple above).
On Mon, Jul 18, 2011 at 10:04 AM, speedy <u...@trustedopinion.com> wrote: