We use resque in combination with redis, backed by AOF and RDB. I had two crashes so far caused by aof files hitting the disk limit. One even got up to 78GB.
1. Do you think AOF files are worth it, technically they make full persistence available, in practice the cause more crashes.
2. Is full persistence via slave a viable option? Advise by Felix from last Sunday:
> The 'fastest' way to deal with this is to turn persistence off on your > production server, and set up a (networked, not on the same hypervisor) > slave to handle persistence. It looks like the iops on your disk is > relatively poor; are there any other VMs on this box that might be hogging > the disk or hitting it hard enough to confuse the situation? > The next-most-pragmatic solution would be to disable aof rewrite > completely and do it either offline or during non-peak hours.
3. What disk space to memory ratio do you use? I have 7GB memory available for redis and prevent for overcommit by no longer processing small jobs into big ones. Our system has small amounts if data coming in [id, id, options], but large amounts going out of resque. I think 14 GB for the aof file might not have been enough in the first place. Especially since a rewrite needs quite a bit itself. Freeing up 1 GB wasn't enough in my case.
On Wed, Oct 3, 2012 at 3:33 AM, ticktricktrack <rai...@incutio.com> wrote:
> We use resque in combination with redis, backed by AOF and RDB. I had two
> crashes so far caused by aof files hitting the disk limit. One even got up
> to 78GB.
> 1. Do you think AOF files are worth it, technically they make full
> persistence available, in practice the cause more crashes.
It depends on your use-case. If you aren't rewriting on a regular
basis automatically, or are writing so much data that your system
can't rewrite it fast enough, then it may not be right for you.
Have you configured the trigger for bgrewriteaof?
> 2. Is full persistence via slave a viable option? Advise by Felix from last
> Sunday:
>> The 'fastest' way to deal with this is to turn persistence off on your
>> production server, and set up a (networked, not on the same hypervisor)
>> slave to handle persistence. It looks like the iops on your disk is
>> relatively poor; are there any other VMs on this box that might be hogging
>> the disk or hitting it hard enough to confuse the situation?
>> The next-most-pragmatic solution would be to disable aof rewrite
>> completely and do it either offline or during non-peak hours.
Using slaves to handle persistence is an option. As is rewriting
during odd hours. It all depends on your use-case, and whether your
disk can store as much data as you will be writing before a rewrite.
> 3. What disk space to memory ratio do you use?
> I have 7GB memory available for redis and prevent for overcommit by no
> longer processing small jobs into big ones. Our system has small amounts if
> data coming in [id, id, options], but large amounts going out of resque.
> I think 14 GB for the aof file might not have been enough in the first
> place. Especially since a rewrite needs quite a bit itself. Freeing up 1 GB
> wasn't enough in my case.
For dumps, my experience and expectation is that in-memory will use
5-10x what the on-disk representation uses. For AOFs, that will depend
on your write volume before your bgrewriteaof triggers. In terms of
main memory to rewrite an AOF, you run into effectively the same issue
as for dumps. Worst-case you will use 2x your main memory, but as a
practical matter, it is typically much smaller. Of course how much
memory it will use during a rewrite will depend on your disk IO
capacity, your processor load, etc., at the time of the rewrite. There
is no simple formula, especially in the cloud. Test, benchmark,
repeat, check multiple instances of the same type, ...
My general recommendation is to actually split your workloads onto
different Redis servers. If you have a resque Redis, don't use it for
caching. Start another Redis server up to run your caching, etc.
If you're already doing this, then my other recommendation is to use
the fastest spinning disk you can find for your AOF. I know, SSDs are
better in many ways, but my back-of-the-envelope calculations tells me
that constant writing of some loads may chew through write cycles
faster than is expected (a commodity 256 gig SSD with 5k write cycles
will last 1 year at 40 megs/second write volume). But if you've got
the money to buy SSDs, or rent them from Amazon, they may be a
reasonable answer.
On Wed, Oct 03, 2012 at 03:33:57AM -0700, ticktricktrack wrote:
> We use resque in combination with redis, backed by AOF and RDB. I had two > crashes so far caused by aof files hitting the disk limit. One even got up > to 78GB.
> 1. Do you think AOF files are worth it, technically they make full > persistence available, in practice the cause more crashes.
They're worth it in the right circumstances. AOF causes more crashes if you
don't manage it properly (and we've learnt that the hard way), but it can be
managed correctly. The issues of disk full can be mitigated by a couple of
patches that a colleague at work sent pull requests for about a month ago:
> 2. Is full persistence via slave a viable option? Advise by Felix from last > Sunday:
Not unless you've got a really solid way to switch to using that slave
"live". Some people like to change client config and point to the
slave-now-master, I prefer to patch Redis to support live reconfiguration of
the listening IP address, and migrate the service IP, a patch for which I
posted to this list a few months ago:
6GB of disk for every 1GB of RAM *allocated to the machine*, assuming a 100%
AOF rewrite percentage and a small change rate compared to rewrite time. We
work on a rule of thumb that a minimal AOF is about twice the RAM used, the
rewrite gets triggered when the AOF is twice the size of the minimal. You
might want some extra headroom to take the extra stuff that'll get written
to both AOFs after the rewrite finishes -- if you're trying to rewrite a
100GB AOF, that'll take a while and you might still fill the disk, so
perhaps give yourself some extra headroom.
If you're also using RDB, or slaving off the instance, or the instance is a
slave itself, then you'll want another RAM's worth of disk for RDB (assuming
you're using compressed RDB and you get a sensible compression ratio on your
data).
> I have 7GB memory available for redis and prevent for overcommit by no > longer processing small jobs into big ones. Our system has small amounts if > data coming in [id, id, options], but large amounts going out of resque.
> I think 14 GB for the aof file might not have been enough in the first > place. Especially since a rewrite needs quite a bit itself. Freeing up 1 GB > wasn't enough in my case.
Yep, 7GB for Redis means you want 42GB of diskspace, minimum. Any less and
you'll get all the problems you've been seeing.
- Matt
-- I seem to have my life in reverse. When I was a wee'un, it seemed perfectly
normal that one could pick up the phone and speak to anybody else in the
world who also has a phone. Now I'm older and more experienced, I'm amazed
that this could possibly work. -- Peter Corlett, in the Monastery
On Wed, Oct 3, 2012 at 3:10 PM, Matthew Palmer <mpal...@hezmatt.org> wrote:
> [...] AOF causes more crashes if you
> don't manage it properly (and we've learnt that the hard way), but it can be
> managed correctly. [...]
? I've never had an AOF instance crash. Apart from running out of
disk space, which is a systems problem rather than a redis problem,
what 'more crashes' are you talking about?
> > 2. Is full persistence via slave a viable option? Advise by Felix from
> > last
> > Sunday:
> Not unless you've got a really solid way to switch to using that slave
> "live". Some people like to change client config and point to the
> slave-now-master, I prefer to patch Redis to support live reconfiguration
> of
> the listening IP address, and migrate the service IP, a patch for which I
> posted to this list a few months ago:
You can also do this at the machine level, which is often preferable and
certainly more standardly supported. But a lot depends on your requirements
for uptime and your confidence level in your systems' uptimes, of course,
as well as whatever other disaster recovery systems you have in store.
For instance, using the system I mentioned, the downtime of my redis
cache is 'in the noise' compared to the usual 'downtime' due to backhoes
that my (globally dispersed) clients see.
- Our aof is set to be rewritten at 100% with a minimum of 1024MB. However, during high loads, a lot of data is goes into redis, temporarily stored for outgoing api calls. We do have 4-7 GB of memory used then. We limit that amount by delaying jobs in the jobchain instead of processing them.
- During high loads - rewrites throttle performance. We know at this point that the will will go up to 10GB or so, since there are several gigabytes of data stored in redis.
- We of course would very much like full persistence, especially since our last server hoster was pretty unreliable under load. It got a lot better with better servers and faster disks.
- With "crashes" I mean a box running out of diskspace and Redis becoming unreachable. The recent case was so bad that we had to delete the old server and setup a new one. Disks with 100% won't resize sadly. The idea of persistence measures to me is to recover from crashes / outages, not to cause them in the first place.
- The first crash with 78GB aof size happened because Redis was never rewriting the file. Once manually triggered, it resized it frequently after that.
I disabled the aof file for now and will see how well we do with just the RDB persistance during the next peak load. I also lowered the redis memory limit we set in Rails to half the available memory.
