AOF rewrite options

[Jay Rolette]

Lately I've been running into issues where my AOF file gets significantly larger than expected. It was a surprise because I expected the AOF file to get rewritten periodically via automatic AOF rewrite.

In one instance, I manually ran BGREWRITEAOF and it dropped from 490 MB down to ~71 MB. In other cases, we've seen it get up to 2GB and even 8GB in one instance.

After thinking about the options available in redis.conf, it looks like the problem is that the automatic AOF rewrite isn't happening on development systems because the trigger is based on percentage growth.

On dev systems, things are getting restarted pretty frequently as code is updated and tested. The AOF file is growing, but not enough to trigger the auto rewrite. And, since the trigger is percentage-based, the problem just gets worse over time because the AOF file was larger each time Redis gets started again.

Any chance I'm missing a config option somewhere that will let me specify auto-aof-rewrite after it gets larger than a specified size (instead of percentage)? There's a min size in bytes, but I didn't see a max.

I don't expect this to be as big of an issue on production systems that aren't being restarted constantly, but there are always dev and test systems to deal with.

Below are the relevant config parameters I found...

Thanks,

Jay

# Automatic rewrite of the append only file.

# Redis is able to automatically rewrite the log file implicitly calling

# BGREWRITEAOF when the AOF log size grows by the specified percentage.

#

# This is how it works: Redis remembers the size of the AOF file after the

# latest rewrite (if no rewrite has happened since the restart, the size of

# the AOF at startup is used).

#

# This base size is compared to the current size. If the current size is

# bigger than the specified percentage, the rewrite is triggered. Also

# you need to specify a minimal size for the AOF file to be rewritten, this

# is useful to avoid rewriting the AOF file even if the percentage increase

# is reached but it is still pretty small.

#

# Specify a percentage of zero in order to disable the automatic AOF

# rewrite feature.

auto-aof-rewrite-percentage 100

auto-aof-rewrite-min-size 64mb

[Matt Palmer]

On Fri, Jan 23, 2015 at 03:10:45PM -0600, Jay Rolette wrote:
> Any chance I'm missing a config option somewhere that will let me specify
> auto-aof-rewrite after it gets larger than a specified size (instead of
> percentage)? There's a min size in bytes, but I didn't see a max.

No, there's no auto-aof-rewrite-max-size directive. It wouldn't be
particularly hard to add, if you felt like getting your hands dirty. It
would *have* to default to "off", because otherwise any other default would
eventually result in a constant stream of AOF rewrites once an AOF got to a
certain size.

Another approach you could take, which wouldn't require any code
modifications, would be to trigger a BGAOFREWRITE immediately on startup.
That would trim any fat from the residual AOF, and would reset the size
measurement so that the percentage measurement would be based on the minimal
AOF post-rewrite, rather than whatever was loaded at startup.

- Matt

[Jay Rolette]

On Fri, Jan 23, 2015 at 3:48 PM, Matt Palmer <mpa...@hezmatt.org> wrote:

On Fri, Jan 23, 2015 at 03:10:45PM -0600, Jay Rolette wrote:
> Any chance I'm missing a config option somewhere that will let me specify
> auto-aof-rewrite after it gets larger than a specified size (instead of
> percentage)? There's a min size in bytes, but I didn't see a max.

No, there's no auto-aof-rewrite-max-size directive.  It wouldn't be
particularly hard to add, if you felt like getting your hands dirty.  It
would *have* to default to "off", because otherwise any other default would
eventually result in a constant stream of AOF rewrites once an AOF got to a
certain size.

I guess what I actually had in mind was more auto-aof-rewrite-max-growth that would trigger a rewrite after the AOF grew by X bytes. I should have described it a little better :)

 

Another approach you could take, which wouldn't require any code
modifications, would be to trigger a BGAOFREWRITE immediately on startup.
That would trim any fat from the residual AOF, and would reset the size
measurement so that the percentage measurement would be based on the minimal
AOF post-rewrite, rather than whatever was loaded at startup.

I started down that path, but my startup is already a little slow because I'm having to do a redis-check-aof at startup to deal with AOF "corruption" cases I've mentioned previously. Running the file 3x on startup would be more time than I want to take right now.

For now, I've just setup a cron job to force a rewrite at least daily. I think that should keep things in check for dev/test systems hopefully.

I mostly wanted to make sure I wasn't missing anything and in case it was something other folks ran into.

Thanks as always for the feedback, Matt!

Jay

[Matt Palmer]

On Fri, Jan 23, 2015 at 04:23:45PM -0600, Jay Rolette wrote:
> On Fri, Jan 23, 2015 at 3:48 PM, Matt Palmer <mpa...@hezmatt.org> wrote:
>
> > On Fri, Jan 23, 2015 at 03:10:45PM -0600, Jay Rolette wrote:
> > > Any chance I'm missing a config option somewhere that will let me specify
> > > auto-aof-rewrite after it gets larger than a specified size (instead of
> > > percentage)? There's a min size in bytes, but I didn't see a max.
> >
> > No, there's no auto-aof-rewrite-max-size directive. It wouldn't be
> > particularly hard to add, if you felt like getting your hands dirty. It
> > would *have* to default to "off", because otherwise any other default would
> > eventually result in a constant stream of AOF rewrites once an AOF got to a
> > certain size.
>
> I guess what I actually had in mind was more auto-aof-rewrite-max-growth
> that would trigger a rewrite after the AOF grew by X bytes. I should have
> described it a little better :)

That would be easy enough to do, too, and would provide more fine-grained
control over AOF rewrite semantics.

> > Another approach you could take, which wouldn't require any code
> > modifications, would be to trigger a BGAOFREWRITE immediately on startup.
> > That would trim any fat from the residual AOF, and would reset the size
> > measurement so that the percentage measurement would be based on the
> > minimal
> > AOF post-rewrite, rather than whatever was loaded at startup.
>
> I started down that path, but my startup is already a little slow because
> I'm having to do a redis-check-aof at startup to deal with AOF "corruption"
> cases I've mentioned previously.

I'm not familiar with the problems you've been having, but I'd hazard a
guess they're probably bugs in Redis that should be fixed.

> Running the file 3x on startup would be more time than I want to take right now.

The BGAOFREWRITE would run in the background, *after* Redis has startup and
is ready to serve requests. It wouldn't impact the time it takes to get
Redis to a point of usefulness.

I've also just thought of the possibility of adding a "rewrite on shutdown"
mode for AOF; that would provide you with a minimal AOF to load next time on
startup -- which would make startup quicker, eliminate corruption problems,
*and* give an accurate indication of file size for the percentage growth
algorithm to chew on. Best of all worlds! (at the cost of a slower shutdown)

- Matt