Amount of disk space need to allocated for aof

[Howard]

We are using aof, and our maxmemory is 4GB, so in our dedicated redis instance, we need to prepare at least disk space of

4GB + auto-aof-rewrite-min-size 

right?

[Josiah Carlson]

On disk use of the AOF will vary depending on your application.

Your best bet would be to actually test on-disk usage of Redis in as
close to the same scenario as possible.

Note that when the AOF is rewritten, Redis will fork the same way that
it forks during a BGSAVE in order to rewrite the AOF. This will create
a second copy of the AOF, which will grow to contain all of your Redis
data, then it will get more recent commands. How often this will occur
will depend on both 'auto-aof-rewrite-percentage' and
'auto-aof-rewrite-min-size', whose meaning can be found in the
configuration file.

If it just so happens that the rewriting repeatedly fails (permissions
and/or disk space is the typical cause), your AOF will actually
continuously grow until you run out of space, which can cause some
very unpleasant results.

So to really answer your question: test it with real data, then
multiply it by 10x. This is a classic time-money tradeoff. By spending
more money on disk space, you can save yourself a lot of time later.

Regards,
- Josiah

> --
> You received this message because you are subscribed to the Google Groups
> "Redis DB" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/redis-db/-/hWf_ikC2OiEJ.
> To post to this group, send email to redi...@googlegroups.com.
> To unsubscribe from this group, send email to
> redis-db+u...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/redis-db?hl=en.

[Howard]

Hi,

On Tuesday, October 30, 2012 11:39:57 PM UTC+8, Josiah Carlson wrote:

On disk use of the AOF will vary depending on your application.

So redis can't just set a max value and auto rotate when it exceed this value?

I am wondering what are the advantage in using  'auto-aof-rewrite-percentage' and 'auto-aof-rewrite-min-size' except more confusion..

Thanks anyway

[Josiah Carlson]

On Oct 31, 2012 7:43 AM, "Howard" <howa...@gmail.com> wrote:
> On Tuesday, October 30, 2012 11:39:57 PM UTC+8, Josiah Carlson wrote:
>> On disk use of the AOF will vary depending on your application.
>>
> So redis can't just set a max value and auto rotate when it exceed this value?

Not really, no. But do you know of another database that offers that
kind of guarantee while also offering the variety of data types and
access patterns that Redis offers?

> I am wondering what are the advantage in using 'auto-aof-rewrite-percentage' and 'auto-aof-rewrite-min-size' except more confusion..

It's not confusing if you read the documentation.

Redis won't automatically rewrite the AOF if it is below
auto-aof-rewrite-min-size, obviously. And Redis won't automatically
rewrite the AOF if since the last rewrite, the file size has not grown
auto-aof-rewrite-percentage larger than before.

So to get the AOF to be automatically rewritten, assuming that
auto-aof-rewrite-min-size is 64M, the last rewrite produced a file of
size X megabytes, and you leave auto-aof-rewrite-percentage at 100,
then you will get a rewrite of the AOF when it reaches max(64, 2*X)
megabytes in size. When it gets that large, it will create a new file
of approximately X megabytes again, plus whatever writes you've
received while the rewrite was happening. So you will likely have a
bit more than 3X on disk before the original 2X AOF is deleted,
dropping you back to ~X before it starts growing again.

Note that the X is relatively unrelated to your in-memory size.
Depending on your data structures, I usually expect that a dump or AOF
will be approximately 1/20 to 1/5 the in-memory size. So if you are
limiting it to 4 gigabytes, your 'X' will likely be around 200M-800M
if your data is anything like mine.

The reason I recommended that you test is because *your data* and
*your access patterns* will determine the actual X related to your 4G.
If you do your calculations right, your on-disk required space may be
substantially less than the 4G that you set aside for it in your first
email.

Regards,
- Josiah

[Howard]

Hi,

On Thursday, November 1, 2012 4:20:10 AM UTC+8, Josiah Carlson wrote:

> I am wondering what are the advantage in using  'auto-aof-rewrite-percentage' and 'auto-aof-rewrite-min-size' except more confusion..

It's not confusing if you read the documentation.

Redis won't automatically rewrite the AOF if it is below
auto-aof-rewrite-min-size, obviously. And Redis won't automatically
rewrite the AOF if since the last rewrite, the file size has not grown
auto-aof-rewrite-percentage larger than before.

Thanks for the detail explanation.

Considering the design, I am wondering why use a percentage instead of a fixed size?

I know you need some flexibility for redis special use case, but you can use 

e.g. 

auto-aof-rewrite-min-size = 64M

auto-aof-rewrite-max-size = 128M (no AOF can grown large than this)

So when people plan their disk space, they can use the baseline 128M multiple by some factors to safety plan the free space.

Not challenge you but just want to learn the design behind.

Thanks.

[Josiah Carlson]

Let's say that the configuration was exactly as you describe it. What
happens if Redis would receive a write that would make the size of the
AOF larger than 128M? Should Redis stop accepting writes? Should it
stop accepting writes until a rewrite can occur and complete? Or what
if the *minimum* size that the AOF can be (due to the data in Redis)
is 127M? Would you start a rewrite operation after it received 1 more
meg of writes, then again after that received 1M of writes? There is
no semantic where a meaningful 'max-size' won't lose data, or won't be
very wasteful (computationally or in disk IO) in a large number of
common situations.

Instead, Redis just says "if the AOF is at least X megabytes, and the
AOF is at least Y% larger than the AOF at the end of the last rewrite,
then do a rewrite now". That means that you choose how much space to
"waste", relative to the amount of data that you *need* to store. That
is also why Y is typically 100 (you waste at most half of the space).

Regards,
- Josiah

[Howard]

Hi,

On Thursday, November 1, 2012 1:40:37 PM UTC+8, Josiah Carlson wrote:

 Or what
if the *minimum* size that the AOF can be (due to the data in Redis)
is 127M? Would you start a rewrite operation after it received 1 more
meg of writes, then again after that received 1M of writes? 

Oh wait..I think my understanding on AOF maybe incorrect. Originally I thought the AOF is just a log, e.g. something like log in RDBMS so when you flush your data into disk (which is persistence and you can safely discard them.

So now you mean the AOF contains all the shortest steps that is required to build an empty DB to the latest DB? Even when I deleted the rdb file and I can rebuild the db using just the AOF?

Thanks for your kind explanation again!

[Josiah Carlson]

Yes, the entire database can be re-created from the contents of the
AOF. BGREWRITEAOF (or when it is automatically executed as a result of
the two configuration options we've been talking about) rewrites the
AOF into a file that only has simple data addition commands (plus any
other write commands that came in since the fork() ).

Unless you are looking for an easy to backup point-in-time dump, if
you have AOF enabled, there really is no need for the rdb (though they
are still created if you have slaves, because they are used for slave
sync).

Regards,
- Josiah

[Howard]

Hi,

Thanks for your kind reply.