Account Options

  1. Sign in
The old Google Groups will be going away soon, but your browser is incompatible with the new version.
Google Groups Home
« Groups Home
Amount of disk space need to allocated for aof
There are currently too many topics in this group that display first. To make this topic appear first, remove this option from another topic.
There was an error processing your request. Please try again.
flag
  9 messages - Collapse all  -  Translate all to Translated (View all originals)
The group you are posting to is a Usenet group. Messages posted to this group will make your email address visible to anyone on the Internet.
Your reply message has not been sent.
Your post was successful
 
From:
To:
Cc:
Followup To:
Add Cc | Add Followup-to | Edit Subject
Subject:
Validation:
For verification purposes please type the characters you see in the picture below or the numbers you hear by clicking the accessibility icon. Listen and type the numbers you hear
 
Howard  
View profile  
 More options Oct 30 2012, 5:10 am
From: Howard <howac...@gmail.com>
Date: Tue, 30 Oct 2012 02:10:52 -0700 (PDT)
Local: Tues, Oct 30 2012 5:10 am
Subject: Amount of disk space need to allocated for aof

We are using aof, and our maxmemory is 4GB, so in our dedicated redis
instance, we need to prepare at least disk space of

4GB + auto-aof-rewrite-min-size

right?


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Josiah Carlson  
View profile  
 More options Oct 30 2012, 11:39 am
From: Josiah Carlson <josiah.carl...@gmail.com>
Date: Tue, 30 Oct 2012 08:39:52 -0700
Local: Tues, Oct 30 2012 11:39 am
Subject: Re: Amount of disk space need to allocated for aof
On disk use of the AOF will vary depending on your application.

Your best bet would be to actually test on-disk usage of Redis in as
close to the same scenario as possible.

Note that when the AOF is rewritten, Redis will fork the same way that
it forks during a BGSAVE in order to rewrite the AOF. This will create
a second copy of the AOF, which will grow to contain all of your Redis
data, then it will get more recent commands. How often this will occur
will depend on both 'auto-aof-rewrite-percentage' and
'auto-aof-rewrite-min-size', whose meaning can be found in the
configuration file.

If it just so happens that the rewriting repeatedly fails (permissions
and/or disk space is the typical cause), your AOF will actually
continuously grow until you run out of space, which can cause some
very unpleasant results.

So to really answer your question: test it with real data, then
multiply it by 10x. This is a classic time-money tradeoff. By spending
more money on disk space, you can save yourself a lot of time later.

Regards,
 - Josiah


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Howard  
View profile  
 More options Oct 31 2012, 10:43 am
From: Howard <howac...@gmail.com>
Date: Wed, 31 Oct 2012 07:43:31 -0700 (PDT)
Local: Wed, Oct 31 2012 10:43 am
Subject: Re: Amount of disk space need to allocated for aof

Hi,

On Tuesday, October 30, 2012 11:39:57 PM UTC+8, Josiah Carlson wrote:

> On disk use of the AOF will vary depending on your application.

So redis can't just set a max value and auto rotate when it exceed this
value?

I am wondering what are the advantage in
using  'auto-aof-rewrite-percentage' and 'auto-aof-rewrite-min-size' except
more confusion..

Thanks anyway


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Josiah Carlson  
View profile  
 More options Oct 31 2012, 4:20 pm
From: Josiah Carlson <josiah.carl...@gmail.com>
Date: Wed, 31 Oct 2012 13:20:06 -0700
Local: Wed, Oct 31 2012 4:20 pm
Subject: Re: Amount of disk space need to allocated for aof
On Oct 31, 2012 7:43 AM, "Howard" <howac...@gmail.com> wrote:

> On Tuesday, October 30, 2012 11:39:57 PM UTC+8, Josiah Carlson wrote:
>> On disk use of the AOF will vary depending on your application.

> So redis can't just set a max value and auto rotate when it exceed this value?

Not really, no. But do you know of another database that offers that
kind of guarantee while also offering the variety of data types and
access patterns that Redis offers?

> I am wondering what are the advantage in using  'auto-aof-rewrite-percentage' and 'auto-aof-rewrite-min-size' except more confusion..

It's not confusing if you read the documentation.

Redis won't automatically rewrite the AOF if it is below
auto-aof-rewrite-min-size, obviously. And Redis won't automatically
rewrite the AOF if since the last rewrite, the file size has not grown
auto-aof-rewrite-percentage larger than before.

So to get the AOF to be automatically rewritten, assuming that
auto-aof-rewrite-min-size is 64M, the last rewrite produced a file of
size X megabytes, and you leave auto-aof-rewrite-percentage at 100,
then you will get a rewrite of the AOF when it reaches max(64, 2*X)
megabytes in size. When it gets that large, it will create a new file
of approximately X megabytes again, plus whatever writes you've
received while the rewrite was happening. So you will likely have a
bit more than 3X on disk before the original 2X AOF is deleted,
dropping you back to ~X before it starts growing again.

Note that the X is relatively unrelated to your in-memory size.
Depending on your data structures, I usually expect that a dump or AOF
will be approximately 1/20 to 1/5 the in-memory size. So if you are
limiting it to 4 gigabytes, your 'X' will likely be around 200M-800M
if your data is anything like mine.

The reason I recommended that you test is because *your data* and
*your access patterns* will determine the actual X related to your 4G.
If you do your calculations right, your on-disk required space may be
substantially less than the 4G that you set aside for it in your first
email.

Regards,
 - Josiah


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Howard  
View profile  
 More options Nov 1 2012, 12:14 am
From: Howard <howac...@gmail.com>
Date: Wed, 31 Oct 2012 21:14:40 -0700 (PDT)
Local: Thurs, Nov 1 2012 12:14 am
Subject: Re: Amount of disk space need to allocated for aof

Hi,

On Thursday, November 1, 2012 4:20:10 AM UTC+8, Josiah Carlson wrote:

> > I am wondering what are the advantage in using
>  'auto-aof-rewrite-percentage' and 'auto-aof-rewrite-min-size' except more
> confusion..

> It's not confusing if you read the documentation.

> Redis won't automatically rewrite the AOF if it is below
> auto-aof-rewrite-min-size, obviously. And Redis won't automatically
> rewrite the AOF if since the last rewrite, the file size has not grown
> auto-aof-rewrite-percentage larger than before.

Thanks for the detail explanation.

Considering the design, I am wondering why use a percentage instead of a
fixed size?

I know you need some flexibility for redis special use case, but you can
use

e.g.

auto-aof-rewrite-min-size = 64M
auto-aof-rewrite-max-size = 128M (no AOF can grown large than this)

So when people plan their disk space, they can use the baseline 128M
multiple by some factors to safety plan the free space.

Not challenge you but just want to learn the design behind.

Thanks.


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Josiah Carlson  
View profile  
 More options Nov 1 2012, 1:40 am
From: Josiah Carlson <josiah.carl...@gmail.com>
Date: Wed, 31 Oct 2012 22:40:32 -0700
Local: Thurs, Nov 1 2012 1:40 am
Subject: Re: Amount of disk space need to allocated for aof

Let's say that the configuration was exactly as you describe it. What
happens if Redis would receive a write that would make the size of the
AOF larger than 128M? Should Redis stop accepting writes? Should it
stop accepting writes until a rewrite can occur and complete? Or what
if the *minimum* size that the AOF can be (due to the data in Redis)
is 127M? Would you start a rewrite operation after it received 1 more
meg of writes, then again after that received 1M of writes? There is
no semantic where a meaningful 'max-size' won't lose data, or won't be
very wasteful (computationally or in disk IO) in a large number of
common situations.

Instead, Redis just says "if the AOF is at least X megabytes, and the
AOF is at least Y% larger than the AOF at the end of the last rewrite,
then do a rewrite now". That means that you choose how much space to
"waste", relative to the amount of data that you *need* to store. That
is also why Y is typically 100 (you waste at most half of the space).

Regards,
 - Josiah


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Howard  
View profile  
 More options Nov 2 2012, 3:40 am
From: Howard <howac...@gmail.com>
Date: Fri, 2 Nov 2012 00:40:28 -0700 (PDT)
Local: Fri, Nov 2 2012 3:40 am
Subject: Re: Amount of disk space need to allocated for aof

Hi,

On Thursday, November 1, 2012 1:40:37 PM UTC+8, Josiah Carlson wrote:

>  Or what
> if the *minimum* size that the AOF can be (due to the data in Redis)
> is 127M? Would you start a rewrite operation after it received 1 more
> meg of writes, then again after that received 1M of writes?

Oh wait..I think my understanding on AOF maybe incorrect. Originally I
thought the AOF is just a log, e.g. something like log in RDBMS so when you
flush your data into disk (which is persistence and you can safely discard
them.

So now you mean the AOF contains all the shortest steps that is required to
build an empty DB to the latest DB? Even when I deleted the rdb file and I
can rebuild the db using just the AOF?

Thanks for your kind explanation again!


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Josiah Carlson  
View profile  
 More options Nov 2 2012, 9:30 am
From: Josiah Carlson <josiah.carl...@gmail.com>
Date: Fri, 2 Nov 2012 06:30:11 -0700
Local: Fri, Nov 2 2012 9:30 am
Subject: Re: Amount of disk space need to allocated for aof

Yes, the entire database can be re-created from the contents of the
AOF. BGREWRITEAOF (or when it is automatically executed as a result of
the two configuration options we've been talking about) rewrites the
AOF into a file that only has simple data addition commands (plus any
other write commands that came in since the fork() ).

Unless you are looking for an easy to backup point-in-time dump, if
you have AOF enabled, there really is no need for the rdb (though they
are still created if you have slaves, because they are used for slave
sync).

Regards,
 - Josiah


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Howard  
View profile  
 More options Nov 3 2012, 12:34 am
From: Howard <howac...@gmail.com>
Date: Fri, 2 Nov 2012 21:34:44 -0700 (PDT)
Local: Sat, Nov 3 2012 12:34 am
Subject: Re: Amount of disk space need to allocated for aof

Hi,

Thanks for your kind reply.


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
End of messages
« Back to Discussions « Newer topic     Older topic »