Redis BGREWRITEAOF fails
Julien
I'm not sure why but one of our server doesn't work anymore with AOF :
(
When I start a background save, here is what the log says :
07 Apr 14:45:02 - Background append only file rewriting started by pid
25535
07 Apr 14:45:24 * Background append only file rewriting terminated by
signal
Also, it creates a temp file : temp-rewriteaof-25535.aof
The problem is that now our server is _very_ slow to restart, and
since we have cron running which generated these temp-rewriteaof
files, we generate a LOT, slowly eating all our disk space :(
is that a known issue?
Thanks!
Salvatore Sanfilippo
> Hello,
>
> I'm not sure why but one of our server doesn't work anymore with AOF :
> (
>
> When I start a background save, here is what the log says :
> 07 Apr 14:45:02 - Background append only file rewriting started by pid
> 25535
> 07 Apr 14:45:24 * Background append only file rewriting terminated by
> signal
Hello Julien,
the background saving process terminates for some reason, the problem
is that we don't know what the reason is... Redis should log the
signal number actually, I'm adding this to the todo list.
In order to understand what happens you should check with "dmesg" if
for some reason the OOM killer is killing the process. Also since
there is a big window from the start of the saving to the killing of
the process, you can try attaching the process with "strace -p" or gdb
(using the attach command) in order to understand what exactly
happens.
This is the first report of this kind I receive so I think we need to
investigate a bit in order to understand what's happening.
Cheers,
Salvatore
--
Salvatore 'antirez' Sanfilippo
http://invece.org
"Once you have something that grows faster than education grows,
you’re always going to get a pop culture.", Alan Kay
Julien
On Apr 7, 5:04 pm, Salvatore Sanfilippo <anti...@gmail.com> wrote:
> On Wed, Apr 7, 2010 at 4:47 PM, Julien <julien.genest...@gmail.com> wrote:
> > Hello,
>
> > I'm not sure why but one of our server doesn't work anymore with AOF :
> > (
>
> > When I start a background save, here is what the log says :
> > 07 Apr 14:45:02 - Background append only file rewriting started by pid
> > 25535
> > 07 Apr 14:45:24 * Background append only file rewriting terminated by
> > signal
>
> Hello Julien,
>
> the background saving process terminates for some reason, the problem
> is that we don't know what the reason is... Redis should log the
> signal number actually, I'm adding this to the todo list.
>
> In order to understand what happens you should check with "dmesg" if
> for some reason the OOM killer is killing the process. Also since
what is dmesg?
> there is a big window from the start of the saving to the killing of
> the process, you can try attaching the process with "strace -p" or gdb
> (using the attach command) in order to understand what exactly
> happens.
Let me try strace -p but I'm ,ot sure what I should be looking for :(
>
> This is the first report of this kind I receive so I think we need to
> investigate a bit in order to understand what's happening.
Good! Let's fix this!
>
> Cheers,
> Salvatore
>
> --
> Salvatore 'antirez' Sanfilippohttp://invece.org
Julien
The AOF worked again!
07 Apr 14:48:32 * Background saving terminated by signal
07 Apr 14:50:10 - Background append only file rewriting started by pid
27671
07 Apr 14:50:35 * Background append only file rewriting terminated by
signal
07 Apr 15:00:03 - Background append only file rewriting started by pid
31796
07 Apr 15:00:48 * Error moving temp append only file on the final
destination: No such file or directory
07 Apr 15:00:49 * Background append only file rewriting error
07 Apr 15:04:15 - Background append only file rewriting started by pid
1100
07 Apr 15:04:57 - SYNC append only file rewrite performed
07 Apr 15:04:58 - Background append only file rewriting terminated
with success
07 Apr 15:04:58 - Parent diff flushed into the new append log file
with success (25940 bytes)
07 Apr 15:04:58 - Append only file successfully rewritten.
07 Apr 15:04:59 - The new append only file was selected for future
appends.
I'll monitor that to make sure this wasn't an error!
On Apr 7, 5:04 pm, Salvatore Sanfilippo <anti...@gmail.com> wrote:
> On Wed, Apr 7, 2010 at 4:47 PM, Julien <julien.genest...@gmail.com> wrote:
> > Hello,
>
> > I'm not sure why but one of our server doesn't work anymore with AOF :
> > (
>
> > When I start a background save, here is what the log says :
> > 07 Apr 14:45:02 - Background append only file rewriting started by pid
> > 25535
> > 07 Apr 14:45:24 * Background append only file rewriting terminated by
> > signal
>
> Hello Julien,
>
> the background saving process terminates for some reason, the problem
> is that we don't know what the reason is... Redis should log the
> signal number actually, I'm adding this to the todo list.
>
> In order to understand what happens you should check with "dmesg" if
> for some reason the OOM killer is killing the process. Also since
> there is a big window from the start of the saving to the killing of
> the process, you can try attaching the process with "strace -p" or gdb
> (using the attach command) in order to understand what exactly
> happens.
>
> This is the first report of this kind I receive so I think we need to
> investigate a bit in order to understand what's happening.
>
> Cheers,
> Salvatore
>
> --
> Salvatore 'antirez' Sanfilippohttp://invece.org
Salvatore Sanfilippo
> Hum... strange.
>
> The AOF worked again!
Yep but there were a few failed tries before. Better to look if there
is enough space on disk, if the OOM killer is killing the process for
some reason and so forth.
Cheers,
Salvatore
> --
> You received this message because you are subscribed to the Google Groups "Redis DB" group.
> To post to this group, send email to redi...@googlegroups.com.
> To unsubscribe from this group, send email to redis-db+u...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/redis-db?hl=en.
>
>
--
Salvatore 'antirez' Sanfilippo
Salvatore Sanfilippo
OOM killer killing the saving process, so it does not looks like a bug
in Redis.
Cheers,
Salvatore