Recovering from crash

[Simão Mata]

Hello,

I am having a problem with redis recovering from a server crash.

Our server ran out of disk space, so the append only file stopped being written to.

After freeing some disk space, I restarted redis but I obtained the following

message in redis.log:

        Bad file format reading the append only file: make a backup of your AOF

        file, then use ./redis-check-aof --fix <filename>

I did just that, and I got the following output:

      # /opt/redis-2.4.7/src/redis-check-aof --fix appendonly.aof

      0xffffffff: Expected to read 28 bytes, got 3 bytes

      AOF is valid

But when I try to start redis, I still get the same message saying the append

only file is in a bad file format.

I was using redis-2.2.5 before the server crashed, but I tried upgrading to

2.4.7 with the same results.

Any ideas?

Thank you very much for your help.

[Salvatore Sanfilippo]

Hello, it's a lame problem, it's not ok that it is still unfixed, but
long story short is because your arch is 32bit. Let's try if I can fix
it in a second and provide you a patch.

Salvatore

2012/2/14 Simão Mata <sim...@gmail.com>:

> --
> You received this message because you are subscribed to the Google Groups
> "Redis DB" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/redis-db/-/yoSXHKSXFzQJ.
> To post to this group, send email to redi...@googlegroups.com.
> To unsubscribe from this group, send email to
> redis-db+u...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/redis-db?hl=en.

--
Salvatore 'antirez' Sanfilippo
open source developer - VMware

http://invece.org
"We are what we repeatedly do. Excellence, therefore, is not an act,
but a habit." -- Aristotele

[Salvatore Sanfilippo]

On Tue, Feb 14, 2012 at 5:55 PM, Salvatore Sanfilippo <ant...@gmail.com> wrote:
> long story short is because your arch is 32bit. Let's try if I can fix

I mean: because your arch is 32bit, and your AOF is > 4GB.

Make sure to use 2.4 that has automatic rewriting!!! ;) (but not an
excuse for this bug).
I'll ping back when a patch is available, was not a matter of overflow
actually, but it needs to be compiled with large file support and I
can't do this right now in real time, so I'll provide a patch in the
next 24h.

Please try this if you need to restart your instance ASAP:

1) Fix the AOF in a 64 bit instance. Not sure it is possible for you.
2) Alternatively... edit aof.c, there is something like:

readerr:
if (feof(fp)) {
redisLog(REDIS_WARNING,"Unexpected end of file reading the
append only file");
} else {
redisLog(REDIS_WARNING,"Unrecoverable error reading the append
only file: %s", strerror(errno));
}
exit(1);
fmterr:
redisLog(REDIS_WARNING,"Bad file format reading the append only

file: make a backup of your AOF file, then use ./redis-check-aof --fix

<filename>");
exit(1);

Where there is exit(1) (there are two instances), replace it with
"return REDIS_OK;"

Compile and use this modified binary to restart Redis for the first
time. rewrite the AOF log with BGREWRITEAOF so that it will take a
convenient amount of space (< 4GB on disk), and then restart with an
unmodified Redis 2.4.

I hope this helps.

Cheers,
Salvatore

[Simão Mata]

Hello  Salvatore,

Thank you for your help.

My aof currently has 2.2 GB so maybe that's not the problem?

I'll try to edit the aof.c file and run that modified version.

Thank you again.

--

Simão

[Salvatore Sanfilippo]

2012/2/14 Simão Mata <sim...@gmail.com>:

> Hello  Salvatore,
>
> Thank you for your help.
>
> My aof currently has 2.2 GB so maybe that's not the problem?

You are right, it is broken with > 2GB files. Actually it WAS broken, I
just committed a fix in the 2.4 branch, now redis-check-aof is sane
for > 2GB files on 32bit systems. Please use this instead of the
modified Redis, much better.

Sorry for this issue, it was in my TODO list since a lot of time, but
this was a critical bug and I made an error not fixing it ASAP.

[Simão Mata]

Hello,

Thanks Salvatore, the patch worked, the aof was fixed and redis is now running again.

Cheers,

Simão

[Salvatore Sanfilippo]

2012/2/15 Simão Mata <sim...@gmail.com>:

> Thanks Salvatore, the patch worked, the aof was fixed and redis is now
> running again.

Cool! Likely this problem will not happen again now that we fixed the
AOF utility.