Due a swapping condition our server crashed during the BGREWRITEAOF
and both the forked and the main redis-server server died.
Since we store a lot of information in Redis (HTML) we end with a
5.4GB file that we believe is pretty much close to the total of
information, however we are unable to load it back.
[28061] 13 Sep 17:23:24 * Server started, Redis version 2.0.1
[28061] 13 Sep 17:24:29 # Bad file format reading the append only file
I found this thread:
http://groups.google.com/group/redis-db/browse_thread/thread/d04777e6d68aed95
But even we found that last MULTI command was incomplete and removed
it, we still get the bad format messages, so there is a chance
something else is happening.
Every attempt to enable "loglevel debug" and get more information was
not working, only when VM was enabled log got noisy, but loading this
file is taking a lot.
Normally before it crash it loads 8.4GB of RAM in a few minutes, but
with VM enabled and default values is taking a lot longer and a log
file of around 350MB...
Any suggestions on tracing this and be able to recover the AOF contents?
Thank you.
--
Luis Lavena
AREA 17
-
Perfection in design is achieved not when there is nothing more to add,
but rather when there is nothing more to take away.
Antoine de Saint-Exupéry
1) Make a backup copy of your AOF file, then..
2) Just use the ./redis-check-aof tool in the Redis distribution
(compiled with the usual "make")
Cheers,
Salvatore
> --
> You received this message because you are subscribed to the Google Groups "Redis DB" group.
> To post to this group, send email to redi...@googlegroups.com.
> To unsubscribe from this group, send email to redis-db+u...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/redis-db?hl=en.
>
>
--
Salvatore 'antirez' Sanfilippo
http://invece.org
"We are what we repeatedly do. Excellence, therefore, is not an act,
but a habit." -- Aristotele
Thank you Salvatore, I give up trying around 11pm (gmt-3)
> But I guess this can be pretty urgent, so:
>
> 1) Make a backup copy of your AOF file, then..
> 2) Just use the ./redis-check-aof tool in the Redis distribution
> (compiled with the usual "make")
0x1576d9ef8: Expected to read 7 bytes, got 0 bytes
AOF is not valid
I already new about AOF not been valid, but what I presume is the hex
position (byte 5761769208) could lead me to the problem, right?
Thank you in advance for your help.
Even if you get it fixed in your file, I bet that it is an important
thing to handle in redis itself.
--
Best Regards,
Konstantin Merenkov
./redis-chck-aof --fix <filename>
with --fix it will fix your file.
Cheers,
Salvatore
Hmn:
appendonly.aof is 5761769208, and the byte indicated by
redis-check-aof seems to be the EOF precisely.
The last command is:
$ tail -n 5 appendonly.aof
*1
$4
exec
*1
$5
According to the protocol file, is expecting 5 bytes after the last command.
Should I truncate it? since is 5.4GB will take a lot to resave the
file just for a test, want to ensure first ;-)
Thank you.
Cheers,
Salvatore
> Even if you get it fixed in your file, I bet that it is an important
> thing to handle in redis itself.
We do this already, as the --fix option of the utility shipped with
the Redis distribution can check and fix the file.
But I think it is important that this is a two step process, that is,
the user needs to understand the file is corrupted, using an external
tool to fix it doing a backup before, possibly checking what is the
difference with the two files with diff.
An AOF corrupted is not something to handle more or less silently I
think, but for sure we need to provide the tools to deal with this
problems :)
Cheers,
Salvatore
Yes, thank you.
$ diff -u appendonly.aof.backup appendonly.aof
--- appendonly.aof.backup 2010-09-14 05:32:55.000000000 -0700
+++ appendonly.aof 2010-09-14 05:59:56.000000000 -0700
@@ -208222098,5 +208222098,3 @@
*1
$4
exec
-*1
-$5
Pretty trivial fix.
Will bring the original file again since I truncated the file manually
following the previous thread that I mentioned.
Salvatore, thank you so much.
> Pretty trivial fix.
>
> Will bring the original file again since I truncated the file manually
> following the previous thread that I mentioned.
>
> Salvatore, thank you so much.
You are welcome!
Btw the last commit in Redis master changes the error message to the following:
"Bad file format reading the append only file: make a backup of your
AOF file, then use ./redis-check-dump --fix <filename>"
Cheers,
Salvatore
Shouldn't it be ./redis-check-aof ?
> --
> You received this message because you are subscribed to the Google Groups "Redis DB" group.
> To post to this group, send email to redi...@googlegroups.com.
> To unsubscribe from this group, send email to redis-db+u...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/redis-db?hl=en.
>
>
--
please can you send the output of tail -50 filename against your
corrupted AOF file?
I want to check how it was fixed in a more detailed way. Thanks!
Cheers,
Salvatore
Will email you the tail contents and the diff directly.
Thank you.
I can confirm the tool is doing the right thing :)
Cheers,
Salvatore