leveldb data corrupt after power down.

[Chunping Ruan]

2 days ago, my leveldb server restart after power switch tripping.

Unfortunately, a 5GB size db corrupted. all keys (10,000,000+) became inavaiable.   

I googled a python script, and tried to repair the db.

#!/usr/local/bin/python

import leveldb

leveldb.RepairDB('/data/leveldb-db1')

I  can see lot's of such info in LOG file:

................

013/07/22-01:16:48.812110 801407400 Table #8527104: 0 entries Corruption: corrupted compressed block contents
2013/07/22-01:16:48.812123 801407400 Table #8527104: ignoring Corruption: corrupted compressed block contents
2013/07/22-01:16:48.812170 801407400 Archiving /data/sleveldb-db1/8527104.sst: OK

............

After "repair" , there's only 2 .sst files left in the directory, and the other 3000+ .sst files were moved to another subdir called "lost"

Is there some useful tools to repair the corrupted data ,or how to repair the db?

OS: FreeBSD 8.2 amd64

Leveldb: 1.8.0

snappy: 1.0.5

[Sanjay Ghemawat]

2 days ago, my leveldb server restart after power switch tripping.

Unfortunately, a 5GB size db corrupted. all keys (10,000,000+) became inavaiable.   

What was the error message when you tried to open the database after the restart?

 

I googled a python script, and tried to repair the db.

#!/usr/local/bin/python

import leveldb

leveldb.RepairDB('/data/leveldb-db1')

I  can see lot's of such info in LOG file:

................

013/07/22-01:16:48.812110 801407400 Table #8527104: 0 entries Corruption: corrupted compressed block contents
2013/07/22-01:16:48.812123 801407400 Table #8527104: ignoring Corruption: corrupted compressed block contents

I presume that your normal DB handling code links in the snappy library?  Perhaps your python bindings that you used for the repair above don't link in snappy?  I would suggest adding a "--repair" type option to your normal application and have it call Repair itself (passing in any special comparator you may be using for normal operation).

Since you already ran repair, you will have to undo what it did.

(a) Move all of the lost/* files back into the db directory

(b) Save away all files that were created during repair (file system modification time should help)

Also, before you run repair again, I would strongly suggest just making a copy of the db directory and running repair on that.  You can use hard-links (the "ln" command) to copy any *.sst files cheaply.

[Chunping Ruan]

Dear sanjay,

On Tuesday, July 23, 2013 11:27:49 PM UTC+8, sanjay wrote:

2 days ago, my leveldb server restart after power switch tripping.

Unfortunately, a 5GB size db corrupted. all keys (10,000,000+) became inavaiable.   

What was the error message when you tried to open the database after the restart? 

 

I googled a python script, and tried to repair the db.

#!/usr/local/bin/python

import leveldb

leveldb.RepairDB('/data/leveldb-db1')

I  can see lot's of such info in LOG file:

................

013/07/22-01:16:48.812110 801407400 Table #8527104: 0 entries Corruption: corrupted compressed block contents
2013/07/22-01:16:48.812123 801407400 Table #8527104: ignoring Corruption: corrupted compressed block contents

I presume that your normal DB handling code links in the snappy library?  Perhaps your python bindings that you used for the repair above don't link in snappy?  I would suggest adding a "--repair" type option to your normal application and have it call Repair itself (passing in any special comparator you may be using for normal operation).

You are right!

I checked my libleveldb.so , snappy was not linked :(

It's very strange, I installed the leveldb from freebsd's ports, and did choose SNAPPY...

Now, I modiffed the build_detect_platform, and install it again.

$ ldd /usr/local/lib/libleveldb.so

/usr/local/lib/libleveldb.so:

        libsnappy.so.2 => /usr/local/lib/libsnappy.so.2 (0x80125d000)

        libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x801463000)

        libm.so.5 => /lib/libm.so.5 (0x801773000)

        libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x801994000)

        libthr.so.3 => /lib/libthr.so.3 (0x801ba1000)

        libc.so.7 => /lib/libc.so.7 (0x80081a000)

And installed py-leveldb again too.

$ldd /usr/local/lib/python2.7/site-packages/leveldb.so

/usr/local/lib/python2.7/site-packages/leveldb.so:

        libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x80120b000)

        libleveldb.so.1 => /usr/local/lib/libleveldb.so.1 (0x80151b000)

        libm.so.5 => /lib/libm.so.5 (0x801778000)

        libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x801999000)

        libc.so.7 => /lib/libc.so.7 (0x80081a000)

        libsnappy.so.2 => /usr/local/lib/libsnappy.so.2 (0x801ba6000)

        libthr.so.3 => /lib/libthr.so.3 (0x801dac000)

(there is a mistake in my last post, my OS is FreeBSD 9.1-RELEASE amd 64)

The db was generated by redis-storage ( redis API and leveldb as backend ,  https://github.com/rchunping/redis-storage )

The snappy and leveldb were static linked into redis-storage, so the data was compressed by snappy.

Since you already ran repair, you will have to undo what it did.

(a) Move all of the lost/* files back into the db directory

(b) Save away all files that were created during repair (file system modification time should help)

Also, before you run repair again, I would strongly suggest just making a copy of the db directory and running repair on that.  You can use hard-links (the "ln" command) to copy any *.sst files cheaply.

I will try these steps, thank you very much!!!!

 

[Chunping Ruan]

Dear sanjay,

Thank you very much,follow your steps, db was fixed.everything is ok now. 

[Junaid Farooq]

I know this is an old post.. but is there any possibility to repair the data but dint recover very old indexs and files?

We used that leveldb.repair and it worked but it not just repaired but recovered so many data which wasnt even needed.. is there any way to clean this up?

Is there any way to repair but the option of dont recover?

We just want to repair the corruption.