LevelDB files are corrupt, How can we repair them?

[Junaid Farooq]

Hi everyone,

We have been using a filesystem build in Golang, seaweedfs, and our own disk's file system was XFS, 

a few days back we got an error as

[   42.236828] XFS (sda5): xfs_do_force_shutdown(0x8) called from line 118 of file /build/linux-lts-wily-JUyKGw/linux-lts-wily-4.2.0/fs/xfs/xfs_bmap_util.c.  Return address = 0xffffffffc02e35e4
[   42.237271] XFS (sda5): Corruption of in-memory data detected.  Shutting down filesystem
[   42.237320] XFS (sda5): Please umount the filesystem and rectify the problem(s)

so, we mounted and unmounted the disk and then It needs cleaning, so we repaired it with

 xfs_repair -L /dev/sda5

there was nothing in lost+found folder but after that, our seaweed master server started to crash, As We asked a question to Chris Lu, who made the seaweedFS, he mentioned the files are corrupt, in filer, actually leveldb files. the error was something like this

I1101 12:42:19 31159 volume.go:110] loading index file /storage/1082.idx readonly false
F1101 12:42:19 31159 filer_server.go:53] Can not start filer in dir /storage/filer : leveldb/storage: corrupted or incomplete meta file
goroutine 21 [running]:
github.com/chrislusf/seaweedfs/weed/glog.stacks(0x1214600, 0x0, 0x0, 0x0)
	/root/go/src/github.com/chrislusf/seaweedfs/weed/glog/glog.go:767 +0xb8
github.com/chrislusf/seaweedfs/weed/glog.(*loggingT).output(0x11f59e0, 0xc800000003, 0xc8200faf00, 0x11cb17f, 0xf, 0x35, 0x0)
	/root/go/src/github.com/chrislusf/seaweedfs/weed/glog/glog.go:718 +0x259
github.com/chrislusf/seaweedfs/weed/glog.(*loggingT).printf(0x11f59e0, 0xc800000003, 0xe57d80, 0x22, 0xc82004dd10, 0x2, 0x2)
	/root/go/src/github.com/chrislusf/seaweedfs/weed/glog/glog.go:656 +0x1d4
github.com/chrislusf/seaweedfs/weed/glog.Fatalf(0xe57d80, 0x22, 0xc82004dd10, 0x2, 0x2)
	/root/go/src/github.com/chrislusf/seaweedfs/weed/glog/glog.go:1149 +0x5d
github.com/chrislusf/seaweedfs/weed/server.NewFilerServer(0xc8200659e0, 0xd8b9b0, 0x7, 0x22b8, 0xc8200eb740, 0x13, 0xc82018d280, 0xe, 0x0, 0x0, ...)




after that, we used a tool to repair leveldb files, https://github.com/rchunping/leveldb-tools, It repaired the files without any error, and seaweedfs Master started with filer, with no errors like the mentioned above, But as when everything is working fine, filer should respond on port `8888`, but when we access port 8888, it gave error like this,

017/11/02 01:50:41 http: panic serving 39.36.53.157:46124: leveldb: internal key "\x00\x01d,", len=4: invalid length
goroutine 2498 [running]:
net/http.(*conn).serve.func1(0xcd3aaf8080)
	/usr/lib/go/src/net/http/server.go:1389 +0xc1
panic(0xaf99a0, 0xcfefca8210)
	/usr/lib/go/src/runtime/panic.go:426 +0x4e9
github.com/syndtr/goleveldb/leveldb.internalKey.assert(0xc8712930ed, 0x4, 0x20)
	/root/go/src/github.com/syndtr/goleveldb/leveldb/key.go:115 +0x22c
github.com/syndtr/goleveldb/leveldb.internalKey.ukey(0xc8712930ed, 0x4, 0x20, 0x0, 0x0, 0x0)
	/root/go/src/github.com/syndtr/goleveldb/leveldb/key.go:120 +0x46
github.com/syndtr/goleveldb/leveldb.(*iComparer).Compare(0xc852e100a0, 0xc8712930ed, 0x4, 0x20, 0xcfefca81d0, 0xc, 0xc, 0xc850f5cf60)
	/root/go/src/github.com/syndtr/goleveldb/leveldb/comparer.go:36 +0x4a
github.com/syndtr/goleveldb/leveldb/table.(*block).seek.func1(0x1, 0x8cce81)
	/root/go/src/github.com/syndtr/goleveldb/leveldb/table/reader.go:68 +0x278
sort.Search(0x3, 0xcfefc44bd0, 0xe8)
	/usr/lib/go/src/sort/search.go:66 +0x52
github.com/syndtr/goleveldb/leveldb/table.(*block).seek(0xc87231c1c0, 0x7f8204327100, 0xc852e100a0, 0x0, 0x3, 0xcfefca81d0, 0xc, 0xc, 0xcdcd40, 0x443815, ...)
	/root/go/src/github.com/syndtr/goleveldb/leveldb/table/reader.go:69 +0xa9
github.com/syndtr/goleveldb/leveldb/table.(*blockIter).Seek(0xcfefb9e5a0, 0xcfefca81d0, 0xc, 0xc, 0x0)
	/root/go/src/github.com/syndtr/goleveldb/leveldb/table/reader.go:229 +0x10e
github.com/syndtr/goleveldb/leveldb/table.(*Reader).newBlockIter(0xc8387461a0, 0xc87231c1c0, 0x7f81f57e7168, 0xcb79534140, 0xc87026d3b0, 0x1, 0x0)
	/root/go/src/github.com/syndtr/goleveldb/leveldb/table/reader.go:736 +0x1db
github.com/syndtr/goleveldb/leveldb/table.(*Reader).NewIterator(0xc8387461a0, 0xc87026d3b0, 0x0, 0x0, 0x0)
	/root/go/src/github.com/syndtr/goleveldb/leveldb/table/reader.go:804 +0x356
github.com/syndtr/goleveldb/leveldb.(*tOps).newIterator(0xc86b6ad350, 0xc85214bf90, 0xc87026d3b0, 0x0, 0x0, 0x0)
	/root/go/src/github.com/syndtr/goleveldb/leveldb/table.go:411 +0x1a5
github.com/syndtr/goleveldb/leveldb.(*version).getIterators(0xc844da5cc0, 0xc87026d3b0, 0x0, 0x0, 0x0, 0x0)
	/root/go/src/github.com/syndtr/goleveldb/leveldb/version.go:253 +0x1a7
github.com/syndtr/goleveldb/leveldb.(*DB).newRawIterator(0xc8201f4780, 0x0, 0x0, 0x0, 0x0, 0xc87026d3b0, 0x0, 0x0, 0x0)
	/root/go/src/github.com/syndtr/goleveldb/leveldb/db_iter.go:41 +0xdc
github.com/syndtr/goleveldb/leveldb.(*DB).newIterator(0xc8201f4780, 0x0, 0x0, 0x0, 0x0, 0x16571aa28, 0xcfefc45550, 0x0, 0x4ee651)
	/root/go/src/github.com/syndtr/goleveldb/leveldb/db_iter.go:79 +0x229
github.com/syndtr/goleveldb/leveldb.(*DB).NewIterator(0xc8201f4780, 0xcfefc45550, 0x0, 0x0, 0x0)
	/root/go/src/github.com/syndtr/goleveldb/leveldb/db.go:882 +0x205
github.com/chrislusf/seaweedfs/weed/filer/embedded_filer.(*FileListInLevelDb).ListFiles(0xc82009c138, 0xcf00000000, 0x0, 0x0, 0x64, 0x0, 0x0, 0x0)
	/root/go/src/github.com/chrislusf/seaweedfs/weed/filer/embedded_filer/files_in_leveldb.go:64 +0x48c
github.com/chrislusf/seaweedfs/weed/filer/embedded_filer.(*FilerEmbedded).ListFiles(0xc849337300, 0xcfefca8144, 0x1, 0x0, 0x0, 0x64, 0x0, 0x0, 0x0, 0x0, ...)
	/root/go/src/github.com/chrislusf/seaweedfs/weed/filer/embedded_filer/filer_embedded.go:63 +0xf1
github.com/chrislusf/seaweedfs/weed/server.(*FilerServer).listDirectoryHandler(0xc8201f6000, 0x7f81e2155190, 0xcbbe8221a0, 0xc820156380)
	/root/go/src/github.com/chrislusf/seaweedfs/weed/server/filer_server_handlers_read.go:40 +0x6b1
github.com/chrislusf/seaweedfs/weed/server.(*FilerServer).GetOrHeadHandler(0xc8201f6000, 0x7f81e2155190, 0xcbbe8221a0, 0xc820156380, 0xffffffffffffff01)
	/root/go/src/github.com/chrislusf/seaweedfs/weed/server/filer_server_handlers_read.go:50 +0x150
github.com/chrislusf/seaweedfs/weed/server.(*FilerServer).filerHandler(0xc8201f6000, 0x7f81e2155190, 0xcbbe8221a0, 0xc820156380)
	/root/go/src/github.com/chrislusf/seaweedfs/weed/server/filer_server_handlers.go:10 +0xf6
github.com/chrislusf/seaweedfs/weed/server.(*FilerServer).(github.com/chrislusf/seaweedfs/weed/server.filerHandler)-fm(0x7f81e2155190, 0xcbbe8221a0, 0xc820156380)
	/root/go/src/github.com/chrislusf/seaweedfs/weed/server/filer_server.go:60 +0x3e
net/http.HandlerFunc.ServeHTTP(0xc82ddf86d0, 0x7f81e2155190, 0xcbbe8221a0, 0xc820156380)
	/usr/lib/go/src/net/http/server.go:1618 +0x3a
net/http.(*ServeMux).ServeHTTP(0xc82017eb40, 0x7f81e2155190, 0xcbbe8221a0, 0xc820156380)
	/usr/lib/go/src/net/http/server.go:1910 +0x17d
net/http.serverHandler.ServeHTTP(0xc852de9f00, 0x7f81e2155190, 0xcbbe8221a0, 0xc820156380)
	/usr/lib/go/src/net/http/server.go:2081 +0x19e
net/http.(*conn).serve(0xcd3aaf8080)
	/usr/lib/go/src/net/http/server.go:1472 +0xf2e
created by net/http.(*Server).Serve
	/usr/lib/go/src/net/http/server.go:2137 +0x44e
2017/11/02 01:51:12 http: panic serving 39.36.53.157:46128: leveldb: internal key "\x00\x01d,", len=4: invalid length
goroutine 2467 [running]:
net/http.(*conn).serve.func1(0xc921cf6780)
	/usr/lib/go/src/net/http/server.go:1389 +0xc1
panic(0xaf99a0, 0xcf62fea290)
	/usr/lib/go/src/runtime/panic.go:426 +0x4e9
github.com/syndtr/goleveldb/leveldb.internalKey.assert(0xc8712930ed, 0x4, 0x20)
	/root/go/src/github.com/syndtr/goleveldb/leveldb/key.go:115 +0x22c
github.com/syndtr/goleveldb/leveldb.internalKey.ukey(0xc8712930ed, 0x4, 0x20, 0x0, 0x0, 0x0)
	/root/go/src/github.com/syndtr/goleveldb/leveldb/key.go:120 +0x46
github.com/syndtr/goleveldb/leveldb.(*iComparer).Compare(0xc852e100a0, 0xc8712930ed, 0x4, 0x20, 0xcf62fea250, 0xc, 0xc, 0x0)
	/root/go/src/github.com/syndtr/goleveldb/leveldb/comparer.go:36 +0x4a
github.com/syndtr/goleveldb/leveldb/table.(*block).seek.func1(0x1, 0x8cce81)
	/root/go/src/github.com/syndtr/goleveldb/leveldb/table/reader.go:68 +0x278
sort.Search(0x3, 0xca5cc78bd0, 0xe8)
	/usr/lib/go/src/sort/search.go:66 +0x52
github.com/syndtr/goleveldb/leveldb/table.(*block).seek(0xc87231c1c0, 0x7f8204327100, 0xc852e100a0, 0x0, 0x3, 0xcf62fea250, 0xc, 0xc, 0xcdcd40, 0x443815, ...)
	/root/go/src/github.com/syndtr/goleveldb/leveldb/table/reader.go:69 +0xa9
github.com/syndtr/goleveldb/leveldb/table.(*blockIter).Seek(0xce24069b30, 0xcf62fea250, 0xc, 0xc, 0x0)
	/root/go/src/github.com/syndtr/goleveldb/leveldb/table/reader.go:229 +0x10e
github.com/syndtr/goleveldb/leveldb/table.(*Reader).newBlockIter(0xc8387461a0, 0xc87231c1c0, 0x7f81f57e7168, 0xcb795cc1f8, 0xc87024edb0, 0x1, 0x0)
	/root/go/src/github.com/syndtr/goleveldb/leveldb/table/reader.go:736 +0x1db
github.com/syndtr/goleveldb/leveldb/table.(*Reader).NewIterator(0xc8387461a0, 0xc87024edb0, 0x0, 0x0, 0x0)
	/root/go/src/github.com/syndtr/goleveldb/leveldb/table/reader.go:804 +0x356
github.com/syndtr/goleveldb/leveldb.(*tOps).newIterator(0xc86b6ad350, 0xc85214bf90, 0xc87024edb0, 0x0, 0x0, 0x0)
	/root/go/src/github.com/syndtr/goleveldb/leveldb/table.go:411 +0x1a5
github.com/syndtr/goleveldb/leveldb.(*version).getIterators(0xc844da5cc0, 0xc87024edb0, 0x0, 0x0, 0x0, 0x0)
	/root/go/src/github.com/syndtr/goleveldb/leveldb/version.go:253 +0x1a7
github.com/syndtr/goleveldb/leveldb.(*DB).newRawIterator(0xc8201f4780, 0x0, 0x0, 0x0, 0x0, 0xc87024edb0, 0x0, 0x0, 0x0)
	/root/go/src/github.com/syndtr/goleveldb/leveldb/db_iter.go:41 +0xdc
github.com/syndtr/goleveldb/leveldb.(*DB).newIterator(0xc8201f4780, 0x0, 0x0, 0x0, 0x0, 0x16571aa28, 0xca5cc79550, 0x0, 0x4ee651)
	/root/go/src/github.com/syndtr/goleveldb/leveldb/db_iter.go:79 +0x229
github.com/syndtr/goleveldb/leveldb.(*DB).NewIterator(0xc8201f4780, 0xca5cc79550, 0x0, 0x0, 0x0)
	/root/go/src/github.com/syndtr/goleveldb/leveldb/db.go:882 +0x205
github.com/chrislusf/seaweedfs/weed/filer/embedded_filer.(*FileListInLevelDb).ListFiles(0xc82009c138, 0xcf00000000, 0x0, 0x0, 0x64, 0x0, 0x0, 0x0)
	/root/go/src/github.com/chrislusf/seaweedfs/weed/filer/embedded_filer/files_in_leveldb.go:64 +0x48c
github.com/chrislusf/seaweedfs/weed/filer/embedded_filer.(*FilerEmbedded).ListFiles(0xc849337300, 0xcf62fea1c4, 0x1, 0x0, 0x0, 0x64, 0x0, 0x0, 0x0, 0x0, ...)
	/root/go/src/github.com/chrislusf/seaweedfs/weed/filer/embedded_filer/filer_embedded.go:63 +0xf1
github.com/chrislusf/seaweedfs/weed/server.(*FilerServer).listDirectoryHandler(0xc8201f6000, 0x7f81e2155190, 0xcdf1bf21a0, 0xc82018c8c0)
	/root/go/src/github.com/chrislusf/seaweedfs/weed/server/filer_server_handlers_read.go:40 +0x6b1
github.com/chrislusf/seaweedfs/weed/server.(*FilerServer).GetOrHeadHandler(0xc8201f6000, 0x7f81e2155190, 0xcdf1bf21a0, 0xc82018c8c0, 0xffffffffffffff01)
	/root/go/src/github.com/chrislusf/seaweedfs/weed/server/filer_server_handlers_read.go:50 +0x150
github.com/chrislusf/seaweedfs/weed/server.(*FilerServer).filerHandler(0xc8201f6000, 0x7f81e2155190, 0xcdf1bf21a0, 0xc82018c8c0)
	/root/go/src/github.com/chrislusf/seaweedfs/weed/server/filer_server_handlers.go:10 +0xf6
github.com/chrislusf/seaweedfs/weed/server.(*FilerServer).(github.com/chrislusf/seaweedfs/weed/server.filerHandler)-fm(0x7f81e2155190, 0xcdf1bf21a0, 0xc82018c8c0)
	/root/go/src/github.com/chrislusf/seaweedfs/weed/server/filer_server.go:60 +0x3e
net/http.HandlerFunc.ServeHTTP(0xc82ddf86d0, 0x7f81e2155190, 0xcdf1bf21a0, 0xc82018c8c0)
	/usr/lib/go/src/net/http/server.go:1618 +0x3a
net/http.(*ServeMux).ServeHTTP(0xc82017eb40, 0x7f81e2155190, 0xcdf1bf21a0, 0xc82018c8c0)
	/usr/lib/go/src/net/http/server.go:1910 +0x17d
net/http.serverHandler.ServeHTTP(0xc852de9f00, 0x7f81e2155190, 0xcdf1bf21a0, 0xc82018c8c0)
	/usr/lib/go/src/net/http/server.go:2081 +0x19e
net/http.(*conn).serve(0xc921cf6780)
	/usr/lib/go/src/net/http/server.go:1472 +0xf2e
created by net/http.(*Server).Serve
	/usr/lib/go/src/net/http/server.go:2137 +0x44e
2017/11/02 01:52:23 http: panic serving 39.36.53.157:46142: leveldb: internal key "\x00\x01d,", len=4: invalid length
goroutine 2456 [running]:
net/http.(*conn).serve.func1(0xc97d935b80)
	/usr/lib/go/src/net/http/server.go:1389 +0xc1
panic(0xaf99a0, 0xce2f857050)
	/usr/lib/go/src/runtime/panic.go:426 +0x4e9
github.com/syndtr/goleveldb/leveldb.internalKey.assert(0xc8712930ed, 0x4, 0x20)
	/root/go/src/github.com/syndtr/goleveldb/leveldb/key.go:115 +0x22c
github.com/syndtr/goleveldb/leveldb.internalKey.ukey(0xc8712930ed, 0x4, 0x20, 0x0, 0x0, 0x0)
	/root/go/src/github.com/syndtr/goleveldb/leveldb/key.go:120 +0x46
github.com/syndtr/goleveldb/leveldb.(*iComparer).Compare(0xc852e100a0, 0xc8712930ed, 0x4, 0x20, 0xce2f857010, 0xc, 0xc, 0x0)
	/root/go/src/github.com/syndtr/goleveldb/leveldb/comparer.go:36 +0x4a
github.com/syndtr/goleveldb/leveldb/table.(*block).seek.func1(0x1, 0x8cce81)
	/root/go/src/github.com/syndtr/goleveldb/leveldb/table/reader.go:68 +0x278
sort.Search(0x3, 0xca5cc78bd0, 0xe8)
	/usr/lib/go/src/sort/search.go:66 +0x52
github.com/syndtr/goleveldb/leveldb/table.(*block).seek(0xc87231c1c0, 0x7f8204327100, 0xc852e100a0, 0x0, 0x3, 0xce2f857010, 0xc, 0xc, 0xcdcd40, 0x443815, ...)
	/root/go/src/github.com/syndtr/goleveldb/leveldb/table/reader.go:69 +0xa9
github.com/syndtr/goleveldb/leveldb/table.(*blockIter).Seek(0xce24069ef0, 0xce2f857010, 0xc, 0xc, 0x0)
	/root/go/src/github.com/syndtr/goleveldb/leveldb/table/reader.go:229 +0x10e
github.com/syndtr/goleveldb/leveldb/table.(*Reader).newBlockIter(0xc8387461a0, 0xc87231c1c0, 0x7f81f57e7168, 0xcb795cc478, 0xc8701be090, 0x1, 0x0)
	/root/go/src/github.com/syndtr/goleveldb/leveldb/table/reader.go:736 +0x1db
github.com/syndtr/goleveldb/leveldb/table.(*Reader).NewIterator(0xc8387461a0, 0xc8701be090, 0x0, 0x0, 0x0)
	/root/go/src/github.com/syndtr/goleveldb/leveldb/table/reader.go:804 +0x356
github.com/syndtr/goleveldb/leveldb.(*tOps).newIterator(0xc86b6ad350, 0xc85214bf90, 0xc8701be090, 0x0, 0x0, 0x0)
	/root/go/src/github.com/syndtr/goleveldb/leveldb/table.go:411 +0x1a5
github.com/syndtr/goleveldb/leveldb.(*version).getIterators(0xc844da5cc0, 0xc8701be090, 0x0, 0x0, 0x0, 0x0)
	/root/go/src/github.com/syndtr/goleveldb/leveldb/version.go:253 +0x1a7
github.com/syndtr/goleveldb/leveldb.(*DB).newRawIterator(0xc8201f4780, 0x0, 0x0, 0x0, 0x0, 0xc8701be090, 0x0, 0x0, 0x0)
	/root/go/src/github.com/syndtr/goleveldb/leveldb/db_iter.go:41 +0xdc
github.com/syndtr/goleveldb/leveldb.(*DB).newIterator(0xc8201f4780, 0x0, 0x0, 0x0, 0x0, 0x16571aa28, 0xca5cc79550, 0x0, 0x4ee651)
	/root/go/src/github.com/syndtr/goleveldb/leveldb/db_iter.go:79 +0x229
github.com/syndtr/goleveldb/leveldb.(*DB).NewIterator(0xc8201f4780, 0xca5cc79550, 0x0, 0x0, 0x0)
	/root/go/src/github.com/syndtr/goleveldb/leveldb/db.go:882 +0x205
github.com/chrislusf/seaweedfs/weed/filer/embedded_filer.(*FileListInLevelDb).ListFiles(0xc82009c138, 0xce00000000, 0x0, 0x0, 0x64, 0x0, 0x0, 0x0)
	/root/go/src/github.com/chrislusf/seaweedfs/weed/filer/embedded_filer/files_in_leveldb.go:64 +0x48c
github.com/chrislusf/seaweedfs/weed/filer/embedded_filer.(*FilerEmbedded).ListFiles(0xc849337300, 0xce2f856f64, 0x1, 0x0, 0x0, 0x64, 0x0, 0x0, 0x0, 0x0, ...)
	/root/go/src/github.com/chrislusf/seaweedfs/weed/filer/embedded_filer/filer_embedded.go:63 +0xf1
github.com/chrislusf/seaweedfs/weed/server.(*FilerServer).listDirectoryHandler(0xc8201f6000, 0x7f81e2155190, 0xcc30caf520, 0xcfef0d87e0)
	/root/go/src/github.com/chrislusf/seaweedfs/weed/server/filer_server_handlers_read.go:40 +0x6b1
github.com/chrislusf/seaweedfs/weed/server.(*FilerServer).GetOrHeadHandler(0xc8201f6000, 0x7f81e2155190, 0xcc30caf520, 0xcfef0d87e0, 0xffffffffffffff01)
	/root/go/src/github.com/chrislusf/seaweedfs/weed/server/filer_server_handlers_read.go:50 +0x150
github.com/chrislusf/seaweedfs/weed/server.(*FilerServer).filerHandler(0xc8201f6000, 0x7f81e2155190, 0xcc30caf520, 0xcfef0d87e0)
	/root/go/src/github.com/chrislusf/seaweedfs/weed/server/filer_server_handlers.go:10 +0xf6
github.com/chrislusf/seaweedfs/weed/server.(*FilerServer).(github.com/chrislusf/seaweedfs/weed/server.filerHandler)-fm(0x7f81e2155190, 0xcc30caf520, 0xcfef0d87e0)
	/root/go/src/github.com/chrislusf/seaweedfs/weed/server/filer_server.go:60 +0x3e
net/http.HandlerFunc.ServeHTTP(0xc82ddf86d0, 0x7f81e2155190, 0xcc30caf520, 0xcfef0d87e0)
	/usr/lib/go/src/net/http/server.go:1618 +0x3a
net/http.(*ServeMux).ServeHTTP(0xc82017eb40, 0x7f81e2155190, 0xcc30caf520, 0xcfef0d87e0)
	/usr/lib/go/src/net/http/server.go:1910 +0x17d
net/http.serverHandler.ServeHTTP(0xc852de9f00, 0x7f81e2155190, 0xcc30caf520, 0xcfef0d87e0)
	/usr/lib/go/src/net/http/server.go:2081 +0x19e
net/http.(*conn).serve(0xc97d935b80)
	/usr/lib/go/src/net/http/server.go:1472 +0xf2e
created by net/http.(*Server).Serve
	/usr/lib/go/src/net/http/server.go:2137 +0x44

So Chris says , the leveldb file seems corrupted.

My question is: 

Is there any way to repair those corrupted files, As I have used the above tool, but I made little bit difference only, are there any good tools which can repair those corrupted files, or we are shooting in the dark and it not even possible.

I don't have any knowledge about Golang or leveldb, So pardon me if the question seems silly to all of you. Purpose of posting here is: If you can see the error and suggest about , how to recover that. :)

[Chris Mumford]

For this I would use the latest release of the official leveldb library, and write a small C/C++ application to do the repair. The problem with Python/Go implementations is that they may not use the leveldb library, may not be the current release of the library, or may have their options hard-coded. Also, make sure to set the paranoid_checks option to true when calling RepairDB. I would also set the info_log option so that you can write out detailed status of the repaired db.

I'm not very familiar with RepairDB, but I don't see how it would restore deleted records intentionally. It does look for manifests (which reference the active tables). If the corruption somehow deleted the most recent manifest then the most recent (highest numbered) one will be used and then you'll be using the old tables during the restoration.

[Junaid Farooq]

Hello, Thanks for your reply, I tried that in a few days back,

#python

import leveldb;
leveldb.RepairDB("path")

I did recover but it really recovered everything else. but not repaired actually.

I understand what you are saying at the moment for c++ application, But is there anyway, or any option in repair, to say not recover just repair?