recovering a corrupted index?

[Dan Kortschak]

I have an ancient instance of camlistore (that ancient) that I want to
check whether I can discard/migrate. When the server starts, it fails
with the following?

I have tried to reindex, but this panics as well. I'm wondering what
the best way of salvaging the blobs would be at least to be able to
look at them to see whether I want to keep them.

thanks
Dan

```
2021/05/08 03:39:05 Starting camlistored version 2016-09-27-3e1edc1; Go
go1.7.1 (linux/amd64)
2021/05/08 03:39:05 TLS enabled, with SHA-256 certificate fingerprint:
7a8f78061f778fd50889
2021/05/08 03:39:05 Starting to listen on https://localhost:3179
2021/05/08 03:39:05 Caught panic installer handlers: error
instantiating storage for prefix "/bs/", type "blobpacked": failed to
setup blobpacked metaIndex: error from "leveldb" KeyValue: leveldb:
manifest corrupted (field 'comparer'): missing [file=MANIFEST-000281]
goroutine 1 [running]:
runtime/debug.Stack(0xc420056050, 0x2, 0xc420315000)
/home/user/go/src/runtime/debug/stack.go:24 +0x79
runtime/debug.PrintStack()
/home/user/go/src/runtime/debug/stack.go:16 +0x22
camlistore.org/pkg/serverinit.(*Config).InstallHandlers.func1(0xc4202df
bc0)
/home/user/src/camlistore.org/tmp/build-gopath-
nosqlite/src/camlistore.org/pkg/serverinit/serverinit.go:525 +0xa8
panic(0xcced60, 0xc420a375e0)
/home/user/go/src/runtime/panic.go:458 +0x243
camlistore.org/pkg/serverinit.(*handlerLoader).setupHandler.func1(0xc42
09ff140, 0xc4201bd000, 0xe3bed7, 0xb)
/home/user/src/camlistore.org/tmp/build-gopath-
nosqlite/src/camlistore.org/pkg/serverinit/serverinit.go:307 +0x1b4
panic(0xcced60, 0xc420a375e0)
/home/user/go/src/runtime/panic.go:458 +0x243
camlistore.org/pkg/serverinit.(*handlerLoader).setupHandler.func1(0xc42
09ff440, 0xc4201bd000, 0xe3ee18, 0xe)
/home/user/src/camlistore.org/tmp/build-gopath-
nosqlite/src/camlistore.org/pkg/serverinit/serverinit.go:307 +0x1b4
panic(0xcced60, 0xc420a375e0)
/home/user/go/src/runtime/panic.go:458 +0x243
camlistore.org/pkg/serverinit.(*handlerLoader).setupHandler.func1(0xc42
09ff740, 0xc4201bd000, 0xe34422, 0x4)
/home/user/src/camlistore.org/tmp/build-gopath-
nosqlite/src/camlistore.org/pkg/serverinit/serverinit.go:307 +0x1b4
panic(0xcced60, 0xc420a375e0)
/home/user/go/src/runtime/panic.go:458 +0x243
camlistore.org/pkg/serverinit.exitFailure(0xe6548f, 0x36, 0xc4202dea90,
0x3, 0x3)
/home/user/src/camlistore.org/tmp/build-gopath-
nosqlite/src/camlistore.org/pkg/serverinit/serverinit.go:276 +0xef
camlistore.org/pkg/serverinit.(*handlerLoader).setupHandler(0xc4201bd00
0, 0xe34422, 0x4)
/home/user/src/camlistore.org/tmp/build-gopath-
nosqlite/src/camlistore.org/pkg/serverinit/serverinit.go:319 +0x9c7
camlistore.org/pkg/serverinit.(*handlerLoader).GetStorage(0xc4201bd000,
0xe34422, 0x4, 0x2, 0x2, 0x0, 0xc4202decc0)
/home/user/src/camlistore.org/tmp/build-gopath-
nosqlite/src/camlistore.org/pkg/serverinit/serverinit.go:250 +0x49
camlistore.org/pkg/blobserver/replica.newFromConfig(0x1507400,
0xc4201bd000, 0xc420a29d40, 0x7, 0xc4205f2308, 0xc4202ded01,
0xc4202ded70)
/home/user/src/camlistore.org/tmp/build-gopath-
nosqlite/src/camlistore.org/pkg/blobserver/replica/replica.go:112
+0x270
camlistore.org/pkg/blobserver.CreateStorage(0xe40b4f, 0x7, 0x1507400,
0xc4201bd000, 0xc420a29d40, 0xe, 0x1, 0x2, 0x7f60c1148750)
/home/user/src/camlistore.org/tmp/build-gopath-
nosqlite/src/camlistore.org/pkg/blobserver/registry.go:111 +0xc3
camlistore.org/pkg/serverinit.(*handlerLoader).setupHandler(0xc4201bd00
0, 0xe3ee18, 0xe)
/home/user/src/camlistore.org/tmp/build-gopath-
nosqlite/src/camlistore.org/pkg/serverinit/serverinit.go:316 +0x24d
camlistore.org/pkg/serverinit.(*handlerLoader).GetStorage(0xc4201bd000,
0xe3ee18, 0xe, 0x14fb080, 0xc42094eb60, 0x1502c80, 0xc42094eb20)
/home/user/src/camlistore.org/tmp/build-gopath-
nosqlite/src/camlistore.org/pkg/serverinit/serverinit.go:250 +0x49
camlistore.org/pkg/jsonsign/signhandler.newJSONSignFromConfig(0x1507400
, 0xc4201bd000, 0xc420a29a40, 0x8, 0xc420001c28, 0xc4202df301,
0xc4202df358)
/home/user/src/camlistore.org/tmp/build-gopath-
nosqlite/src/camlistore.org/pkg/jsonsign/signhandler/sig.go:109 +0x6e0
camlistore.org/pkg/blobserver.CreateHandler(0xe39517, 0x8, 0x1507400,
0xc4201bd000, 0xc420a29a40, 0xb, 0x0, 0x1, 0x0)
/home/user/src/camlistore.org/tmp/build-gopath-
nosqlite/src/camlistore.org/pkg/blobserver/registry.go:140 +0xc3
camlistore.org/pkg/serverinit.(*handlerLoader).setupHandler(0xc4201bd00
0, 0xe3bed7, 0xb)
/home/user/src/camlistore.org/tmp/build-gopath-
nosqlite/src/camlistore.org/pkg/serverinit/serverinit.go:365 +0xa96
camlistore.org/pkg/serverinit.(*handlerLoader).setupAll(0xc4201bd000)
/home/user/src/camlistore.org/tmp/build-gopath-
nosqlite/src/camlistore.org/pkg/serverinit/serverinit.go:230 +0x9a
camlistore.org/pkg/serverinit.(*Config).InstallHandlers(0xc420a07e80,
0x14f7e80, 0xc4201bc280, 0xc4209aeba0, 0x16, 0x0, 0x0, 0x0, 0x0, 0x0,
...)
/home/user/src/camlistore.org/tmp/build-gopath-
nosqlite/src/camlistore.org/pkg/serverinit/serverinit.go:591 +0x451
main.Main(0x0, 0x0)
/home/user/src/camlistore.org/tmp/build-gopath-
nosqlite/src/camlistore.org/server/camlistored/camlistored.go:421
+0x5d3
main.main()
/home/user/src/camlistore.org/tmp/build-gopath-
nosqlite/src/camlistore.org/server/camlistored/camlistored.go:354 +0x33
Error parsing config: Caught panic: error instantiating storage for
prefix "/bs/", type "blobpacked": failed to setup blobpacked metaIndex:
error from "leveldb" KeyValue: leveldb: manifest corrupted (field
'comparer'): missing [file=MANIFEST-000281]
```

[tgulacsi78]

blobpacked has a SetRecovery (in latest Perkeep), but I don't see a mode to set it.

The easiest would be the following patch, then `go install ./server/perkeepd` and then `env RECOVERY=1 perkeepd` or `env RECOVERY=2 perkeepd`.

diff --git a/pkg/blobserver/blobpacked/blobpacked.go b/pkg/blobserver/blobpacked/blobpacked.go
index 7fc364fc7..f401c0f15 100644
--- a/pkg/blobserver/blobpacked/blobpacked.go
+++ b/pkg/blobserver/blobpacked/blobpacked.go
@@ -163,6 +163,11 @@ var (
        recoveryMu sync.Mutex
        recovery   = NoRecovery
 )
+func init() {
+       if n, err := strconv.Atoi(os.Getenv("RECOVERY")); err == nil && n != 0 {
+               SetRecovery(RecoveryMode(n))
+       }
+}
 

 // TODO(mpl): make SetRecovery a method of type storage if we ever export it.

[Dan Kortschak]

On Sat, 2021-05-08 at 11:05 -0700, tgulacsi78 wrote:
> blobpacked has a SetRecovery (in latest Perkeep), but I don't see a
> mode to set it.
> The easiest would be the following patch, then `go install
> ./server/perkeepd` and then `env RECOVERY=1 perkeepd` or `env
> RECOVERY=2 perkeepd`.
>
> diff --git a/pkg/blobserver/blobpacked/blobpacked.go
> b/pkg/blobserver/blobpacked/blobpacked.go
> index 7fc364fc7..f401c0f15 100644
> --- a/pkg/blobserver/blobpacked/blobpacked.go
> +++ b/pkg/blobserver/blobpacked/blobpacked.go
> @@ -163,6 +163,11 @@ var (
> recoveryMu sync.Mutex
> recovery = NoRecovery
> )
> +func init() {
> + if n, err := strconv.Atoi(os.Getenv("RECOVERY")); err == nil
> && n != 0 {
> + SetRecovery(RecoveryMode(n))
> + }
> +}
>
> // TODO(mpl): make SetRecovery a method of type storage if we ever
> export it.

This is set by the recovery flag here,
https://github.com/perkeep/perkeep/blob/dd14e9ea063054c9230218c2311cea15ba7f9284/server/perkeepd/perkeepd.go#L364-L371

The output that I showed (I should have posted this) comes from running
camlistored with recover=2 and reindex=true.

```
$ bin/camlistored -recovery 2 -reindex
2021/05/08 21:17:54 Starting camlistored version 2016-09-27-3e1edc1; Go
go1.7.1 (linux/amd64)
2021/05/08 21:17:54 TLS enabled, with SHA-256 certificate fingerprint:
7a8f78061f778fd50889
2021/05/08 21:17:54 Starting to listen on https://localhost:3179
2021/05/08 21:17:54 Caught panic installer handlers: error
instantiating handler for prefix "/sync/", type "sync": error from

"leveldb" KeyValue: leveldb: manifest corrupted (field 'comparer'):

missing [file=MANIFEST-000283]
goroutine 1 [running]:
runtime/debug.Stack(0xc420056050, 0x2, 0xc4209c5450)

/home/user/go/src/runtime/debug/stack.go:24 +0x79
runtime/debug.PrintStack()
/home/user/go/src/runtime/debug/stack.go:16 +0x22

camlistore.org/pkg/serverinit.(*Config).InstallHandlers.func1(0xc420251

bc0)
/home/user/src/camlistore.org/tmp/build-gopath-
nosqlite/src/camlistore.org/pkg/serverinit/serverinit.go:525 +0xa8

panic(0xcced60, 0xc42046cd60)
/home/user/go/src/runtime/panic.go:458 +0x243
camlistore.org/pkg/serverinit.(*handlerLoader).setupHandler.func1(0xc42
0115950, 0xc42007fd80, 0xe35f06, 0x6)
/home/user/src/camlistore.org/tmp/build-gopath-
nosqlite/src/camlistore.org/pkg/serverinit/serverinit.go:307 +0x1b4
panic(0xcced60, 0xc42046cd60)
/home/user/go/src/runtime/panic.go:458 +0x243
camlistore.org/pkg/serverinit.exitFailure(0xe65459, 0x36, 0xc420251630,

0x3, 0x3)
/home/user/src/camlistore.org/tmp/build-gopath-
nosqlite/src/camlistore.org/pkg/serverinit/serverinit.go:276 +0xef

camlistore.org/pkg/serverinit.(*handlerLoader).setupHandler(0xc42007fd8
0, 0xe35f06, 0x6)
/home/user/src/camlistore.org/tmp/build-gopath-
nosqlite/src/camlistore.org/pkg/serverinit/serverinit.go:368 +0x1272
camlistore.org/pkg/serverinit.(*handlerLoader).setupAll(0xc42007fd80)
/home/user/src/camlistore.org/tmp/build-gopath-
nosqlite/src/camlistore.org/pkg/serverinit/serverinit.go:230 +0x9a
camlistore.org/pkg/serverinit.(*Config).InstallHandlers(0xc420278fc0,
0x14f7e80, 0xc42007f000, 0xc420477c40, 0x16, 0x0, 0x0, 0x0, 0x0, 0x0,

...)
/home/user/src/camlistore.org/tmp/build-gopath-
nosqlite/src/camlistore.org/pkg/serverinit/serverinit.go:591 +0x451
main.Main(0x0, 0x0)
/home/user/src/camlistore.org/tmp/build-gopath-
nosqlite/src/camlistore.org/server/camlistored/camlistored.go:421
+0x5d3
main.main()
/home/user/src/camlistore.org/tmp/build-gopath-
nosqlite/src/camlistore.org/server/camlistored/camlistored.go:354 +0x33

Error parsing config: Caught panic: error instantiating handler for
prefix "/sync/", type "sync": error from "leveldb" KeyValue: leveldb:
manifest corrupted (field 'comparer'): missing [file=MANIFEST-000283]
```

[Tamás Gulácsi]

What if you first start it with "-recovery 2" only (no reindex)?

If that fails, you may have to remove the blobpacked's leveldb dir
(but I doubt, it should be wiped automatically).

If that passes, then you should start the reindex.

If not, you may have to simplify your config first (I see some traces of
replica for example), maybe even opt out from the index, to stabilize
the storage, and add the bells and whistles back one at a time.

Camlistore in its entirety is quite a bit of magic, but, fortunately,
it's been built up from wooden blocks :)

> --
> You received this message because you are subscribed to the Google
> Groups "Perkeep" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to perkeep+u...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/perkeep/f8608748904de9f07c534e7f6306dcf6c7051438.camel%40kortschak.io.

[Dan Kortschak]

On Sun, 2021-05-09 at 06:55 +0200, Tamás Gulácsi wrote:
> What if you first start it with "-recovery 2" only (no reindex)?

This also fails.

> If that fails, you may have to remove the blobpacked's leveldb dir
> (but I doubt, it should be wiped automatically).

This too.

> If that passes, then you should start the reindex.
>
> If not, you may have to simplify your config first (I see some traces
> of
> replica for example),

The server config it pretty vanilla and there have never been any
replicas. Here is it with secrets smooshed.

{
"auth": "userpass:user:pass:vivify=otherpass:+localhost",
"listen": ":3179",
"identity": "nnnnnnnnn",
"identitySecretRing": "/path/to/.config/camlistore/identity-
secring.gpg",
"blobPath": "/path/to/var/camlistore/blobs",
"packRelated": true,
"levelDB": "/path/to/var/camlistore/index.leveldb",
"dbNames": null,
"https": true,
"httpsCert": "/path/to/.config/camlistore/camlistore.crt",
"httpsKey": "/path/to/.config/camlistore/camlistore.key"

}

> maybe even opt out from the index, to stabilize
> the storage, and add the bells and whistles back one at a time.

How do I run it without the index.

thanks

[Tamás Gulácsi]

https://perkeep.org/doc/server-config

runIndex: false

> --
> You received this message because you are subscribed to the Google
> Groups "Perkeep" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to perkeep+u...@googlegroups.com.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/perkeep/4ec57d87ed683f57d91a37595a07302d672de933.camel%40kortschak.io.