bdb broken, recovery does not fix everything

[Adrian Gschwend]

Yesterday night our openldap-server stopped working properly, we had to
kill the process hard (-9). After the next start bdb didn't start up
properly anymore, so I checked the database files with db_verify. I had
several

Page 1: out-of-order key at entry xxx

messages in some of the bdb-files. After some googling I found the
OpenLDAP FAQ-O-Matic entry
(http://www.openldap.org/faq/data/cache/738.html) with the link to the BDB
docs at http://www.sleepycat.com/docs/ref/transapp/recovery.html

So far so good, I followed the db_recovery hints and with a catastrophic
recovery I could get openldap to start again (it didn't work without
catastropic recovery, even if I didn't run out of diskspace or somthing
alike on this box).

Now OpenLDAP seems to work again for simple stuff, however, I cannot
replicate the full server anymore. The CPU usage goes up to 100% and I
don't get results anyway. So I checked with db_verify again and noticed
that most of the bdb-files are still complaining (out-of-order key...).
db_recover doesn't seem to do anything more however.

Do I miss something here? I think db_verify is correct that the files are
still broken or not?

thanks

Adrian

--
Adrian Gschwend
System Administrator
Berne University of Applied Sciences
Biel, Switzerland

[Adrian Gschwend]

On Mon, 08 Aug 2005 16:00:55 +0200, Adrian Gschwend wrote:

>
> Yesterday night our openldap-server stopped working properly, we had to
> kill the process hard (-9). After the next start bdb didn't start up
> properly anymore, so I checked the database files with db_verify. I had
> several

[...]

What I forgot, I'm running the following versions:
- OpenLDAP 2.2.15
- BDB 4.2.52
- FreeBSD 5.3

[Quanah Gibson-Mount]

Adrian Gschwend <adrian....@bfh.ch> writes:

> On Mon, 08 Aug 2005 16:00:55 +0200, Adrian Gschwend wrote:
>>
>> Yesterday night our openldap-server stopped working properly, we had to
>> kill the process hard (-9). After the next start bdb didn't start up
>> properly anymore, so I checked the database files with db_verify. I had
>> several
> [...]

> What I forgot, I'm running the following versions:
> - OpenLDAP 2.2.15
> - BDB 4.2.52
> - FreeBSD 5.3

I assume then you know that it is recommended to use the *latest* version
of OpenLDAP (Say, 2.2.26 (stable) or 2.2.27) for the 2.2 tree; and that to
use BDB 4.2.52, you need to patch it with the patches provided by
sleepycat.

Any time you hard kill OpenLDAP with a -9, you *must* run db_recover
before restarting slapd (This is fixed in OpenLDAP 2.3, which has an auto
recover detector).

--Quanah

--
Quanah Gibson-Mount
Principal Software Developer
ITSS/Shared Services
Stanford University
GnuPG Public Key: http://www.stanford.edu/~quanah/pgp.html

[Adrian Gschwend]

On Mon, 08 Aug 2005 09:38:05 -0700, Quanah Gibson-Mount wrote:

> I assume then you know that it is recommended to use the *latest* version
> of OpenLDAP (Say, 2.2.26 (stable) or 2.2.27) for the 2.2 tree; and that to
> use BDB 4.2.52, you need to patch it with the patches provided by
> sleepycat.

yes, I just updated OpenLDAP 2.2.27 and BDB 4.2.52 last night. Didn't fix
more however.

> Any time you hard kill OpenLDAP with a -9, you *must* run db_recover
> before restarting slapd (This is fixed in OpenLDAP 2.3, which has an auto
> recover detector).

ah ok, thanks for the hint. But what about db_verify, how can it be that
it still reports errors after db_recover?

Thanks

Adrian

[Adrian Gschwend]

On Mon, 08 Aug 2005 16:00:55 +0200, Adrian Gschwend wrote:

[...]

> So far so good, I followed the db_recovery hints and with a catastrophic
> recovery I could get openldap to start again (it didn't work without
> catastropic recovery, even if I didn't run out of diskspace or somthing
> alike on this box).

some more information on that: The OpenLDAP server is back online (2.2.27
now, BDB is 4.2.52). Unfortunately db_verify still complains in quite some
tables, most of the time it's something like this:

Page xxx: out-of-order key at entry yyy

db_recover does nothing at all, db_recover -c neither, it does fix
stuff after I have to kill slapd with -9, but that doesn't seem to fix the
out-of-order key stuff.

So I tried another approach: db_dump of each bdb file, db_load of the
dump-file and recreation of a new bdb file. I did both the dump and the
load without any specific options.
There was no error message or anything so I assumed that the recreation of
the files went just fine. When I load those bdb files into openldap I
won't get much however, all user entries are missing and half of the OUs
are missing too. The id2entry.bdb file however is 4.6MB, while it was 5.4
before (did all of this on a backup for sure).
db_verify does not complain anymore however.

so, does anyone see any chance to get my DB back in a usable state or do I
have to recreate the data?

btw I couldn't find the -r option in db_load, is this a new feature of
4.3? I thought I want to give that option a try.

[michael...@gmail.com]

Hi Adrian,

You'll need to run db_verify with the "-o" flag:

http://www.sleepycat.com/docs/utility/db_verify.html

-o
Skip the database checks for btree and duplicate sort order and for
hashing.
If the file being verified contains databases with non-default
comparison or
hashing configurations, calling the db_verify utility without the -o
flag will
usually return failure. The -o flag causes db_verify to ignore
database sort
or hash ordering and allows db_verify to be used on these files. To
fully
verify these files, verify them explicitly using the DB->verify
method, after
configuring the correct comparison or hashing functions.

Regards,
Michael.

[Howard Chu]

Also note, in OpenLDAP 2.3 we have eliminated all use of custom sort
functions, so that db_verify / db_dump / db_load can all be used safely,
and the database files are now fully byte-order independent.

But OpenLDAP 2.2 still has a number of these dependencies.

--
-- Howard Chu
Chief Architect, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc
OpenLDAP Core Team http://www.openldap.org/project/