Bug#598361: slapd: slapcat gives "unclean shutdown detected; attempting recovery" after squeeze upgrade
Bryan K. Walton
Version: 2.4.23-6
Severity: normal
I've recently upgraded slapd from Lenny to Squeeze. Nightly, I run
a slapcat to dump to directory to an LDIF file for backup. Since
the upgrade, I now get the following error:
bdb_db_open: database "dc=i-clic,dc=uihc,dc=uiowa,dc=edu": unclean
shutdown detected; attempting recovery.
bdb_db_open: database "dc=i-clic,dc=uihc,dc=uiowa,dc=edu": recovery
skipped in read-only mode. Run manual recovery if errors are
encountered.
I get this error every time. I also get this error when I run slaptest.
After doing some research online, I have installed db4.8-util and have
run:
db4.8-recover
db4.8-checkpoint -1
These commands seem to run fine. I have verified that my slapd instance
is using 4.8 (4.8.30). However, I can immediately run the slapcat or
slaptest after running the above commands and the same error repeats
itself.
-- System Information:
Debian Release: squeeze/sid
APT prefers testing
APT policy: (500, 'testing')
Architecture: amd64 (x86_64)
Kernel: Linux 2.6.32.22 (SMP w/4 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Versions of packages slapd depends on:
ii adduser 3.112 add and remove users and groups
ii coreutils 8.5-1 GNU core utilities
ii debconf [debconf-2.0] 1.5.35 Debian configuration management sy
ii libc6 2.11.2-6 Embedded GNU C Library: Shared lib
ii libdb4.8 4.8.30-2 Berkeley v4.8 Database Libraries [
ii libgnutls26 2.8.6-1 the GNU TLS library - runtime libr
ii libldap-2.4-2 2.4.23-6 OpenLDAP libraries
ii libltdl7 2.2.6b-2 A system independent dlopen wrappe
ii libperl5.10 5.10.1-14 shared Perl library
ii libsasl2-2 2.1.23.dfsg1-6 Cyrus SASL - authentication abstra
ii libslp1 1.2.1-7.8 OpenSLP libraries
ii libwrap0 7.6.q-19 Wietse Venema's TCP wrappers libra
ii lsb-base 3.2-23.1 Linux Standard Base 3.2 init scrip
ii perl [libmime-base64-perl 5.10.1-14 Larry Wall's Practical Extraction
ii psmisc 22.11-1 utilities that use the proc file s
ii unixodbc 2.2.14p2-1 ODBC tools libraries
Versions of packages slapd recommends:
ii libsasl2-modules 2.1.23.dfsg1-6 Cyrus SASL - pluggable authenticat
Versions of packages slapd suggests:
ii ldap-utils 2.4.23-6 OpenLDAP utilities
-- Configuration Files:
/etc/default/slapd changed:
SLAPD_CONF="/etc/ldap/slapd.conf"
SLAPD_USER="openldap"
SLAPD_GROUP="openldap"
SLAPD_PIDFILE=
SLAPD_SERVICES="ldap:/// ldaps:/// ldapi:///"
SLAPD_SENTINEL_FILE=/etc/ldap/noslapd
SLAPD_OPTIONS=""
-- debconf information:
slapd/tlsciphersuite:
slapd/password_mismatch:
slapd/invalid_config: true
shared/organization: i-clic.uihc.uiowa.edu
slapd/upgrade_slapcat_failure:
slapd/slurpd_obsolete:
slapd/backend: HDB
slapd/dump_database: when needed
slapd/allow_ldap_v2: false
slapd/no_configuration: false
slapd/move_old_database: true
slapd/suffix_change: false
slapd/dump_database_destdir: /var/backups/slapd-VERSION
slapd/purge_database: false
slapd/domain: i-clic.uihc.uiowa.edu
--
To UNSUBSCRIBE, email to debian-bugs-...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org
Matthijs Mohlmann
> Package: slapd
> Version: 2.4.23-6
> Severity: normal
>
>
> I've recently upgraded slapd from Lenny to Squeeze. Nightly, I run
> a slapcat to dump to directory to an LDIF file for backup. Since
> the upgrade, I now get the following error:
>
> bdb_db_open: database "dc=i-clic,dc=uihc,dc=uiowa,dc=edu": unclean
> shutdown detected; attempting recovery.
> bdb_db_open: database "dc=i-clic,dc=uihc,dc=uiowa,dc=edu": recovery
> skipped in read-only mode. Run manual recovery if errors are
> encountered.
>
> I get this error every time. I also get this error when I run slaptest.
>
> After doing some research online, I have installed db4.8-util and have
> run:
>
> db4.8-recover
> db4.8-checkpoint -1
>
> These commands seem to run fine. I have verified that my slapd instance
> is using 4.8 (4.8.30). However, I can immediately run the slapcat or
> slaptest after running the above commands and the same error repeats
> itself.
Hi,
Do you before you backup, shutdown the slapd process ?
/etc/init.d/slapd stop
slapcat ...
/etc/init.d/slapd start
If you don't, then this is probably the reason for this error.
Regards,
Matthijs Möhlmann
Walton, Bryan K
> Do you before you backup, shutdown the slapd process ?
>
> /etc/init.d/slapd stop
> slapcat ...
> /etc/init.d/slapd start
>
> If you don't, then this is probably the reason for this error.
Hi Matthijs,
Thanks for the reply. You are correct. If I shutdown slapd first, no
error is generated. I can start doing this. However, I didn't think
this was necessary. From the slapcat man page:
"For some backend types, your slapd(8) should not be running (at least,
not in read-write mode) when you do this to ensure consistency of the
database. It is always safe to run slapcat with the slapd-bdb(5),
slapd-hdb(5), and slapd-null(5) backends."
Is this no longer accurate?
-Bryan
Quanah Gibson-Mount
<bryan-...@uiowa.edu> wrote:
> Thanks for the reply. You are correct. If I shutdown slapd first, no
> error is generated. I can start doing this. However, I didn't think
> this was necessary. From the slapcat man page:
>
> "For some backend types, your slapd(8) should not be running (at least,
> not in read-write mode) when you do this to ensure consistency of the
> database. It is always safe to run slapcat with the slapd-bdb(5),
> slapd-hdb(5), and slapd-null(5) backends."
>
> Is this no longer accurate?
If you run db_recover while slapd is running, you'll likely corrupt your
database. However, it is perfectly fine to run slapcat while slapd is
running.
--Quanah
--
Quanah Gibson-Mount
Principal Software Engineer
Zimbra, Inc
--------------------
Zimbra :: the leader in open source messaging and collaboration
Matthijs Möhlmann
> On Tue, 2010-09-28 at 15:58 +0200, Matthijs Mohlmann wrote:
>
>> Do you before you backup, shutdown the slapd process ?
>>
>> /etc/init.d/slapd stop
>> slapcat ...
>> /etc/init.d/slapd start
>>
>> If you don't, then this is probably the reason for this error.
>
> Hi Matthijs,
>
> Thanks for the reply. You are correct. If I shutdown slapd first, no
> error is generated. I can start doing this. However, I didn't think
> this was necessary. From the slapcat man page:
>
> "For some backend types, your slapd(8) should not be running (at least,
> not in read-write mode) when you do this to ensure consistency of the
> database. It is always safe to run slapcat with the slapd-bdb(5),
> slapd-hdb(5), and slapd-null(5) backends."
>
> Is this no longer accurate?
>
> -Bryan
>
Bryan:
Asking one of the OpenLDAP developers.
Quanah, can you comment on this ?
Regards,
Matthijs Mohlmann
Quanah Gibson-Mount
<qua...@zimbra.com> wrote:
> If you run db_recover while slapd is running, you'll likely corrupt your
> database. However, it is perfectly fine to run slapcat while slapd is
> running.
Finally, I would note I receive no such error while using slapcat with my
own build of OpenLDAP 2.4.23:
zimbra@zre-ldap002:~$ /opt/zimbra/openldap/sbin/slapcat -F
/opt/zimbra/data/ldap/config -l /tmp/test.out -b ''
zimbra@zre-ldap002:~$
Walton, Bryan K
> Finally, I would note I receive no such error while using slapcat with my
> own build of OpenLDAP 2.4.23:
>
> zimbra@zre-ldap002:~$ /opt/zimbra/openldap/sbin/slapcat -F
> /opt/zimbra/data/ldap/config -l /tmp/test.out -b ''
> zimbra@zre-ldap002:~$
Hi Quanah, I'll acknowledge that I this doesn't seem universal. This is
only happening on our master ldap server. On the slave, we are get no
such error. Both are running the current slapd that exists in Squeeze.
Still, the master does give the error. Running db4.8_recover doesn't
fix whatever the problem is (if there is really a problem). Any ideas
how this can be fixed? Is this possibly a bug in BDB 4.8 rather than
slapd?
Thanks,
Bryan
Quanah Gibson-Mount
<bryan-...@uiowa.edu> wrote:
> On Tue, 2010-09-28 at 08:29 -0700, Quanah Gibson-Mount wrote:
>
>> Finally, I would note I receive no such error while using slapcat with
>> my own build of OpenLDAP 2.4.23:
>>
>> zimbra@zre-ldap002:~$ /opt/zimbra/openldap/sbin/slapcat -F
>> /opt/zimbra/data/ldap/config -l /tmp/test.out -b ''
>> zimbra@zre-ldap002:~$
>
> Hi Quanah, I'll acknowledge that I this doesn't seem universal. This is
> only happening on our master ldap server. On the slave, we are get no
> such error. Both are running the current slapd that exists in Squeeze.
>
> Still, the master does give the error. Running db4.8_recover doesn't
> fix whatever the problem is (if there is really a problem). Any ideas
> how this can be fixed? Is this possibly a bug in BDB 4.8 rather than
> slapd?
Well, the server I ran slapcat on is a master server, so I don't think
that's relevant. ;) Also, numerous people run OpenLDAP with bdb 4.8 and
haven't reported such an error. So it's unlikely it is a 4.8 issue,
although it could be possible.
Is there any difference in file system between the master and replicas?
I.e., nfs or ext4 vs ext3 etc?
--Quanah
--
Quanah Gibson-Mount
Principal Software Engineer
Zimbra, Inc
--------------------
Zimbra :: the leader in open source messaging and collaboration
--
Walton, Bryan K
>
> Is there any difference in file system between the master and replicas?
> I.e., nfs or ext4 vs ext3 etc?
Yes. The one giving the error is on an ext3. The one without an error
is xfs. I just took the liberty of unmounting the ext3 filesystem on
the machine in question and forcing a filesystem check which came back
clean. After staring slapd up again, I still get the error.
-Bryan
Quanah Gibson-Mount
<bryan-...@uiowa.edu> wrote:
> On Tue, 2010-09-28 at 09:00 -0700, Quanah Gibson-Mount wrote:
>
>>
>> Is there any difference in file system between the master and replicas?
>> I.e., nfs or ext4 vs ext3 etc?
>
> Yes. The one giving the error is on an ext3. The one without an error
> is xfs. I just took the liberty of unmounting the ext3 filesystem on
> the machine in question and forcing a filesystem check which came back
> clean. After staring slapd up again, I still get the error.
Please try the following:
(a) stop slapd
(b) run db_recover on the database
(c) remove the alock file in the database directory
(d) start slapd
(e) run slapcat
Thanks,
Quanah
--
Quanah Gibson-Mount
Principal Software Engineer
Zimbra, Inc
--------------------
Zimbra :: the leader in open source messaging and collaboration
--
Walton, Bryan K
> Please try the following:
>
> (a) stop slapd
> (b) run db_recover on the database
> (c) remove the alock file in the database directory
> (d) start slapd
> (e) run slapcat
I just completed this. Still get the same error.
-Bryan
Quanah Gibson-Mount
<bryan-...@uiowa.edu> wrote:
> On Tue, 2010-09-28 at 09:27 -0700, Quanah Gibson-Mount wrote:
>
>> Please try the following:
>>
>> (a) stop slapd
>> (b) run db_recover on the database
>> (c) remove the alock file in the database directory
>> (d) start slapd
>> (e) run slapcat
>
> I just completed this. Still get the same error.
Ok, well, that rules out a corrupted alock file (which I've had happen a
few times).
The last thing I can think of, just to verify the database didn't get into
an odd place, would be to reload the master. I'd spot check the LDIF file
to make sure it looked fine.
--Quanah
--
Quanah Gibson-Mount
Principal Software Engineer
Zimbra, Inc
--------------------
Zimbra :: the leader in open source messaging and collaboration
--
Walton, Bryan K
>
> The last thing I can think of, just to verify the database didn't get into
> an odd place, would be to reload the master. I'd spot check the LDIF file
> to make sure it looked fine.
Thanks. That didn't seem to make any difference. LDIF file seems
perfectly correct. I dumped both directories (master and slave), and
LDIFs were identical. Yet, slapcat errors on the master and not the
slave. Anyway, I repopulated the directory and still get the error.
BTW, there is one other difference between the two servers, the master
is 64-bit, while the slave is 32-bit.
The other thing I find odd is that the error only occurs when slapd is
running. If there was really a problem, wouldn't the error also occur
when doing a slapcat while slapd is stopped?
Thanks,
Bryan
Quanah Gibson-Mount
<bryan-...@uiowa.edu> wrote:
> On Tue, 2010-09-28 at 09:42 -0700, Quanah Gibson-Mount wrote:
>
>>
>> The last thing I can think of, just to verify the database didn't get
>> into an odd place, would be to reload the master. I'd spot check the
>> LDIF file to make sure it looked fine.
>
> Thanks. That didn't seem to make any difference. LDIF file seems
> perfectly correct. I dumped both directories (master and slave), and
> LDIFs were identical. Yet, slapcat errors on the master and not the
> slave. Anyway, I repopulated the directory and still get the error.
>
> BTW, there is one other difference between the two servers, the master
> is 64-bit, while the slave is 32-bit.
>
> The other thing I find odd is that the error only occurs when slapd is
> running. If there was really a problem, wouldn't the error also occur
> when doing a slapcat while slapd is stopped?
Yeah, I really have no idea why you are seeing this. I would note that a
32-bit binary would not work well with a 64-bit built database (or vice
versa), but I assume both slapd and slapcat in your case are 64-bit.
--Quanah
--
Quanah Gibson-Mount
Principal Software Engineer
Zimbra, Inc
--------------------
Zimbra :: the leader in open source messaging and collaboration
--
Walton, Bryan K
>
> Yeah, I really have no idea why you are seeing this. I would note that a
> 32-bit binary would not work well with a 64-bit built database (or vice
> versa), but I assume both slapd and slapcat in your case are 64-bit.
Yes. The box is running Debian Squeeze with all packages coming from the
AMD64 Debian release. Both slapd and slapcat binaries come from this deb
package:
slapd_2.4.23-6_amd64.deb
-Bryan