KDC 1.15 startup error: Invalid credentials - while initializing database

[Jaap Winius]

Hi folks,

My plan is to migrate away from three older Debian wheezy systems
running MIT Kerberos 1.10.1+dfsg-5+deb7u7 with an OpenLDAP
2.4.31-2+deb7u2 backend to Debian stretch. The idea it to start by
adding a slave system based on MIT Kerberos 1.15-1 and OpenLDAP
2.4.44+dfsg-3. Only, there's this problem... :-)

Setting up the OpenLDAP backend on the stretch system went fine and a
copy of the DIT, which includes a fresh copy of the Kerberos database,
is present. But, when I attempt to start up the new KDC it fails with:

krb5kdc: cannot initialize realm EXAMPLE.COM - see log file for details

The Kerberos log says:

krb5kdc: Cannot bind to LDAP server 'ldapi://' as
'cn=kdc-srv,ou=krb5,dc=example,dc=com':
Invalid credentials - while initializing database for realm EXAMPLE.COM

The Kerberos master is kls1.example.com and the new slave is
kls4.example.com. The Kerberos configuration on the latter is
essentially the same as on the older slaves, kls2 and kls3. Here's the
/etc/krb5.conf on kls4:

[libdefaults]
default_realm = EXAMPLE.COM
forwardable = true
proxiable = true
allow_weak_crypto = true

[realms]
EXAMPLE.COM = {
kdc = kls4.example.com
admin_server = klsm.example.com
database_module = openldap_ldapconf
}

[domain_realm]
.example.com = EXAMPLE.COM
example.com = EXAMPLE.COM

[dbdefaults]
ldap_kerberos_container_dn = ou=krb5,dc=example,dc=com

[dbmodules]
openldap_ldapconf = {
db_library = kldap
ldap_kdc_dn = cn=kdc-srv,ou=krb5,dc=example,dc=com
ldap_service_password_file = /etc/krb5kdc/service.keyfile
ladap_conns_per_server = 5
disable_last_success = true
disable_lockout = true
}

[logging]
kdc = FILE:/var/log/krb5/kdc.log


And here's /etc/krb5kdc/kdc.conf on kls4:

[kdcdefaults]
kdc_ports = 750,88

[realms]
EXAMPLE.COM = {
key_stash_file = /etc/krb5kdc/stash
kdc_ports = 750,88
max_life = 1d 0h 0m 0s
max_renewable_life = 90d 0h 0m 0s
master_key_type = des3-hmac-sha1
supported_enctypes = aes256-cts:normal \
arcfour-hmac:normal des3-hmac-sha1:normal \
des-cbc-crc:normal des:normal des:v4 des:norealm \
des:onlyrealm des:afs3
default_principal_flags = +preauth
}

The credentials for cn=kdc-srv, the LDAP account for the KDC service,
are stored in /etc/krb5kdc/service.keyfile. This file, together with
the 'stash' file containing the KDC database master key were simply
copied from the old systems. The service.keyfile has a line in it that
looks like:

cn=kdc-srv,ou=krb5,dc=example,dc=com#{HEX}3c9264892e086756

Finally, kls4.example.com has forward and reverse DNS entries that
match (for both IPv4 and IPv6) and time is synchronized with the
master, kls1.

Any idea what could be causing the aforementioned error? Have the
configuration requirements for Kerberos v1.15 changed since v1.10?

Thanks,

Jaap

[Pallissard, Matthew]

What does your olcSyncrepl line for dc=example,dc=com look like?

Matt Pallissard

> ________________________________________________
> Kerberos mailing list           Kerb...@mit.edu
> https://mailman.mit.edu/mailman/listinfo/kerberos

[t Seeger]

Hello,
please check what URI value is in '/etc/ldap/ldap.conf'. Are both set two ldapi:///?

Thorsten

Von meinem iPhone gesendet

[Jaap Winius]

Quoting "Pallissard, Matthew" <k...@pallissard.net>:

> What does your olcSyncrepl line for dc=example,dc=com look like?

olcSyncrepl: {0}rid=123 provider="ldap://klsm.example.com:389/" type=refreshAn
dPersist retry="60 30 300 +" searchbase="dc=example,dc=com" bindmethod=sasl s
aslmech=gssapi

The OpenLDAP configuration works, as far as I can tell, since I was
able to pull in a copy of the DIT, which includes the Kerberos database.

Cheers,

Jaap

[Pallissard, Matthew]

Hmm,

Do your cn=config databases match?

Do you know what that hashed password actually is? Can you manually bind with that username/pw and ldapsearch?

Matt Pallissard

[Jaap Winius]

Quoting t Seeger <tseeg...@gmail.com>:

> please check what URI value is in '/etc/ldap/ldap.conf'. Are both
> set two ldapi:///?

Both? I had the URI value in /etc/ldap/ldap.conf set like this:

URI ldap://kls4.example.com/

So I tried it with:

URI ldapi:///

And I also tried these variations:

URI ldapi:/// ldapi:///
URI ldapi:/// ldap://kls4.example.com/
URI ldapi:/// ldapi://kls4.example.com/
URI ldapi://kls4.example.com/ ldapi:///
URI ldap://kls4.example.com/ ldapi:///
URI ldapi://kls4.example.com/ ldapi:///

But, none of them made any difference.

Thanks,

Jaap

[Jaap Winius]

Quoting "Pallissard, Matthew" <k...@pallissard.net>:

> Do your cn=config databases match?

Almost. The main difference is that the databases on the old systems
are in an hdb format and the new one uses mdb, so there are a few
olcDbConfig lines on the old systems that are not present in the new
system.

> Do you know what that hashed password actually is? Can you manually
> bind with that username/pw and ldapsearch?

Regrettably, no, I don't have the passwords. I copied the
'service.keyfile 'and 'stash' files from the old systems hoped it
would work. Could it be that the required format or key type of one or
both of these files has changed? If so, then unless I can decrypt that
HEX value it will probably be necessary to create a new realm. If not,
then it does make troubleshooting a bit more difficult.

Thanks,

Jaap

[Pallissard, Matthew]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

> Could it be that the required format or key type of one or both of these files has changed?

That I do not know.

> If so, then unless I can decrypt that HEX value it will probably be necessary to create a new realm.

I don't think that a new realm would be necessary. You could probably generate a new password/hash and reset it in LDAP.

You could also try pointing your new KDC to your old LDAP server to see whether or not the issue is with your LDAP instance or the KDC config.


You should check your slapd logs as well.


Also, now that I'm looking at config you originally posted a little more closely, it looks like you're missing the 'ldap_servers' line and that you've misspelled 'ladap_conns_per_server'.

FWIW here's a stripped down working config that I've used. I don't know if it follows best practice or not but it works for me. (I also just stick everything in /etc/krb5.conf)

[libdefaults]
default_realm = EXAMPLE.COM
dns_lookup_realm = 0
dns_lookup_kdc = 0
ticket_lifetime = 24h
renew_lifetime = 7d
fowardable = true



[realms]
EXAMPLE.COM = {
admin_server = server.example.com
kdc = server.example.com
acl_file = /etc/krb5kdc/example.com/kadm5.acl
default_domain = example.com
database_module = LDAP.example.com
key_stash_file = /etc/krb5kdc/example.com/example.com.sf
admin_keytab = /etc/krb5kdc/example.com/kadm.keytab
}

[dbdefaults]

[dbmodules]
LDAP.example.com = {
db_library = kldap
ldap_kdc_dn = "cn=kdc,dc=authentication"
ldap_kadmind_dn = "cn=adm,dc=authentication"
ldap_service_password_file = /etc/krb5kdc/example.com/example.com.keyfile
ldap_servers = ldapi://
ldap_kerberos_container_dn = "cn=krb,dc=authentication"

}

[domain_realm]
.example.com = EXAMPLE.COM
example.com = EXAMPLE.COM

[kdcdefaults]
kdc_ports = 88
kdc_tcp_ports = 88

[logging]
kdc = SYSLOG:debug:local1
admin-server = SYSLOG:debug:local2
default = SYSLOG:debug:auth


Matt Pallissard

-----BEGIN PGP SIGNATURE-----

iHUEARYKAB0WIQTvIUMPApUGn6YFkXl1uof+t048SQUCWO+MhAAKCRB1uof+t048
SSkVAQDTJdjwnaRZDolKfUEUzN4twMGjfwjjrRmmeIZ/gYWbLAD9Fb/nhgZLadQ0
etOJ1/cCNCbU1tjZqGjEAvXiaEb9zgE=
=BtWz
-----END PGP SIGNATURE-----

[Greg Hudson]

On 04/13/2017 09:13 AM, Jaap Winius wrote:
> Regrettably, no, I don't have the passwords. I copied the
> 'service.keyfile 'and 'stash' files from the old systems hoped it
> would work. Could it be that the required format or key type of one or
> both of these files has changed? If so, then unless I can decrypt that
> HEX value it will probably be necessary to create a new realm. If not,
> then it does make troubleshooting a bit more difficult.

To my knowledge the format of that file has not changed, so I don't know
why the 1.15 KDC isn't able to bind the LDAP server when the 1.10 KDCs can.

The HEX value is not encrypted. It's just encoded in hex. So "3c" is
the ASCII value 60 which is the character '<', and so forth.

[Jaap Winius]

Quoting "Pallissard, Matthew" <k...@pallissard.net>:

> You could also try pointing your new KDC to your old LDAP server to
> see whether or not the issue is with your LDAP instance or the KDC
> config.

That worked. In other words, the problem is with the new slapd server.

> You should check your slapd logs as well.

Nothing new there. Hold on! How can I have missed this?

slapd[560]: GSSAPI Error: Unspecified GSS failure. \
Minor code may provide more information \
(Server ldap/loca...@EXAMPLE.COM not found in Kerberos database)

So, it's attempting to authenticate to the Kerberos master as
'localhost'... and it turns out that I had not successfully replicated
the DIT after all. Doh!

> Also, now that I'm looking at config you originally posted a little

> more closely, it looks like you're missing the 'ldap_servers' line ...

Omitting that line causes it to connect to ldapi:///. It probably
doesn't make a difference, since I don't use it elsewhere, but I'll
keep an eye on it.

> and that you've misspelled 'ladap_conns_per_server'.

Thanks for spotting that. It's a mistake I made years ago and never
noticed. But, in this case fixing it made no difference.

> FWIW here's a stripped down working config that I've used.

I'll check it out later after I've fixed the localhost problem.

Thanks!

Jaap

[Jaap Winius]

Quoting Jaap Winius <jwi...@umrk.nl>:

> slapd[560]: GSSAPI Error: Unspecified GSS failure. \
> Minor code may provide more information \
> (Server ldap/loca...@EXAMPLE.COM not found in Kerberos database)

Invalid credentials? It's because of this. Slapd should discover its
identity by reading its keytab, the location for which can be found in
the value for KRB5_KTNAME (set in /etc/default/slapd), but that's not
happening. This is starting to look like a bug, perhaps in
libsasl2-modules-gssapi-mit.

Cheers,

Jaap

[Pallissard, Matthew]

Is it slapd reading its key tab incorrectly or is the hostname being derived incorrectly.  Is this a host file issue?

Matt Pallissard

[Jaap Winius]

Quoting "Pallissard, Matthew" <k...@pallissard.net>:

> Is it slapd reading its key tab incorrectly or is the hostname being
> derived incorrectly.  Is this a host file issue?

IMO, this is slapd not reading its key table, as the host file does
not give information about the Kerberos principal needed for
authentication. I started out using a separate keytab file like on the
other systems, using this line in /etc/default/slapd:

export KRB5_KTNAME=/etc/ldap/krb5-ldap.keytab

It's important to ensure that the openldap group has read access to
it. I've also tried using the default keytab file instead, applying
the same group access, but slapd continues to attempt to authenticate
with 'ldap/loca...@EXAMPLE.COM'.

Furthermore, /etc/hostname is fine, 'hostnamectl status' checks out
okay, there's nothing funny in /etc/hosts and the DNS forward and
reverse records are consistent.

So, this looks like a bug to me, but in what part of the software:
Kerberos, slapd, or some library, like libsasl2-modules-gssapi-mit?
I'm leaning towards the latter.

Cheers,

Jaap