problem sending initial data to slave Kerberos server

[Dave Steiner]

I'm havin problems adding a slave to an existing test cluster. The
output is slightly sanitized. I've researched this and can't find out
what I'm missing. The keytabs have the correct kvnos. trace and
debug mode on the kpropd don't seem to show anything wrong. What do I
need to check that I'm missing?

master$ /usr/local/kerberos/sbin/kprop -r REALM -d -P 754 -f slave_datatrans slave.rutgers.edu
/usr/local/kerberos/sbin/kprop: Server rejected authentication (during sendauth exchange) while authenticating to server
/usr/local/kerberos/sbin/kprop: Decrypt integrity check failed signalled from server
Error text from server: Decrypt integrity check failed

master$ /usr/local/kerberos/bin/ktutil
ktutil: rkt /etc/krb5.keytab
ktutil: l
slot KVNO Principal
---- ---- ---------------------------------------------------------------------
1 4 host/master@REALM
2 7 host/master.rutgers.edu@REALM


slave$ /usr/local/kerberos/bin/ktutil
ktutil: rkt /etc/krb5.keytab
ktutil: l
slot KVNO Principal
---- ---- ---------------------------------------------------------------------
1 2 host/slave@REALM
2 2 host/slave.rutgers.edu@REALM


I need both of these entries due to the way out Unix support sets up
the hostname. The "resolve" test program doesn't find any issues.


master$ /usr/local/kerberos/sbin/kadmin.local -r REALM
Authenticating as principal krbadm/admin@REALM with password.
kadmin.local: getprinc host/slave
Principal: host/slave@REALM
Expiration date: [never]
Last password change: Tue Jan 28 17:13:06 EST 2014
Password expiration date: [none]
Maximum ticket life: 0 days 10:00:00
Maximum renewable life: 7 days 00:00:00
Last modified: Tue Jan 28 17:13:06 EST 2014 (krbadm/admin@REALM)
Last successful authentication: [never]
Last failed authentication: [never]
Failed password attempts: 0
Number of keys: 1
Key: vno 2, des-cbc-crc, no salt
MKey: vno 1
Attributes: REQUIRES_PRE_AUTH
Policy: default
kadmin.local: getprinc host/slave.rutgers.edu
Principal: host/slave.rutgers.edu@REALM
Expiration date: [never]
Last password change: Tue Jan 28 17:13:06 EST 2014
Password expiration date: [none]
Maximum ticket life: 0 days 10:00:00
Maximum renewable life: 7 days 00:00:00
Last modified: Tue Jan 28 17:13:06 EST 2014 (krbadm/admin@REALM)
Last successful authentication: [never]
Last failed authentication: [never]
Failed password attempts: 0
Number of keys: 1
Key: vno 2, des-cbc-crc, no salt
MKey: vno 1
Attributes: REQUIRES_PRE_AUTH
Policy: default
kadmin.local: getprinc host/master
Principal: host/master@REALM
Expiration date: [never]
Last password change: Tue Jan 28 14:40:49 EST 2014
Password expiration date: [none]
Maximum ticket life: 0 days 10:00:00
Maximum renewable life: 7 days 00:00:00
Last modified: Tue Jan 28 14:40:49 EST 2014 (steiner/admin@REALM)
Last successful authentication: [never]
Last failed authentication: [never]
Failed password attempts: 0
Number of keys: 1
Key: vno 3, des-cbc-crc, no salt
MKey: vno 1
Attributes: REQUIRES_PRE_AUTH
Policy: default
kadmin.local: getprinc host/master.rutgers.edu
Principal: host/master.rutgers.edu@REALM
Expiration date: [never]
Last password change: Tue Jan 28 18:52:10 EST 2014
Password expiration date: [none]
Maximum ticket life: 0 days 10:00:00
Maximum renewable life: 7 days 00:00:00
Last modified: Tue Jan 28 18:52:10 EST 2014 (krbadm/admin@REALM)
Last successful authentication: [never]
Last failed authentication: [never]
Failed password attempts: 0
Number of keys: 1
Key: vno 7, des-cbc-crc, no salt
MKey: vno 1
Attributes: REQUIRES_PRE_AUTH
Policy: default

kpropd running on the slave:

/usr/local/kerberos/sbin/kpropd -r REALM -f /u01/krb/data/REALM/from_master -F /u01/krb/data/REALM/principal -P 754 -S -d

debug output from kpropd:

Connection from master.rutgers.edu
krb5_recvauth(6, kprop5_01, host/slave@REALM, ...)
Database load process for full propagation completed.
waiting for a kprop connection

trace output from kpropd:

[4318] 1390947375.656260: Convert service host (service with host as instance) on host (null) to principal
[4318] 1390947375.657065: Remote host after forward canonicalization: slave
[4318] 1390947375.657102: Remote host after reverse DNS processing: slave
[4318] 1390947375.657114: Get host realm for slave
[4318] 1390947375.657131: Use local host slave to get host realm
[4318] 1390947375.657140: Look up slave in the domain_realm map
[4318] 1390947375.657155: Got realm for host slave
[4318] 1390947375.657201: Got service principal host/slave@
[4319] 1390947385.303114: Retrieving host/slave@REALM from FILE:/etc/krb5.keytab (vno 2, enctype des-cbc-crc) with result: 0/Success
[5029] 1390947902.449116: Retrieving host/slave@REALM from FILE:/etc/krb5.keytab (vno 2, enctype des-cbc-crc) with result: 0/Success
[5046] 1390947929.179913: Retrieving host/slave@REALM from FILE:/etc/krb5.keytab (vno 2, enctype des-cbc-crc) with result: 0/Success
[8676] 1390950188.191260: Retrieving host/slave@REALM from FILE:/etc/krb5.keytab (vno 2, enctype des-cbc-crc) with result: 0/Success
[8831] 1390950354.193759: Retrieving host/slave@REALM from FILE:/etc/krb5.keytab (vno 2, enctype des-cbc-crc) with result: 0/Success
[12984] 1390952933.79323: Retrieving host/slave@REALM from FILE:/etc/krb5.keytab (vno 2, enctype des-cbc-crc) with result: 0/Success
[13422] 1390953199.426489: Retrieving host/slave@REALM from FILE:/etc/krb5.keytab (vno 2, enctype des-cbc-crc) with result: 0/Success


Thanks for any help!
-ds

[Dave Steiner]

In looking over this message, I must have grabbed an older output from kadmin... here's the latest for host/master.. the knvo matches the keytab.

master$ /usr/local/kerberos/sbin/kadmin.local -r REALM
Authenticating as principal krbadm/admin@REALM with password.

kadmin.local: getprinc host/master
Principal: host/master@REALM
Expiration date: [never]

Last password change: Tue Jan 28 18:52:10 EST 2014
Password expiration date: [none]
Maximum ticket life: 0 days 10:00:00
Maximum renewable life: 7 days 00:00:00
Last modified: Tue Jan 28 18:52:10 EST 2014 (krbadm/admin@REALM)
Last successful authentication: [never]
Last failed authentication: [never]
Failed password attempts: 0
Number of keys: 1

Key: vno 4, des-cbc-crc, no salt

[Dave Steiner]

[I posted this to the comp.protocols.kerberos newsgroup but don't see it in the
mailing list archives. Please forgive it this gets duplicated. -ds]

I'm havin problems adding a slave to an existing test cluster. The output is
slightly sanitized. I've researched this and can't find out what I'm missing.
The keytabs have the correct kvnos. trace and debug mode on the kpropd don't
seem to show anything wrong. What do I need to check that I'm missing?

master$ /usr/local/kerberos/sbin/kprop -r REALM -d -P 754 -f slave_datatrans

slave.rutgers.edu <http://slave.rutgers.edu>

/usr/local/kerberos/sbin/kprop: Server rejected authentication (during sendauth
exchange) while authenticating to server
/usr/local/kerberos/sbin/kprop: Decrypt integrity check failed signalled from
server
Error text from server: Decrypt integrity check failed

master$ /usr/local/kerberos/bin/ktutil
ktutil: rkt /etc/krb5.keytab
ktutil: l
slot KVNO Principal
---- ---- ---------------------------------------------------------------------
1 4 host/master@REALM
2 7 host/master.rutgers.edu@REALM


slave$ /usr/local/kerberos/bin/ktutil
ktutil: rkt /etc/krb5.keytab
ktutil: l
slot KVNO Principal
---- ---- ---------------------------------------------------------------------
1 2 host/slave@REALM
2 2 host/slave.rutgers.edu@REALM


I need both of these entries due to the way out Unix support sets up the
hostname. The "resolve" test program doesn't find any issues.

master$ /usr/local/kerberos/sbin/kadmin.local -r REALM
Authenticating as principal krbadm/admin@REALM with password.

kadmin.local: getprinc host/slave
Principal: host/slave@REALM
Expiration date: [never]
Last password change: Tue Jan 28 17:13:06 EST 2014

Password expiration date: [none]
Maximum ticket life: 0 days 10:00:00
Maximum renewable life: 7 days 00:00:00

Last modified: Tue Jan 28 17:13:06 EST 2014 (krbadm/admin@REALM)

Last successful authentication: [never]
Last failed authentication: [never]
Failed password attempts: 0
Number of keys: 1

Key: vno 2, des-cbc-crc, no salt

MKey: vno 1
Attributes: REQUIRES_PRE_AUTH
Policy: default

kadmin.local: getprinc host/slave.rutgers.edu <http://slave.rutgers.edu>
Principal: host/slave.rutgers.edu@REALM
Expiration date: [never]
Last password change: Tue Jan 28 17:13:06 EST 2014

Password expiration date: [none]
Maximum ticket life: 0 days 10:00:00
Maximum renewable life: 7 days 00:00:00

Last modified: Tue Jan 28 17:13:06 EST 2014 (krbadm/admin@REALM)

Last successful authentication: [never]
Last failed authentication: [never]
Failed password attempts: 0
Number of keys: 1

Key: vno 2, des-cbc-crc, no salt

MKey: vno 1
Attributes: REQUIRES_PRE_AUTH
Policy: default
kadmin.local: getprinc host/master
Principal: host/master@REALM
Expiration date: [never]
Last password change: Tue Jan 28 18:52:10 EST 2014
Password expiration date: [none]
Maximum ticket life: 0 days 10:00:00
Maximum renewable life: 7 days 00:00:00
Last modified: Tue Jan 28 18:52:10 EST 2014 (krbadm/admin@REALM)
Last successful authentication: [never]
Last failed authentication: [never]
Failed password attempts: 0
Number of keys: 1
Key: vno 4, des-cbc-crc, no salt
MKey: vno 1
Attributes: REQUIRES_PRE_AUTH
Policy: default

kadmin.local: getprinc host/master.rutgers.edu <http://master.rutgers.edu>
Principal: host/master.rutgers.edu@REALM

Expiration date: [never]
Last password change: Tue Jan 28 18:52:10 EST 2014
Password expiration date: [none]
Maximum ticket life: 0 days 10:00:00
Maximum renewable life: 7 days 00:00:00
Last modified: Tue Jan 28 18:52:10 EST 2014 (krbadm/admin@REALM)
Last successful authentication: [never]
Last failed authentication: [never]
Failed password attempts: 0
Number of keys: 1

Key: vno 7, des-cbc-crc, no salt

MKey: vno 1
Attributes: REQUIRES_PRE_AUTH
Policy: default

kpropd running on the slave:

/usr/local/kerberos/sbin/kpropd -r REALM -f /u01/krb/data/REALM/from_master -F
/u01/krb/data/REALM/principal -P 754 -S -d

debug output from kpropd:

Connection from master.rutgers.edu <http://master.rutgers.edu>

[Greg Hudson]

On 01/29/2014 01:44 PM, Dave Steiner wrote:
> I'm havin problems adding a slave to an existing test cluster.

kpropd should be syslogging at LOG_ERROR; finding the relevant syslog
should help figure out what step is failing. The failure might be
coming from rd_req during authentication, or rd_safe during transmission
of the size, or rd_priv during transmission of the database contents.
Whatever it is, it seems to be causing an AP_ERR_BAD_INTEGRITY code
getting sent sent back from kpropd to kprop.

My guess is that the failure is coming from rd_safe or rd_priv, since
rd_req can't produce an AP_ERR_BAD_INTEGRITY error at this point (it
produces AP_WRONG_PRINC instead). But I'm not sure what would cause a
decryption or checksum failure for a KRB-SAFE or KRB-PRIV message, to be
honest. A NAT between master and slave could cause an AP_ERR_BADADDR
error, but we're not seeing that.

The fact that you need host/slave and host/slave.rutgers.edu principals
is troubling, but is most likely just a confounding variable, not the
cause of this particular problem.

[Tom Yu]

Greg Hudson <ghu...@MIT.EDU> writes:

> My guess is that the failure is coming from rd_safe or rd_priv, since
> rd_req can't produce an AP_ERR_BAD_INTEGRITY error at this point (it
> produces AP_WRONG_PRINC instead). But I'm not sure what would cause a
> decryption or checksum failure for a KRB-SAFE or KRB-PRIV message, to be
> honest. A NAT between master and slave could cause an AP_ERR_BADADDR
> error, but we're not seeing that.

I'm fairly sure it's coming from rd_req (via recvauth) in the kpropd.
The "signalled from server" text from kprop that accompanies "during
sendauth" only happens if there's a non-generic error code in the
KRB-ERROR from the server's recvauth.

> The fact that you need host/slave and host/slave.rutgers.edu principals
> is troubling, but is most likely just a confounding variable, not the
> cause of this particular problem.

That might depend on the krb5 version on the slave.

[Dave Steiner]

On 1/29/2014 3:59 PM, Greg Hudson wrote:
> On 01/29/2014 01:44 PM, Dave Steiner wrote:
>> I'm havin problems adding a slave to an existing test cluster.
> kpropd should be syslogging at LOG_ERROR; finding the relevant syslog
> should help figure out what step is failing. The failure might be
> coming from rd_req during authentication, or rd_safe during transmission
> of the size, or rd_priv during transmission of the database contents.
> Whatever it is, it seems to be causing an AP_ERR_BAD_INTEGRITY code
> getting sent sent back from kpropd to kprop.
>

> My guess is that the failure is coming from rd_safe or rd_priv, since
> rd_req can't produce an AP_ERR_BAD_INTEGRITY error at this point (it
> produces AP_WRONG_PRINC instead). But I'm not sure what would cause a
> decryption or checksum failure for a KRB-SAFE or KRB-PRIV message, to be
> honest. A NAT between master and slave could cause an AP_ERR_BADADDR
> error, but we're not seeing that.
>

> The fact that you need host/slave and host/slave.rutgers.edu principals
> is troubling, but is most likely just a confounding variable, not the
> cause of this particular problem.

Hi Greg,

I literally just fixed this problem before I saw your email. It was the host
issue. In /etc/hosts, our Unix group only put the short hostname and let DNS
handle the full hostname. I requested that they also include the full hostname
(first) in /etc/hosts, restarted things on both master and slave, and now things
work.

Thanks for getting back to me.

take care,
ds