"Checksum failed" error

[light...@gmail.com]

Hello,

I originally posted a question about Kerberos here:
https://groups.google.com/a/cloudera.org/forum/#!topic/cdh-user/JAREHVXxSTA
attempting to use Kerberos in a Hadoop environment.

I currently have 3 systems up:
- krb-auth that is Ubuntu 10.04.4, the kerberos kdc and admin server, version 1.8.1 and runs bind9 with proper DNS resolution. I switched to this krb5 1.8.1 because it's known to be supported (http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH4/latest/CDH4-Security-Guide/cdh4sg_topic_3.html)
- host, that is Ubuntu 12.04 and runs a namenode and datanode
- node1, that is also Ubuntu 12.04 and runs a datanode and secondary namenode.

The datanode on the namenode can authenticate and connect from within 'host', but the datanode and secondary namenode cannot with this exception on the namenode:

2013-04-23 14:04:22,786 WARN SecurityLogger.org.apache.hadoop.ipc.Server: Auth failed for 192.168.56.102:33363:null
2013-04-23 14:04:22,794 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 8020: readAndProcess threw exception javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: Failure unspecified a
t GSS-API level (Mechanism level: Checksum failed)] from client 192.168.56.102. Count of bytes read: 0
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: Failure unspecified at GSS-API level (Mechanism level: Checksum failed)]
at com.sun.security.sasl.gsskerb.GssKrb5Server.evaluateResponse(GssKrb5Server.java:177)
at org.apache.hadoop.ipc.Server$Connection.saslReadAndProcess(Server.java:1202)
at org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:1396)
at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:711)
at org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:510)
at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:485)
Caused by: GSSException: Failure unspecified at GSS-API level (Mechanism level: Checksum failed)
at sun.security.jgss.krb5.Krb5Context.acceptSecContext(Krb5Context.java:788)
at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:342)
at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:285)
at com.sun.security.sasl.gsskerb.GssKrb5Server.evaluateResponse(GssKrb5Server.java:155)
... 5 more
Caused by: KrbException: Checksum failed
at sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType.decrypt(Aes256CtsHmacSha1EType.java:102)
at sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType.decrypt(Aes256CtsHmacSha1EType.java:94)
at sun.security.krb5.EncryptedData.decrypt(EncryptedData.java:177)
at sun.security.krb5.KrbApReq.authenticate(KrbApReq.java:278)
at sun.security.krb5.KrbApReq.<init>(KrbApReq.java:144)
at sun.security.jgss.krb5.InitSecContextToken.<init>(InitSecContextToken.java:108)
at sun.security.jgss.krb5.Krb5Context.acceptSecContext(Krb5Context.java:771)
... 8 more
Caused by: java.security.GeneralSecurityException: Checksum failed
at sun.security.krb5.internal.crypto.dk.AesDkCrypto.decryptCTS(AesDkCrypto.java:451)
at sun.security.krb5.internal.crypto.dk.AesDkCrypto.decrypt(AesDkCrypto.java:272)
at sun.security.krb5.internal.crypto.Aes256.decrypt(Aes256.java:76)
at sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType.decrypt(Aes256CtsHmacSha1EType.java:100)
... 14 more


I've turned on kerberos debugging (via -Dsun.security.krb5.debug=true) and see this output when connecting on 'host':
Found KeyTab
Found KerberosKey for hdfs/host@DOMAIN
Found KerberosKey for hdfs/host@DOMAIN
Found KerberosKey for hdfs/host@DOMAIN
Found KerberosKey for hdfs/host@DOMAIN
Entered Krb5Context.acceptSecContext with state=STATE_NEW
Added key: 1version: 2
Added key: 16version: 2
Added key: 23version: 2
Added key: 18version: 2
Ordering keys wrt default_tkt_enctypes list
Using builtin default etypes for default_tkt_enctypes
default etypes for default_tkt_enctypes: 18 17 16 23 1 3.
>>> EType: sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType

Which looks ordinary where it finds the four keys of different encryption types and orders them by encryption type.

I've tried different versions of java (1.6.0_43, 1.6.0_31, 1.7.0_21) across all machines, made sure the unlimited strength JCE policy files are in place, can use kinit with keytabs and able authenticate with the kerberos kdc fine, generating keytabs on the kerberos server, and everything else that I can think of.

What else could I do to fix this problem?

Thanks for your time.

[soole...@gmail.com]

Hello,

I am facing the exact same error and have no idea why. Can you share how was the issue resolved for you ? Thank you.