does Titan work with security enabled HBase tables?
314 views
Skip to first unread message
Frank Luo
Apr 7, 2014, 12:36:44 PM4/7/14
to aureliu...@googlegroups.com
My code works with a non-secured HBase cluster but not with a security-enabled one. I am using HBase remote connection. Here is the trace:
-sh-4.1$ 14/04/07 10:08:39 ERROR security.UserGroupInformation: PriviledgedActio nException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl. SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:08:39 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:08:41 ERROR security.UserGroupInformation: PriviledgedActionExceptio n as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslExcep tion: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:08:41 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:08:43 ERROR security.UserGroupInformation: PriviledgedActionExceptio n as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslExcep tion: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:08:43 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:08:45 ERROR security.UserGroupInformation: PriviledgedActionExceptio n as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslExcep tion: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:08:45 WARN ipc.Client: Couldn't setup connection for jl...@PCLC0.MERK LE.LOCAL to hdfs/hqhd02nm01.pcl...@HQHD02.HADOOP.LOCAL
14/04/07 10:08:45 ERROR security.UserGroupInformation: PriviledgedActionExceptio n as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:java.io.IOException: Couldn't setup connection for jl...@PCLC0.MERKLE.LOCAL to hdfs/hqhd02nm01.pclc0.merkle.lo c...@HQHD02.HADOOP.LOCAL
14/04/07 10:08:45 WARN retry.RetryInvocationHandler: Exception while invoking ge tFileInfo of class ClientNamenodeProtocolTranslatorPB after 3 fail over attempts . Trying to fail over immediately.
14/04/07 10:08:45 ERROR security.UserGroupInformation: PriviledgedActionExceptio n as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslExcep tion: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:08:45 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:08:46 ERROR security.UserGroupInformation: PriviledgedActionExceptio n as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslExcep tion: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:08:46 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:08:48 ERROR security.UserGroupInformation: PriviledgedActionExceptio n as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslExcep tion: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:08:48 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
-sh-4.1$ 14/04/07 10:08:53 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:08:53 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:08:56 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:08:56 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:08:57 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:08:57 WARN ipc.Client: Couldn't setup connection for jl...@PCLC0.MERKLE.LOCAL to hdfs/hqhd02nm02.pcl...@HQHD02.HADOOP.LOCAL
14/04/07 10:08:57 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:java.io.IOException: Couldn't setup connection for jl...@PCLC0.MERKLE.LOCAL to hdfs/hqhd02nm02.pcl...@HQHD02.HADOOP.LOCAL
14/04/07 10:08:57 WARN retry.RetryInvocationHandler: Exception while invoking getFileInfo of class ClientNamenodeProtocolTranslatorPB after 4 fail over attempts. Trying to fail over immediately.
14/04/07 10:08:57 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:08:57 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:09:01 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:09:01 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:09:02 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:09:02 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:09:03 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:09:03 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:09:04 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:09:04 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:09:05 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:09:05 WARN ipc.Client: Couldn't setup connection for jl...@PCLC0.MERKLE.LOCAL to hdfs/hqhd02nm01.pcl...@HQHD02.HADOOP.LOCAL
14/04/07 10:09:05 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:java.io.IOException: Couldn't setup connection for jl...@PCLC0.MERKLE.LOCAL to hdfs/hqhd02nm01.pcl...@HQHD02.HADOOP.LOCAL
14/04/07 10:09:05 WARN retry.RetryInvocationHandler: Exception while invoking getFileInfo of class ClientNamenodeProtocolTranslatorPB after 5 fail over attempts. Trying to fail over immediately.
14/04/07 10:09:05 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:09:05 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:09:08 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:09:08 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:09:09 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:09:09 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:09:10 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:09:10 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:09:11 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:09:11 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:09:15 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:09:15 WARN ipc.Client: Couldn't setup connection for jl...@PCLC0.MERKLE.LOCAL to hdfs/hqhd02nm02.pcl...@HQHD02.HADOOP.LOCAL
14/04/07 10:09:15 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:java.io.IOException: Couldn't setup connection for jl...@PCLC0.MERKLE.LOCAL to hdfs/hqhd02nm02.pcl...@HQHD02.HADOOP.LOCAL
14/04/07 10:09:15 WARN retry.RetryInvocationHandler: Exception while invoking getFileInfo of class ClientNamenodeProtocolTranslatorPB after 6 fail over attempts. Trying to fail over immediately.
14/04/07 10:09:15 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:09:15 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:09:16 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:09:16 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:09:20 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:09:20 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:09:22 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:09:22 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:09:25 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:09:25 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:09:29 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:09:29 WARN ipc.Client: Couldn't setup connection for jl...@PCLC0.MERKLE.LOCAL to hdfs/hqhd02nm01.pcl...@HQHD02.HADOOP.LOCAL
14/04/07 10:09:29 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:java.io.IOException: Couldn't setup connection for jl...@PCLC0.MERKLE.LOCAL to hdfs/hqhd02nm01.pcl...@HQHD02.HADOOP.LOCAL
14/04/07 10:09:29 WARN retry.RetryInvocationHandler: Exception while invoking getFileInfo of class ClientNamenodeProtocolTranslatorPB after 7 fail over attempts. Trying to fail over immediately.
14/04/07 10:09:29 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:09:29 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:09:32 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:09:32 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:09:35 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:09:35 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:09:35 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:09:35 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:09:37 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:09:37 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:09:38 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:09:38 WARN ipc.Client: Couldn't setup connection for jl...@PCLC0.MERKLE.LOCAL to hdfs/hqhd02nm02.pcl...@HQHD02.HADOOP.LOCAL
14/04/07 10:09:38 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:java.io.IOException: Couldn't setup connection for jl...@PCLC0.MERKLE.LOCAL to hdfs/hqhd02nm02.pcl...@HQHD02.HADOOP.LOCAL
14/04/07 10:09:38 WARN retry.RetryInvocationHandler: Exception while invoking getFileInfo of class ClientNamenodeProtocolTranslatorPB after 8 fail over attempts. Trying to fail over immediately.
14/04/07 10:09:38 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:09:38 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:09:40 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:09:40 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:09:42 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:09:42 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:09:46 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:09:46 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:09:48 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:09:48 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:09:50 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:09:50 WARN ipc.Client: Couldn't setup connection for jl...@PCLC0.MERKLE.LOCAL to hdfs/hqhd02nm01.pcl...@HQHD02.HADOOP.LOCAL
14/04/07 10:09:50 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:java.io.IOException: Couldn't setup connection for jl...@PCLC0.MERKLE.LOCAL to hdfs/hqhd02nm01.pcl...@HQHD02.HADOOP.LOCAL
14/04/07 10:09:50 WARN retry.RetryInvocationHandler: Exception while invoking getFileInfo of class ClientNamenodeProtocolTranslatorPB after 9 fail over attempts. Trying to fail over immediately.
14/04/07 10:09:50 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:09:50 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:09:54 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:09:54 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:09:56 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:09:56 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:09:59 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:09:59 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:10:00 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:10:00 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:10:05 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:10:05 WARN ipc.Client: Couldn't setup connection for jl...@PCLC0.MERKLE.LOCAL to hdfs/hqhd02nm02.pcl...@HQHD02.HADOOP.LOCAL
14/04/07 10:10:05 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:java.io.IOException: Couldn't setup connection for jl...@PCLC0.MERKLE.LOCAL to hdfs/hqhd02nm02.pcl...@HQHD02.HADOOP.LOCAL
14/04/07 10:10:05 WARN retry.RetryInvocationHandler: Exception while invoking getFileInfo of class ClientNamenodeProtocolTranslatorPB after 10 fail over attempts. Trying to fail over immediately.
14/04/07 10:10:05 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:10:05 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:10:08 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:10:08 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:10:09 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:10:09 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:10:10 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:10:10 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:10:14 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:10:14 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:10:15 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:10:15 WARN ipc.Client: Couldn't setup connection for jl...@PCLC0.MERKLE.LOCAL to hdfs/hqhd02nm01.pcl...@HQHD02.HADOOP.LOCAL
14/04/07 10:10:15 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:java.io.IOException: Couldn't setup connection for jl...@PCLC0.MERKLE.LOCAL to hdfs/hqhd02nm01.pcl...@HQHD02.HADOOP.LOCAL
14/04/07 10:10:15 WARN retry.RetryInvocationHandler: Exception while invoking getFileInfo of class ClientNamenodeProtocolTranslatorPB after 11 fail over attempts. Trying to fail over immediately.
14/04/07 10:10:15 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:10:15 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:10:17 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:10:17 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:10:22 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:10:22 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:10:27 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:10:27 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:10:30 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:10:30 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:10:31 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:10:31 WARN ipc.Client: Couldn't setup connection for jl...@PCLC0.MERKLE.LOCAL to hdfs/hqhd02nm02.pcl...@HQHD02.HADOOP.LOCAL
14/04/07 10:10:31 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:java.io.IOException: Couldn't setup connection for jl...@PCLC0.MERKLE.LOCAL to hdfs/hqhd02nm02.pcl...@HQHD02.HADOOP.LOCAL
14/04/07 10:10:31 WARN retry.RetryInvocationHandler: Exception while invoking getFileInfo of class ClientNamenodeProtocolTranslatorPB after 12 fail over attempts. Trying to fail over immediately.
14/04/07 10:10:31 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:10:31 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:10:33 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:10:33 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:10:35 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:10:35 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:10:40 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:10:40 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:10:41 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:10:41 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:10:44 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:10:44 WARN ipc.Client: Couldn't setup connection for jl...@PCLC0.MERKLE.LOCAL to hdfs/hqhd02nm01.pcl...@HQHD02.HADOOP.LOCAL
14/04/07 10:10:44 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:java.io.IOException: Couldn't setup connection for jl...@PCLC0.MERKLE.LOCAL to hdfs/hqhd02nm01.pcl...@HQHD02.HADOOP.LOCAL
14/04/07 10:10:44 WARN retry.RetryInvocationHandler: Exception while invoking getFileInfo of class ClientNamenodeProtocolTranslatorPB after 13 fail over attempts. Trying to fail over immediately.
14/04/07 10:10:44 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:10:44 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:10:48 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:10:48 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:10:53 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:10:53 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:10:56 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:10:56 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:11:00 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:11:00 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:11:00 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:11:00 WARN ipc.Client: Couldn't setup connection for jl...@PCLC0.MERKLE.LOCAL to hdfs/hqhd02nm02.pcl...@HQHD02.HADOOP.LOCAL
14/04/07 10:11:00 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:java.io.IOException: Couldn't setup connection for jl...@PCLC0.MERKLE.LOCAL to hdfs/hqhd02nm02.pcl...@HQHD02.HADOOP.LOCAL
14/04/07 10:11:00 WARN retry.RetryInvocationHandler: Exception while invoking getFileInfo of class ClientNamenodeProtocolTranslatorPB after 14 fail over attempts. Trying to fail over immediately.
14/04/07 10:11:00 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:11:00 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:11:04 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:11:04 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:11:05 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:11:05 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:11:06 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:11:06 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:11:08 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:11:08 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:11:10 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:11:10 WARN ipc.Client: Couldn't setup connection for jl...@PCLC0.MERKLE.LOCAL to hdfs/hqhd02nm01.pcl...@HQHD02.HADOOP.LOCAL
14/04/07 10:11:10 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:java.io.IOException: Couldn't setup connection for jl...@PCLC0.MERKLE.LOCAL to hdfs/hqhd02nm01.pcl...@HQHD02.HADOOP.LOCAL
14/04/07 10:11:10 WARN retry.RetryInvocationHandler: Exception while invoking class org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo. Not retrying because failovers (15) exceeded maximum allowed (15)
java.io.IOException: Failed on local exception: java.io.IOException: Couldn't setup connection for jl...@PCLC0.MERKLE.LOCAL to hdfs/hqhd02nm01.pcl...@HQHD02.HADOOP.LOCAL; Host Details : local host is: "hqhd02ed02.pclc0.merkle.local/10.129.8.12"; destination host is: "hqhd02nm01.pclc0.merkle.local":8020;
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:763)
at org.apache.hadoop.ipc.Client.call(Client.java:1242)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202)
at com.sun.proxy.$Proxy9.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:629)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:164)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:83)
at com.sun.proxy.$Proxy10.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1545)
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:820)
at org.apache.hadoop.fs.FileSystem.globStatusInternal(FileSystem.java:1681)
at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1627)
at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1602)
at org.apache.hadoop.fs.shell.PathData.expandAsGlob(PathData.java:271)
at org.apache.hadoop.fs.shell.Command.expandArgument(Command.java:224)
at org.apache.hadoop.fs.shell.Command.expandArguments(Command.java:207)
at org.apache.hadoop.fs.shell.Command.processRawArguments(Command.java:190)
at org.apache.hadoop.fs.shell.Command.run(Command.java:154)
at org.apache.hadoop.fs.FsShell.run(FsShell.java:254)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at org.apache.hadoop.fs.FsShell.main(FsShell.java:304)
Caused by: java.io.IOException: Couldn't setup connection for jl...@PCLC0.MERKLE.LOCAL to hdfs/hqhd02nm01.pcl...@HQHD02.HADOOP.LOCAL
at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:574)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:546)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:630)
at org.apache.hadoop.ipc.Client$Connection.access$2100(Client.java:252)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1291)
at org.apache.hadoop.ipc.Client.call(Client.java:1209)
... 24 more
Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212)
at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:137)
at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:457)
at org.apache.hadoop.ipc.Client$Connection.access$1400(Client.java:252)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:623)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:620)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:619)
... 27 more
Caused by: GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)
at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:710)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:248)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:193)
... 36 more
Caused by: KrbException: Fail to create credential. (63) - No service creds
at sun.security.krb5.internal.CredentialsUtil.acquireServiceCreds(CredentialsUtil.java:301)
at sun.security.krb5.Credentials.acquireServiceCreds(Credentials.java:449)
at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:641)
... 39 more
rm: Failed on local exception: java.io.IOException: Couldn't setup connection for jl...@PCLC0.MERKLE.LOCAL to hdfs/hqhd02nm01.pcl...@HQHD02.HADOOP.LOCAL; Host Details : local host is: "hqhd02ed02.pclc0.merkle.local/10.129.8.12"; destination host is: "hqhd02nm01.pclc0.merkle.local":8020;
14/04/07 10:08:39 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:08:41 ERROR security.UserGroupInformation: PriviledgedActionExceptio n as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslExcep tion: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:08:41 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:08:43 ERROR security.UserGroupInformation: PriviledgedActionExceptio n as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslExcep tion: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:08:43 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:08:45 ERROR security.UserGroupInformation: PriviledgedActionExceptio n as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslExcep tion: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:08:45 WARN ipc.Client: Couldn't setup connection for jl...@PCLC0.MERK LE.LOCAL to hdfs/hqhd02nm01.pcl...@HQHD02.HADOOP.LOCAL
14/04/07 10:08:45 ERROR security.UserGroupInformation: PriviledgedActionExceptio n as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:java.io.IOException: Couldn't setup connection for jl...@PCLC0.MERKLE.LOCAL to hdfs/hqhd02nm01.pclc0.merkle.lo c...@HQHD02.HADOOP.LOCAL
14/04/07 10:08:45 WARN retry.RetryInvocationHandler: Exception while invoking ge tFileInfo of class ClientNamenodeProtocolTranslatorPB after 3 fail over attempts . Trying to fail over immediately.
14/04/07 10:08:45 ERROR security.UserGroupInformation: PriviledgedActionExceptio n as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslExcep tion: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:08:45 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:08:46 ERROR security.UserGroupInformation: PriviledgedActionExceptio n as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslExcep tion: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:08:46 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:08:48 ERROR security.UserGroupInformation: PriviledgedActionExceptio n as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslExcep tion: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:08:48 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
-sh-4.1$ 14/04/07 10:08:53 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:08:53 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:08:56 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:08:56 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:08:57 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:08:57 WARN ipc.Client: Couldn't setup connection for jl...@PCLC0.MERKLE.LOCAL to hdfs/hqhd02nm02.pcl...@HQHD02.HADOOP.LOCAL
14/04/07 10:08:57 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:java.io.IOException: Couldn't setup connection for jl...@PCLC0.MERKLE.LOCAL to hdfs/hqhd02nm02.pcl...@HQHD02.HADOOP.LOCAL
14/04/07 10:08:57 WARN retry.RetryInvocationHandler: Exception while invoking getFileInfo of class ClientNamenodeProtocolTranslatorPB after 4 fail over attempts. Trying to fail over immediately.
14/04/07 10:08:57 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:08:57 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:09:01 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:09:01 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:09:02 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:09:02 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:09:03 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:09:03 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:09:04 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:09:04 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:09:05 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:09:05 WARN ipc.Client: Couldn't setup connection for jl...@PCLC0.MERKLE.LOCAL to hdfs/hqhd02nm01.pcl...@HQHD02.HADOOP.LOCAL
14/04/07 10:09:05 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:java.io.IOException: Couldn't setup connection for jl...@PCLC0.MERKLE.LOCAL to hdfs/hqhd02nm01.pcl...@HQHD02.HADOOP.LOCAL
14/04/07 10:09:05 WARN retry.RetryInvocationHandler: Exception while invoking getFileInfo of class ClientNamenodeProtocolTranslatorPB after 5 fail over attempts. Trying to fail over immediately.
14/04/07 10:09:05 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:09:05 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:09:08 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:09:08 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:09:09 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:09:09 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:09:10 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:09:10 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:09:11 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:09:11 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:09:15 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:09:15 WARN ipc.Client: Couldn't setup connection for jl...@PCLC0.MERKLE.LOCAL to hdfs/hqhd02nm02.pcl...@HQHD02.HADOOP.LOCAL
14/04/07 10:09:15 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:java.io.IOException: Couldn't setup connection for jl...@PCLC0.MERKLE.LOCAL to hdfs/hqhd02nm02.pcl...@HQHD02.HADOOP.LOCAL
14/04/07 10:09:15 WARN retry.RetryInvocationHandler: Exception while invoking getFileInfo of class ClientNamenodeProtocolTranslatorPB after 6 fail over attempts. Trying to fail over immediately.
14/04/07 10:09:15 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:09:15 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:09:16 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:09:16 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:09:20 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:09:20 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:09:22 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:09:22 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:09:25 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:09:25 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:09:29 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:09:29 WARN ipc.Client: Couldn't setup connection for jl...@PCLC0.MERKLE.LOCAL to hdfs/hqhd02nm01.pcl...@HQHD02.HADOOP.LOCAL
14/04/07 10:09:29 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:java.io.IOException: Couldn't setup connection for jl...@PCLC0.MERKLE.LOCAL to hdfs/hqhd02nm01.pcl...@HQHD02.HADOOP.LOCAL
14/04/07 10:09:29 WARN retry.RetryInvocationHandler: Exception while invoking getFileInfo of class ClientNamenodeProtocolTranslatorPB after 7 fail over attempts. Trying to fail over immediately.
14/04/07 10:09:29 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:09:29 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:09:32 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:09:32 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:09:35 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:09:35 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:09:35 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:09:35 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:09:37 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:09:37 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:09:38 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:09:38 WARN ipc.Client: Couldn't setup connection for jl...@PCLC0.MERKLE.LOCAL to hdfs/hqhd02nm02.pcl...@HQHD02.HADOOP.LOCAL
14/04/07 10:09:38 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:java.io.IOException: Couldn't setup connection for jl...@PCLC0.MERKLE.LOCAL to hdfs/hqhd02nm02.pcl...@HQHD02.HADOOP.LOCAL
14/04/07 10:09:38 WARN retry.RetryInvocationHandler: Exception while invoking getFileInfo of class ClientNamenodeProtocolTranslatorPB after 8 fail over attempts. Trying to fail over immediately.
14/04/07 10:09:38 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:09:38 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:09:40 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:09:40 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:09:42 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:09:42 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:09:46 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:09:46 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:09:48 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:09:48 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:09:50 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:09:50 WARN ipc.Client: Couldn't setup connection for jl...@PCLC0.MERKLE.LOCAL to hdfs/hqhd02nm01.pcl...@HQHD02.HADOOP.LOCAL
14/04/07 10:09:50 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:java.io.IOException: Couldn't setup connection for jl...@PCLC0.MERKLE.LOCAL to hdfs/hqhd02nm01.pcl...@HQHD02.HADOOP.LOCAL
14/04/07 10:09:50 WARN retry.RetryInvocationHandler: Exception while invoking getFileInfo of class ClientNamenodeProtocolTranslatorPB after 9 fail over attempts. Trying to fail over immediately.
14/04/07 10:09:50 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:09:50 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:09:54 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:09:54 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:09:56 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:09:56 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:09:59 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:09:59 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:10:00 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:10:00 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:10:05 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:10:05 WARN ipc.Client: Couldn't setup connection for jl...@PCLC0.MERKLE.LOCAL to hdfs/hqhd02nm02.pcl...@HQHD02.HADOOP.LOCAL
14/04/07 10:10:05 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:java.io.IOException: Couldn't setup connection for jl...@PCLC0.MERKLE.LOCAL to hdfs/hqhd02nm02.pcl...@HQHD02.HADOOP.LOCAL
14/04/07 10:10:05 WARN retry.RetryInvocationHandler: Exception while invoking getFileInfo of class ClientNamenodeProtocolTranslatorPB after 10 fail over attempts. Trying to fail over immediately.
14/04/07 10:10:05 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:10:05 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:10:08 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:10:08 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:10:09 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:10:09 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:10:10 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:10:10 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:10:14 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:10:14 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:10:15 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:10:15 WARN ipc.Client: Couldn't setup connection for jl...@PCLC0.MERKLE.LOCAL to hdfs/hqhd02nm01.pcl...@HQHD02.HADOOP.LOCAL
14/04/07 10:10:15 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:java.io.IOException: Couldn't setup connection for jl...@PCLC0.MERKLE.LOCAL to hdfs/hqhd02nm01.pcl...@HQHD02.HADOOP.LOCAL
14/04/07 10:10:15 WARN retry.RetryInvocationHandler: Exception while invoking getFileInfo of class ClientNamenodeProtocolTranslatorPB after 11 fail over attempts. Trying to fail over immediately.
14/04/07 10:10:15 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:10:15 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:10:17 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:10:17 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:10:22 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:10:22 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:10:27 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:10:27 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:10:30 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:10:30 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:10:31 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:10:31 WARN ipc.Client: Couldn't setup connection for jl...@PCLC0.MERKLE.LOCAL to hdfs/hqhd02nm02.pcl...@HQHD02.HADOOP.LOCAL
14/04/07 10:10:31 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:java.io.IOException: Couldn't setup connection for jl...@PCLC0.MERKLE.LOCAL to hdfs/hqhd02nm02.pcl...@HQHD02.HADOOP.LOCAL
14/04/07 10:10:31 WARN retry.RetryInvocationHandler: Exception while invoking getFileInfo of class ClientNamenodeProtocolTranslatorPB after 12 fail over attempts. Trying to fail over immediately.
14/04/07 10:10:31 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:10:31 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:10:33 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:10:33 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:10:35 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:10:35 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:10:40 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:10:40 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:10:41 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:10:41 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:10:44 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:10:44 WARN ipc.Client: Couldn't setup connection for jl...@PCLC0.MERKLE.LOCAL to hdfs/hqhd02nm01.pcl...@HQHD02.HADOOP.LOCAL
14/04/07 10:10:44 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:java.io.IOException: Couldn't setup connection for jl...@PCLC0.MERKLE.LOCAL to hdfs/hqhd02nm01.pcl...@HQHD02.HADOOP.LOCAL
14/04/07 10:10:44 WARN retry.RetryInvocationHandler: Exception while invoking getFileInfo of class ClientNamenodeProtocolTranslatorPB after 13 fail over attempts. Trying to fail over immediately.
14/04/07 10:10:44 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:10:44 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:10:48 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:10:48 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:10:53 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:10:53 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:10:56 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:10:56 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:11:00 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:11:00 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:11:00 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:11:00 WARN ipc.Client: Couldn't setup connection for jl...@PCLC0.MERKLE.LOCAL to hdfs/hqhd02nm02.pcl...@HQHD02.HADOOP.LOCAL
14/04/07 10:11:00 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:java.io.IOException: Couldn't setup connection for jl...@PCLC0.MERKLE.LOCAL to hdfs/hqhd02nm02.pcl...@HQHD02.HADOOP.LOCAL
14/04/07 10:11:00 WARN retry.RetryInvocationHandler: Exception while invoking getFileInfo of class ClientNamenodeProtocolTranslatorPB after 14 fail over attempts. Trying to fail over immediately.
14/04/07 10:11:00 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:11:00 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:11:04 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:11:04 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:11:05 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:11:05 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:11:06 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:11:06 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:11:08 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:11:08 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
14/04/07 10:11:10 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
14/04/07 10:11:10 WARN ipc.Client: Couldn't setup connection for jl...@PCLC0.MERKLE.LOCAL to hdfs/hqhd02nm01.pcl...@HQHD02.HADOOP.LOCAL
14/04/07 10:11:10 ERROR security.UserGroupInformation: PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS) cause:java.io.IOException: Couldn't setup connection for jl...@PCLC0.MERKLE.LOCAL to hdfs/hqhd02nm01.pcl...@HQHD02.HADOOP.LOCAL
14/04/07 10:11:10 WARN retry.RetryInvocationHandler: Exception while invoking class org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo. Not retrying because failovers (15) exceeded maximum allowed (15)
java.io.IOException: Failed on local exception: java.io.IOException: Couldn't setup connection for jl...@PCLC0.MERKLE.LOCAL to hdfs/hqhd02nm01.pcl...@HQHD02.HADOOP.LOCAL; Host Details : local host is: "hqhd02ed02.pclc0.merkle.local/10.129.8.12"; destination host is: "hqhd02nm01.pclc0.merkle.local":8020;
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:763)
at org.apache.hadoop.ipc.Client.call(Client.java:1242)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202)
at com.sun.proxy.$Proxy9.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:629)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:164)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:83)
at com.sun.proxy.$Proxy10.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1545)
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:820)
at org.apache.hadoop.fs.FileSystem.globStatusInternal(FileSystem.java:1681)
at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1627)
at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1602)
at org.apache.hadoop.fs.shell.PathData.expandAsGlob(PathData.java:271)
at org.apache.hadoop.fs.shell.Command.expandArgument(Command.java:224)
at org.apache.hadoop.fs.shell.Command.expandArguments(Command.java:207)
at org.apache.hadoop.fs.shell.Command.processRawArguments(Command.java:190)
at org.apache.hadoop.fs.shell.Command.run(Command.java:154)
at org.apache.hadoop.fs.FsShell.run(FsShell.java:254)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at org.apache.hadoop.fs.FsShell.main(FsShell.java:304)
Caused by: java.io.IOException: Couldn't setup connection for jl...@PCLC0.MERKLE.LOCAL to hdfs/hqhd02nm01.pcl...@HQHD02.HADOOP.LOCAL
at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:574)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:546)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:630)
at org.apache.hadoop.ipc.Client$Connection.access$2100(Client.java:252)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1291)
at org.apache.hadoop.ipc.Client.call(Client.java:1209)
... 24 more
Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212)
at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:137)
at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:457)
at org.apache.hadoop.ipc.Client$Connection.access$1400(Client.java:252)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:623)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:620)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:619)
... 27 more
Caused by: GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)
at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:710)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:248)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:193)
... 36 more
Caused by: KrbException: Fail to create credential. (63) - No service creds
at sun.security.krb5.internal.CredentialsUtil.acquireServiceCreds(CredentialsUtil.java:301)
at sun.security.krb5.Credentials.acquireServiceCreds(Credentials.java:449)
at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:641)
... 39 more
rm: Failed on local exception: java.io.IOException: Couldn't setup connection for jl...@PCLC0.MERKLE.LOCAL to hdfs/hqhd02nm01.pcl...@HQHD02.HADOOP.LOCAL; Host Details : local host is: "hqhd02ed02.pclc0.merkle.local/10.129.8.12"; destination host is: "hqhd02nm01.pclc0.merkle.local":8020;
Dan LaRocque
Apr 7, 2014, 6:48:25 PM4/7/14
to aureliu...@googlegroups.com
Hi Frank,
Are you running ZK configured for SASL auth with JAAS? IOW, are you
doing HBase user/password authentication via Kerberos principals? Or
just running HBase with keytabs that let it authenticate to a Kerberized
HDFS?
Would appreciate any other background on your config you deem appropriate.
thanks,
Dan
--
Dan LaRocque
http://www.thinkaurelius.com/
Are you running ZK configured for SASL auth with JAAS? IOW, are you
doing HBase user/password authentication via Kerberos principals? Or
just running HBase with keytabs that let it authenticate to a Kerberized
HDFS?
Would appreciate any other background on your config you deem appropriate.
thanks,
Dan
Dan LaRocque
http://www.thinkaurelius.com/
Frank Luo
Apr 8, 2014, 1:44:01 PM4/8/14
to aureliu...@googlegroups.com
Here is the answer from my IT expert:
> I think we are doing the former, except we are using trusted kerberos principals.
> hdfs/hqhd02nm01.pclc0.merkle.lo...@HQHD02.HADOOP.LOCAL
> hdfs/hqhd02nm02.pclc0.merkle.lo...@HQHD02.HADOOP.LOCAL
> 14/04/07 10:08:57 ERROR security.UserGroupInformation:
> PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS)
> cause:java.io.IOException: Couldn't setup connection for
> jl...@PCLC0.MERKLE.LOCAL to
> hdfs/hqhd02nm02.pclc0.merkle.lo...@HQHD02.HADOOP.LOCAL
> hdfs/hqhd02nm01.pclc0.merkle.lo...@HQHD02.HADOOP.LOCAL
> 14/04/07 10:09:05 ERROR security.UserGroupInformation:
> PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS)
> cause:java.io.IOException: Couldn't setup connection for
> jl...@PCLC0.MERKLE.LOCAL to
> hdfs/hqhd02nm01.pclc0.merkle.lo...@HQHD02.HADOOP.LOCAL
> hdfs/hqhd02nm02.pclc0.merkle.lo...@HQHD02.HADOOP.LOCAL
> 14/04/07 10:09:15 ERROR security.UserGroupInformation:
> PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS)
> cause:java.io.IOException: Couldn't setup connection for
> jl...@PCLC0.MERKLE.LOCAL to
> hdfs/hqhd02nm02.pclc0.merkle.lo...@HQHD02.HADOOP.LOCAL
> hdfs/hqhd02nm01.pclc0.merkle.lo...@HQHD02.HADOOP.LOCAL
> 14/04/07 10:09:29 ERROR security.UserGroupInformation:
> PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS)
> cause:java.io.IOException: Couldn't setup connection for
> jl...@PCLC0.MERKLE.LOCAL to
> hdfs/hqhd02nm01.pclc0.merkle.lo...@HQHD02.HADOOP.LOCAL
> hdfs/hqhd02nm02.pclc0.merkle.lo...@HQHD02.HADOOP.LOCAL
> 14/04/07 10:09:38 ERROR security.UserGroupInformation:
> PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS)
> cause:java.io.IOException: Couldn't setup connection for
> jl...@PCLC0.MERKLE.LOCAL to
> hdfs/hqhd02nm02.pclc0.merkle.lo...@HQHD02.HADOOP.LOCAL
> hdfs/hqhd02nm01.pclc0.merkle.lo...@HQHD02.HADOOP.LOCAL
> 14/04/07 10:09:50 ERROR security.UserGroupInformation:
> PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS)
> cause:java.io.IOException: Couldn't setup connection for
> jl...@PCLC0.MERKLE.LOCAL to
> hdfs/hqhd02nm01.pclc0.merkle.lo...@HQHD02.HADOOP.LOCAL
> hdfs/hqhd02nm02.pclc0.merkle.lo...@HQHD02.HADOOP.LOCAL
> 14/04/07 10:10:05 ERROR security.UserGroupInformation:
> PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS)
> cause:java.io.IOException: Couldn't setup connection for
> jl...@PCLC0.MERKLE.LOCAL to
> hdfs/hqhd02nm02.pclc0.merkle.lo...@HQHD02.HADOOP.LOCAL
> hdfs/hqhd02nm01.pclc0.merkle.lo...@HQHD02.HADOOP.LOCAL
> 14/04/07 10:10:15 ERROR security.UserGroupInformation:
> PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS)
> cause:java.io.IOException: Couldn't setup connection for
> jl...@PCLC0.MERKLE.LOCAL to
> hdfs/hqhd02nm01.pclc0.merkle.lo...@HQHD02.HADOOP.LOCAL
> hdfs/hqhd02nm02.pclc0.merkle.lo...@HQHD02.HADOOP.LOCAL
> 14/04/07 10:10:31 ERROR security.UserGroupInformation:
> PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS)
> cause:java.io.IOException: Couldn't setup connection for
> jl...@PCLC0.MERKLE.LOCAL to
> hdfs/hqhd02nm02.pclc0.merkle.lo...@HQHD02.HADOOP.LOCAL
> hdfs/hqhd02nm01.pclc0.merkle.lo...@HQHD02.HADOOP.LOCAL
> 14/04/07 10:10:44 ERROR security.UserGroupInformation:
> PriviledgedActionException as:jl...@PCLC0.MERKLE.LOCAL (auth:KERBEROS)
> cause:java.io.IOException: Couldn't setup connection for
> jl...@PCLC0.MERKLE.LOCAL to
> hdfs/hqhd02nm01.pclc0.merkle.lo...@HQHD02.HADOOP.LOCAL
> cause:javax.security.sasl.SaslException: GSS initiate fa...
Morten Lileng
Apr 10, 2014, 11:23:07 AM4/10/14
to aureliu...@googlegroups.com
Hello Dan, We are stuck on this item and time is quickly running out for our proof of concept with Titan. Can you please confirm if Titan supports any form of Kerberos? Is there a place where we see examples of how this is implemented?
Thanks, Morten
On Tuesday, April 8, 2014 12:44:01 PM UTC-5, Frank Luo wrote:
Here is the answer from my IT expert:
> I think we are doing the former, except we are using trusted kerberos principals.
On Monday, April 7, 2014 5:48:25 PM UTC-5, Dan LaRocque wrote:
Hi Frank,
Are you running ZK configured for SASL auth with JAAS? IOW, are you
doing HBase user/password authentication via Kerberos principals? Or
just running HBase with keytabs that let it authenticate to a Kerberized
HDFS?
Would appreciate any other background on your config you deem appropriate.
thanks,
Dan
On 04/07/2014 12:36 PM, Frank Luo wrote:
> My code works with a non-secured HBase cluster but not with a
> security-enabled one. I am using HBase remote connection. Here is the
> trace:
> ...
Dan LaRocque
Apr 10, 2014, 12:57:05 PM4/10/14
to aureliu...@googlegroups.com
On 04/10/2014 11:23 AM, Morten Lileng wrote:
> Hello Dan, We are stuck on this item and time is quickly running out for
> our proof of concept with Titan. Can you please confirm if Titan
> supports any form of Kerberos? Is there a place where we see examples of
> how this is implemented?
>
> Thanks, Morten
>
Hi Morten,
> Hello Dan, We are stuck on this item and time is quickly running out for
> our proof of concept with Titan. Can you please confirm if Titan
> supports any form of Kerberos? Is there a place where we see examples of
> how this is implemented?
>
> Thanks, Morten
>
Titan is agnostic to HBase's authentication.
Titan's HBase client config accepts arbitrary keys from hbase-site.xml
(if it's on the CLASSPATH) and any keys in Titan's properties file with
the "storage.hbase-config." prefix. The key
hbase.security.authentication=kerberos could be specified in either place.
If you're also running ZK with SASL (e.g. a SASL -> GSSAPI -> Krb5
stack), then the steps in chapter 17 of the HBase manual will also
matter on Titan hosts running the HBase client.
I don't know if you're using Rexster, but we do not pass a Rexster
username and password (if configured) down to Titan. Whatever
user/princ Titan runs with will be what it authenticates to HBase as,
independent of any authentication that might be happening at the Rexster
layer.
Hope this helps.
thanks,
Dan
--
Dan LaRocque
http://www.thinkaurelius.com
Reply all
Reply to author
Forward
0 new messages