Issue: repeated request

[Luca Rea]

Hi,
we have scripts in crontab that fork parallel jobs, these ones call "kinit" and run commands that sometimes fail returning the error below:


15/05/14 06:28:35 ERROR security.UserGroupInformation: PriviledgedActionException as:clabhdfs cause:GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)
2015-05-14 06:28:35 SEVERE org.apache.thrift.transport.TTransportException: org.apache.http.client.ClientProtocolException
java.sql.SQLException: org.apache.thrift.transport.TTransportException: org.apache.http.client.ClientProtocolException
at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:311)
at com.contactlab.connection.ConnectToMetaStore.executePrepararation(ConnectToMetaStore.java:169)
at com.contactlab.connection.ConnectToMetaStore.main(ConnectToMetaStore.java:67)
Caused by: org.apache.thrift.transport.TTransportException: org.apache.http.client.ClientProtocolException
at org.apache.thrift.transport.THttpClient.flushUsingHttpClient(THttpClient.java:281)
at org.apache.thrift.transport.THttpClient.flush(THttpClient.java:297)
at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:65)
at org.apache.hive.service.cli.thrift.TCLIService$Client.send_GetOperationStatus(TCLIService.java:403)
at org.apache.hive.service.cli.thrift.TCLIService$Client.GetOperationStatus(TCLIService.java:395)
at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:280)
... 2 more
Caused by: org.apache.http.client.ClientProtocolException
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:909)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:827)
at org.apache.thrift.transport.THttpClient.flushUsingHttpClient(THttpClient.java:235)
... 7 more
Caused by: org.apache.http.HttpException: Unknown exception in doAs
at org.apache.hive.jdbc.HttpKerberosRequestInterceptor.process(HttpKerberosRequestInterceptor.java:69)
at org.apache.http.protocol.ImmutableHttpProcessor.process(ImmutableHttpProcessor.java:109)
at org.apache.http.protocol.HttpRequestExecutor.preProcess(HttpRequestExecutor.java:176)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:518)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
... 9 more
Caused by: java.lang.reflect.UndeclaredThrowableException: Unknown exception in doAs
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1203)
at org.apache.hive.service.auth.HttpAuthUtils.getKerberosServiceTicket(HttpAuthUtils.java:67)
at org.apache.hive.jdbc.HttpKerberosRequestInterceptor.process(HttpKerberosRequestInterceptor.java:63)
... 13 more
Caused by: java.security.PrivilegedActionException: GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
... 15 more
Caused by: GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)
at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:710)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:248)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
at org.apache.hive.service.auth.HttpAuthUtils$HttpKerberosClientAction.run(HttpAuthUtils.java:111)
at org.apache.hive.service.auth.HttpAuthUtils$HttpKerberosClientAction.run(HttpAuthUtils.java:79)
... 18 more
Caused by: KrbException: Fail to create credential. (63) - No service creds
at sun.security.krb5.internal.CredentialsUtil.acquireServiceCreds(CredentialsUtil.java:299)
at sun.security.krb5.Credentials.acquireServiceCreds(Credentials.java:454)
at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:641)
... 22 more



In krb5kdc.log of the kdc server I see the following line:

May 14 13:13:15 <hostname> krb5kdc[23330](info): DISPATCH: repeated (retransmitted?) request from <IP of server running scripts> during request processing, dropping repeated request
krb5kdc: The KDC should discard this request - while dispatching (udp)


How can I avoid the issue? Any idea or suggestion?


Regards,
Luca Rea

[Luca Rea]

The correct entry related to the specific application error:

May 14 06:27:35 <hostname> krb5kdc[23330](info): DISPATCH: repeated (retransmitted?) request from <IP of server running scripts> during request processing, dropping repeated request

krb5kdc: The KDC should discard this request - while dispatching (udp)

Just one minute (60s timeout?)before the fail returned from the command.