(Kerberos) Cascading and Delegation tokens
321 views
Skip to first unread message
Eric Fulton
Oct 22, 2014, 2:38:59 PM10/22/14
to cascadi...@googlegroups.com
We've written our own application master that runs cascading flows. We are also taking care of delegation tokens ourself, and we've noticed an error where cascading SEEMS to be trying to deal the the delegation tokens itself. Is there a way to disable cascading's efforts? Is this even happening?
Chris K Wensel
Oct 22, 2014, 3:36:30 PM10/22/14
to cascadi...@googlegroups.com
I'm going to need a lot more context. Preferably it includes where this is actually happening in the Cascading code (and what versions of things etc).
ckw
On Oct 22, 2014, at 11:38 AM, Eric Fulton <efu...@snaplogic.com> wrote:
We've written our own application master that runs cascading flows. We are also taking care of delegation tokens ourself, and we've noticed an error where cascading SEEMS to be trying to deal the the delegation tokens itself. Is there a way to disable cascading's efforts? Is this even happening?
--
You received this message because you are subscribed to the Google Groups "cascading-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cascading-use...@googlegroups.com.
To post to this group, send email to cascadi...@googlegroups.com.
Visit this group at http://groups.google.com/group/cascading-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/cascading-user/7335bfb5-1210-4132-9650-9972699041a1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Eric Fulton
Oct 22, 2014, 7:48:28 PM10/22/14
to cascadi...@googlegroups.com
OK so in my code I call:
SecurityUtil.setAuthenticationMethod(UserGroupInformation.AuthenticationMethod.KERBEROS, config);
When cascading calls getSplits it eventually reaches FSNameSystem.getDelegationToken(). In that function it checks to see if the the above is true (in isAllowedDelegationTokenOp()) and finds it to be something else. I'm wondering, not sure, if it's possible cascading resets this value at some point in the execution intervening the two points int he code.
My stack trace looks something like this:
2014-10-22T21:02:20,973 E MR_JOB_9242a6eb-8e50-4efb-bf54-9048208889a8 MRPipelineRunner.java [ ] Unable to launch hadoop job for 9242a6eb-8e50-4efb-bf54-9048208889a8 cascading.flow.planner.PlannerException: could not build flow from assembly: [unable to read fields from tap: Hfs["TextLine[['line']->[ALL]]"]["hdfs://cdh-dev-1.clouddev.snaplogic.com:8020/user/ec2-user/snapreduce/city_zip_codes_1000.csv"]]
at cascading.flow.planner.FlowPlanner.handleExceptionDuringPlanning(FlowPlanner.java:577) ~[cascading-core-2.5.5.jar:?]
at cascading.flow.hadoop.planner.HadoopPlanner.buildFlow(HadoopPlanner.java:286) ~[cascading-hadoop2-mr1-2.5.5.jar:?]
at cascading.flow.hadoop.planner.HadoopPlanner.buildFlow(HadoopPlanner.java:80) ~[cascading-hadoop2-mr1-2.5.5.jar:?]
at cascading.flow.FlowConnector.connect(FlowConnector.java:459) ~[cascading-core-2.5.5.jar:?]
at com.snaplogic.cc.yarn.MRJobManager.getFlow(MRJobManager.java:203) ~[jcc-4.0.jar:?]
at com.snaplogic.cc.pipeline.MRPipelineRunner.run(MRPipelineRunner.java:86) [jcc-4.0.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [?:1.7.0_55]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [?:1.7.0_55]
at java.lang.Thread.run(Thread.java:745) [?:1.7.0_55]
Caused by: cascading.tap.TapException: unable to read fields from tap: Hfs["TextLine[['line']->[ALL]]"]["hdfs://cdh-dev-1.clouddev.snaplogic.com:8020/user/ec2-user/snapreduce/city_zip_codes_1000.csv"]
at cascading.scheme.util.DelimitedParser.parseFirstLine(DelimitedParser.java:319) ~[cascading-core-2.5.5.jar:?]
at cascading.scheme.hadoop.TextDelimited.retrieveSourceFields(TextDelimited.java:968) ~[cascading-hadoop2-mr1-2.5.5.jar:?]
at cascading.tap.Tap.retrieveSourceFields(Tap.java:361) ~[cascading-core-2.5.5.jar:?]
at cascading.flow.BaseFlow.retrieveSourceFields(BaseFlow.java:215) ~[cascading-core-2.5.5.jar:?]
at cascading.flow.BaseFlow.<init>(BaseFlow.java:181) ~[cascading-core-2.5.5.jar:?]
at cascading.flow.hadoop.HadoopFlow.<init>(HadoopFlow.java:98) ~[cascading-hadoop2-mr1-2.5.5.jar:?]
at cascading.flow.hadoop.planner.HadoopPlanner.createFlow(HadoopPlanner.java:231) ~[cascading-hadoop2-mr1-2.5.5.jar:?]
at cascading.flow.hadoop.planner.HadoopPlanner.buildFlow(HadoopPlanner.java:244) ~[cascading-hadoop2-mr1-2.5.5.jar:?]
... 7 more
Caused by: org.apache.hadoop.ipc.RemoteException: Delegation Token can be issued only with kerberos or web authentication
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getDelegationToken(FSNamesystem.java:6388)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getDelegationToken(NameNodeRpcServer.java:461)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getDelegationToken(ClientNamenodeProtocolServerSideTranslatorPB.java:906)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1026)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1986)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1982)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1980)
at org.apache.hadoop.ipc.Client.call(Client.java:1410) ~[hadoop-common-2.4.0.jar:?]
at org.apache.hadoop.ipc.Client.call(Client.java:1363) ~[hadoop-common-2.4.0.jar:?]
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) ~[hadoop-common-2.4.0.jar:?]
at com.sun.proxy.$Proxy92.getDelegationToken(Unknown Source) ~[?:?]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.7.0_55]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[?:1.7.0_55]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.7.0_55]
at java.lang.reflect.Method.invoke(Method.java:606) ~[?:1.7.0_55]
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:190) ~[hadoop-common-2.4.0.jar:?]
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:103) ~[hadoop-common-2.4.0.jar:?]
at com.sun.proxy.$Proxy92.getDelegationToken(Unknown Source) ~[?:?]
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getDelegationToken(ClientNamenodeProtocolTranslatorPB.java:854) ~[hadoop-hdfs-2.4.0.jar:?]
at org.apache.hadoop.hdfs.DFSClient.getDelegationToken(DFSClient.java:924) ~[hadoop-hdfs-2.4.0.jar:?]
at org.apache.hadoop.hdfs.DistributedFileSystem.getDelegationToken(DistributedFileSystem.java:1336) ~[hadoop-hdfs-2.4.0.jar:?]
at org.apache.hadoop.fs.FileSystem.collectDelegationTokens(FileSystem.java:527) ~[hadoop-common-2.4.0.jar:?]
at org.apache.hadoop.fs.FileSystem.addDelegationTokens(FileSystem.java:505) ~[hadoop-common-2.4.0.jar:?]
at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:121) ~[hadoop-mapreduce-client-core-2.4.0.jar:?]
at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:100) ~[hadoop-mapreduce-client-core-2.4.0.jar:?]
at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:80) ~[hadoop-mapreduce-client-core-2.4.0.jar:?]
at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:205) ~[hadoop-mapreduce-client-core-2.4.0.jar:?]
at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:304) ~[hadoop-mapreduce-client-core-2.4.0.jar:?]
at cascading.tap.hadoop.io.MultiRecordReaderIterator.initialize(MultiRecordReaderIterator.java:102) ~[cascading-hadoop2-mr1-2.5.5.jar:?]
at cascading.tap.hadoop.io.MultiRecordReaderIterator.<init>(MultiRecordReaderIterator.java:80) ~[cascading-hadoop2-mr1-2.5.5.jar:?]
at cascading.tap.hadoop.io.HadoopTupleEntrySchemeIterator.makeIterator(HadoopTupleEntrySchemeIterator.java:57) ~[cascading-hadoop2-mr1-2.5.5.jar:?]
at cascading.tap.hadoop.io.HadoopTupleEntrySchemeIterator.<init>(HadoopTupleEntrySchemeIterator.java:44) ~[cascading-hadoop2-mr1-2.5.5.jar:?]
at cascading.tap.hadoop.Hfs.openForRead(Hfs.java:503) ~[cascading-hadoop2-mr1-2.5.5.jar:?]
at cascading.tap.hadoop.Hfs.openForRead(Hfs.java:109) ~[cascading-hadoop2-mr1-2.5.5.jar:?]
at cascading.tap.Tap.openForRead(Tap.java:270) ~[cascading-core-2.5.5.jar:?]
at cascading.scheme.util.DelimitedParser.parseFirstLine(DelimitedParser.java:297) ~[cascading-core-2.5.5.jar:?]
at cascading.scheme.hadoop.TextDelimited.retrieveSourceFields(TextDelimited.java:968) ~[cascading-hadoop2-mr1-2.5.5.jar:?]
at cascading.tap.Tap.retrieveSourceFields(Tap.java:361) ~[cascading-core-2.5.5.jar:?]
at cascading.flow.BaseFlow.retrieveSourceFields(BaseFlow.java:215) ~[cascading-core-2.5.5.jar:?]
at cascading.flow.BaseFlow.<init>(BaseFlow.java:181) ~[cascading-core-2.5.5.jar:?]
at cascading.flow.hadoop.HadoopFlow.<init>(HadoopFlow.java:98) ~[cascading-hadoop2-mr1-2.5.5.jar:?]
at cascading.flow.hadoop.planner.HadoopPlanner.createFlow(HadoopPlanner.java:231) ~[cascading-hadoop2-mr1-2.5.5.jar:?]
at cascading.flow.hadoop.planner.HadoopPlanner.buildFlow(HadoopPlanner.java:244) ~[cascading-hadoop2-mr1-2.5.5.jar:?]
I'm not sure if this is enough context to go on. Please let me know if i should add more. It's more than possible that this has nothing to do with cascading, but I figured I'd ask just in case. I'll post whatever i find out in the end regardless (for posterity).
Eric Fulton
Oct 23, 2014, 3:58:49 PM10/23/14
to cascadi...@googlegroups.com
It seems like this only happens when you specify Fields.ALL for fields with TextDelimited on a kerberos enabled cluster. Doesn't seem to happen when you explicitly set the field names.
Eric Fulton
Oct 23, 2014, 5:04:46 PM10/23/14
to cascadi...@googlegroups.com
So i think what's happening is that when we specify Fields.ALL, at plan time it tries to retrieve the headers. During that process it manages mess up dealing with the tokens. I'll keep this updated if i find anything else. MultiRecordReaderIterator seems to be the culprit.
Chris K Wensel
Oct 23, 2014, 6:32:37 PM10/23/14
to cascadi...@googlegroups.com
This was added in 2.5.5
any recommendations on what to do better would be appreciated.
ckw
On Oct 23, 2014, at 2:04 PM, Eric Fulton <efu...@snaplogic.com> wrote:
So i think what's happening is that when we specify Fields.ALL, at plan time it tries to retrieve the headers. During that process it manages mess up dealing with the tokens. I'll keep this updated if i find anything else. MultiRecordReaderIterator seems to be the culprit.
--
You received this message because you are subscribed to the Google Groups "cascading-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cascading-use...@googlegroups.com.
To post to this group, send email to cascadi...@googlegroups.com.
Visit this group at http://groups.google.com/group/cascading-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/cascading-user/ee2c7dea-d92b-44e0-a289-7e76c8a6099d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Eric Fulton
Oct 23, 2014, 7:17:32 PM10/23/14
to cascadi...@googlegroups.com
So we managed to fix the issue by running the:
flowConnector.connect(flowDef)
command inside of a doAs done as a ProxyUser to the UserGroupInformation.getLoginUser() with the same name as the kerberos principal. So, down the line, isAllowedDelegationTokenOp() inside of FSNamesystem will return true.
Mathieu Chataigner
Dec 11, 2014, 3:32:40 PM12/11/14
to cascadi...@googlegroups.com
Hello,
I have an issue with Kerberos since the 2.5.5.
I create a HadoopFlowConnector, connect it to the source sink trap and the pipe.
I may not use it correctly. It worked with the 2.5.4.
Could you help me? Thanks.
Here is the stack trace:
2014-12-11 20:27:22,917 FATAL [IPC Server handler 4 on 56669] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Task: attempt_1418040941248_66263_m_000066_0 - exited : java.io.IOException: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host is: "***************/*.*.*.*"; destination host is: "***************/*.*.*.*":8020;
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:763)
at org.apache.hadoop.ipc.Client.call(Client.java:1242)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202)
at com.sun.proxy.$Proxy10.getBlockLocations(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB.java:155)
at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:164)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:83)
at com.sun.proxy.$Proxy11.getBlockLocations(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:970)
at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:960)
at org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:239)
at org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:206)
at org.apache.hadoop.hdfs.DFSInputStream.<init>(DFSInputStream.java:199)
at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1117)
at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:249)
at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:82)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:746)
at org.apache.hadoop.mapred.LineRecordReader.<init>(LineRecordReader.java:106)
at org.apache.hadoop.mapred.TextInputFormat.getRecordReader(TextInputFormat.java:67)
at cascading.tap.hadoop.io.MultiInputFormat$1.operate(MultiInputFormat.java:253)
at cascading.tap.hadoop.io.MultiInputFormat$1.operate(MultiInputFormat.java:248)
at cascading.util.Util.retry(Util.java:762)
at cascading.tap.hadoop.io.MultiInputFormat.getRecordReader(MultiInputFormat.java:247)
at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.<init>(MapTask.java:166)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:407)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:160)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1438)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:155)
Caused by: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:582)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1438)
at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:546)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:630)
at org.apache.hadoop.ipc.Client$Connection.access$2100(Client.java:252)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1291)
at org.apache.hadoop.ipc.Client.call(Client.java:1209)
... 32 more
Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212)
at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:137)
at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:457)
at org.apache.hadoop.ipc.Client$Connection.access$1400(Client.java:252)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:623)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:620)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1438)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:619)
... 35 more
Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121)
at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:223)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:193)
... 44 more
Chris K Wensel
Dec 12, 2014, 2:37:20 PM12/12/14
to cascadi...@googlegroups.com, Gian Merlino
Sorry, I don’t have any insights.
I’m including the author of the patch that was introduced to this thread.
ckw
To view this discussion on the web visit https://groups.google.com/d/msgid/cascading-user/65303300-b111-4a4d-bd84-8102b5c82086%40googlegroups.com.
Reply all
Reply to author
Forward
0 new messages