how to use HDFS API in play Framework in a Kerberos-authentication hadoop cluster

[sindy...@gmail.com]

I encounter a problem while going through play framework to use HDFS API, and the cluster we use is hadoop-2.5.0 with Kerberos authentication.  

First I logged in using kinit from CLI to get a Kerberos ticket cache. Then I use HDFS API to get the UserGroupInfomation(ugi). Thereafter I launched the play application that send a request. The request is to create a file in HDFS. While sending the request, I saw the log (is not paste below) showed that I have got the ticket cache successfully, but I got following error later:

org.apache.hadoop.security.AccessControlException: SIMPLE authentication is not enabled.  Available:[TOKEN, KERBEROS]

        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)

        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

        at java.lang.reflect.Constructor.newInstance(Constructor.java:525)

        at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)

        at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73)

        at org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1616)

        at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1488)

        at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1413)

        at org.apache.hadoop.hdfs.DistributedFileSystem$6.doCall(DistributedFileSystem.java:387)

        at org.apache.hadoop.hdfs.DistributedFileSystem$6.doCall(DistributedFileSystem.java:383)

        at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)

        at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:383)

        at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:327)

        at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:906)

        at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:887)

        at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:784)

        at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:773)

 

The error seems that the way of authentication it auto use is SIMPLE, not Kerberos, so I set the authentication way in the conf of ugi, but it doesn’t work.

 

Then I setup the play framework in a hadoop cluster without Kerberos. It works well.

So I want to know how to use HDFS API in play Framework in a Kerberos-authentication hadoop cluster?

Thanks.

[Marius Soutier]

What does this have to do with Play?

--
You received this message because you are subscribed to the Google Groups "play-framework" group.
To unsubscribe from this group and stop receiving emails from it, send an email to play-framewor...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[sindy...@gmail.com]

If I put aside PLAY and run the code in the way "hadoop jar XXX.jar" , it works well. However in play it pops up error.

在 2014年9月12日星期五UTC+8下午5时04分05秒，Marius Soutier写道：

[Gianmario Spacagna]

I am facing the same issue using the spark-notebook https://github.com/andypetrella/spark-notebook which uses Akka and play.

I run kinit before to start my server and then when I try to access any file using sc.textFile("hdfs://hostname/path") it fails with:

org.apache.hadoop.security.AccessControlException: SIMPLE authentication is not enabled. Available:[TOKEN, KERBEROS]

But if I run the same command from the spark-shell it works.

Have you managed to find any solution?

Thanks,

Gianmario

[Igmar Palsenberg]

 

Then I setup the play framework in a hadoop cluster without Kerberos. It works well.

So I want to know how to use HDFS API in play Framework in a Kerberos-authentication hadoop cluster?

Thanks.

That ticket is probably stored in a thread-local variable. That doesn't work in Play, you need to attach it to the request context, or store is for example in cache.

Igmar 

[Gianmario Spacagna]

I solved running:

export HADOOP_CONF_DIR=/etc/hadoop/conf

It looks like when you are not running spark using spark-submit or spark-shell you have to manually export those variables. Then Hadoop will be able to pick the right Kerberos configuration, supposed you have properly set it up.

--
You received this message because you are subscribed to a topic in the Google Groups "play-framework" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/play-framework/o3PQVVdp1cc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to play-framewor...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Gianmario Spacagna

Data Scientist

Analysis, Development and Infrastructure of Big Data applications

# London
# +44 7463525302
# www.sqldilemma.com