Alluxio doesnt work with HDFS thru auth

109 views
Skip to first unread message

Hakata J

unread,
Sep 7, 2016, 9:55:19 AM9/7/16
to Alluxio Users
I tried introduce alluxio to speed up the processing, our HDFS need kerberos auth. There are quite a few kinds of kerberos configuration re kerberos on Alluxio. From official site, it documented to create a config file named alluxio-site.properties with params:

alluxio.master.keytab.file=<YOUR_HDFS_KEYTAB_FILE_PATH>
alluxio.master.principal=hdfs/<_HOST>@<REALM>
alluxio.worker.keytab.file=<YOUR_HDFS_KEYTAB_FILE_PATH>
alluxio.worker.principal=hdfs/<_HOST>@<REALM>
 
my config file is like:
alluxio.master.keytab.file=/root/hdfs.keytab
alluxio.master.principal=hd...@SAIC.COM
alluxio.worker.keytab.file=/root/hdfs.keytab
alluxio.worker.principal=hd...@SAIC.COM

Does not work.

Others config from web:
Add java opts into alluxio-env.sh

ALLUXIO_JAVA_OPTS+="
 -Djava.security.krb5.realm=SAIC.COM 
 -Djava.security.krb5.kdc=10.32.47.201:88
 -Dalluxio.master.keytab.file=file:///root/mxsdev.keytab
 -Dalluxio.master.principal=mxs...@SAIC.COM
 -Dalluxio.worker.keytab.file=file:///root/mxsdev.keytab
 -Dalluxio.worker.principal=mxs...@SAIC.COM"

Does not work as well. Are the the same config, confused!


Log of worker:
Caused by: java.lang.IllegalArgumentException: Illegal principal name hd...@SAIC.COM: org.apache.hadoop.security.authentication.util.KerberosName$NoMatchingRule: No rules applied to hd...@SAIC.COM
at org.apache.hadoop.security.User.<init>(User.java:50)
at org.apache.hadoop.security.User.<init>(User.java:43)
at org.apache.hadoop.security.UserGroupInformation$HadoopLoginModule.commit(UserGroupInformation.java:197)
... 30 more
Caused by: org.apache.hadoop.security.authentication.util.KerberosName$NoMatchingRule: No rules applied to hd...@SAIC.COM
at org.apache.hadoop.security.authentication.util.KerberosName.getShortName(KerberosName.java:389)
at org.apache.hadoop.security.User.<init>(User.java:48)
... 32 more
2016-09-07 19:39:59,628 ERROR logger.type (AlluxioWorker.java:<init>) - Failed to initialize alluxio.worker.AlluxioWorker
java.lang.IllegalArgumentException: All eligible Under File Systems were unable to create an instance for the given path: hdfs://10.32.47.105:8020/tmp/
java.lang.RuntimeException: java.io.IOException: failure to login


How to solve it?

Hakata J

unread,
Sep 7, 2016, 9:56:11 AM9/7/16
to Alluxio Users
BTW： I am using CDH 5.7, Alluxio 1.2

在 2016年9月7日星期三 UTC+8下午9:55:19，Hakata J写道：
I tried introduce alluxio to speed up the processing, our HDFS need kerberos auth. There are quite a few kinds of kerberos configuration re kerberos on Alluxio. From official site, it documented to create a config file named alluxio-site.properties with params:

alluxio.master.keytab.file=<YOUR_HDFS_KEYTAB_FILE_PATH>
alluxio.master.principal=hdfs/<_HOST>@<REALM>
alluxio.worker.keytab.file=<YOUR_HDFS_KEYTAB_FILE_PATH>
alluxio.worker.principal=hdfs/<_HOST>@<REALM>
 
my config file is like:
alluxio.master.keytab.file=/root/hdfs.keytab
alluxio.master.principal=hdfs@SAIC.COM
alluxio.worker.keytab.file=/root/hdfs.keytab
alluxio.worker.principal=hdfs@SAIC.COM

Does not work.

Others config from web:
Add java opts into alluxio-env.sh

ALLUXIO_JAVA_OPTS+="
 -Djava.security.krb5.realm=SAIC.COM 
 -Djava.security.krb5.kdc=10.32.47.201:88
 -Dalluxio.master.keytab.file=file:///root/mxsdev.keytab
 -Dalluxio.master.principal=mxsd...@SAIC.COM
 -Dalluxio.worker.keytab.file=file:///root/mxsdev.keytab
 -Dalluxio.worker.principal=mxsd...@SAIC.COM"

Chaomin Yu

unread,
Sep 12, 2016, 3:50:37 PM9/12/16
to Hakata J, Alluxio Users
Hi,

The error is due to no Kerberos name rules applied to hd...@SAIC.COM.
By default Hadoop takes the full Kerberos principal with the format: <NAME>/<_HOST>@<REALM>

Can you please try to use the principal name with "/<_HOST>" field? e.g. hdfs/<your_host_name_here>@SAIC.COM

Hope this helps,
Chaomin

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to alluxio-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.



--
Cheers,
Chaomin

Hakata J

unread,
Sep 18, 2016, 5:09:48 AM9/18/16
to Alluxio Users, tanshe...@163.com
So, the kerberos user name must be hdfs, or can be any other users?

在 2016年9月13日星期二 UTC+8上午3:50:37，Chaomin Yu写道：
Hi,

The error is due to no Kerberos name rules applied to hd...@SAIC.COM.
By default Hadoop takes the full Kerberos principal with the format: <NAME>/<_HOST>@<REALM>

Can you please try to use the principal name with "/<_HOST>" field? e.g. hdfs/<your_host_name_here>@SAIC.COM

Hope this helps,
Chaomin
On Wed, Sep 7, 2016 at 6:56 AM, Hakata J <tanshe...@163.com> wrote:
BTW： I am using CDH 5.7, Alluxio 1.2

在 2016年9月7日星期三 UTC+8下午9:55:19，Hakata J写道：
I tried introduce alluxio to speed up the processing, our HDFS need kerberos auth. There are quite a few kinds of kerberos configuration re kerberos on Alluxio. From official site, it documented to create a config file named alluxio-site.properties with params:

alluxio.master.keytab.file=<YOUR_HDFS_KEYTAB_FILE_PATH>
alluxio.master.principal=hdfs/<_HOST>@<REALM>
alluxio.worker.keytab.file=<YOUR_HDFS_KEYTAB_FILE_PATH>
alluxio.worker.principal=hdfs/<_HOST>@<REALM>
 
my config file is like:
alluxio.master.keytab.file=/root/hdfs.keytab
alluxio.master.principal=hd...@SAIC.COM
alluxio.worker.keytab.file=/root/hdfs.keytab
alluxio.worker.principal=hd...@SAIC.COM
To unsubscribe from this group and stop receiving emails from it, send an email to alluxio-user...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Hakata J

unread,
Sep 18, 2016, 5:23:18 AM9/18/16
to Alluxio Users, tanshe...@163.com
I change the alluxio-env.sh like this:
export ALLUXIO_JAVA_OPTS+="
 -Djava.security.krb5.realm=SAIC.COM 
 -Djava.security.krb5.kdc=10.32.47.201:88
 -Dalluxio.master.keytab.file=/root/hdfs.keytab
 -Dalluxio.master.principal=hdfs/saic...@SAIC.COM
 -Dalluxio.worker.keytab.file=/root/hdfs.keytab
 -Dalluxio.worker.principal=hdfs/saic...@SAIC.COM"

it throws out the error as the same:
Caused by: javax.security.auth.login.LoginException: java.lang.IllegalArgumentException: Illegal principal name hd...@SAIC.COM: org.apache.hadoop.security.authentication.util.KerberosName$NoMatchingRule: No rules applied to hd...@SAIC.COM
at org.apache.hadoop.security.UserGroupInformation$HadoopLoginModule.commit(UserGroupInformation.java:199)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at javax.security.auth.login.LoginContext.invoke(LoginContext.java:762)
at javax.security.auth.login.LoginContext.access$000(LoginContext.java:203)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:690)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:688)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:687)
at javax.security.auth.login.LoginContext.login(LoginContext.java:596)
at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:812)
... 17 more
Caused by: java.lang.IllegalArgumentException: Illegal principal name hd...@SAIC.COM: org.apache.hadoop.security.authentication.util.KerberosName$NoMatchingRule: No rules applied to hd...@SAIC.COM

在 2016年9月13日星期二 UTC+8上午3:50:37，Chaomin Yu写道：
Hi,

The error is due to no Kerberos name rules applied to hd...@SAIC.COM.
By default Hadoop takes the full Kerberos principal with the format: <NAME>/<_HOST>@<REALM>

Can you please try to use the principal name with "/<_HOST>" field? e.g. hdfs/<your_host_name_here>@SAIC.COM

Hope this helps,
Chaomin
On Wed, Sep 7, 2016 at 6:56 AM, Hakata J <tanshe...@163.com> wrote:
BTW： I am using CDH 5.7, Alluxio 1.2

在 2016年9月7日星期三 UTC+8下午9:55:19，Hakata J写道：
I tried introduce alluxio to speed up the processing, our HDFS need kerberos auth. There are quite a few kinds of kerberos configuration re kerberos on Alluxio. From official site, it documented to create a config file named alluxio-site.properties with params:

alluxio.master.keytab.file=<YOUR_HDFS_KEYTAB_FILE_PATH>
alluxio.master.principal=hdfs/<_HOST>@<REALM>
alluxio.worker.keytab.file=<YOUR_HDFS_KEYTAB_FILE_PATH>
alluxio.worker.principal=hdfs/<_HOST>@<REALM>
 
my config file is like:
alluxio.master.keytab.file=/root/hdfs.keytab
alluxio.master.principal=hd...@SAIC.COM
alluxio.worker.keytab.file=/root/hdfs.keytab
alluxio.worker.principal=hd...@SAIC.COM
To unsubscribe from this group and stop receiving emails from it, send an email to alluxio-user...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Hakata J

unread,
Sep 18, 2016, 5:27:19 AM9/18/16
to Alluxio Users, tanshe...@163.com
BTW: It confused to me that the kerberos configure should be on alluxio server side(include master and worker), or should be passed from alluxio client. As the document says that: "If you use Alluxio shell, you can add to ALLUXIO_JAVA_OPTS in conf/alluxio-env.sh. ALLUXIO_JAVA_OPTS+=" -Djava.security.krb5.realm=<YOUR_KERBEROS_REALM> -Djava.security.krb5.kdc=<YOUR_KERBEROS_KDC_ADDRESS>", it seem that the kerberos configuration should be on client side. Isnt it? Or it mean that, alluxio server side should use kerberos user "HDFS" and client also need pass it own kerberos user?

在 2016年9月18日星期日 UTC+8下午5:23:18，Hakata J写道：

Chaomin Yu

unread,
Sep 19, 2016, 12:31:26 PM9/19/16
to Hakata J, Alluxio Users
Please see detailed comments inline.

Hope this helps.
Chaomin

On Sun, Sep 18, 2016 at 2:27 AM, Hakata J <tanshe...@163.com> wrote:
BTW: It confused to me that the kerberos configure should be on alluxio server side(include master and worker), or should be passed from alluxio client. As the document says that: "If you use Alluxio shell, you can add to ALLUXIO_JAVA_OPTS in conf/alluxio-env.sh. ALLUXIO_JAVA_OPTS+=" -Djava.security.krb5.realm=<YOUR_KERBEROS_REALM> -Djava.security.krb5.kdc=<YOUR_KERBEROS_KDC_ADDRESS>", it seem that the kerberos configuration should be on client side. Isnt it? Or it mean that, alluxio server side should use kerberos user "HDFS" and client also need pass it own kerberos user?

Both server-side and client-side Kerberos configurations are required to interact with underlying secure HDFS.

The server-side configs are the followings, which will not take effect until you restart the cluster.
"-Dalluxio.master.keytab.file=/root/hdfs.keytab
 -Dalluxio.master.principal=hdfs/saic...@SAIC.COM
 -Dalluxio.worker.keytab.file=/root/hdfs.keytab
 -Dalluxio.worker.principal=hdfs/saic...@SAIC.COM" 

The client-side Kerberos login ticket cache is also required because client sometimes talks with under file system directly.
So it's suggested to add ALLUXIO_JAVA_OPTS+=" -Djava.security.krb5.realm=<YOUR_KERBEROS_REALM> -Djava.security.krb5.kdc=<YOUR_KERBEROS_KDC_ADDRESS>" in conf/alluxio-env.sh. This is a client configuration. 

在 2016年9月18日星期日 UTC+8下午5:23:18，Hakata J写道：
I change the alluxio-env.sh like this:
export ALLUXIO_JAVA_OPTS+="
 -Djava.security.krb5.realm=SAIC.COM 
 -Djava.security.krb5.kdc=10.32.47.201:88
 -Dalluxio.master.keytab.file=/root/hdfs.keytab
 -Dalluxio.master.principal=hdfs/saic...@SAIC.COM
 -Dalluxio.worker.keytab.file=/root/hdfs.keytab
 -Dalluxio.worker.principal=hdfs/saic...@SAIC.COM"

On the server side, it's recommended to use the same service Kerberos user as the HDFS admin. Typically it is hdfs/<your_hostname>@<YOUR_REALM>. But it's not required to be "hdfs" if your hdfs cluster is managed by another Kerberos user name. 


it throws out the error as the same:
Caused by: javax.security.auth.login.LoginException: java.lang.IllegalArgumentException: Illegal principal name hd...@SAIC.COM: org.apache.hadoop.security.authentication.util.KerberosName$NoMatchingRule: No rules applied to hd...@SAIC.COM
at org.apache.hadoop.security.UserGroupInformation$HadoopLoginModule.commit(UserGroupInformation.java:199)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at javax.security.auth.login.LoginContext.invoke(LoginContext.java:762)
at javax.security.auth.login.LoginContext.access$000(LoginContext.java:203)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:690)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:688)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:687)
at javax.security.auth.login.LoginContext.login(LoginContext.java:596)
at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:812)
... 17 more
Caused by: java.lang.IllegalArgumentException: Illegal principal name hd...@SAIC.COM: org.apache.hadoop.security.authentication.util.KerberosName$NoMatchingRule: No rules applied to hd...@SAIC.COM

According to the error log, from the HDFS perspective, the actual Kerberos login user is still hd...@SAIC.COM, rather than hdfs/saic...@SAIC.COM. Can you please double check your client ticket cache by running a "klist" command, to see whether the login user is "hd...@SAIC.COM" or "hdfs/saic...@SAIC.COM" ?

To unsubscribe from this group and stop receiving emails from it, send an email to alluxio-users+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Chaomin Yu

unread,
Oct 20, 2016, 8:24:13 PM10/20/16
to Hakata J, Alluxio Users
Hi Hakata,

Were you able to resolve the problem?

Best,
Chaomin

谈晟

unread,
Oct 25, 2016, 2:34:58 AM10/25/16
to Chaomin Yu, Alluxio Users
no still struggling.


发自网易邮箱手机版


Chaomin Yu

unread,
Nov 18, 2016, 1:15:05 PM11/18/16
to 谈晟, Alluxio Users
Hi,

Did the previous comments make sense for your questions?
Can you please provide the detailed information/logs for the latest problem you are seeing?

Thanks!
Chaomin
Reply all
Reply to author
Forward
0 new messages