Mounting Secured and Unsecured hdfs in Alluxio

[Siva]

Hi  

     I have two hdfs cluster one is secured(Kerberos) and the other is unsecured  It seems hdfs client in alluxio is  always getting initialized with security configuration of the first mount and if the security configuration of the subsequent mount is different , it is not working.

In my case, i am trying to mount  Secured hdfs first, it works and if i try to mount a unsecured one , it fails similarly if i mount unsecured first not able to mount secured after that )

Any one faced similar issue, please let me know if there are any work around?

[cha...@alluxio.com]

Hi Nathan,

To mount a Non-Secure HDFS and a Secure-HDFS simultaneously, each mount point needs a separate UFS configuration.

The HDFS authentication type is different for those two HDFS types, one is SIMPLE and the other one is KERBEROS.

You can specify `alluxio.underfs.hdfs.configuration` for the second mount point, with a separate core-site.xml (or hdfs-site.xml).

Please refer to this page for how to specify per mount point option.

For instance:

./bin/alluxio fs mount --option alluxio.underfs.hdfs.configuration=<your_secure_hdfs_conf> /mnt/secure-hdfs hdfs://<secure-HDFS-namenode>:8020/data/

Hope this helps,

Chaomin

[Nathan]

Hi Chaomin,

    Thanks for the response, yes I have been using two UFS configuration., Unsecured hdfs, i have defined as understore in alluxio-site.properties and  when i start alluxio it is reading the configuration and i can able to browse the data from unstructured hdfs site. 

Now when i try to mount the secured HDFS in Command line(I am passing the hdfs configuration in --option alluxio.underfs.hdfs.configuration)  but I am getting the error

Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]

similarly when initialize alluxio with understore as secure and try to pass hdfs configurations for unsecured mount ,  i am getting error

java.io.IOException: Failed on local exception: java.io.IOException: Server asks us to fall back to SIMPLE auth, but this client is configured to only allow secure connections.

So it is kind of weird and not working , i have tried this in 1.5 and 1.6 version of alluxio

Thanks,

Nathan

[Chaomin Yu]

Looks like the second mount point option "alluxio.underfs.hdfs.configuration" is not consumed property somehow.

Alluxio is trying to use SIMPLE to talk with mounted Secure-HDFS.

Can you please paste the config files detail (core-site.xml) under your specified dir in "--option alluxio.underfs.hdfs.configuration=<dir>"?

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to alluxio-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Cheers,

Chaomin

Alluxio Inc

[Nathan]

Hi Chaomin,

  I am using the following configurations in site.properties

alluxio-site.properties

alluxio.master.hostname=localhost

alluxio.underfs.address=hdfs://securecluster:8020/

alluxio.master.keytab.file=/home/user/alluxio-1.6.0/conf/alluxio.keytab

alluxio.master.principal=all...@XYZ.COM

alluxio.worker.keytab.file=/home/user/alluxio-1.6.0/conf/alluxio.keytab

alluxio.worker.principal=all...@XYZ.COM

alluxio.underfs.hdfs.configuration=/home/user/alluxio-1.6.0/conf/secure/core-site.xml:/home/user/alluxio-1.6.0/conf/secure/hdfs-site.xml

# Worker properties

alluxio.worker.memory.size=1GB

alluxio.worker.tieredstore.levels=1

alluxio.worker.tieredstore.level0.alias=MEM

alluxio.worker.tieredstore.level0.dirs.path=/mnt/ramdisk

# User properties

alluxio.user.file.readtype.default=CACHE_PROMOTE

alluxio.user.file.writetype.default=MUST_CACHE

mount Command

bin/alluxio fs mount --option alluxio.underfs.hdfs.configuration=/home/user/alluxio-1.6.0/conf/unsecure/core-site.xml:/home/user/alluxio-1.6.0/conf/unsecure/hdfs-site.xml /unsecuredcluster hdfs://unsecuredcluster:8020/

Basically it is taking the first security configuration and it is not allowing to override it for the second mount , in this case it is first mounted with kerberos and when I try to mount an unsecured one it fails with java.io.IOException: Failed on local exception: java.io.IOException: Server asks us to fall back to SIMPLE auth, but this client is configured to only allow secure connections.  and when i swap the underfs as unsecured and try mounting secured from cli it gives Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS] error..  

I even tried to make the underfs as underFSStorage alluxio.underfs.address=${alluxio.work.dir}/underFSStorage  and tried to mount both from cli still no luck getting the same error based on the order i run the mount command.

Thanks, 

Nathan

To unsubscribe from this group and stop receiving emails from it, send an email to alluxio-user...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Chaomin Yu]

Hi Nathan,

I was able to reproduce the same issue...

Just double checked the source code and it is because there is only a single UFS object in Alluxio open source edition.

This makes it impossible to talk with both secure and insecure HDFS at the same time.

Unfortunately this use case is not supported yet in Alluxio open source edition.

Sorry for the confusion.

On Fri, Sep 29, 2017 at 2:37 PM, Nathan <r.sivasu...@gmail.com> wrote:

Hi Chaomin,

  I am using the following configurations in site.properties

alluxio-site.properties

alluxio.master.hostname=localhost

alluxio.underfs.address=hdfs://securecluster:8020/

alluxio.master.keytab.file=/home/user/alluxio-1.6.0/conf/alluxio.keytab

alluxio.master.principal=alluxi...@XYZ.COM

alluxio.worker.keytab.file=/home/user/alluxio-1.6.0/conf/alluxio.keytab

alluxio.worker.principal=alluxi...@XYZ.COM

mount Command

To unsubscribe from this group and stop receiving emails from it, send an email to alluxio-users+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Nathan]

Hi Chaomin,  

     Thanks for checking it out, but do you think this is available in any other version today like community/ enterprise ? or can this be opened as a issue in GitHub?

 

Thanks

Nathan