CDAP with Apache Ranger

[Mehul Thakkar]

Hello,

I installed Apache Ranger on AWS EC2 instance. The Apache Ranger plugin is installed on AWS EMR cluster. The CDAP installation is also on AWS EMR cluster. I am using LDAP ( using AWS directory service ). The users configured in LDAP are cdapuser1 and cdapuser2. Using Hue, I can view hive table on EMR cluster. The policies I can access from ranger server ( For example: http://10.0.0.1:6080/login.jsp ). Now I would like to control datasets, pipeline etc in CDAP using Apache Ranger.

I am following the document located at https://docs.cask.co/cdap/4.3.1/en/integrations/apache-ranger.html

It seems the information is misleading.

The instructions are as follows.

Installing CDAP Lookup in Ranger

1. Create a new folder called cdap under your Ranger plugins directory. Typically on Ambari clusters it is: /usr/hdp/current/ranger-admin/ews/webapp/WEB-INF/classes/ranger-plugins

mkdir cdap

cd cdap

 I don't understand which Ranger plugins directory under EMR. I have Apache Ranger installed on EC2 instance. All I need is ranger-usersync and ranger-admin.

On EMR cluster under /usr/lib/ranger directory, I have two directories ranger-0.7.1-hdfs-plugin and ranger-0.7.1-hive-plugin. 

Please let me know under which directory I have to add cdap ranger plugin. 

Also while adding user in ranger admin screen, it is asking cdap username. I tried using cdapuser1 which is in LDAP but it didn't work.

Thanks,

Mehul

[Mehul Thakkar]

I found the path for plugin under EMR. It is under /usr/lib/ranger/ranger-0.7.1-admin/ews/webapp/WEB-INF/classes/ranger-plugins. Now only problem is connecting to CDAP from Ranger EC2 instance.

Thanks,

Mehul

[yao...@cask.co]

Hi Mehul,

Did you follow the documentation after you found the ranger plugin path? Can you tell us a bit more about what problem are you having to connect CDAP from Ranger so that we can debug the issue for you?

Thanks,

Yaojie

[Mehul Thakkar]

The document assumes that Apache Ranger is installed on Hadoop cluster but actually it is on separate cluster. I resolved all issues except LDAP connectivity.

From Apache Ranger EC2 instance, I cannot connect to CDAP on EMR cluster to set policies. The LDAP is Simple AD on AWS ( AWS directory service ). I set following parameters in  

/etc/cdap/conf/cdap-site.xml and started cdap services.

security.authentication.handler.useLdaps = true

security.authentication.handler.bindDn = bind...@corp.emr.poc

security.authentication.handler.bindPassword=Bind@User1

security.authentication.handler.userBaseDn=

dc=corp,dc=emr,dc=poc

security.authentication.handler.userRdnAttribute=cn

security.authentication.handler.hostname=10.0.0.1

security.authentication.handler.port=32456

security.authentication.handler.userIdAttribute=sAMAccountName

I enabled debugging too but I cannot find out the root cause. The Ranger cannot connect to CDAP on EMR. I can see that CDAP is working fine using UI. 

Thanks,

Mehul

--
You received this message because you are subscribed to the Google Groups "CDAP User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cdap-user+...@googlegroups.com.
To post to this group, send email to cdap...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cdap-user/7fa7e3dc-af81-4f42-8a1c-608b25cae204%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[yao...@cask.co]

Hi Mekul,

Can you provide the following information:

1. Is authorization enabled on CDAP on the EMR cluster? Check the value of security.authorization.enabled in cdap-site.xml.

2. Can you telnet from the EC2 node that's running Ranger to the EMR node that's running CDAP over port 11015?

Thanks,

Yaojie

[Mehul Thakkar]

Yes I can telnet from EC2 node to EMR node over port 11015. It is enabled in security group. 

Following are other settings.

security.authorization.enabled = true

security.authorization.extension.extra.classpath=/usr/local/ranger-cdap-conf

security.authorization.extension.jar.path=/opt/cdap/master/ext/security/cdap-Ranger-binding-0.1.1.jar

Thanks,

Mehul

To view this discussion on the web visit https://groups.google.com/d/msgid/cdap-user/4de5df35-e2c9-45b6-bfd6-56d0c2605dca%40googlegroups.com.

[yao...@cask.co]

Hi Mehul,

Can you provide us the following information:

1. Which step can you able to get to in the documentation: https://docs.cask.co/cdap/4.3.1/en/integrations/apache-ranger.html#adding-cdap-as-a-service-in-ranger? Are you not able to test connection from Ranger to CDAP (step 4)?

2. The authorization policy will take effect as long as CDAP can connect to Ranger. Can you add a policy in Ranger for CDAP service and check if the policy takes effect in CDAP?

Thanks, 

Yaojie

[Mehul Thakkar]

Yes problem is with step 4. Since there is no connectivity with CDAP, I cannot test Authorization on CDAP datasets. If you want to reproduce this problem quickly then you can follow instructions on  https://aws.amazon.com/blogs/big-data/implementing-authorization-and-auditing-using-apache-ranger-on-amazon-emr/ 

web page and try to use LDAP user "analyst1" from Ranger EC2 instance while trying to connect to CDAP. After setting up EMR using AWS Cloudformation, you can install CDAP on master cluster on EMR.

Thanks,

Mehul

 

To unsubscribe from this group and stop receiving emails from it, send an email to cdap-user+unsubscribe@googlegroups.com.

To post to this group, send email to cdap...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/cdap-user/2694fe31-d7fe-4663-821f-c26f9db34178%40googlegroups.com.

[yao...@cask.co]

Hi Mehul,

Based on the context, kerberos is not enabled on the cluster. To enable authorization and be able to pass the authentication process, kerberos must be configured on the cluster.

Thanks,

Yaojie