LDAP Authentication in Livy

[Janki Akhani]

Hi,

I am Janki Akhani from LinkedIn. We are planning to use Livy for Jupyter Notebook. Currently, Livy doesn't support LDAP Authentication. We need to add LDAP authentication to use it. Can you please share your thoughts on the same? Is there any existing ticket or PR for the same?

--

Thanks & Regards,

Janki Akhani
Data Analytics and Infrastructure

+91 96200 51508
jak...@linkedin.com
linkedin.com/in/jakhani

[Saisai Shao]

I think currently Livy only supports Kerberos authentication, and there's no existing ticket or PR to handle this.

I'm not sure if you could use Knox in front of Livy to do authentication and proxy, since Knox support different authentications so we don't need to achieve this inside Livy.

Thanks

Jerry

--
You received this message because you are subscribed to the Google Groups "Livy Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to livy-dev+unsubscribe@cloudera.org.

[Janki Akhani]

I don't have much idea about Knox. Can it work as authentication service in between Sparkmagic and Livy? I found below mentioned steps to configure Knox gateway with Livy.

https://community.hortonworks.com/articles/70499/adding-livy-server-as-service-to-apache-knox.html

Probably we won't be able to use Knox. In this case, we need to add new authentication type in Livy which is LDAP. Does it look ok? Please let us know in case of any issue.

Thanks,

Janki Akhani

[Saisai Shao]

I'm OK with adding new authentication type if it is a simple and less maintenance overhead. Since we don't have much knowledge on LDAP, it would be better to keep it isolated and easy to maintain.

[Janki Akhani]

Sure. Thanks.

To unsubscribe from this group and stop receiving emails from it, send an email to livy-dev+u...@cloudera.org.

--

Thanks & Regards,

Janki Akhani
Data Analytics and Infrastructure

+91 96200 51508
jak...@linkedin.com
linkedin.com/in/jakhani

[Janki Akhani]

Hi Saisai,

I have implemented LDAP authentication in Livy and incorporated couple of review comments mentioned in this PR. While testing it, authentication part is working fine but once authentication is successful, it is not able to create interactive session. It is not able to create Spark application in cluster. Can you please help to look into this issue?

Thanks,

Janki Akhani

Sure. Thanks.

To unsubscribe from this group and stop receiving emails from it, send an email to livy-dev+unsubscribe@cloudera.org.

--

Thanks & Regards,

Janki Akhani
Data Analytics and Infrastructure

+91 96200 51508
jak...@linkedin.com
linkedin.com/in/jakhani

--

Thanks & Regards,

Janki Akhani
Data Analytics and Infrastructure

+91 96200 51508
jak...@linkedin.com
linkedin.com/in/jakhani

[Saisai Shao]

What is the issue you met, do you have exception log?

In the meanwhile please rebase your patch with latest master branch to test again, we recently fixed a binary incompatible issue which will lead to session creation failure.

[Janki Akhani]

I am not getting any exception. It is just not creating session after authentication is completed. I will try to rebase my code and test it once again. Meanwhile can you please try to pull my PR and test with LDAP auth type if possible?

Thanks,

Janki Akhani

[Saisai Shao]

Sorry I don't have such facility to test LDAP, can you please verify if the issue is due to your LDAP fix or not? If not, are you building livy with default Spark version (spark 1.6.3), but running with Spark2? If yes I think you should rebase your code first and test again.

[Janki Akhani]

Hi Saisai,

This issue has been resolved. Basically livy.server.auth.token.max-inactive-interval property needs to be set to keep authentication token active. Once this is set, from sparkmagic for subsequent calls, we need to pass hadoop.auth cookie. hadoop.auth cookie is provided by livy in response to authentication call.

Thanks,

Janki Akhani