how to isolate the spark session for each user on jupyterhub

26 views

Skip to first unread message

Kevin Zhu

unread,

Dec 23, 2018, 9:16:57 AM12/23/18

to Project Jupyter

hi,

I have a AD plugin connected to my jupyterhub installation so that users will be authenticated on jupyterhub via AD connector. The issue is if different users submit their spark job through jupyter notebook on jupyterhub to the shared underlying spark engine, is there a way that each user will have their isolated spark session instead of the shared one?

thanks very much in advance!

Luciano Resende

unread,

Dec 23, 2018, 11:15:17 AM12/23/18

to jup...@googlegroups.com

Use a combination of JupyterHub and Jupyter Enterprise gateway with user impersonation enable (suggested Kerberos)

Some more info at:
https://blog.jupyter.org/on-demand-notebooks-with-jupyterhub-jupyter-enterprise-gateway-and-kubernetes-e8e423695cbf

https://jupyter-enterprise-gateway.readthedocs.io/en/latest/ (see some details about Kerberos on security mode)

And the following ansible script have a bunch of roles to create a kerberized cluster from bare os

https://github.com/lresende/ansible-spark-cluster

> --
> You received this message because you are subscribed to the Google Groups "Project Jupyter" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to jupyter+u...@googlegroups.com.
> To post to this group, send email to jup...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/jupyter/0c9dc82f-1075-4c33-b133-113f0c6cf5bd%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

--
Luciano Resende
http://twitter.com/lresende1975
http://lresende.blogspot.com/