How to run parallel jobs in Jupyterhub

[Sergei Ponomarev]

Dear Developers,

I have three questions:

1) Is there a way to run parallel jobs in Jupyterhub (via Jupyter Notebook)? For example, can a user submit his/her job to the queue (UGE/SGE) from inside Jupyterhub GUI? Should Jupyterhub in this case be hosted on the head node?

2) Depending on 1) do you recommend running Jupyterhub on the head node and/or regular node? We may have 20-30 students running Jypyterhub at the same time. Would that affect head node performance, or would this be ok considering 1) is implemented?

and

3) Can users share notebooks, and, if so, how can this be achieved? We are using LDAP on our HPC cluster, if that helps.

Thank you in advance!

Sergei

[MinRK]

On Tue, Feb 21, 2017 at 3:31 PM, Sergei Ponomarev <sergei.y....@gmail.com> wrote:

Dear Developers,

I have three questions:

1) Is there a way to run parallel jobs in Jupyterhub (via Jupyter Notebook)? For example, can a user submit his/her job to the queue (UGE/SGE) from inside Jupyterhub GUI? Should Jupyterhub in this case be hosted on the head node?

2) Depending on 1) do you recommend running Jupyterhub on the head node and/or regular node? We may have 20-30 students running Jypyterhub at the same time. Would that affect head node performance, or would this be ok considering 1) is implemented?

Yes, you can do this. There are varying levels of implementations of submitting jobs more conveniently from notebooks, but you can always resort to `!qsub`.

Putting JupyterHub on a head node is one such approach. You can run notebooks locally or on work nodes via a Spawner. If you use BatchSpawner, you can run JupyterHub on the login node, and each user's notebook will run on a work node as a real job. This makes the load on the login node very low, because the JupyterHub process itself doesn't do very much other than submit jobs. Some clusters have dedicated a single node to JupyterHub, running notebooks on that node, where users can submit jobs from their notebooks.

-Min

 

and

3) Can users share notebooks, and, if so, how can this be achieved? We are using LDAP on our HPC cluster, if that helps.

Thank you in advance!

Sergei

--
You received this message because you are subscribed to the Google Groups "Project Jupyter" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jupyter+unsubscribe@googlegroups.com.
To post to this group, send email to jup...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jupyter/6899fcb4-ac97-4852-9332-b0f79f513610%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Sergei Ponomarev]

Thanks, Min! Regarding 3) notebook sharing we will wait for version 0.8 of Jupyterhub. We appreciate your help!

Best regards,

Sergei

--
You received this message because you are subscribed to a topic in the Google Groups "Project Jupyter" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jupyter/eMR9oFC83FI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jupyter+unsubscribe@googlegroups.com.

To post to this group, send email to jup...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jupyter/CAHNn8BWbQ_7iQakKjf6CxNSzf0XRzYX%3Dyeaa6AuZEAhGQOYn0A%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

*****************************************************************************************
Sincerely Yours,

Sergei Y. Ponomarev (NIH/NIAID) [C], PhD, Contractor
Senior Systems Administrator - Linux and High Performance Computing Team
SRA International, Inc., A CSRA Company
Operations and Engineering Branch
Email: sergei.p...@nih.gov
Work phone: 301-761-6731, Mobile: 301-502-1673
5601 Fishers Lane, 6A66, Rockville, MD 20852
NIAID, National Institutes of Health, DHHS
******************************************************************************************