best practice for enabling cluster tab in jupyterhub/dockerspawner

Chris Kees

unread,

Mar 12, 2016, 10:42:10 AM3/12/16

to Project Jupyter

I'd like to enable the cluster tab for a simple MPI profile in the singleuser instances spawned via Dockerspawner. This is for teaching, not serious computing, so I just want to enable mpich within the same docker container. I've verified that the container and the jovyan user can start/use clusters via the mpi profile, but it looks like per-user config is disabled for the notebook so adding the following to my Dockerfile doesn't seem to work for enabling the cluster tab:

RUN ipython profile create mpi --parallel && ipcluster nbextension enable

Should I enable user configs in the jupyterhub-singleuser script I'm using, or is there a better way to enable the extensions? While I'm configuring, would any of the others who are teaching and/or supercomputing have any recommendations on extensions to enable?

Thanks,

Chris

MinRK

unread,

Mar 12, 2016, 6:02:06 PM3/12/16

to jup...@googlegroups.com

Per-user config should be loading; this is a bug in the ipcluster nbextension command, where it doesn’t write the config to the correct place. The nature of the bug means that it should do the right thing if you cd to the ~/.jupyter directory before running it. But a simpler workaround might be to add the config manually (the 4.x, way):

echo 'c.NotebookApp.server_extensions.append("ipyparallel.nbextension")' >> ~/.jupyter/jupyter_notebook_config.py

which effectively accomplishes the same thing.

You can actually do this in /etc/jupyter/jupyter_notebook_config.py, if you want, and it should affect all users.

-MinRK

Thanks,
Chris

--
You received this message because you are subscribed to the Google Groups "Project Jupyter" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jupyter+u...@googlegroups.com.
To post to this group, send email to jup...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jupyter/058ce065-2f28-4ce7-940e-c088a4ac6d2f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Chris Kees

unread,

Mar 13, 2016, 9:54:46 AM3/13/16

to Project Jupyter

Thanks, Min. I'm still not getting the cluster tab to show up. Here's what I have now (I also copied this to /etc/jupyter):

jovyan@ce721be2c025:~/.jupyter$ more jupyter_notebook_config.py

# Copyright (c) Jupyter Development Team.

from jupyter_core.paths import jupyter_data_dir

import subprocess

import os

import errno

import stat

PEM_FILE = os.path.join(jupyter_data_dir(), 'notebook.pem')

c = get_config()

c.NotebookApp.ip = '*'

c.NotebookApp.port = 8888

c.NotebookApp.open_browser = False

# Set a certificate if USE_HTTPS is set to any value

if 'USE_HTTPS' in os.environ:

if not os.path.isfile(PEM_FILE):

# Ensure PEM_FILE directory exists

dir_name = os.path.dirname(PEM_FILE)

try:

os.makedirs(dir_name)

except OSError as exc: # Python >2.5

if exc.errno == errno.EEXIST and os.path.isdir(dir_name):

pass

else: raise

# Generate a certificate if one doesn't exist on disk

subprocess.check_call(['openssl', 'req', '-new',

'-newkey', 'rsa:2048', '-days', '365', '-nodes', '-x509',

'-subj', '/C=XX/ST=XX/L=XX/O=generated/CN=generated',

'-keyout', PEM_FILE, '-out', PEM_FILE])

# Restrict access to PEM_FILE

os.chmod(PEM_FILE, stat.S_IRUSR | stat.S_IWUSR)

c.NotebookApp.certfile = PEM_FILE

# Set a password if PASSWORD is set

if 'PASSWORD' in os.environ:

from IPython.lib import passwd

c.NotebookApp.password = passwd(os.environ['PASSWORD'])

del os.environ['PASSWORD']

c.NotebookApp.server_extensions.append("ipyparallel.nbextension")

MinRK

unread,

Mar 13, 2016, 3:49:50 PM3/13/16

to jup...@googlegroups.com

Can you set `c.Spawner.debug = True` and show the logs from a single-user server's startup?

--
You received this message because you are subscribed to the Google Groups "Project Jupyter" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jupyter+u...@googlegroups.com.
To post to this group, send email to jup...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jupyter/4f7f636e-affb-47c8-8cc9-aa3436dfb26d%40googlegroups.com.

Chris Kees

unread,

Mar 13, 2016, 8:34:06 PM3/13/16

to Project Jupyter

Here is the log after restarting jupyterhub and starting my server:

root@proteus:/var/log# more jupyterhub.log

/usr/local/lib/python3.4/dist-packages/IPython/utils/localinterfaces.py:3: UserWarning: IPython.utils.localinterfaces has moved to jupyter_cli

ent.localinterfaces

warn("IPython.utils.localinterfaces has moved to jupyter_client.localinterfaces")

[I 2016-03-14 00:30:12.066 JupyterHub app:563] Loading cookie_secret from /srv/jupyterhub/jupyterhub_cookie_secret

[I 2016-03-14 00:30:12.275 JupyterHub dockerspawner:332] Container 'jupyter-cekees' is gone

[W 2016-03-14 00:30:12.276 JupyterHub dockerspawner:303] container not found

[I 2016-03-14 00:30:12.417 JupyterHub app:1119] Hub API listening on http://66.216.100.66:8081/hub/

[I 2016-03-14 00:30:12.431 JupyterHub app:865] Starting proxy @ http://*:443/

00:30:12.875 - info: [ConfigProxy] Proxying https://*:443 to http://66.216.100.66:8081

00:30:12.885 - info: [ConfigProxy] Proxy API at http://127.0.0.1:444/api/routes

[I 2016-03-14 00:30:12.955 JupyterHub app:1142] JupyterHub is now running at http://127.0.0.1:443/

[I 2016-03-14 00:30:32.056 JupyterHub log:100] 302 GET / (@172.15.130.90) 4.96ms

[I 2016-03-14 00:30:32.117 JupyterHub log:100] 302 GET /hub (@172.15.130.90) 1.65ms

[I 2016-03-14 00:30:32.211 JupyterHub log:100] 302 GET /hub/ (cek...@172.15.130.90) 12.41ms

[I 2016-03-14 00:30:32.387 JupyterHub log:100] 200 GET /hub/home (cek...@172.15.130.90) 115.24ms

[I 2016-03-14 00:30:46.541 JupyterHub log:100] 302 GET /hub/spawn (cek...@172.15.130.90) 16.00ms

[I 2016-03-14 00:30:46.576 JupyterHub log:100] 302 GET /user/cekees (@172.15.130.90) 1.36ms

[I 2016-03-14 00:30:46.638 JupyterHub dockerspawner:332] Container 'jupyter-cekees' is gone

[W 2016-03-14 00:30:46.640 JupyterHub dockerspawner:303] container not found

[I 2016-03-14 00:30:46.722 JupyterHub dockerspawner:332] Container 'jupyter-cekees' is gone

[I 2016-03-14 00:30:46.902 JupyterHub dockerspawner:394] Created container 'jupyter-cekees' (id: 4d78977) from image cekees/singleuser:latest

[I 2016-03-14 00:30:46.902 JupyterHub dockerspawner:404] Starting container 'jupyter-cekees' (id: 4d78977)

[I 2016-03-14 00:30:49.578 JupyterHub base:301] User cekees server took 2.937 seconds to start

[I 2016-03-14 00:30:49.579 JupyterHub orm:179] Adding user cekees to proxy /user/cekees => http://127.0.0.1:32781

[I 2016-03-14 00:30:49.634 JupyterHub log:100] 302 GET /hub/user/cekees (cek...@172.15.130.90) 2998.45ms

[I 2016-03-14 00:30:49.811 JupyterHub log:100] 200 GET /hub/api/authorizations/cookie/jupyter-hub-token-cekees/[secret] (cek...@172.17.0.2) 43

.51ms

On Saturday, March 12, 2016 at 9:42:10 AM UTC-6, Chris Kees wrote:

MinRK

unread,

Mar 14, 2016, 7:56:42 AM3/14/16

to jup...@googlegroups.com

Sorry, I meant the single-user logs:

docker logs jupyter-cekees

-MinRK

--
You received this message because you are subscribed to the Google Groups "Project Jupyter" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jupyter+u...@googlegroups.com.
To post to this group, send email to jup...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jupyter/50a24742-9607-4a67-9969-8ddf96a96af2%40googlegroups.com.

Chris Kees

unread,

Mar 14, 2016, 10:41:54 AM3/14/16

to Project Jupyter

Ah, OK. Here is the tail end. I'm pretty sure it's just repeating this, and that this explains the problem anyway. Looks like I need to be sure to install ipyparallel and any other extensions in the base ipython and not just my python2 kernel:

[W 2016-03-14 01:59:13.627 cekees configurable:159] Config option `open_browser` not recognized by `SingleUserNotebookApp`, do you mean : `browser`

[W 2016-03-14 01:59:13.640 cekees configurable:159] Config option `open_browser` not recognized by `SingleUserNotebookApp`, do you mean : `browser`

[W 2016-03-14 01:59:13.657 cekees configurable:159] Config option `open_browser` not recognized by `SingleUserNotebookApp`, do you mean : `browser`

[W 2016-03-14 01:59:13.664 cekees loader:410] Unrecognized JSON config file version, assuming version 1

[W 2016-03-14 01:59:13.671 cekees configurable:159] Config option `open_browser` not recognized by `SingleUserNotebookApp`, do you mean : `browser`

[W 2016-03-14 01:59:13.678 cekees loader:410] Unrecognized JSON config file version, assuming version 1

[W 2016-03-14 01:59:13.686 cekees configurable:159] Config option `open_browser` not recognized by `SingleUserNotebookApp`, do you mean : `browser`

[W 2016-03-14 01:59:13.697 cekees configurable:159] Config option `open_browser` not recognized by `SingleUserNotebookApp`, do you mean : `browser`

[W 2016-03-14 01:59:14.568 cekees notebookapp:253] ipywidgets package not installed. Widgets are unavailable.

[W 2016-03-14 01:59:14.616 cekees notebookapp:1013] Error loading server extension ipyparallel.nbextension

Traceback (most recent call last):

File "/usr/local/lib/python3.4/dist-packages/notebook/notebookapp.py", line 1005, in init_server_extensions

mod = importlib.import_module(modulename)

File "/usr/lib/python3.4/importlib/__init__.py", line 109, in import_module

return _bootstrap._gcd_import(name[level:], package, level)

File "<frozen importlib._bootstrap>", line 2254, in _gcd_import

File "<frozen importlib._bootstrap>", line 2237, in _find_and_load

File "<frozen importlib._bootstrap>", line 2212, in _find_and_load_unlocked

File "<frozen importlib._bootstrap>", line 321, in _call_with_frames_removed

File "<frozen importlib._bootstrap>", line 2254, in _gcd_import

File "<frozen importlib._bootstrap>", line 2237, in _find_and_load

File "<frozen importlib._bootstrap>", line 2224, in _find_and_load_unlocked

ImportError: No module named 'ipyparallel'

[I 2016-03-14 01:59:14.621 cekees notebookapp:1079] Serving notebooks from local directory: /home/jovyan/work

[I 2016-03-14 01:59:14.622 cekees notebookapp:1079] 0 active kernels

[I 2016-03-14 01:59:14.622 cekees notebookapp:1079] The Jupyter Notebook is running at: http://0.0.0.0:8888/user/cekees/

[I 2016-03-14 01:59:14.623 cekees notebookapp:1080] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).

[I 2016-03-14 01:59:14.649 cekees log:47] 302 GET /user/cekees (172.17.0.1) 3.29ms

[I 2016-03-14 01:59:14.754 cekees log:47] 302 GET /user/cekees/ (172.15.130.90) 2.14ms

[W 2016-03-14 01:59:16.155 cekees log:47] 404 GET /user/cekees/clusters?_=1457920753154 (172.15.130.90) 27.21ms referer=https://66.216.100.66/user/cekees/tree

On Saturday, March 12, 2016 at 9:42:10 AM UTC-6, Chris Kees wrote:

Chris Kees

unread,

Mar 14, 2016, 11:31:55 AM3/14/16

to Project Jupyter

OK, installing ipyparallel into the python3 installation enables the cluster tab. Now the only problem is that I want the cluster to use python2 engines.

On Saturday, March 12, 2016 at 9:42:10 AM UTC-6, Chris Kees wrote:

Chris Kees

unread,

Mar 14, 2016, 12:48:26 PM3/14/16

to Project Jupyter

I thought I might be able to just set ipcluster_cmd in the profile configs, but it looks like that config option went away. Do I need to set c.IPKernelApp.kernel_class by somehow getting the list of kernels registered with jupyter?

MinRK

unread,

Mar 15, 2016, 5:04:29 AM3/15/16

to jup...@googlegroups.com

Ah, if you want to start the cluster with Python 2, but the notebook server with Python 3 that’s a little bit tricky, but doable.

You need to install ipyparallel with both Python 2 and 3, because it needs to be available to the notebook server for instantiating the IPCluster classes it uses to launch everything, and again available to Python 2 for running the actual processes.

Set the config to launch controller and engines with Python 2, regardless of the current interpreter:

# ~/.ipython/profile_default/ipcluster_config.py (or profile_mpi, etc.)
# (you may need an absolute path for 'python2')
c.IPClusterStart.controller_cmd = ['python2', '-m', 'ipyparallel.controller']
c.IPClusterEngines.engine_cmd = ['python2', '-m', 'ipyparallel.engine']

The ipcluster_cmd doesn’t come up for the clusters tab because the server extension doesn’t call ipcluster as a subprocess, it instantiates the IPCluster classes locally, which then start the controller(s) and engines.

-MinRK

--
You received this message because you are subscribed to the Google Groups "Project Jupyter" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jupyter+u...@googlegroups.com.
To post to this group, send email to jup...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jupyter/6a7dd80d-222b-415c-8979-3b238ccbdc19%40googlegroups.com.

Chris Kees

unread,

Mar 15, 2016, 6:15:18 AM3/15/16

to Project Jupyter

I don't necessarily want to start the notebook server with Python3, that's just the way it's set up with the DockerSpawner example. I believe it's the jupyterhub-singleuser script that uses Python3. Should I try to switch that to Python2?

I tried configuring the commands below and a few permutations, but it looks like something's not quite right. I can't seem to figure out where the engine_cmd value is located.

$ /usr/local/bin/ipcluster start -n 4 --profile=mpi

2016-03-15 10:07:21.111 [IPClusterStart] WARNING | Config option `engine_cmd` not recognized by `IPClusterStart`, do you mean : `en

gine_launcher`

2016-03-15 10:07:21.116 [IPClusterStart] WARNING | Config option `controller_cmd` not recognized by `IPClusterStart`, do you mean o

ne of : `controller_ip ,controller_launcher ,controller_launcher_class`

2016-03-15 10:07:21.123 [IPClusterStart] WARNING | Config option `engine_cmd` not recognized by `IPClusterStart`, do you mean : `en

gine_launcher`

MinRK

unread,

Mar 15, 2016, 7:08:01 AM3/15/16

to jup...@googlegroups.com

On Tue, Mar 15, 2016 at 11:15 AM, Chris Kees <cek...@gmail.com> wrote:

I don't necessarily want to start the notebook server with Python3, that's just the way it's set up with the DockerSpawner example. I believe it's the jupyterhub-singleuser script that uses Python3. Should I try to switch that to Python2?

You can do that, but I wouldn’t recommend it. It requires a bit of fiddling and changing what we install, so let’s try to get it working with the defaults.

I tried configuring the commands below and a few permutations, but it looks like something's not quite right. I can't seem to figure out where the engine_cmd value is located.

$ /usr/local/bin/ipcluster start -n 4 --profile=mpi
2016-03-15 10:07:21.111 [IPClusterStart] WARNING | Config option `engine_cmd` not recognized by `IPClusterStart`, do you mean : `en
gine_launcher`
2016-03-15 10:07:21.116 [IPClusterStart] WARNING | Config option `controller_cmd` not recognized by `IPClusterStart`, do you mean o
ne of : `controller_ip ,controller_launcher ,controller_launcher_class`
2016-03-15 10:07:21.123 [IPClusterStart] WARNING | Config option `engine_cmd` not recognized by `IPClusterStart`, do you mean : `en
gine_launcher`

Whoops, sorry about that (hooray for messages when config is ignored!). It should be:

c.LocalControllerLauncher.controller_cmd
c.MPIEngineSetLauncher.engine_cmd

Not IPClusterX.

-MinRK

To view this discussion on the web visit https://groups.google.com/d/msgid/jupyter/a3a01af7-8bf6-43fa-a54c-2b246c2d4367%40googlegroups.com.

Chris Kees

unread,

Mar 15, 2016, 11:37:13 AM3/15/16

to Project Jupyter

Great! That works except I had to do MPIEngine -> LocalEngine.

c.LocalControllerLauncher.controller_cmd = ['python2', '-m', 'ipyparallel.controller']

c.LocalEngineSetLauncher.engine_cmd = ['python2', '-m', 'ipyparallel.engine']

...

Fernando Perez

unread,

Mar 15, 2016, 7:01:50 PM3/15/16

to Project Jupyter

ps - could we get a summary of this awesome little exchange captured in the docs somewhere? There's a lot of knowledge buried in there that I'm sure could be useful to others...

To view this discussion on the web visit https://groups.google.com/d/msgid/jupyter/5c5eacbe-6366-416d-aad6-9c1fc46a72f3%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Fernando Perez (@fperez_org; http://fperez.org)
fperez.net-at-gmail: mailing lists only (I ignore this when swamped!)
fernando.perez-at-berkeley: contact me here for any direct mail

Chris Kees

unread,

Mar 16, 2016, 12:23:38 PM3/16/16

to Project Jupyter

I'll make a wiki page. Should I put it on the IPython wiki? Maybe 'Cookbook: Parallel IPython config for jupyterhub'?

Fernando Perez

unread,

Mar 16, 2016, 2:44:33 PM3/16/16

to Project Jupyter

On Wed, Mar 16, 2016 at 9:23 AM, Chris Kees <cek...@gmail.com> wrote:

I'll make a wiki page. Should I put it on the IPython wiki? Maybe 'Cookbook: Parallel IPython config for jupyterhub'?

That's a perfect start, unless Carol has a better suggestion? As she makes steady progress on the structure and layout of our docs it might get moved elsewhere, but if nothing else, having the content there is better than not :)

Thanks!!

f

Carol Willing

unread,

Mar 16, 2016, 11:27:29 PM3/16/16

to Project Jupyter, cek...@gmail.com

Hi Chris,

That’s great that you are willing to write something up and pass your
knowledge along to others. Why don’t you start by making the wiki page
on either IPython or JupyterHub. I’m willingc on GitHub, and please do
ask questions and let me know when you have a draft. I’ve opened an
issue on the JupyterHub repo as a place for us to communicate about this
https://github.com/jupyter/jupyterhub/issues/483

All,

If you have information on JupyterHub deployment that you think others
would find useful, please let me know, and I would be happy to assist
you.

Warmly,

Carol

P.S. Thanks Fernando/

> --
> You received this message because you are subscribed to the Google
> Groups "Project Jupyter" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to jupyter+u...@googlegroups.com.
> To post to this group, send email to jup...@googlegroups.com.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/jupyter/CAHAreOq_q30fr4rXBqUULNSCn1_v8k%3DOsWUDUHY4mbVw-9epaw%40mail.gmail.com.

> For more options, visit https://groups.google.com/d/optout.

Carol Willing
Research Software Engineer, Project Jupyter @ Cal Poly
Director, Python Software Foundation

Reply all

Reply to author

Forward