How to configure custom spawner?

[Michael Gilbert]

Hi,

I'm trying to create a custom spawner for Jupyterhub (installed on a cluster) that creates spawners on different nodes as opposed to the head node. I've been using the DockerSpawner example as a reference and am having some trouble figuring out how to get this configured properly.

Without my custom spawner, Jupyterhub is working great, and I can create and run notebooks just fine. When I load my spawner and try to login to Jupyterhub, I can see in the logs that the spawner is in fact running instances of jupyterhub-singleuser on a different node. The problem is that the compute node doesn't seem to communicate any information back to the head node. Here's the command my spawner is submiting to the compute node in the start() method:

export JPY_API_TOKEN="ef98aec6e6bb47ab9181b325feb5c6e3; jupyterhub-singleuser --user=mkg52 --port=33911 --cookie-name=jupyter-hub-token-mkg52 --base-url=/user/mkg52 --hub-prefix=/hub/ --hub-api-url=http://172.16.2.1:54321/hub/api --ip=127.0.0.1

Once this is submitted, I'm seeing in the logs a couple of 302 GET responses, a 200, then a constant string of 304s. So it's obviously not communicating to the node. This brings me to my main question:

Would someone be able to explain how I need to setup the compute nodes and/or configuration file in order to talk back to the head node via jupyterhub-singleuser? I have jupyterhub installed on the compute nodes, but do I also need all the node.js dependencies and configuration files and whatnot? How do the ports need to be configured? I'm just a little lost at this point, so any help would be greatly appreciated!

Thank you!

Mike

[Jess Hamrick]

Hi Michael,

In your spawner, are you setting the IP of the spawned servers after they are spawned? That might be causing you problems because if you don’t set the IP, the hub won’t know how to talk to the user servers. You might have a look at my SwarmSpawner for an example, which spawns servers on a cluster of nodes using Docker swarm:

https://github.com/compmodels/jupyterhub-deploy/blob/master/roles/jupyterhub/files/swarmspawner.py#L52

The user servers also need access to the hub, which requires setting the hub_ip. For example, here’s the corresponding jupyterhub_config.py that I’m using:

https://github.com/compmodels/jupyterhub-deploy/blob/master/roles/jupyterhub/templates/jupyterhub_config.py#L27

I don’t think you should need to install anything on the nodes except the jupyterhub-singleuser script (as well as IPython itself).

Cheers,

Jess

--
You received this message because you are subscribed to the Google Groups "Project Jupyter" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jupyter+u...@googlegroups.com.
To post to this group, send email to jup...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jupyter/9d6e51ad-30a1-4dcb-b631-931facf5deec%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Michael Kit Gilbert]

Jess,

Thank you very much for your help! So, I'm assuming what you've adding in your config like with {{ servicenet_ip }} is jinja2? If so, is servicenet_ip something you defined or is it already defined by jupyterhub? 

Thanks again,

Mike

--
You received this message because you are subscribed to a topic in the Google Groups "Project Jupyter" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jupyter/ESAMbMWevB4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jupyter+u...@googlegroups.com.

To post to this group, send email to jup...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jupyter/5E5D4B57-5E04-483D-97D6-BA9E44C56B22%40gmail.com.

[Jess Hamrick]

Hi Mike,

Yes, it’s Jinja (that repository contains an Ansible deployment, which defines some variables and then uses Jinja to render them into templates you define). My deployment was run on Rackspace, so the servicenet_ip is the Rackspace service net IP address of the hub. It can be any IP address of the hub, though, just as long as the nodes can connect to that IP.

Cheers,

Jess

To view this discussion on the web visit https://groups.google.com/d/msgid/jupyter/CADPBQyQ4CtR5YOg81nUWPynZevkuqzAHOA77VDAst_GrPrqD0g%40mail.gmail.com.

[Michael Kit Gilbert]

Thanks Jess. Yes, I have the IP set to the head node's internal IP. I can ping that ip from the compute nodes just fine. I also turned off iptables temporarily to make sure there wasn't a port blocking issue and I still was getting the 304 get messages once I try to login.

One thing that might be a problem is my versions of the different programs. I'm running the dev versions of ipyparallel, ipython, and jupyter-notebook, but the released version of jupyterhub. Do you think this could be the issue? Would you be willing to show me a printout of your python environment? Or at least tell me which versions of which program you're using? Here's my environment:

# packages in environment at /miniconda3/envs/ipython-slurm:

#

decorator                 3.4.2                     <pip>

ipykernel (/home/mkg52/ipykernel-dev/ipykernel) 4.0.0.dev0                <pip>

ipyparallel (/home/mkg52/ipyparallel-dev/ipyparallel) 4.0.0.dev0                <pip>

ipython (/home/mkg52/IPython-dev/ipython) 4.0.0.dev0                <pip>

ipython-genutils          0.1.0                     <pip>

ipywidgets (/home/mkg52/Jupyter/ipywidgets) 4.0.0.dev0                <pip>

jinja2                    2.7.3                     <pip>

jsonschema                2.5.1                     <pip>

jupyter-client            4.0.0                     <pip>

jupyter-core              4.0.1                     <pip>

jupyterhub (/home/mkg52/Jupyter/jupyterhub) 0.2.0.dev0                <pip>

markupsafe                0.23                      <pip>

mistune                   0.5.1                     <pip>

nbconvert                 4.0.0                     <pip>

nbformat                  4.0.0                     <pip>

nose                      1.3.7                    py34_0

notebook (/home/mkg52/Jupyter/notebook) 4.0.0.dev0                <pip>

openssl                   1.0.1k                        1

path.py                   7.4                       <pip>

pexpect                   3.3                       <pip>

pickleshare               0.5                       <pip>

pip                       7.1.0                    py34_0

ptyprocess                0.5                       <pip>

pygments                  2.0.2                     <pip>

python                    3.4.3                         1

pyzmq                     14.7.0                    <pip>

readline                  6.2                           2

requests                  2.7.0                     <pip>

setuptools                18.0.1                   py34_0

simplegeneric             0.8.1                     <pip>

simplepam                 0.1.5                     <pip>

slurmspawner              0.0.1.dev0                <pip>

sqlalchemy                1.0.6                     <pip>

sqlite                    3.8.4.1                       1

system                    5.8                           2

terminado                 0.5                       <pip>

tk                        8.5.18                        0

tornado                   4.2                       <pip>

traitlets                 4.0.0                     <pip>

xz                        5.0.5                         0

zlib                      1.2.8                         0

Thanks again!

Mike

To view this discussion on the web visit https://groups.google.com/d/msgid/jupyter/A0464559-A105-41B8-83FC-0D0F0FC30DF3%40gmail.com.

[Michael Kit Gilbert]

OK so I finally got it to work! My problem was in my custom spawner. When I was starting the server on the node, I was starting it on 127.0.0.1. Once I changed this to 0.0.0.0, it worked just fine. Thanks again for your help :)

Mike

[Prateek Naik]

@Jess Hamrick, The link which you have mentioned is no longer exists. can you please update it

[Jessica B. Hamrick]

Hi Prateek,

The link to the jupyterhub_config.py is:

https://github.com/compmodels/jupyterhub/blob/master/jupyterhub_config.py

SwarmSpawner no longer exists because it is not necessary, please see: https://github.com/jupyterhub/dockerspawner#using-docker-swarm

Cheers,

Jess

To unsubscribe from this group and stop receiving emails from it, send an email to jupyter+unsubscribe@googlegroups.com.

To post to this group, send email to jup...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jupyter/a2631f13-bd4b-44ca-81b6-0cb6a8d499de%40googlegroups.com.