JupyterHub Swarm Spawner - Exeception in Future (tornado error)

[Kellen O'Connor]

So I'm relatively new to jupyterhub, and I wanted to try out Swarm Spawner across two separate nodes using docker swarm. I've already got a working swarm set up, and I can start services that get distributed across the worker nodes just fine in docker. Jupyterhub even has no problem creating services, too. It's just that for some reason the Hub itself can't seem to see the services that the spawner is creating. I'm posting this here because I think I've seen this error for other spawners, and I wasn't sure if it had a common fix, regardless of spawner.

Here's the error:

[I 2017-09-27 00:32:17.595 JupyterHub login:91] User logged in: kel
[W 2017-09-27 00:32:47.910 JupyterHub user:278] kel's server never showed up at http://jupyter-d37aebc5ce74fd9e4e744bdb9a9ba06b-1:8888/user/kel after 30 seconds. Giving up
[I 2017-09-27 00:32:47.922 JupyterHub swarmspawner:368] Stopping and removing Docker service jupyter-d37aebc5ce74fd9e4e744bdb9a9ba06b-1 (id: aebmhx7)
[I 2017-09-27 00:32:47.957 JupyterHub swarmspawner:372] Docker service jupyter-d37aebc5ce74fd9e4e744bdb9a9ba06b-1 (id: aebmhx7) removed
[E 2017-09-27 00:32:48.052 JupyterHub gen:914] Exception in Future <tornado.concurrent.Future object at 0x7fb40584beb8> after timeout
    Traceback (most recent call last):
      File "/usr/lib64/python3.4/site-packages/tornado/gen.py", line 910, in error_callback
        future.result()
      File "/usr/lib/python3.4/site-packages/jupyterhub/user.py", line 294, in spawn
        raise e
      File "/usr/lib/python3.4/site-packages/jupyterhub/user.py", line 270, in spawn
        yield self.server.wait_up(http=True, timeout=spawner.http_timeout)
      File "/usr/lib/python3.4/site-packages/jupyterhub/orm.py", line 108, in wait_up
        yield wait_for_http_server(self.url, timeout=timeout)
      File "/usr/lib/python3.4/site-packages/jupyterhub/utils.py", line 94, in wait_for_http_server
        **locals()
    TimeoutError: Server at http://jupyter-d37aebc5ce74fd9e4e744bdb9a9ba06b-1:8888/user/kel didn't respond in 30 seconds

Here are the events from docker:

2017-09-27T00:32:07.579789623-04:00 service create aebmhx7uitx4a8btoquyvknz6 (name=jupyter-d37aebc5ce74fd9e4e744bdb9a9ba06b-1)
2017-09-27T00:32:07.593433646-04:00 service update aebmhx7uitx4a8btoquyvknz6 (name=jupyter-d37aebc5ce74fd9e4e744bdb9a9ba06b-1)
2017-09-27T00:32:07.930763651-04:00 network create h3j59x5lve5npxv6hc6z2rn1b (name=jupyterhub, type=overlay)
2017-09-27T00:32:08.162563411-04:00 image pull jupyter/base-notebook:latest (name=jupyter/base-notebook)
2017-09-27T00:32:08.281409070-04:00 container create f62ea2bf0f64d052313894ac90b6a9066a94c9a4fa69e66b69de9749e75bf16b (com.docker.swarm.node.id=t0r8a6997efajop7hmkez2oav, com.docker.swarm.service.id=aebmhx7uitx4a8btoquyvknz6, com.docker.swarm.service.name=jupyter-d37aebc5ce74fd9e4e744bdb9a9ba06b-1, com.docker.swarm.task=, com.docker.swarm.task.id=w9wczjt08gask9klknohj0nrn, com.docker.swarm.task.name=jupyter-d37aebc5ce74fd9e4e744bdb9a9ba06b-1.1.w9wczjt08gask9klknohj0nrn, image=jupyter/base-notebook:latest, name=jupyter-d37aebc5ce74fd9e4e744bdb9a9ba06b-1.1.w9wczjt08gask9klknohj0nrn)
2017-09-27T00:32:08.636546214-04:00 network connect h3j59x5lve5npxv6hc6z2rn1b (container=f62ea2bf0f64d052313894ac90b6a9066a94c9a4fa69e66b69de9749e75bf16b, name=jupyterhub, type=overlay)
2017-09-27T00:32:08.662711254-04:00 volume mount jupyterhub-user-d37aebc5ce74fd9e4e744bdb9a9ba06b (container=f62ea2bf0f64d052313894ac90b6a9066a94c9a4fa69e66b69de9749e75bf16b, destination=/home/joyvan/work, driver=local, propagation=, read/write=true)
2017-09-27T00:32:08.968032911-04:00 container start f62ea2bf0f64d052313894ac90b6a9066a94c9a4fa69e66b69de9749e75bf16b (com.docker.swarm.node.id=t0r8a6997efajop7hmkez2oav, com.docker.swarm.service.id=aebmhx7uitx4a8btoquyvknz6, com.docker.swarm.service.name=jupyter-d37aebc5ce74fd9e4e744bdb9a9ba06b-1, com.docker.swarm.task=, com.docker.swarm.task.id=w9wczjt08gask9klknohj0nrn, com.docker.swarm.task.name=jupyter-d37aebc5ce74fd9e4e744bdb9a9ba06b-1.1.w9wczjt08gask9klknohj0nrn, image=jupyter/base-notebook:latest, name=jupyter-d37aebc5ce74fd9e4e744bdb9a9ba06b-1.1.w9wczjt08gask9klknohj0nrn)
2017-09-27T00:32:47.955744716-04:00 service remove aebmhx7uitx4a8btoquyvknz6 (name=jupyter-d37aebc5ce74fd9e4e744bdb9a9ba06b-1)
2017-09-27T00:32:50.070579901-04:00 container kill f62ea2bf0f64d052313894ac90b6a9066a94c9a4fa69e66b69de9749e75bf16b (com.docker.swarm.node.id=t0r8a6997efajop7hmkez2oav, com.docker.swarm.service.id=aebmhx7uitx4a8btoquyvknz6, com.docker.swarm.service.name=jupyter-d37aebc5ce74fd9e4e744bdb9a9ba06b-1, com.docker.swarm.task=, com.docker.swarm.task.id=w9wczjt08gask9klknohj0nrn, com.docker.swarm.task.name=jupyter-d37aebc5ce74fd9e4e744bdb9a9ba06b-1.1.w9wczjt08gask9klknohj0nrn, image=jupyter/base-notebook:latest, name=jupyter-d37aebc5ce74fd9e4e744bdb9a9ba06b-1.1.w9wczjt08gask9klknohj0nrn, signal=15)
2017-09-27T00:32:50.215360352-04:00 container die f62ea2bf0f64d052313894ac90b6a9066a94c9a4fa69e66b69de9749e75bf16b (com.docker.swarm.node.id=t0r8a6997efajop7hmkez2oav, com.docker.swarm.service.id=aebmhx7uitx4a8btoquyvknz6, com.docker.swarm.service.name=jupyter-d37aebc5ce74fd9e4e744bdb9a9ba06b-1, com.docker.swarm.task=, com.docker.swarm.task.id=w9wczjt08gask9klknohj0nrn, com.docker.swarm.task.name=jupyter-d37aebc5ce74fd9e4e744bdb9a9ba06b-1.1.w9wczjt08gask9klknohj0nrn, exitCode=0, image=jupyter/base-notebook:latest, name=jupyter-d37aebc5ce74fd9e4e744bdb9a9ba06b-1.1.w9wczjt08gask9klknohj0nrn)
2017-09-27T00:32:50.610724063-04:00 network disconnect h3j59x5lve5npxv6hc6z2rn1b (container=f62ea2bf0f64d052313894ac90b6a9066a94c9a4fa69e66b69de9749e75bf16b, name=jupyterhub, type=overlay)
2017-09-27T00:32:50.622303293-04:00 volume unmount jupyterhub-user-d37aebc5ce74fd9e4e744bdb9a9ba06b (container=f62ea2bf0f64d052313894ac90b6a9066a94c9a4fa69e66b69de9749e75bf16b, driver=local)
2017-09-27T00:32:50.668470503-04:00 container stop f62ea2bf0f64d052313894ac90b6a9066a94c9a4fa69e66b69de9749e75bf16b (com.docker.swarm.node.id=t0r8a6997efajop7hmkez2oav, com.docker.swarm.service.id=aebmhx7uitx4a8btoquyvknz6, com.docker.swarm.service.name=jupyter-d37aebc5ce74fd9e4e744bdb9a9ba06b-1, com.docker.swarm.task=, com.docker.swarm.task.id=w9wczjt08gask9klknohj0nrn, com.docker.swarm.task.name=jupyter-d37aebc5ce74fd9e4e744bdb9a9ba06b-1.1.w9wczjt08gask9klknohj0nrn, image=jupyter/base-notebook:latest, name=jupyter-d37aebc5ce74fd9e4e744bdb9a9ba06b-1.1.w9wczjt08gask9klknohj0nrn)
2017-09-27T00:32:50.669027724-04:00 network destroy h3j59x5lve5npxv6hc6z2rn1b (name=jupyterhub, type=overlay)
2017-09-27T00:32:50.721176231-04:00 container destroy f62ea2bf0f64d052313894ac90b6a9066a94c9a4fa69e66b69de9749e75bf16b (com.docker.swarm.node.id=t0r8a6997efajop7hmkez2oav, com.docker.swarm.service.id=aebmhx7uitx4a8btoquyvknz6, com.docker.swarm.service.name=jupyter-d37aebc5ce74fd9e4e744bdb9a9ba06b-1, com.docker.swarm.task=, com.docker.swarm.task.id=w9wczjt08gask9klknohj0nrn, com.docker.swarm.task.name=jupyter-d37aebc5ce74fd9e4e744bdb9a9ba06b-1.1.w9wczjt08gask9klknohj0nrn, image=jupyter/base-notebook:latest, name=jupyter-d37aebc5ce74fd9e4e744bdb9a9ba06b-1.1.w9wczjt08gask9klknohj0n

And lastly, here is my JupyterHub_Configy.py. Some of options are commented out because I've been tweaking this as I test, but this is how I have it set up currently.



#Configuration file for JupyterHub
import os
import subprocess
import os
import errno
import stat

c = get_config()
#pwd = os.path.dirname(__file__)


c.JupyterHub.ssl_cert = '/srv/jupyterhub/jupyter.hub.crt'
c.JupyterHub.ssl_key = '/srv/jupyterhub/jupyter.hub.key'


# TLS config
c.JupyterHub.ip = '0.0.0.0'
#c.JupyterHub.port = 443
c.JupyterHub.hub_ip = '0.0.0.0'
c.Authenticator.whitelist = {'kel','kellen'}
c.JupyterHub.spawner_class = 'cassinyspawner.SwarmSpawner'
c.JupyterHub.cleanup_servers = False
#c.SwarmSpawner.start_timeout = 300
c.SwarmSpawner.service_image = 'jupyter/base-notebook'
c.SwarmSpawner.jupyterhub_service_name = 'jupyterhub_jupyterhub'
#c.SwarmSpawner.service_prefix = "jupyterhub"
c.SwarmSpawner.networks = ["jupyterhub"]
#notebook_dir = os.environ.get('NOTEBOOK_DIR')
notebook_dir = '/home/jovyan/work'
c.SwarmSpawner.use_user_options = True
c.SwarmSpawner.notebook_dir = notebook_dir   
c.SwarmSpawner.container_spec = {
            'command' : ['start-singleuser.sh'],
            'Image' :'jupyter/base-notebook:latest',
            'mounts' : [{'type' : 'volume',
            'source' : 'jupyterhub-user-{username}',
            'target' : '/home/joyvan/work'}]
          }