Jupyterhub, dockerspawner, and docker swarm

[Ian Stuart]

I am trying to get JupyterHub to talk to an external Docker Swarm - and by this, I mean that the swarm is hosted on a [private] cloud whilst the 'Hub is on an external machine.

Environment:
* jupyterhub 0.7.2 running on CentOS 7
  (initially instantiated inside a docker container on the host, then as an application directly on the host)
* OpenStack cloud in a private space
   (run by another department within this University)
* Using DockerSpawner to start NoteBooks

I can happily fire up the hub, and log in to the hub.
The hub will start & stop containers within the cloud - and I can even name them as I please.
Once I log into the hub, I get presented with the expected "start my server" button
.... so all that is good

jupyterhub_config.py:

>>>: cat config/jupyterhub_config.py | egrep -v ^# | grep .
c.JupyterHub.spawner_class = 'dockerspawner.DockerSpawner'
c.DockerSpawner.container_ip = "0.0.0.0"
c.DockerSpawner.use_internal_ip = False
c.DockerSpawner.hub_ip_connect = '172.16.48.141'
c.DockerSpawner.container_prefix = 'mystring'

The problem:
The problem is I can't actually connect to the server. Here's the error in the log:

[D 2017-03-31 10:33:42.404 JupyterHub dockerspawner:355] Getting container 'mystring-kiz'
[I 2017-03-31 10:33:42.410 JupyterHub dockerspawner:437] Found existing container 'mystring-kiz' (id: 4a5fa6b)
[I 2017-03-31 10:33:42.410 JupyterHub dockerspawner:449] Starting container 'mystring-kiz' (id: 4a5fa6b)
[D 2017-03-31 10:33:42.904 JupyterHub spawner:584] Polling subprocess every 30s
[D 2017-03-31 10:33:52.406 JupyterHub dockerspawner:355] Getting container 'mystring-kiz'
[D 2017-03-31 10:33:52.414 JupyterHub dockerspawner:341] Container 4a5fa6b status: {'Dead': False,
     'Error': '',
     'ExitCode': 0,
     'FinishedAt': '2017-03-31T09:29:19.838356006Z',
     'OOMKilled': False,
     'Paused': False,
     'Pid': 23279,
     'Restarting': False,
     'Running': True,
     'StartedAt': '2017-03-31T09:33:42.668175671Z',
     'Status': 'running'}
[W 2017-03-31 10:33:52.415 JupyterHub base:347] User kiz's server is slow to become responsive (timeout=10)
[D 2017-03-31 10:33:52.415 JupyterHub base:348] Expecting server for kiz at: http://127.0.0.1:32828/user/kiz
[D 2017-03-31 10:33:52.415 JupyterHub base:236] Setting cookie for kiz: jupyter-hub-token-kiz, {}
[I 2017-03-31 10:33:52.429 JupyterHub log:100] 302 GET /hub/user/kiz (kiz@::1) 10440.67ms
[I 2017-03-31 10:33:52.510 JupyterHub log:100] 302 GET /user/kiz (@::1) 1.02ms
[I 2017-03-31 10:33:52.553 JupyterHub log:100] 200 GET /hub/user/kiz (kiz@::1) 7.57ms
[D 2017-03-31 10:33:52.854 JupyterHub log:100] 304 GET /favicon.ico (@::1) 0.65ms
[I 2017-03-31 10:33:57.804 JupyterHub log:100] 200 GET /hub/user/kiz (kiz@::1) 8.85ms
[D 2017-03-31 10:33:57.899 JupyterHub log:100] 304 GET /hub/static/css/style.min.css?v=d96e0760e0c2b7356ce89635b646c350 (@::1) 0.72ms
[D 2017-03-31 10:33:57.903 JupyterHub log:100] 304 GET /hub/static/components/requirejs/require.js?v=6da8be361b9ee26c5e721e76c6d4afce (@::1) 0.74ms
[D 2017-03-31 10:33:58.048 JupyterHub log:100] 304 GET /hub/logo (@::1) 0.71ms
[D 2017-03-31 10:33:58.109 JupyterHub log:100] 304 GET /hub/static/components/jquery/jquery.min.js?v=(& (@::1) 0.77ms
[D 2017-03-31 10:33:58.209 JupyterHub log:100] 304 GET /favicon.ico (@::1) 0.66ms
[D 2017-03-31 10:34:03.212 JupyterHub log:100] 304 GET /hub/user/kiz (kiz@::1) 7.23ms
[D 2017-03-31 10:34:03.336 JupyterHub log:100] 304 GET /hub/static/components/requirejs/require.js?v=6da8be361b9ee26c5e721e76c6d4afce (@::1) 0.73ms
[D 2017-03-31 10:34:03.337 JupyterHub log:100] 304 GET /hub/static/css/style.min.css?v=d96e0760e0c2b7356ce89635b646c350 (@::1) 0.68ms
[D 2017-03-31 10:34:03.455 JupyterHub log:100] 304 GET /hub/logo (@::1) 0.60ms
[D 2017-03-31 10:34:03.457 JupyterHub log:100] 304 GET /hub/static/components/jquery/jquery.min.js?v=(& (@::1) 0.65ms
[D 2017-03-31 10:34:03.565 JupyterHub log:100] 304 GET /favicon.ico (@::1) 0.63ms
[D 2017-03-31 10:34:08.598 JupyterHub log:100] 304 GET /hub/user/kiz (kiz@::1) 6.32ms
[D 2017-03-31 10:34:08.709 JupyterHub log:100] 304 GET /hub/static/css/style.min.css?v=d96e0760e0c2b7356ce89635b646c350 (@::1) 1.18ms
[D 2017-03-31 10:34:08.711 JupyterHub log:100] 304 GET /hub/static/components/requirejs/require.js?v=6da8be361b9ee26c5e721e76c6d4afce (@::1) 0.64ms
[D 2017-03-31 10:34:08.825 JupyterHub log:100] 304 GET /hub/logo (@::1) 0.63ms
[D 2017-03-31 10:34:08.896 JupyterHub log:100] 304 GET /hub/static/components/jquery/jquery.min.js?v=(& (@::1) 0.74ms
[D 2017-03-31 10:34:08.988 JupyterHub log:100] 304 GET /favicon.ico (@::1) 0.64ms
[D 2017-03-31 10:34:12.906 JupyterHub dockerspawner:355] Getting container 'mystring-kiz'
[D 2017-03-31 10:34:12.915 JupyterHub dockerspawner:341] Container 4a5fa6b status: {'Dead': False,
     'Error': '',
     'ExitCode': 0,
     'FinishedAt': '2017-03-31T09:29:19.838356006Z',
     'OOMKilled': False,
     'Paused': False,
     'Pid': 23279,
     'Restarting': False,
     'Running': True,
     'StartedAt': '2017-03-31T09:33:42.668175671Z',
     'Status': 'running'}
[W 2017-03-31 10:34:13.172 JupyterHub user:278] kiz's server never showed up at http://127.0.0.1:32828/user/kiz after 30 seconds. Giving up
[D 2017-03-31 10:34:13.172 JupyterHub dockerspawner:355] Getting container 'mystring-kiz'
[D 2017-03-31 10:34:13.180 JupyterHub dockerspawner:341] Container 4a5fa6b status: {'Dead': False,
     'Error': '',
     'ExitCode': 0,
     'FinishedAt': '2017-03-31T09:29:19.838356006Z',
     'OOMKilled': False,
     'Paused': False,
     'Pid': 23279,
     'Restarting': False,
     'Running': True,
     'StartedAt': '2017-03-31T09:33:42.668175671Z',
     'Status': 'running'}
[I 2017-03-31 10:34:13.180 JupyterHub dockerspawner:519] Stopping container mystring-kiz (id: 4a5fa6b)
[E 2017-03-31 10:34:13.742 JupyterHub gen:878] Exception in Future <tornado.concurrent.Future object at 0x7f12988be3c8> after timeout
    Traceback (most recent call last):
      File "/usr/lib64/python3.4/site-packages/tornado/gen.py", line 874, in error_callback
        future.result()
      File "/usr/lib/python3.4/site-packages/jupyterhub/user.py", line 294, in spawn
        raise e
      File "/usr/lib/python3.4/site-packages/jupyterhub/user.py", line 270, in spawn
        yield self.server.wait_up(http=True, timeout=spawner.http_timeout)
      File "/usr/lib/python3.4/site-packages/jupyterhub/orm.py", line 108, in wait_up
        yield wait_for_http_server(self.url, timeout=timeout)
      File "/usr/lib/python3.4/site-packages/jupyterhub/utils.py", line 94, in wait_for_http_server
        **locals()
    TimeoutError: Server at http://127.0.0.1:32828/user/kiz didn't respond in 30 seconds

So in the initial Expecting server for kiz at: http://127.0.0.1:32828/user/kiz the 32828 matches the published port from the Docker on OpenStack - but surely the 127.0.0.1 is wrong... isn't that looking for the Docker Image on the same host as the Hub?

... how do I correct that? How do I get it to talk to the Swarm on 172.16.48.141

(and yes, I've set the environment variable to get docker to talk to the swarm:
 >>>: export DOCKER_HOST=172.16.48.141:2375
)

In anticipation.... many thanks

--
Ian