Using SwarmSpawner with DockerSpawner

[Justin Vrooman]

I have successfully setup DockerSpawner (https://github.com/jupyter/dockerspawner) to launch instances of my Jupyter setup. This is working quite well. I am now attempting to integrate the SwarmSpawner from (https://github.com/rgbkrk/jupyterhub-compmodels-deploy/blob/92c75b10cfd77d1116b8ce43342e6ccb106bc31d/roles/jupyterhub/files/swarmspawner.py) into my setup. My current swarm is just a docker-machine swarm setup using these instructions: https://docs.docker.com/swarm/install-w-machine/ on the same host that I am attempting to run JupyterHub on. Right now my jupyterhub_config.py is pretty simple:

# Configuration file for jupyterhub.
c = get_config()
import os
import sys
c.JupyterHub.cookie_secret_file = '/root/jupyterhub/jupyterhub_cookie_secret'
c.JupyterHub.ssl_key = '/root/jupyterhub/ssl/cert.key'
c.JupyterHub.ssl_cert = '/root/jupyterhub/ssl/cert.crt'
c.JupyterHub.proxy_auth_token = 'a_generated_proxy_auth_token'
c.JupyterHub.spawner_class = 'swarmspawner.SwarmSpawner'
c.DockerSpawner.container_image = 'vroomanj/systemuser'
c.DockerSpawner.hub_ip_connect = 'ip.addr.of.hub'
c.DockerSpawner.tls_cert = os.environ['DOCKER_TLS_CERT']
c.DockerSpawner.tls_key = os.environ['DOCKER_TLS_KEY']
c.Authenticator.admin_users = {'admin'}
c.JupyterHub.proxy_api_ip = '0.0.0.0'
c.JupyterHub.hub_ip = '0.0.0.0'

 

I have DOCKER_TLS_CERT and DOCKER_TLS_KEY as well as some other environmental variables set as follows:

DOCKER_TLS_CERT=/root/.docker/machine/machines/swarm-master/cert.pem
DOCKER_HOST=https://192.168.99.101:2375
DOCKER_TLS_VERIFY=1
DOCKER_TLS=1
DOCKER_TLS_KEY=/root/.docker/machine/machines/swarm-master/key.pem
DOCKER_CERT_PATH=/root/.docker/machine/machines/swarm-master

And have tried various values for DOCKER_HOST with varying negative results. Does anyone have any suggestions? I think my problem is in the TLS verification so I may attempt a swarm setup without TLS and see if that works but I would eventually need TLS verification working. Can anyone offer any assistance in the area of DockerSpawner/SwarmSpawner?

Thanks!

- Justin

PS. Let me know if you need any additional information.

[Kacper Kowalik]

Hi Justin,
could you also paste what's the problem? Do you have anything in the
logs on either jupyterhub, or swarm nodes?

I'm running pretty much similar config using slightly modified version
of swarmSpawner [1]. One difference is I have:

c.Spawner.tls_assert_hostname = False

in jupyterhub_config.py. Without it docker-py couldn't connect to nodes.
Cheers,
Kacper

[1] https://github.com/jupyter/dockerspawner/pull/54/

[Justin Vrooman]

Depending on what I use for DOCKER_HOST I get a couple of results. This is the most recent:

[root@villain jupyterhub]# jupyterhub
[I 2016-01-16 13:43:12.110 JupyterHub app:523] Loading cookie_secret from /root/jupyterhub/jupyterhub_cookie_secret
[I 2016-01-16 13:43:12.204 JupyterHub app:657] Not using whitelist. Any authenticated user will be allowed.
[E 2016-01-16 13:43:12.235 JupyterHub app:1101]
    Traceback (most recent call last):
      File "/root/jupyterhub/jupyterhub/app.py", line 1098, in launch_instance_async
        yield self.initialize(argv)
      File "/root/jupyterhub/jupyterhub/app.py", line 918, in initialize
        yield self.init_spawners()
      File "/root/jupyterhub/jupyterhub/app.py", line 717, in init_spawners
        status = yield spawner.poll()
      File "/usr/local/lib/python3.4/site-packages/dockerspawner/dockerspawner.py", line 259, in poll
        container = yield self.get_container()
      File "/usr/local/lib/python3.4/site-packages/dockerspawner/dockerspawner.py", line 285, in get_container
        'inspect_container', self.container_name
      File "/usr/local/lib/python3.4/concurrent/futures/_base.py", line 395, in result
        return self.__get_result()
      File "/usr/local/lib/python3.4/concurrent/futures/_base.py", line 354, in __get_result
        raise self._exception
      File "/usr/local/lib/python3.4/concurrent/futures/thread.py", line 54, in run
        result = self.fn(*self.args, **self.kwargs)
      File "/usr/local/lib/python3.4/site-packages/dockerspawner/dockerspawner.py", line 246, in _docker
        m = getattr(self.client, method)
      File "/usr/local/lib/python3.4/site-packages/dockerspawner/dockerspawner.py", line 69, in client
        client = docker.Client(base_url=docker_host, tls=tls_config, version='auto')
      File "/usr/local/lib/python3.4/site-packages/docker/client.py", line 77, in __init__
        self._version = self._retrieve_server_version()
      File "/usr/local/lib/python3.4/site-packages/docker/client.py", line 97, in _retrieve_server_version
        'Error while fetching server API version: {0}'.format(e)
    docker.errors.DockerException: Error while fetching server API version: HTTPSConnectionPool(host='192.168.99.101', port=2375): Max retries exceeded with url: /version (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7fc466ee24a8>: Failed to establish a new connection: [Errno 111] Connection refused',))

I will attempt to add the line you mentioned and post the results when I get back to my workstation.

Thanks!

- Justin

[Kacper Kowalik]

On 01/16/2016 12:58 PM, Justin Vrooman wrote:
> version: HTTPSConnectionPool(host='192.168.99.101', port=2375): Max retries
>>exceeded with url: /version (Caused by
>>NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection
>>object at 0x7fc466ee24a8>: Failed to establish a new connection: [Errno
>>111] Connection refused',))

Doesn't docker-machine set up swarm on 2376 port by default? Can you
double check that your env points to swarm agent rather that docker
daemon without TLS?
Cheers,
Kacper

[Justin Vrooman]

So I got it to start by changing the port and using your suggestion. Now I am back to getting this error:

[E 2016-01-16 14:08:37.429 JupyterHub user:189] Unhandled error starting justin's server: invalid literal for int() with base 10: 'true'
[I 2016-01-16 14:08:37.437 JupyterHub dockerspawner:290] Container 'jupyter-justin' is gone
[W 2016-01-16 14:08:37.438 JupyterHub dockerspawner:261] container not found
[E 2016-01-16 14:08:37.475 JupyterHub web:1524] Uncaught exception POST /hub/login?next= (35.9.44.94)
    HTTPServerRequest(protocol='https', host='villain.egr.msu.edu:8000', method='POST', uri='/hub/login?next=', version='HTTP/1.1', remote_ip='35.9.44.94', headers={'Content-Type': 'application/x-www-form-urlencoded', 'Referer': 'https://villain.egr.msu.edu:8000/hub/login', 'X-Forwarded-Port': '8000', 'X-Forwarded-Proto': 'https', 'Content-Length': '33', 'X-Forwarded-For': '35.9.44.94', 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8', 'Origin': 'https://villain.egr.msu.edu:8000', 'Cache-Control': 'max-age=0', 'Accept-Encoding': 'gzip, deflate', 'Accept-Language': 'en-US,en;q=0.8', 'Connection': 'close', 'Host': 'villain.egr.msu.edu:8000', 'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36', 'Upgrade-Insecure-Requests': '1'})

    Traceback (most recent call last):

      File "/usr/local/lib/python3.4/site-packages/tornado/web.py", line 1445, in _execute
        result = yield result
      File "/root/jupyterhub/jupyterhub/handlers/login.py", line 72, in post
        yield self.spawn_single_user(user)
      File "/root/jupyterhub/jupyterhub/handlers/base.py", line 296, in spawn_single_user
        yield gen.with_timeout(timedelta(seconds=self.slow_spawn_timeout), f)
      File "/root/jupyterhub/jupyterhub/user.py", line 199, in spawn
        raise e
      File "/root/jupyterhub/jupyterhub/user.py", line 180, in spawn
        yield gen.with_timeout(timedelta(seconds=spawner.start_timeout), f)
      File "/root/jupyterhub/swarmspawner.py", line 32, in start
        num_nodes = int(info['DriverStatus'][3][1])
    ValueError: invalid literal for int() with base 10: 'true'

I'll try setting something up without TLS but docker-machine sets up TLS by default and I'm not sure how to disable it on a docker-machine cluster. Maybe I need a better test environment.

[Justin Vrooman]

Holdup, using port 3376 and your suggested line I think it is working (only it can't access the home directory for the user which would make sense in my scenario).

What are the implications of: 
c.Spawner.tls_assert_hostname = False 

[Kacper Kowalik]

What version of docker and swarm are you using? There's a chance that
info format changed.
Could you add debug statement somewhere before line 32 in
/root/jupyterhub/swarmspawner.py and dump whole info['DriverStatus']
part of the dictionary?

Cheers,
Kacper

[Jess Hamrick]

Hi Justin,

Just a quick note — the most recent version of SwarmSpawner is https://github.com/compmodels/jupyterhub/blob/master/swarmspawner.py. That’s probably pretty similar to the version Kacper linked to but is definitely a lot more recent than https://github.com/rgbkrk/jupyterhub-compmodels-deploy/blob/92c75b10cfd77d1116b8ce43342e6ccb106bc31d/roles/jupyterhub/files/swarmspawner.py and includes some important bugfixes.

Cheers,

Jess

--
You received this message because you are subscribed to the Google Groups "Project Jupyter" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jupyter+u...@googlegroups.com.
To post to this group, send email to jup...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jupyter/6adcb838-973b-4e68-81e3-db7cd63720f8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Kacper Kowalik]

Simpler way to do that outside the jupyterhub (since you already have
everything in env):

import docker
from docker.utils import kwargs_from_env
kwargs = kwargs_from_env(assert_hostname=False)
client = docker.Client(version='auto', **kwargs)
print(client.info()['DriverStatus'])

Cheers,
Kacper

[Kacper Kowalik]

I pushed change to PR 54, which allow SwarmSpawner to uses regex to find
nodes' names rather than relying on position in array. Justin, could you
port it[1] to you version of swarmspawner.py and see if it fixes the issue?

[1]
https://github.com/Xarthisius/dockerspawner/commit/e478222cbb3fb2f9cbc1bcb943bc11d3ee4725fa

[Justin Vrooman]

Will try this in a couple of hours. Thanks!

--
You received this message because you are subscribed to a topic in the Google Groups "Project Jupyter" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jupyter/NQ6JrEdwBNE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jupyter+u...@googlegroups.com.

To post to this group, send email to jup...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jupyter/569AA6B9.5020808%40gmail.com.

[Justin Vrooman]

I think things are working. I just need to setup a "real" test swarm environment where I can mount my home volume since it doesn't seem like I can in docker-machine swarm agents.

[Justin Vrooman]

Thank you Jess! I know you are busy.