nodebooks does not start with SwarmSpawner.

Peter Zoltan Keresztes

unread,

Feb 13, 2020, 4:19:52 PM2/13/20

to jup...@googlegroups.com

hello,

I am trying to convert the dockerspawner setup into swarmspawner. All I am getting is timeout.

The configuration I am using looking like this:

# Configuration file for JupyterHub.*

c.Application.log_level = 30

c.JupyterHub.ssl_key = '/etc/letsencrypt/live/jhub.z0z0.tk/privkey.pem'

c.JupyterHub.ssl_cert = '/etc/letsencrypt/live/jhub.z0z0.tk/fullchain.pem'

c.JupyterHub.port = 443

#from oauthenticator.github import LocalGitHubOAuthenticator

from oauthenticator.github import GitHubOAuthenticator

c.JupyterHub.authenticator_class = GitHubOAuthenticator

#c.LocalGitHubOAuthenticator.create_system_users = True

import os

c.JupyterHub.spawner_class = 'dockerspawner.SwarmSpawner'

c.SwarmSpawner.network_name = 'jupyterhub_network'

from jupyter_client.localinterfaces import public_ips

c.JupyterHub.hub_ip = '0.0.0.0'

c.JupyterHub.proxy_api_ip = '0.0.0.0'

#c.SwarmSpawner.extra_placement_spec = { 'constraints' : ['node.role==worker'] }

while I can see the service starting:

root@jupyterhub:~# docker service ls

ID NAME MODE REPLICAS IMAGE PORTS

wtq7n8hvymwf jupyter-zozo6015 replicated 0/1 jupyterhub/singleuser:0.9

This is what I get in the logs:

[W 2020-02-13 16:15:12.977 JupyterHub utils:192] Failed to connect to http://jupyter-zozo6015:8888/user/zozo6015/ ([Errno -2] Name or service not known)

[W 2020-02-13 16:15:13.035 JupyterHub utils:192] Failed to connect to http://jupyter-zozo6015:8888/user/zozo6015/ ([Errno -2] Name or service not known)

[W 2020-02-13 16:15:13.214 JupyterHub utils:192] Failed to connect to http://jupyter-zozo6015:8888/user/zozo6015/ ([Errno -2] Name or service not known)

[W 2020-02-13 16:15:13.461 JupyterHub utils:192] Failed to connect to http://jupyter-zozo6015:8888/user/zozo6015/ ([Errno -2] Name or service not known)

[W 2020-02-13 16:15:13.781 JupyterHub utils:192] Failed to connect to http://jupyter-zozo6015:8888/user/zozo6015/ ([Errno -2] Name or service not known)

[W 2020-02-13 16:15:14.293 JupyterHub base:744] User zozo6015 is slow to become responsive (timeout=10)

[W 2020-02-13 16:15:15.134 JupyterHub utils:192] Failed to connect to http://jupyter-zozo6015:8888/user/zozo6015/ ([Errno -2] Name or service not known)

[W 2020-02-13 16:15:18.796 JupyterHub utils:192] Failed to connect to http://jupyter-zozo6015:8888/user/zozo6015/ ([Errno -2] Name or service not known)

[W 2020-02-13 16:15:21.634 JupyterHub utils:192] Failed to connect to http://jupyter-zozo6015:8888/user/zozo6015/ ([Errno -2] Name or service not known)

[W 2020-02-13 16:15:26.651 JupyterHub utils:192] Failed to connect to http://jupyter-zozo6015:8888/user/zozo6015/ ([Errno -2] Name or service not known)

[W 2020-02-13 16:15:31.664 JupyterHub utils:192] Failed to connect to http://jupyter-zozo6015:8888/user/zozo6015/ ([Errno -2] Name or service not known)

[W 2020-02-13 16:15:36.683 JupyterHub utils:192] Failed to connect to http://jupyter-zozo6015:8888/user/zozo6015/ ([Errno -2] Name or service not known)

[W 2020-02-13 16:15:41.701 JupyterHub utils:192] Failed to connect to http://jupyter-zozo6015:8888/user/zozo6015/ ([Errno -2] Name or service not known)

[W 2020-02-13 16:15:44.048 JupyterHub utils:192] Failed to connect to http://jupyter-zozo6015:8888/user/zozo6015/ ([Errno -2] Name or service not known)

[W 2020-02-13 16:15:45.070 JupyterHub utils:192] Failed to connect to http://jupyter-zozo6015:8888/user/zozo6015/ ([Errno -2] Name or service not known)

[W 2020-02-13 16:15:45.071 JupyterHub user:510] zozo6015's server never showed up at http://jupyter-zozo6015:8888/user/zozo6015/ after 30 seconds. Giving up

[E 2020-02-13 16:15:45.229 JupyterHub gen:974] Exception in Future <Task finished coro=<BaseHandler.spawn_single_user.<locals>.finish_user_spawn() done, defined at /srv/.jupyterhub/lib/python3.5/site-packages/jupyterhub/handlers/base.py:629> exception=TimeoutError("Server at http://jupyter-zozo6015:8888/user/zozo6015/ didn't respond in 30 seconds",)> after timeout

Traceback (most recent call last):

File "/srv/.jupyterhub/lib/python3.5/site-packages/tornado/gen.py", line 970, in error_callback

future.result()

File "/usr/lib/python3.5/asyncio/futures.py", line 293, in result

raise self._exception

File "/usr/lib/python3.5/asyncio/tasks.py", line 239, in _step

result = coro.send(None)

File "/srv/.jupyterhub/lib/python3.5/site-packages/jupyterhub/handlers/base.py", line 636, in finish_user_spawn

await spawn_future

File "/srv/.jupyterhub/lib/python3.5/site-packages/jupyterhub/user.py", line 528, in spawn

raise e

File "/srv/.jupyterhub/lib/python3.5/site-packages/jupyterhub/user.py", line 502, in spawn

resp = await server.wait_up(http=True, timeout=spawner.http_timeout)

File "/srv/.jupyterhub/lib/python3.5/site-packages/jupyterhub/utils.py", line 197, in wait_for_http_server

timeout=timeout

File "/srv/.jupyterhub/lib/python3.5/site-packages/jupyterhub/utils.py", line 155, in exponential_backoff

raise TimeoutError(fail_message)

TimeoutError: Server at http://jupyter-zozo6015:8888/user/zozo6015/ didn't respond in 30 seconds

16:17:39.648 [ConfigProxy] info: 200 GET /api/routes

Can I get any help to make this up and running?

Regards,

Peter

Peter Zoltan Keresztes

unread,

Feb 18, 2020, 5:29:27 AM2/18/20

to Project Jupyter

Is there anyone in here to help me with this issue I am struggling for more than a week with?

Jason Anderson

unread,

Feb 18, 2020, 11:30:38 AM2/18/20

to jup...@googlegroups.com

Hi Peter,

I don't have much experience with Docker Swarm, but this line jumped out to me:

root@jupyterhub:~# docker service ls

ID NAME MODE REPLICAS IMAGE PORTS

wtq7n8hvymwf jupyter-zozo6015 replicated 0/1 jupyterhub/singleuser:0.9

Replicas still show that 0 are up. I would try to check out the containers that are supposed to be running (`docker container ls -a`) to see what is happening. I did a quick StackOverflow search, you might find some help/ideas here[1].

Long story short, I am not sure it is a problem specifically with the JupyterHub setup. Though, I'm not sure what parts need to be configured for Swarm to work. Another thing the docs[2] mention is that the network needs to exist. Is it created already?

docker network create --driver overlay jupyterhub_network

Hope that helps,
/Jason

[1]: https://stackoverflow.com/questions/39284233/docker-service-replicas-remain-0-1
[2]: https://github.com/jupyterhub/dockerspawner/blob/master/README.md#swarmspawner

--
You received this message because you are subscribed to the Google Groups "Project Jupyter" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jupyter+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jupyter/57a3ad96-a24f-4925-964c-3b911ed596a5%40googlegroups.com.

Peter Zoltan Keresztes

unread,

Feb 18, 2020, 12:55:29 PM2/18/20

to Project Jupyter

Already created the overlay network, I get the services up but for a very short time when the services ends with the following error:

[W 2020-02-18 09:51:34.551 JupyterHub utils:192] Failed to connect to http://jupyter-zozo6015:8888/user/zozo6015/ ([Errno -2] Name or service not known)
[W 2020-02-18 09:51:39.566 JupyterHub utils:192] Failed to connect to http://jupyter-zozo6015:8888/user/zozo6015/ ([Errno -2] Name or service not known)
[W 2020-02-18 09:51:41.553 JupyterHub utils:192] Failed to connect to http://jupyter-zozo6015:8888/user/zozo6015/ ([Errno -2] Name or service not known)
[W 2020-02-18 09:51:41.555 JupyterHub user:510] zozo6015's server never showed up at http://jupyter-zozo6015:8888/user/zozo6015/ after 30 seconds. Giving up
[D 2020-02-18 09:51:41.557 JupyterHub swarmspawner:144] Getting task of service 'jupyter-zozo6015'
[D 2020-02-18 09:51:41.558 JupyterHub dockerspawner:777] Getting container 'jupyter-zozo6015'
[D 2020-02-18 09:51:41.593 JupyterHub swarmspawner:133] Service 9ni1q9z status: {'ContainerStatus': {'ContainerID': '9480724f2d31d9fe8a784c99a17441f024ceba683951ad9be4e71766cf8499b7',
                         'ExitCode': 0,
                         'PID': 18817},
     'Message': 'started',
     'PortStatus': {},
     'State': 'running',
     'Timestamp': '2020-02-18T14:51:10.859533481Z'}
[I 2020-02-18 09:51:41.596 JupyterHub dockerspawner:1107] Stopping service jupyter-zozo6015 (id: 9ni1q9z)
[I 2020-02-18 09:51:41.597 JupyterHub swarmspawner:232] Removing service 9ni1q9zsp99khlvpbnl3x4qyx
[D 2020-02-18 09:51:41.653 JupyterHub user:578] Deleting oauth client jupyterhub-user-zozo6015
[E 2020-02-18 09:51:41.674 JupyterHub gen:974] Exception in Future <Task finished coro=<BaseHandler.spawn_single_user.<locals>.finish_user_spawn() done, defined at /srv/.jhub/lib/python3.5/site-packages/jupyterhub/handlers/base.py:629> exception=TimeoutError("Server at http://jupyter-zozo6015:8888/user/zozo6015/ didn't respond in 30 seconds",)> after timeout


    Traceback (most recent call last):

      File "/srv/.jhub/lib/python3.5/site-packages/tornado/gen.py", line 970, in error_callback
        future.result()
      File "/usr/lib/python3.5/asyncio/futures.py", line 293, in result
        raise self._exception
      File "/usr/lib/python3.5/asyncio/tasks.py", line 239, in _step
        result = coro.send(None)
      File "/srv/.jhub/lib/python3.5/site-packages/jupyterhub/handlers/base.py", line 636, in finish_user_spawn
        await spawn_future
      File "/srv/.jhub/lib/python3.5/site-packages/jupyterhub/user.py", line 528, in spawn
        raise e
      File "/srv/.jhub/lib/python3.5/site-packages/jupyterhub/user.py", line 502, in spawn


        resp = await server.wait_up(http=True, timeout=spawner.http_timeout)

      File "/srv/.jhub/lib/python3.5/site-packages/jupyterhub/utils.py", line 197, in wait_for_http_server
        timeout=timeout
      File "/srv/.jhub/lib/python3.5/site-packages/jupyterhub/utils.py", line 155, in exponential_backoff


        raise TimeoutError(fail_message)
    TimeoutError: Server at http://jupyter-zozo6015:8888/user/zozo6015/ didn't respond in 30 seconds

Regards,

Peter

To unsubscribe from this group and stop receiving emails from it, send an email to jup...@googlegroups.com.

Jason Anderson

unread,

Feb 18, 2020, 1:56:50 PM2/18/20

to jup...@googlegroups.com

Hi Peter, those are the logs for JupyterHub itself, but do not contain logs for the individual container (jupyter-zozo6015) - you will probably have to use `docker logs` to get those. Maybe that will contain some additional information. It looks like JupyterHub is doing its job ok, but the user container is immediately exiting (ExitCode: 0 in your posted logs.)

/Jason

On 2/18/20 11:55 AM, Peter Zoltan Keresztes wrote:

Already created the overlay network, I get the services up but for a very short time when the services ends with the following error:

[W 2020-02-18 09:51:34.551 JupyterHub utils:192] Failed to connect to http://jupyter-zozo6015:8888/user/zozo6015/ ([Errno -2] Name or service not known) [W 2020-02-18 09:51:39.566 JupyterHub utils:192] Failed to connect to http://jupyter-zozo6015:8888/user/zozo6015/ ([Errno -2] Name or service not known) [W 2020-02-18 09:51:41.553 JupyterHub utils:192] Failed to connect to http://jupyter-zozo6015:8888/user/zozo6015/ ([Errno -2] Name or service not known)

[W 2020-02-18 09:51:41.555 JupyterHub user:510] zozo6015's server never showed up at http://(:8888/user/zozo6015/ after 30 seconds. Giving up

jupyter-zozo6015'

ContainerStatus': {'ContainerID': '9480724f2d31d9fe8a784c99a17441f024ceba683951ad9be4e71766cf8499b7', 'ExitCode': 0, 'PID': 18817}, 'Message': 'started', 'PortStatus': {}, 'State': 'running', 'Timestamp': '2020-02-18T14:51:10.859533481Z'}

t respond in 30 seconds",)> after timeout

/srv/.jhub/lib/python3.5/site-packages/tornado/gen.py", line 970, in error_callback future.result() File "/usr/lib/python3.5/asyncio/futures.py", line 293, in result raise self._exception File "/usr/lib/python3.5/asyncio/tasks.py", line 239, in _step result = coro.send(None) File "/srv/.jhub/lib/python3.5/site-packages/jupyterhub/handlers/base.py", line 636, in finish_user_spawn await spawn_future File "/srv/.jhub/lib/python3.5/site-packages/jupyterhub/user.py", line 528, in spawn raise e File "/srv/.jhub/lib/python3.5/site-packages/jupyterhub/user.py", line 502, in spawn

/srv/.jhub/lib/python3.5/site-packages/jupyterhub/utils.py", line 197, in wait_for_http_server timeout=timeout File "/srv/.jhub/lib/python3.5/site-packages/jupyterhub/utils.py", line 155, in exponential_backoff

To unsubscribe from this group and stop receiving emails from it, send an email to jupyter+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jupyter/3c901a13-6b0e-4d8b-93f7-6d2e45c95a13%40googlegroups.com.

Peter Zoltan Keresztes

unread,

Feb 18, 2020, 2:02:05 PM2/18/20

to Project Jupyter

The container is shutting down because the jupyterhub it is issuing the shutdown command after it fails connect to the container for 30 seconds.Here are the logs for the container.

root@jupyterhub:/srv/jupyterhub# docker logs -f jupyter-zozo6015.1.u5030s66gvd7rkt4af4bn6kux 
Executing the command: jupyterhub-singleuser --ip="0.0.0.0" --port=8888
[W 2020-02-18 14:27:50.746 SingleUserNotebookApp configurable:168] Config option `open_browser` not recognized by `SingleUserNotebookApp`.  Did you mean `browser`?
[I 2020-02-18 14:27:52.760 SingleUserNotebookApp extension:158] JupyterLab extension loaded from /opt/conda/lib/python3.7/site-packages/jupyterlab
[I 2020-02-18 14:27:52.761 SingleUserNotebookApp extension:159] JupyterLab application directory is /opt/conda/share/jupyter/lab
[I 2020-02-18 14:27:53.666 SingleUserNotebookApp singleuser:406] Starting jupyterhub-singleuser server version 0.9.6
[I 2020-02-18 14:27:53.694 SingleUserNotebookApp notebookapp:1924] Serving notebooks from local directory: /home/jovyan
[I 2020-02-18 14:27:53.696 SingleUserNotebookApp notebookapp:1924] The Jupyter Notebook is running at:
[I 2020-02-18 14:27:53.697 SingleUserNotebookApp notebookapp:1924] http://6ea2c6d31f52:8888/user/zozo6015/
[I 2020-02-18 14:27:53.697 SingleUserNotebookApp notebookapp:1925] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 2020-02-18 14:28:20.462 SingleUserNotebookApp notebookapp:1615] received signal 15, stopping
[I 2020-02-18 14:28:20.472 SingleUserNotebookApp notebookapp:1784] Shutting down 0 kernels

Regards,

Peter

To view this discussion on the web visit https://groups.google.com/d/msgid/jupyter/3c901a13-6b0e-4d8b-93f7-6d2e45c95a13%40googlegroups.com.

Jason Anderson

unread,

Feb 18, 2020, 2:30:12 PM2/18/20

to jup...@googlegroups.com

OK, then I would suggest to try raising the timeout and seeing if that helps. Perhaps it is just taking longer than we would expect to spawn the container. I think the settings are Spawner.http_timeout (default=30) and Spawner.start_timeout (default=60). Try setting them to 5 minutes or something quite high and see if that helps. If it does, then maybe the spawn system is just taking a while to get everything going.

/Jason

To unsubscribe from this group and stop receiving emails from it, send an email to jupyter+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jupyter/8d6df650-600a-460d-92bd-7bd17f98f57f%40googlegroups.com.

Peter Zoltan Keresztes

unread,

Feb 18, 2020, 2:45:28 PM2/18/20

to jup...@googlegroups.com

Same error:

[W 2020-02-18 14:43:52.180 JupyterHub user:510] zozo6015's server never showed up at http://jupyter-zozo6015:8888/user/zozo6015/ after 300 seconds. Giving up
[E 2020-02-18 14:43:52.309 JupyterHub gen:974] Exception in Future <Task finished coro=<BaseHandler.spawn_single_user.<locals>.finish_user_spawn() done, defined at /srv/.jhub/lib/python3.5/site-packages/jupyterhub/handlers/base.py:629> exception=TimeoutError("Server at http://jupyter-zozo6015:8888/user/zozo6015/ didn't respond in 300 seconds",)> after timeout

Traceback (most recent call last):
File "/srv/.jhub/lib/python3.5/site-packages/tornado/gen.py", line 970, in error_callback
future.result()
File "/usr/lib/python3.5/asyncio/futures.py", line 293, in result
raise self._exception
File "/usr/lib/python3.5/asyncio/tasks.py", line 239, in _step
result = coro.send(None)
File "/srv/.jhub/lib/python3.5/site-packages/jupyterhub/handlers/base.py", line 636, in finish_user_spawn
await spawn_future
File "/srv/.jhub/lib/python3.5/site-packages/jupyterhub/user.py", line 528, in spawn
raise e
File "/srv/.jhub/lib/python3.5/site-packages/jupyterhub/user.py", line 502, in spawn
resp = await server.wait_up(http=True, timeout=spawner.http_timeout)
File "/srv/.jhub/lib/python3.5/site-packages/jupyterhub/utils.py", line 197, in wait_for_http_server
timeout=timeout
File "/srv/.jhub/lib/python3.5/site-packages/jupyterhub/utils.py", line 155, in exponential_backoff
raise TimeoutError(fail_message)

TimeoutError: Server at http://jupyter-zozo6015:8888/user/zozo6015/ didn't respond in 300 seconds

To view this discussion on the web visit https://groups.google.com/d/msgid/jupyter/4e28377a-e856-c6e4-234f-b4b3a35801a6%40uchicago.edu.

Jason Anderson

unread,

Feb 18, 2020, 2:58:20 PM2/18/20

to jup...@googlegroups.com

Having never used Swarm, I'm not sure how to help at this point, though it feels like something Swarm-related. I would keep the timeout quite high and do some inspecting of the containers being started by the swarmspawner. Make sure they are connected to the same network as the JupyterHub container. You can also try deleting the service and the network, then re-creating the network. Sometimes I have seen issues that are only resolvable by re-creating a Docker network, whether due to iptables or Linux bridges getting weird settings, I don't know.

If you recreate the network with the --attachable flag you can also attach other containers to it for debugging. You could use an attached container to see if you can indeed hit the port on the Jupyter user container. Given that the error appears to be DNS-related ("Name or service not known"), I'd suspect that really it's something with the containers not actually being connected to the same network, or the network is misbehaving.

Hope that helps.
/Jason

To view this discussion on the web visit https://groups.google.com/d/msgid/jupyter/da65a48b-6e17-4408-8010-110ba169c3cf%40Spark.

Peter Zoltan Keresztes

unread,

Feb 18, 2020, 3:01:51 PM2/18/20

to jup...@googlegroups.com

There is a mistake in here. The jupyterhub is not running in a container. It is running natively on the master node of the swarm cluster.

Regards,

Peter

To view this discussion on the web visit https://groups.google.com/d/msgid/jupyter/ec9c318b-29dd-3df5-22d4-1c871794d300%40uchicago.edu.

Jason Anderson

unread,

Feb 18, 2020, 3:09:49 PM2/18/20

to jup...@googlegroups.com

Ah yes, that would be the problem. The docs [1] give an example docker-compose reference for the JupyterHub swarm service. It needs to have access to the Docker socket and be placed on a manager node. You should be able to just copy from the README.

Cheers,
/Jason

[1]: https://github.com/jupyterhub/dockerspawner#swarmspawner

To view this discussion on the web visit https://groups.google.com/d/msgid/jupyter/11779b81-0e24-4d7f-b1ef-3cdcf5cddf6c%40Spark.

Peter Zoltan Keresztes

unread,

Feb 18, 2020, 3:12:40 PM2/18/20

to jup...@googlegroups.com

But docker-compose it’s not something that we would want to run in production. This is a development environment and the request is so that it would be as close to the production as a setup as possible. Therefore docker-compose it out of the question.

Regards,

Peter

To view this discussion on the web visit https://groups.google.com/d/msgid/jupyter/2f2db2ce-af37-f164-6c3d-a7eff140604d%40uchicago.edu.

Reply all

Reply to author

Forward