Failed starting a notebook in docker swarm environment

[Peter Zoltan Keresztes]

Hello,

I am running jupyterhub in a docker swarm environment. I can manage start up the jupyterhub container and authenticate to it however when I click start server it hangs for a while then I get the error that:

 

`Spawn failed: Server at http://<contaner if>:30030/user/jovyan/didn't respond in 300 seconds`

In the docker service logs -f <servicename> I can see the following:

jupyter-jovyan.1.yr4fsrhsxdef@vps147843    | Executing the command: jupyterhub-singleuser --ip=0.0.0.0 --port=8888 --NotebookApp.default_url=/lab

 

jupyter-jovyan.1.yr4fsrhsxdef@vps147843    | [W 2020-03-26 21:12:21.487 SingleUserNotebookApp configurable:168] Config option `open_browser` not recognized by `SingleUserNotebookApp`.  Did you mean `browser`?
jupyter-jovyan.1.yr4fsrhsxdef@vps147843    | [I 2020-03-26 21:12:23.502 SingleUserNotebookApp extension:158] JupyterLab extension loaded from /opt/conda/lib/python3.7/site-packages/jupyterlab
jupyter-jovyan.1.yr4fsrhsxdef@vps147843    | [I 2020-03-26 21:12:23.503 SingleUserNotebookApp extension:159] JupyterLab application directory is /opt/conda/share/jupyter/lab
jupyter-jovyan.1.yr4fsrhsxdef@vps147843    | [I 2020-03-26 21:12:25.226 SingleUserNotebookApp singleuser:561] Starting jupyterhub-singleuser server version 1.0.1dev
jupyter-jovyan.1.yr4fsrhsxdef@vps147843    | [I 2020-03-26 21:12:25.237 SingleUserNotebookApp notebookapp:1825] Serving notebooks from local directory: /home/jovyan
jupyter-jovyan.1.yr4fsrhsxdef@vps147843    | [I 2020-03-26 21:12:25.238 SingleUserNotebookApp notebookapp:1825] The Jupyter Notebook is running at:
jupyter-jovyan.1.yr4fsrhsxdef@vps147843    | [I 2020-03-26 21:12:25.238 SingleUserNotebookApp notebookapp:1825] http://08c2db387720:8888/user/jovyan/
jupyter-jovyan.1.yr4fsrhsxdef@vps147843    | [I 2020-03-26 21:12:25.238 SingleUserNotebookApp notebookapp:1826] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
jupyter-jovyan.1.yr4fsrhsxdef@vps147843    | [I 2020-03-26 21:12:25.246 SingleUserNotebookApp singleuser:542] Updating Hub with activity every 300 seconds

If I try connecting to the url manually I am getting a 404 not found and in the logs the following would appear:

jupyter-jovyan.1.yr4fsrhsxdef@vps147843    | [I 2020-03-26 21:14:04.026 SingleUserNotebookApp log:174] 302 GET /user/jovyan -> /user/jovyan/lab? (@10.0.0.2) 2.85ms
jupyter-jovyan.1.yr4fsrhsxdef@vps147843    | [I 2020-03-26 21:14:04.155 SingleUserNotebookApp log:174] 302 GET /user/jovyan/lab? -> /hub/api/oauth2/authorize?client_id=jupyterhub-user-jovyan&redirect_uri=%2Fuser%2Fjovyan%2Foauth_callback&response_type=code&state=[secret] (@10.0.0.2) 6.47ms
jupyter-jovyan.1.yr4fsrhsxdef@vps147843    | [E 2020-03-26 21:14:04.324 SingleUserNotebookApp web:2991] Could not open static file ''
jupyter-jovyan.1.yr4fsrhsxdef@vps147843    | [W 2020-03-26 21:14:04.326 SingleUserNotebookApp log:174] 404 GET /hub/api/oauth2/authorize?client_id=jupyterhub-user-jovyan&redirect_uri=%2Fuser%2Fjovyan%2Foauth_callback&response_type=code&state=[secret] (@10.0.0.2) 44.85ms
jupyter-jovyan.1.yr4fsrhsxdef@vps147843    | [W 2020-03-26 21:14:04.636 SingleUserNotebookApp log:174] 404 GET /user/jovyan/static/components/react/react-dom.production.min.js (@10.0.0.2) 2.42ms

The docker-compose.yml looks like this:

version: "3"

services:
  proxy:
    env_file: .env
    image: jupyterhub/configurable-http-proxy:3.1.1
    networks:
      - jupyterhub-net
    # expose the proxy to the world
    ports:
      - "80:8000"
    command:
      - configurable-http-proxy
      - '--error-target'
      - 'http://hub/hub/error'

  hub:
    # build an image with SwarmSpawner and our jupyterhub_config.py
    env_file: .env
    image: quay.io/redcapcloud.com/jupyterhub:27237066
#    build:
#      context: "."
#      dockerfile: "dockerfiles/Dockerfile"
    # mount the docker socket
    ports:
      - 8081
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock"
    networks:
      - jupyterhub-net

networks:
  jupyterhub-net:
    driver: overlay

and the jupyter_config.py that runs in the jupyterhub container looks like this:

c.ConfigurableHTTPProxy.should_start = False
c.ConfigurableHTTPProxy.api_url = 'http://proxy:8001'
c.JupyterHub.authenticator_class = 'jwtauthenticator.jwtauthenticator.JSONWebTokenLocalAuthenticator'
c.JSONWebTokenLocalAuthenticator.secret = 'G-KaPdSgVkYp3s6v8y/B?E(H+MbQeThW'
c.SwarmSpawner.http_timeout = 300
c.SwarmSpawner.start_timeout = 300
c.JupyterHub.spawner_class = 'dockerspawner.SwarmSpawner'
c.JupyterHub.hub_ip = '0.0.0.0'

# this is the name of the 'service' in docker-compose.yml
#This was set to hub but it did not worked
c.JupyterHub.hub_connect_ip = '172.18.0.1'
# this is the network name for jupyterhub in docker-compose.yml
# with a leading 'swarm_' that docker-compose adds
c.SwarmSpawner.network_name = 'swarm_jupyterhub-net'
# start jupyterlab
c.Spawner.cmd = ["jupyter", "labhub"]
# debug-logging for testing
import logging
c.JupyterHub.log_level = logging.DEBUG

Any idea how to make this work?