Trouble Installing JupyterHub as a service on RHEL 7

[Michael Metts]

Hello,

I would like to try to get some help with our JupyterHub installation.  The problem is that we're unable to get it to function as service on RHEL 7 such that it start/stops when the system does and otherwise provides normal operations to our users.  In our attempts to set it up that way, we get an error saying that it's unable to spawn the a single-user server.  But this behavior is not found when we simply launch the server in a shell.

This is a conda installation with both Python 2.7 and Python 3.5.  The mission of the server is such that we prefer to keep Python 2.7 as the default.

To launch the server as a systemd service, I created the file:  jupyterhub-server.service in /etc/systemd/system.  It looks like this:

[root@crmpred1m system]# cat jupyterhub-server.service

[Unit]

Description=JupyterHub Server

[Service]

Type=simple

User=root

ExecStart=/srv/jupyterhub/run_jupyterhub.sh

Restart=on-failure

RestartSec=30s

[Install]

WantedBy=multi-user.target

The shell script it calls looks like this:

[root@crmpred1m jupyterhub]# cat run_jupyterhub.sh

#!/usr/bin/env bash

#source activate py3

export PATH=/opt/anaconda2/envs/py3/bin:$PATH

export CONDA_ENV_PATH=/opt/anaconda2/envs/py3

export CONDA_DEFAULT_ENV=py3

/opt/anaconda2/envs/py3/bin/jupyterhub -f /srv/jupyterhub/jupyterhub_config.py

I did it this way as I was having no luck with just plain 

/opt/anaconda2/envs/py3/bin/jupyterhub -f /srv/jupyterhub/jupyterhub_config.py

as the ExecStart target because jupyterhub appears to need /opt/anaconda2/envs/py3/bin in the path to function.  I wrapped the startup command in a .sh hoping that that would be a way to give the server the environment it needed to run.  But still no luck getting it to run as a service.

I am able run start and run the server in a shell with the .sh above or just with the straight jupyterhub call so long as I have /opt/anaconda2/envs/py3/bin in my path.

My goal here is to have JupyterHub be a service running under root that starts automatically when the right conditions on the system exist (multi-user run mode etc.).  In any case.  With the server running via systemd using the file jupyter-server.service shown above, what I get is the login prompt when I go to ourmachine.cornerstone.com:8000  ... and I make it through that prompt but then get a message that my server failed to start.  Looking at the log snip below you can see where the failure occurs (look for ">>" on the left).

I guess at this point I'm pretty confused about permission denied since it running as root and I can run the very same startup script in a bash shell having SUed to root.  When I do that, the log spills down my screen with no errors.  The thing is, I can't leave it like this.  I need it to run as a proper service on our Red Hat box.  

A couple of other comments:

1) I don't want to make Python 3 the system-wide default and that's why I really want the JupyterHub service to get a proper path definition so it and the proxy server can run but I don't want to the to change default user behavior to do that.

2) I have attached our config file after the log file (below).

3) My apologies if my questions are ill posed or misplaced.  In that event, please send me in the right direction.

Many thanks,

Mike  

Log snippet:

[root@crmpred1m jupyterhub]# tail -f /var/log/jupyterhub.log

[I 2016-04-13 13:13:36.776 JupyterHub app:1042] ...done

[I 2016-04-13 13:14:07.231 JupyterHub app:558] Loading cookie_secret from /jupyterhub_cookie_secret

[W 2016-04-13 13:14:07.256 JupyterHub app:685] No admin users, admin interface will be unavailable.

[W 2016-04-13 13:14:07.256 JupyterHub app:686] Add any administrative users to `c.Authenticator.admin_users` in config.

[I 2016-04-13 13:14:07.256 JupyterHub app:712] Not using whitelist. Any authenticated user will be allowed.

[I 2016-04-13 13:14:07.269 JupyterHub app:1113] Hub API listening on http://127.0.0.1:8081/hub/

[W 2016-04-13 13:14:07.271 JupyterHub app:851] Running JupyterHub without SSL. There better be SSL termination happening somewhere else...

[I 2016-04-13 13:14:07.272 JupyterHub app:860] Starting proxy @ http://*:8000/

[I 2016-04-13 13:14:07.375 JupyterHub app:1136] JupyterHub is now running at http://127.0.0.1:8000/

[I 2016-04-13 13:14:45.333 JupyterHub log:100] 302 GET /user/mmetts/api/sessions?_=1460565540031 (@10.188.200.26) 1.38ms

[I 2016-04-13 13:14:45.336 JupyterHub log:100] 302 GET /user/mmetts/api/terminals?_=1460565540032 (@10.188.200.26) 0.72ms

[W 2016-04-13 13:14:45.338 JupyterHub base:157] Invalid or expired cookie token

[W 2016-04-13 13:14:45.338 JupyterHub base:157] Invalid or expired cookie token

[W 2016-04-13 13:14:45.341 JupyterHub base:157] Invalid or expired cookie token

[I 2016-04-13 13:14:45.341 JupyterHub log:100] 302 GET /hub/user/mmetts/api/sessions?_=1460565540031 (@10.188.200.26) 1.96ms

[W 2016-04-13 13:14:45.342 JupyterHub base:157] Invalid or expired cookie token

[W 2016-04-13 13:14:45.343 JupyterHub base:157] Invalid or expired cookie token

[W 2016-04-13 13:14:45.345 JupyterHub base:157] Invalid or expired cookie token

[I 2016-04-13 13:14:45.346 JupyterHub log:100] 302 GET /hub/user/mmetts/api/terminals?_=1460565540032 (@10.188.200.26) 1.58ms

[I 2016-04-13 13:14:45.369 JupyterHub log:100] 200 GET /hub/login?next=%2Fhub%2Fuser%2Fmmetts%2Fapi%2Fsessions%3F_%3D1460565540031 (@10.188.200.26) 20.73ms

[I 2016-04-13 13:14:45.372 JupyterHub log:100] 200 GET /hub/login?next=%2Fhub%2Fuser%2Fmmetts%2Fapi%2Fterminals%3F_%3D1460565540032 (@10.188.200.26) 1.00ms

[I 2016-04-13 13:14:56.245 JupyterHub log:100] 302 GET / (@10.188.200.26) 1.21ms

[I 2016-04-13 13:14:56.258 JupyterHub log:100] 302 GET /hub (@10.188.200.26) 0.36ms

[I 2016-04-13 13:14:56.274 JupyterHub log:100] 302 GET /hub/ (@10.188.200.26) 0.63ms

[I 2016-04-13 13:14:56.279 JupyterHub log:100] 200 GET /hub/login (@10.188.200.26) 1.24ms

[I 2016-04-13 13:15:10.935 JupyterHub spawner:436] Spawning jupyterhub-singleuser --user=mmetts --port=37326 --cookie-name=jupyter-hub-token-mmetts --base-url=/user/mmetts --hub-host= --hub-prefix=/hub/ --hub-api-url=http://127.0.0.1:8081/hub/api --ip=127.0.0.1

**>>**[E 2016-04-13 13:15:11.183 JupyterHub user:235] Unhandled error starting mmetts's server: [Errno 13] Permission denied

[E 2016-04-13 13:15:11.239 JupyterHub web:1524] Uncaught exception POST /hub/login?next= (10.188.200.26)

   HTTPServerRequest(protocol='http', host='crmpred1.cornerstone.com:8000', method='POST', uri='/hub/login?next=', version='HTTP/1.1', remote_ip='10.188.200.26', headers={'Cookie': 'user-id=mmetts|Thu%2C%2012%20May%202016%2017%3A07%3A27%20GMT|fP23OPRMyFelyirjUHiWKKsvURgMnvAWsSBIAkG%2B3D4%3D', 'Referer': 'http://crmpred1.cornerstone.com:8000/hub/login', 'Accept-Encoding': 'gzip, deflate', 'X-Forwarded-Port': '8000', 'X-Forwarded-Proto': 'http', 'X-Forwarded-For': '10.188.200.26', 'Host': 'crmpred1.cornerstone.com:8000', 'Content-Length': '35', 'Connection': 'close', 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8', 'Content-Type': 'application/x-www-form-urlencoded', 'Accept-Language': 'en-US,en;q=0.5', 'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0'})

   Traceback (most recent call last):

     File "/opt/anaconda2/envs/py3/lib/python3.5/site-packages/tornado/web.py", line 1445, in _execute

       result = yield result

     File "/opt/anaconda2/envs/py3/lib/python3.5/site-packages/jupyterhub/handlers/login.py", line 72, in post

       yield self.spawn_single_user(user)

     File "/opt/anaconda2/envs/py3/lib/python3.5/site-packages/jupyterhub/handlers/base.py", line 306, in spawn_single_user

       yield gen.with_timeout(timedelta(seconds=self.slow_spawn_timeout), f)

     File "/opt/anaconda2/envs/py3/lib/python3.5/site-packages/jupyterhub/user.py", line 245, in spawn

       raise e

     File "/opt/anaconda2/envs/py3/lib/python3.5/site-packages/jupyterhub/user.py", line 226, in spawn

       yield gen.with_timeout(timedelta(seconds=spawner.start_timeout), f)

     File "/opt/anaconda2/envs/py3/lib/python3.5/types.py", line 243, in wrapped

       coro = func(*args, **kwargs)

     File "/opt/anaconda2/envs/py3/lib/python3.5/site-packages/jupyterhub/spawner.py", line 439, in start

       start_new_session=True, # don't forward signals

     File "/opt/anaconda2/envs/py3/lib/python3.5/subprocess.py", line 950, in __init__

       restore_signals, start_new_session)

     File "/opt/anaconda2/envs/py3/lib/python3.5/subprocess.py", line 1544, in _execute_child

       raise child_exception_type(errno_num, err_msg)

   PermissionError: [Errno 13] Permission denied

[E 2016-04-13 13:15:11.250 JupyterHub log:99] {

     "Cookie": "user-id=mmetts|Thu%2C%2012%20May%202016%2017%3A07%3A27%20GMT|fP23OPRMyFelyirjUHiWKKsvURgMnvAWsSBIAkG%2B3D4%3D",

     "Referer": "http://crmpred1.cornerstone.com:8000/hub/login",

     "Accept-Encoding": "gzip, deflate",

     "X-Forwarded-Port": "8000",

     "X-Forwarded-Proto": "http",

     "X-Forwarded-For": "10.188.200.26",

     "Host": "crmpred1.cornerstone.com:8000",

     "Content-Length": "35",

     "Connection": "close",

     "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",

     "Content-Type": "application/x-www-form-urlencoded",

     "Accept-Language": "en-US,en;q=0.5",

     "User-Agent": "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"

   }

[E 2016-04-13 13:15:11.250 JupyterHub log:100] 500 POST /hub/login?next= (@10.188.200.26) 559.78ms

Config file:

[root@crmpred1m jupyterhub]# cat jupyterhub_config.py

# Configuration file for jupyterhub.

#------------------------------------------------------------------------------

# Configurable configuration

#------------------------------------------------------------------------------

#------------------------------------------------------------------------------

# LoggingConfigurable configuration

#------------------------------------------------------------------------------

# A parent class for Configurables that log.

#

# Subclasses have a log trait, and the default behavior is to get the logger

# from the currently running Application.

#------------------------------------------------------------------------------

# SingletonConfigurable configuration

#------------------------------------------------------------------------------

# A configurable that only allows one instance.

#

# This class is for classes that should only have one instance of itself or

# *any* subclass. To create and retrieve such a class use the

# :meth:`SingletonConfigurable.instance` method.

#------------------------------------------------------------------------------

# Application configuration

#------------------------------------------------------------------------------

# This is an application.

# The date format used by logging formatters for %(asctime)s

#c.Application.log_datefmt = '%Y-%m-%d %H:%M:%S'

# The Logging format template

#c.Application.log_format = '[%(name)s]%(highlevel)s %(message)s'

# Set the log level by value or name.

#c.Application.log_level = 30

#------------------------------------------------------------------------------

# JupyterHub configuration

#------------------------------------------------------------------------------

# An Application for starting a Multi-User Jupyter Notebook server.

# Grant admin users permission to access single-user servers.

#

# Users should be properly informed if this is enabled.

c.JupyterHub.admin_access = False

# DEPRECATED, use Authenticator.admin_users instead.

# c.JupyterHub.admin_users = set()

# Answer yes to any questions (e.g. confirm overwrite)

c.JupyterHub.answer_yes = False

# Class for authenticating users.

#

# This should be a class with the following form:

#

# - constructor takes one kwarg: `config`, the IPython config object.

#

# - is a tornado.gen.coroutine

# - returns username on success, None on failure

# - takes two arguments: (handler, data),

#   where `handler` is the calling web.RequestHandler,

#   and `data` is the POST form data from the login page.

#c.JupyterHub.authenticator_class = 'jupyterhub.auth.PAMAuthenticator'

# The base URL of the entire application

#c.JupyterHub.base_url = '/'

# Whether to shutdown the proxy when the Hub shuts down.

#

# Disable if you want to be able to teardown the Hub while leaving the proxy

# running.

#

# Only valid if the proxy was starting by the Hub process.

#

# If both this and cleanup_servers are False, sending SIGINT to the Hub will

# only shutdown the Hub, leaving everything else running.

#

# The Hub should be able to resume from database state.

c.JupyterHub.cleanup_proxy = True

# Whether to shutdown single-user servers when the Hub shuts down.

#

# Disable if you want to be able to teardown the Hub while leaving the single-

# user servers running.

#

# If both this and cleanup_proxy are False, sending SIGINT to the Hub will only

# shutdown the Hub, leaving everything else running.

#

# The Hub should be able to resume from database state.

c.JupyterHub.cleanup_servers = True

# The config file to load

# c.JupyterHub.config_file = 'jupyterhub_config.py'

# Confirm that JupyterHub should be run without SSL. This is **NOT RECOMMENDED**

# unless SSL termination is being handled by another layer.

c.JupyterHub.confirm_no_ssl = True

# Number of days for a login cookie to be valid. Default is two weeks.

# c.JupyterHub.cookie_max_age_days = 14

# The cookie secret to use to encrypt cookies.

#

# Loaded from the JPY_COOKIE_SECRET env variable by default.

# c.JupyterHub.cookie_secret = b''

# File in which to store the cookie secret.

# c.JupyterHub.cookie_secret_file = 'jupyterhub_cookie_secret'

# The location of jupyterhub data files (e.g. /usr/local/share/jupyter/hub)

c.JupyterHub.data_files_path = '/opt/anaconda2/envs/py3/share/jupyter/hub'

# Include any kwargs to pass to the database connection. See

# sqlalchemy.create_engine for details.

# c.JupyterHub.db_kwargs = {}

# url for the database. e.g. `sqlite:///jupyterhub.sqlite`

# c.JupyterHub.db_url = 'sqlite:///jupyterhub.sqlite'

# log all database transactions. This has A LOT of output

# c.JupyterHub.debug_db = False

# show debug output in configurable-http-proxy

# c.JupyterHub.debug_proxy = False

# Set a logging.FileHandler on this file.

# c.JupyterHub.extra_log_file = ''

c.JupyterHub.extra_log_file = '/var/log/jupyterhub.log'

# Extra log handlers to set on JupyterHub logger

# c.JupyterHub.extra_log_handlers = []

# Generate default config file

# c.JupyterHub.generate_config = False

# The ip for this process

# c.JupyterHub.hub_ip = '127.0.0.1'

# The port for this process

# c.JupyterHub.hub_port = 8081

# The prefix for the hub server. Must not be '/'

# c.JupyterHub.hub_prefix = '/hub/'

# The public facing ip of the whole application (the proxy)

# c.JupyterHub.ip = ''

# Supply extra arguments that will be passed to Jinja environment.

# c.JupyterHub.jinja_environment_options = {}

# Interval (in seconds) at which to update last-activity timestamps.

# c.JupyterHub.last_activity_interval = 300

# Specify path to a logo image to override the Jupyter logo in the banner.

# c.JupyterHub.logo_file = ''

# File to write PID Useful for daemonizing jupyterhub.

# c.JupyterHub.pid_file = ''

# The public facing port of the proxy

# c.JupyterHub.port = 8000

# The ip for the proxy API handlers

# c.JupyterHub.proxy_api_ip = '127.0.0.1'

# The port for the proxy API handlers

# c.JupyterHub.proxy_api_port = 0

# The Proxy Auth token.

#

# Loaded from the CONFIGPROXY_AUTH_TOKEN env variable by default.

c.JupyterHub.proxy_auth_token = '6962f76eb967b47aca21ab95100c921e62e6292aab37d30915afc83b9af32075'

# Interval (in seconds) at which to check if the proxy is running.

# c.JupyterHub.proxy_check_interval = 30

# The command to start the http proxy.

#

# Only override if configurable-http-proxy is not on your PATH

# c.JupyterHub.proxy_cmd = ['configurable-http-proxy']

# Purge and reset the database.

# c.JupyterHub.reset_db = False

# The class to use for spawning single-user servers.

#

# Should be a subclass of Spawner.

# c.JupyterHub.spawner_class = 'jupyterhub.spawner.LocalProcessSpawner'

# Path to SSL certificate file for the public facing interface of the proxy

#

# Use with ssl_key

# c.JupyterHub.ssl_cert = ''

# Path to SSL key file for the public facing interface of the proxy

#

# Use with ssl_cert

# c.JupyterHub.ssl_key = ''

# Run single-user servers on subdomains of this host.

#

# This should be the full https://hub.domain.tld[:port]

#

# Provides additional cross-site protections for javascript served by single-

# user servers.

#

# Requires <username>.hub.domain.tld to resolve to the same host as

# hub.domain.tld.

#

# In general, this is most easily achieved with wildcard DNS.

#

# When using SSL (i.e. always) this also requires a wildcard SSL certificate.

# c.JupyterHub.subdomain_host = ''

# Paths to search for jinja templates.

# c.JupyterHub.template_paths = []

#

# c.JupyterHub.tornado_settings = {}

#------------------------------------------------------------------------------

# Spawner configuration

#------------------------------------------------------------------------------

# Base class for spawning single-user notebook servers.

#

# Subclass this, and override the following methods:

#

# - load_state - get_state - start - stop - poll

# Extra arguments to be passed to the single-user server

# c.Spawner.args = []

# The command used for starting notebooks.

# c.Spawner.cmd = ['jupyterhub-singleuser']

# Enable debug-logging of the single-user server

# c.Spawner.debug = False

# The default URL for the single-user server.

#

# Can be used in conjunction with --notebook-dir=/ to enable  full filesystem

# traversal, while preserving user's homedir as landing page for notebook

#

# `%U` will be expanded to the user's username

# c.Spawner.default_url = ''

# Disable per-user configuration of single-user servers.

#

# This prevents any config in users' $HOME directories from having an effect on

# their server.

# c.Spawner.disable_user_config = False

# Whitelist of environment variables for the subprocess to inherit

# c.Spawner.env_keep = ['PATH', 'PYTHONPATH', 'CONDA_ROOT', 'CONDA_DEFAULT_ENV', 'VIRTUAL_ENV', 'LANG', 'LC_ALL']

# Timeout (in seconds) before giving up on a spawned HTTP server

#

# Once a server has successfully been spawned, this is the amount of time we

# wait before assuming that the server is unable to accept connections.

# c.Spawner.http_timeout = 30

# The IP address (or hostname) the single-user server should listen on

# c.Spawner.ip = '127.0.0.1'

# The notebook directory for the single-user server

#

# `~` will be expanded to the user's home directory `%U` will be expanded to the

# user's username

# c.Spawner.notebook_dir = ''

# An HTML form for options a user can specify on launching their server. The

# surrounding `<form>` element and the submit button are already provided.

#

# For example:

#

#     Set your key:

#     <input name="key" val="default_key"></input>

#     <br>

#     Choose a letter:

#     <select name="letter" multiple="true">

#       <option value="A">The letter A</option>

#       <option value="B">The letter B</option>

#     </select>

# c.Spawner.options_form = ''

# Interval (in seconds) on which to poll the spawner.

# c.Spawner.poll_interval = 30

# Timeout (in seconds) before giving up on the spawner.

#

# This is the timeout for start to return, not the timeout for the server to

# respond. Callers of spawner.start will assume that startup has failed if it

# takes longer than this. start should return when the server process is started

# and its location is known.

# c.Spawner.start_timeout = 60

#------------------------------------------------------------------------------

# LocalProcessSpawner configuration

#------------------------------------------------------------------------------

# A Spawner that just uses Popen to start local processes as users.

#

# Requires users to exist on the local system.

#

# This is the default spawner for JupyterHub.

# Seconds to wait for process to halt after SIGINT before proceeding to SIGTERM

# c.LocalProcessSpawner.INTERRUPT_TIMEOUT = 10

# Seconds to wait for process to halt after SIGKILL before giving up

# c.LocalProcessSpawner.KILL_TIMEOUT = 5

# Seconds to wait for process to halt after SIGTERM before proceeding to SIGKILL

# c.LocalProcessSpawner.TERM_TIMEOUT = 5

#------------------------------------------------------------------------------

# Authenticator configuration

#------------------------------------------------------------------------------

# A class for authentication.

#

# The primary API is one method, `authenticate`, a tornado coroutine for

# authenticating users.

# set of usernames of admin users

#

# If unspecified, only the user that launches the server will be admin.

# c.Authenticator.admin_users = set()

# Dictionary mapping authenticator usernames to JupyterHub users.

#

# Can be used to map OAuth service names to local users, for instance.

#

# Used in normalize_username.

# c.Authenticator.username_map = {}

# Regular expression pattern for validating usernames.

#

# If not defined: allow any username.

# c.Authenticator.username_pattern = ''

# Username whitelist.

#

# Use this to restrict which users can login. If empty, allow any user to

# attempt login.

# c.Authenticator.whitelist = set()

#------------------------------------------------------------------------------

# LocalAuthenticator configuration

#------------------------------------------------------------------------------

# Base class for Authenticators that work with local Linux/UNIX users

#

# Checks for local users, and can attempt to create them if they exist.

# The command to use for creating users as a list of strings.

#

# For each element in the list, the string USERNAME will be replaced with the

# user's username. The username will also be appended as the final argument.

#

# For Linux, the default value is:

#

#     ['adduser', '-q', '--gecos', '""', '--disabled-password']

#

# To specify a custom home directory, set this to:

#

#     ['adduser', '-q', '--gecos', '""', '--home', '/customhome/USERNAME', '--

# disabled-password']

#

# This will run the command:

#

# adduser -q --gecos "" --home /customhome/river --disabled-password river

#

# when the user 'river' is created.

# c.LocalAuthenticator.add_user_cmd = []

# If a user is added that doesn't exist on the system, should I try to create

# the system user?

# c.LocalAuthenticator.create_system_users = False

# Automatically whitelist anyone in this group.

# c.LocalAuthenticator.group_whitelist = set()

#------------------------------------------------------------------------------

# PAMAuthenticator configuration

#------------------------------------------------------------------------------

# Authenticate local Linux/UNIX users with PAM

# The encoding to use for PAM

#c.PAMAuthenticator.encoding = 'utf8'

# The PAM service to use for authentication.

#c.PAMAuthenticator.service = 'login'

##end##

[testb...@gmail.com]

Hi,
Could you please let me know whether you got this resolved? If so, could you send me the resolution/fix? I'm also getting the same error. Thanks!

[Abhinandan Dubey]

Hi,

I have been successful in my attempts installing on RHEL 6.

The reason you're getting this error is that JupyterHub's LocalProcessSpawner tries to spawn subprocesses as root/another user. So either you run jupyterhub as root, or you apply a patch to LocalProcessSpawner's start() method to spawn processes as you (mmetts in your case I guess), there's also a method named set_user_set_uid or something, have a look at it.