Issue: I don't have permission to verify cookies, my auth token may have expired

[Kuldeep Yadav]

Hello Everyone,

I am facing a wierd issue and hope somebody can help me in debugging this. I have setup JupyterHub with DockerSpawner ( to spawn single user server) on a remote Ubuntu machine with PAM authentication. The things are broadly running fine, but some users experience '500 error: I don't have permission to verify cookies, my auth token may have expired', when they login using JupyterHub and it tries to spawn a single user server. On restarting the individual server, it works fine again. Can you please help? I am putting down my docker logs which are experiencing this issue.

[I 2017-05-12 16:51:48.590 group11 notebookapp:1366] The Jupyter Notebook is running at: http://0.0.0.0:8888/user/group11/

[I 2017-05-12 16:51:48.590 group11 notebookapp:1367] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).

[I 2017-05-12 16:51:48.650 group11 log:47] 302 GET /user/group11 (172.17.0.1) 0.75ms

[E 2017-05-12 16:51:48.988 group11 auth:180] I don't have permission to verify cookies, my auth token may have expired: [403] Forbidden

[W 2017-05-12 16:51:48.989 group11 web:1545] 500 GET /user/group11/api/sessions?_=1494607776779 (::ffff:122.178.70.26): Permission failure checking authorization, I may need a new token

[E 2017-05-12 16:51:49.016 group11 log:46] {

      "X-Forwarded-Proto": "https",

      "X-Forwarded-Host": "programming.videoken.com:8000",

      "Host": "programming.videoken.com:8000",

      "Accept": "application/json, text/javascript, */*; q=0.01",

      "X-Forwarded-Port": "8000",

      "X-Forwarded-For": "::ffff:122.178.70.26",

      "Referer": "https://programming.videoken.com:8000/user/group11/tree?",

      "Cookie": "jupyter-hub-token-group11=\"2|1:0|10:1494607908|25:jupyter-hub-token-group11|44:MDRhNDE1OWQ5NDU0NDEwYmFjNjQzMGFiNjM1MTI4Y2E=|5ccc6b50db715168e0f114185d882ca445bdf2c985ab96a5649b7cf653d111b9\"; _xsrf=2|7a6575c8|24031a0226b8948118c2242552861ef2|1494607773",

      "Accept-Encoding": "gzip, deflate, br",

      "X-Requested-With": "XMLHttpRequest",

      "X-Xsrftoken": "2|7a6575c8|24031a0226b8948118c2242552861ef2|1494607773",

      "Connection": "close",

      "Accept-Language": "en-US,en;q=0.5",

      "User-Agent": "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:54.0) Gecko/20100101 Firefox/54.0"

My juputerhub_conf.py is as follows:

from dockerspawner import DockerSpawner

c.JupyterHub.spawner_class = DockerSpawner

c.DockerSpawner.container_image = 'tensorflow-keras'

c.DockerSpawner.remove_containers = True

#c.DockerSpawner.host_homedir_format_string = '/home/ubuntu/jupyterusers/{username}'

c.DockerSpawner.extra_create_kwargs.update({

        'command': '/usr/local/bin/start-singleuser.sh'

})

notebook_dir = os.environ.get('DOCKER_NOTEBOOK_DIR') or '/home/jovyan/work'

c.DockerSpawner.notebook_dir = notebook_dir

# Mount the real user's Docker volume on the host to the notebook user's

# notebook directory in the container

c.DockerSpawner.volumes = { '/datadrive/users/{username}': notebook_dir }

import netifaces

docker0 = netifaces.ifaddresses('docker0')

docker0_ipv4 = docker0[netifaces.AF_INET][0]

c.JupyterHub.hub_ip = docker0_ipv4['addr']

[MinRK]

Can you report your jupyterhub and dockerspawner versions?

When JupyterHub launches a single-user server, it creates an auth token and stores a record of that token (not the token itself) in its database. It then passes the token to the single-user servers it starts. When JupyterHub *stops* a single-user server, it deletes the record of that token unless the Spawner has set a `.will_resume` flag. This error generally means that the single-user server (in your case a docker container) still has an auth token that has been deleted fro the Hub's database. This could be caused by the Hub's database being cleared altogether, or the token being revoked at some point without recreating the docker container.

Since you are using `.remove_containers = True`, containers shouldn't be reusing tokens. If you *changed* this value, some containers may still exist in a stopped state, and will need to be deleted first. If you ever clear your `jupyterhub.sqlite` database, that will require that all your user containers be deleted because they will have stale auth tokens.

-Min

--
You received this message because you are subscribed to the Google Groups "Project Jupyter" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jupyter+unsubscribe@googlegroups.com.
To post to this group, send email to jup...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jupyter/5944cf79-3ba6-4ea3-a4e5-42044ed594da%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.