SSL certificate errors from Jupyter/Docker

Ted Liefeld

unread,

Sep 30, 2016, 4:29:43 PM9/30/16

to Project Jupyter

Hi

I am playing around with running a JupyterHub that uses the Docker singleuser image. I have it up and running happily, but I cannot seem to connect to any HTTPS websites from it unless I turn off certificate validation. In every case it complains that the certificate does not match the hostname, where the displayed hostname is the hostname of the hub (jupyterhubtest.genomespace.org) and not of the site I am trying to connect to.

e.g.

import requests
requests.get('https://github.com')

CertificateError: hostname 'github.com' doesn't match 'jupyterhubtest.genomespace.org'

Now the code below works, but I would rather not have to turn off verification for every request that is made from the hub

requests.get('https://github.com', verify=False)

Now I know that the kernel in Docker is proxying through the hub and that's probably the cause of the error, but I haven't been able to figure out a fix yet. Has anyone else encountered this (and better yet figured out a fix?)

Thanks

MinRK

unread,

Oct 3, 2016, 5:13:30 AM10/3/16

to Project Jupyter

That's exceedingly strange, and I have not seen it before. The proxy is involved when your browser connects to the notebook server, but it shouldn't be involved at all when the kernel connects to the outside world. Can you check `os.environ` to see if there are any relevant environment variables that might be sneaking in here? You might also check things like `nslookup github.com` to make sure DNS isn't confused.

-Min

--
You received this message because you are subscribed to the Google Groups "Project Jupyter" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jupyter+unsubscribe@googlegroups.com.
To post to this group, send email to jup...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jupyter/9be93e9b-d486-4546-ac60-c29ceebf7c3d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Ted Liefeld

unread,

Oct 3, 2016, 4:37:41 PM10/3/16

to Project Jupyter

Min

here is what I see

environ({
'HOSTNAME': '0955ba141195', 
'GS_CODE_ROOT_DIR': '/combined', 
'LANGUAGE': 'en_US.UTF-8', 
'PATH': '/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', 
'LANG': 'en_US.UTF-8', 
'CLICOLOR': '1', 
'JPY_HUB_API_URL': 'http://10.0.2.188:8081/hub/api', 
'JPY_BASE_URL': '/user/ted', 
'HOME': '/home/jovyan', 
'JPY_COOKIE_NAME': 'jupyter-hub-token-ted', 
'PWD': '/home/jovyan/work', 
'JPY_HUB_PREFIX': '/hub/', 
'NB_USER': 'jovyan', 
'CONDA_DIR': '/opt/conda', 
'JPY_USER': 'ted', 
'GIT_PAGER': 'cat', 
'PYTHONPATH': '/combined/jupyter-notebook/profile/extensions', 
'DEBIAN_FRONTEND': 'noninteractive', 
'JPY_PARENT_PID': '5', 
'TERM': 'xterm-color', 
'GS_NOTEBOOK_EXTENSION_DIR': '/combined/jupyter-notebook/profile/extensions', 
'PAGER': 'cat', 
'NB_UID': '1000', 
'LC_ALL': 'en_US.UTF-8', 
'SHELL': '/bin/bash'})

and the nslookup is the same in both the docker container and the host AMI

$ nslookup github.com
Server:		10.0.0.2
Address:	10.0.0.2#53

Non-authoritative answer:
Name:	github.com
Address: 192.30.253.112

FWIW this is running on the ubuntu AMI on AWS taht was published with Jupyterhub in it (

ubuntu/images/hvm-ssd/ubuntu-trusty-14.04-amd64-server-20160627 (ami-2d39803a)

)

MinRK

unread,

Oct 4, 2016, 7:18:59 AM10/4/16

to Project Jupyter

What do you see in

!curl -v https://github.com > /dev/null

?

It’s certainly odd that requests is connecting to something funny.

--
You received this message because you are subscribed to the Google Groups "Project Jupyter" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jupyter+unsubscribe@googlegroups.com.
To post to this group, send email to jup...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jupyter/a25894e4-82af-4a9e-a993-0426ad6a2437%40googlegroups.com.

Ted Liefeld

unread,

Oct 4, 2016, 12:43:13 PM10/4/16

to Project Jupyter

On the hub AMI I see the correct behavior for this. However if I shell into one of the jupyterhub/singleuser docker containers its getting the certificate for the hub. From this post it looks like there might be something odd in the iptables

http://stackoverflow.com/questions/37970990/ssl-certificate-verification-fails-inside-docker-container-on-specific-server

however when I look at the container it has no iptables and on the hub I can't quite see anything that explains it (included below the curl output below)

I am going to try to setup another hub to see if this repeats since I am running out of other ideas at this point. I think I'll try Amazon linux instead of ubuntu for this second try

jovyan@0955ba141195:~/work$ curl -v https://github.com > /dev/null
* Rebuilt URL to: https://github.com/
* Hostname was NOT found in DNS cache
% Total    % Received % Xferd Average Speed   Time    Time     Time Current
                                 Dload Upload   Total   Spent    Left Speed
0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 192.30.253.113...
* Connected to github.com (192.30.253.113) port 443 (#0)
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
} [data not shown]
* SSLv3, TLS handshake, Server hello (2):
{ [data not shown]
* SSLv3, TLS handshake, CERT (11):
{ [data not shown]
* SSLv3, TLS handshake, Server finished (14):
{ [data not shown]
* SSLv3, TLS handshake, Client key exchange (16):
} [data not shown]
* SSLv3, TLS change cipher, Client hello (1):
} [data not shown]
* SSLv3, TLS handshake, Finished (20):
} [data not shown]
* SSLv3, TLS change cipher, Client hello (1):
{ [data not shown]
* SSLv3, TLS handshake, Finished (20):
{ [data not shown]
* SSL connection using TLSv1.2 / AES256-GCM-SHA384
* Server certificate:
*     subject: CN=jupyterhubtest.genomespace.org
*     start date: 2016-08-01 19:53:00 GMT
*     expire date: 2016-10-30 19:53:00 GMT
*     subjectAltName does not match github.com
* SSL: no alternative certificate subject name matches target host name 'github.com'
* Closing connection 0
* SSLv3, TLS alert, Client hello (1):
} [data not shown]
curl: (51) SSL: no alternative certificate subject name matches target host name 'github.com'

========= iptable on the hub ============

sudo iptables --list

Chain INPUT (policy ACCEPT)