[Help] nginx reverse proxy redirect loop with jupyterhub

[Edward Leonard Jr]

I've got a working jupyterhub setup that works when accessed at https://host:8000. I'm trying quite hard to make it so I can just go to https://host and access the same exact thing using nginx as a reverse proxy. 

The jupyterhub_config.py file isn't too interesting; all I've done is changed the values for ssl_key, ssl_cert, and cookie_secret_file to fit my needs. I have tried changing a variety of things within this config file to stop the redirect, but if that's the answer I just haven't found the right recipe yet.

My server block in the sites-enabled forwarding server for nginx looks like so (with HOSTNAME replaced with the fully qualified domain name of the server):

server {

        listen 443;

        client_max_body_size 50M;

        server_name HOSTNAME;

        ssl on;

        ssl_certificate /etc/letsencrypt/live/HOSTNAME/fullchain.pem;

        ssl_certificate_key /etc/letsencrypt/live/HOSTNAME/privkey.pem;

        ssl_ciphers "AES128+EECDH:AES128+EDH";

        ssl_protocols TLSv1 TLSv1.1 TLSv1.2;

        ssl_prefer_server_ciphers on;

#        ssl_session_cache shared:SSL:10m;

        add_header Strict-Transport-Security "max-age=63072000; includeSubDomains";

        add_header X-Content-Type-Options nosniff;

        ssl_stapling on;

        ssl_stapling_verify on;

        resolver_timeout 5s;

        location / {

            proxy_pass http://127.0.0.1:8081;

            proxy_set_header X-Real-IP $remote_addr;

            proxy_set_header Host $host;

            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

            proxy_set_header X-NginX-Proxy true;

        }
}

I've found several other people that have had this issue and have tried a whole slue of different options for the location / proxy pass all to no avail in removing the loop.

The forward does seem to work at the outset, at least, as when I go to https://HOSTNAME, I am taken to the hub login page. If I enter incorrect credentials, I'm told as much. If I enter correct credentials, I'm put into a redirect loop and the server output looks like so:

[I 2016-04-09 01:09:09.200 JupyterHub spawner:436] Spawning jupyterhub-singleuser --user=eleonard --port=59940 --cookie-name=jupyter-hub-token-eleonard --base-url=/user/eleonard --hub-host= --hub-prefix=/hub/ --hub-api-url=http://127.0.0.1:8081/hub/api --ip=127.0.0.1

[W 2016-04-09 01:09:09.876 eleonard notebookapp:253] ipywidgets package not installed.  Widgets are unavailable.

[I 2016-04-09 01:09:09.890 eleonard notebookapp:1079] Serving notebooks from local directory: /afs/physics.wisc.edu/home/eleonard

[I 2016-04-09 01:09:09.891 eleonard notebookapp:1079] 0 active kernels

[I 2016-04-09 01:09:09.891 eleonard notebookapp:1079] The Jupyter Notebook is running at: http://127.0.0.1:59940/user/eleonard/

[I 2016-04-09 01:09:09.891 eleonard notebookapp:1080] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).

[I 2016-04-09 01:09:09.952 eleonard log:47] 302 GET /user/eleonard (127.0.0.1) 2.94ms

[I 2016-04-09 01:09:09.954 JupyterHub base:301] User eleonard server took 1.416 seconds to start

[I 2016-04-09 01:09:09.954 JupyterHub orm:159] Adding user eleonard to proxy /user/eleonard => http://127.0.0.1:59940

[I 2016-04-09 01:09:09.979 JupyterHub log:100] 302 POST /hub/login?next= (@69.58.186.114) 1535.77ms

[I 2016-04-09 01:09:09.979 JupyterHub login:79] User logged in: eleonard

[I 2016-04-09 01:09:10.002 JupyterHub log:100] 302 GET /hub/ (eleo...@69.58.186.114) 7.47ms

[I 2016-04-09 01:09:10.018 JupyterHub log:100] 302 GET /user/eleonard (@69.58.186.114) 1.17ms

[I 2016-04-09 01:09:10.037 JupyterHub log:100] 302 GET /hub/user/eleonard (eleo...@69.58.186.114) 6.13ms

[I 2016-04-09 01:09:10.052 JupyterHub log:100] 302 GET /user/eleonard (@69.58.186.114) 0.99ms

[I 2016-04-09 01:09:10.072 JupyterHub log:100] 302 GET /hub/user/eleonard (eleo...@69.58.186.114) 6.11ms

[I 2016-04-09 01:09:10.086 JupyterHub log:100] 302 GET /user/eleonard (@69.58.186.114) 0.85ms

[I 2016-04-09 01:09:10.109 JupyterHub log:100] 302 GET /hub/user/eleonard (eleo...@69.58.186.114) 8.36ms

[I 2016-04-09 01:09:10.122 JupyterHub log:100] 302 GET /user/eleonard (@69.58.186.114) 0.89ms

[I 2016-04-09 01:09:10.182 JupyterHub log:100] 302 GET /hub/user/eleonard (eleo...@69.58.186.114) 5.67ms

[I 2016-04-09 01:09:10.187 JupyterHub log:100] 302 GET /user/eleonard (@69.58.186.114) 1.43ms

[I 2016-04-09 01:09:10.206 JupyterHub log:100] 302 GET /hub/user/eleonard (eleo...@69.58.186.114) 5.16ms

[I 2016-04-09 01:09:10.220 JupyterHub log:100] 302 GET /user/eleonard (@69.58.186.114) 0.77ms

[I 2016-04-09 01:09:10.242 JupyterHub log:100] 302 GET /hub/user/eleonard (eleo...@69.58.186.114) 5.59ms

[I 2016-04-09 01:09:10.254 JupyterHub log:100] 302 GET /user/eleonard (@69.58.186.114) 0.85ms

[I 2016-04-09 01:09:10.277 JupyterHub log:100] 302 GET /hub/user/eleonard (eleo...@69.58.186.114) 5.24ms

[I 2016-04-09 01:09:10.291 JupyterHub log:100] 302 GET /user/eleonard (@69.58.186.114) 0.84ms

[I 2016-04-09 01:09:10.313 JupyterHub log:100] 302 GET /hub/user/eleonard (eleo...@69.58.186.114) 5.67ms

[I 2016-04-09 01:09:10.326 JupyterHub log:100] 302 GET /user/eleonard (@69.58.186.114) 0.65ms

[I 2016-04-09 01:09:10.344 JupyterHub log:100] 302 GET /hub/user/eleonard (eleo...@69.58.186.114) 4.33ms

[I 2016-04-09 01:09:10.359 JupyterHub log:100] 302 GET /user/eleonard (@69.58.186.114) 0.72ms

[I 2016-04-09 01:09:10.460 JupyterHub log:100] 302 GET /hub/user/eleonard (eleo...@69.58.186.114) 6.23ms

[I 2016-04-09 01:09:10.473 JupyterHub log:100] 302 GET /user/eleonard (@69.58.186.114) 1.39ms

[I 2016-04-09 01:09:10.493 JupyterHub log:100] 302 GET /hub/user/eleonard (eleo...@69.58.186.114) 6.63ms

[I 2016-04-09 01:09:10.507 JupyterHub log:100] 302 GET /user/eleonard (@69.58.186.114) 1.34ms

[I 2016-04-09 01:09:10.527 JupyterHub log:100] 302 GET /hub/user/eleonard (eleo...@69.58.186.114) 6.46ms

[I 2016-04-09 01:09:10.540 JupyterHub log:100] 302 GET /user/eleonard (@69.58.186.114) 0.84ms

[I 2016-04-09 01:09:10.561 JupyterHub log:100] 302 GET /hub/user/eleonard (eleo...@69.58.186.114) 5.55ms

[I 2016-04-09 01:09:10.574 JupyterHub log:100] 302 GET /user/eleonard (@69.58.186.114) 0.87ms

[I 2016-04-09 01:09:10.597 JupyterHub log:100] 302 GET /hub/user/eleonard (eleo...@69.58.186.114) 4.88ms

[I 2016-04-09 01:09:10.611 JupyterHub log:100] 302 GET /user/eleonard (@69.58.186.114) 0.98ms

[I 2016-04-09 01:09:10.634 JupyterHub log:100] 302 GET /hub/user/eleonard (eleo...@69.58.186.114) 5.55ms

[I 2016-04-09 01:09:10.644 JupyterHub log:100] 302 GET /user/eleonard (@69.58.186.114) 0.82ms

[I 2016-04-09 01:09:10.665 JupyterHub log:100] 302 GET /hub/user/eleonard (eleo...@69.58.186.114) 5.87ms

[I 2016-04-09 01:09:10.677 JupyterHub log:100] 302 GET /user/eleonard (@69.58.186.114) 0.82ms

[I 2016-04-09 01:09:10.697 JupyterHub log:100] 302 GET /hub/user/eleonard (eleo...@69.58.186.114) 6.18ms

[I 2016-04-09 01:09:10.713 JupyterHub log:100] 302 GET /user/eleonard (@69.58.186.114) 1.17ms

[I 2016-04-09 01:09:10.733 JupyterHub log:100] 302 GET /hub/user/eleonard (eleo...@69.58.186.114) 5.74ms

[I 2016-04-09 01:09:10.746 JupyterHub log:100] 302 GET /user/eleonard (@69.58.186.114) 0.70ms

[I 2016-04-09 01:09:10.766 JupyterHub log:100] 302 GET /hub/user/eleonard (eleo...@69.58.186.114) 5.61ms

[I 2016-04-09 01:09:10.780 JupyterHub log:100] 302 GET /user/eleonard (@69.58.186.114) 0.75ms

[I 2016-04-09 01:09:10.802 JupyterHub log:100] 302 GET /hub/user/eleonard (eleo...@69.58.186.114) 6.25ms

I can post more log files if they'll help. I'm hoping someone else has conquered this issue and can share the victory story with me soon.

Thanks in advance!

[MinRK]

It looks like you have pointed nginx directly at the Hub (port 8081), rather than the configurable-http-proxy (port 8000). All access to JupyterHub must be made through the proxy, so that’s what you should point nginx to.

The result of pointing nginx at the Hub is that the redirects never work, because they rely on the proxy routing /user/NAME to the single-user server instead of the Hub. If you point directly a the Hub, it just keeps sending you to the URL path (host is not modified) of the single-user server, but since the proxy is bypassed, the Hub gets both requests and keeps redirecting to itself.

-MinRK

​

--
You received this message because you are subscribed to the Google Groups "Project Jupyter" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jupyter+u...@googlegroups.com.
To post to this group, send email to jup...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jupyter/ad44047d-e9d8-4454-8d05-fd186d29caea%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Edward Leonard Jr]

MinRK,

Thanks for this info. It's quite handy to have, but I think I now realize a different problem that is probably the root.

I have two domains that point to the same server and I only want the hub to be available on one of them. Both the hub domain and the non-hub have their own SSL certs. 

Here's the non-hub server file for nginx:

server {

        listen 80;

        server_name NO_HUB.DOMAIN.TLD;

        return 301 https://$host$request_uri;

}

server {

        listen 443 ssl;

        listen [::]:443 ssl;

        root /var/www/html;

        index index.html index.htm index.nginx-debian.html;

        server_name NO_HUB.DOMAIN.TLD;

        ssl_certificate /etc/letsencrypt/live/NO_HUB.DOMAIN.TLD/fullchain.pem;

        ssl_certificate_key /etc/letsencrypt/live/NO_HUB.DOMAIN.TLD/privkey.pem;

        ssl_protocols TLSv1 TLSv1.1 TLSv1.2;

        ssl_prefer_server_ciphers on;

        ssl_dhparam /etc/ssl/certs/dhparam.pem;

        ssl_ciphers 'ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA';

        ssl_session_timeout 1d;

        ssl_session_cache shared:SSL:50m;

        ssl_stapling on;

        ssl_stapling_verify on;

        add_header Strict-Transport-Security max-age=15768000;

        location / {

                try_files $uri $uri/ =404;

        }

        location ~ /.well-known {

                allow all;

        }

}

And here's the entire hub server file (I only gave the one block in the OP):

server {

        listen 80;

        server_name HUB.DOMAIN.TLD;

        return 302 https://$host$request_uri;

}

server {

        listen 443;

        ssl on;

        server_name HUB.DOMAIN.TLD;

        ssl_certificate /etc/letsencrypt/live/HUB.DOMAIN.TLD/fullchain.pem;

        ssl_certificate_key /etc/letsencrypt/live/HUB.DOMAIN.TLD/privkey.pem;

        ssl_protocols TLSv1 TLSv1.1 TLSv1.2;

        ssl_prefer_server_ciphers on;

        ssl_dhparam /etc/ssl/certs/dhparam.pem;

        ssl_ciphers 'ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA';

        ssl_session_timeout 1d;

        ssl_session_cache shared:SSL:50m;

        ssl_stapling on;

        ssl_stapling_verify on;

        add_header Strict-Transport-Security max-age=15768000;

        location / {

                proxy_pass http://127.0.0.1:8000;

                proxy_buffering off;

                proxy_set_header Host $host;

                proxy_set_header X-NginX-Proxy true;

        }

        location ~ /.well-known {

                allow all;

        }

}

So here are my current things that I need to get figured out to make this a reality:

1. I don't want to be able to go to NO_HUB.DOMAIN.TLD or NO_HUB.DOMAIN.TLD:8000 and get to the hub. It should only be available at HUB.DOMAIN.TLD and/or HUB.DOMAIN.TLD:8000 (I don't care if the 8000 is there or not, but the non-8000 needs to work).
2. With the current setup, when I visit HUB.DOMAIN.TLD, I get a 502 bad gateway error. Reason unknown at present. When I go to HUB.DOMAIN.TLD:8000, I can hit the hub just fine.

My initial thoughts on both:

1. I feel like I have to change some settings in configurable-http-proxy (looks like the --host-routing flag), but I'm unsure on where to do that.
2. This would be all avoidable, I think, if item 1. was a reality because then I would just tell jupyterhub to host on HUB.DOMAIN.TLD:443 and no reverse proxy would be needed. This is currently not possible because "something else is using port 443", which I'm taking to mean that configurable-http-proxy sees NO_HUB.DOMAIN.TLD:443 being used as a conflict with *:443 as attempted by the proxy.

Maybe I'm crazy. If not, please send help :)

Thanks!

[MinRK]

For this, I suspect that you only want CHP to listen on localhost, rather than all IPs, which is the default. This will make sure that the Hub can only be accessed from outside through nginx:

# jupyterhub_config.py
c.JupyterHub.ip = '127.0.0.1'

I don’t think you want extra host-based routing options for CHP, it seems like all the configuration you want to do ought to be in nginx. That is, only route to CHP iff the correct domain is used. This tutorial illustrates routing by server_name, so that NO_HUB should never forward requests to JupyterHub.

As you said, you can eliminate the nginx reverse-proxy if you can make CHP the public-facing service on 443, but this isn’t going to work if you are running nginx on 443 for other reasons. While you do have separate domains, you can’t have multiple services listening on the port unless they are on different IP addresses (e.g. hub.domain is eth0 and nohub.domain is eth1).

-MinRK

​

To view this discussion on the web visit https://groups.google.com/d/msgid/jupyter/8fec6b73-a1e2-43ee-81ef-3925b3fed487%40googlegroups.com.

[Edward M Leonard Jr]

Amazing. Works like a charm. Thank you very kindly for taking the time to respond.

You received this message because you are subscribed to a topic in the Google Groups "Project Jupyter" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jupyter/tPcQKr8bznY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jupyter+u...@googlegroups.com.

To post to this group, send email to jup...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jupyter/CAHNn8BV_oDo%2BY8q-Js6HaWpp_Xn8rijRd_OwW4%3DC2pr6PzECrg%40mail.gmail.com.

[Fernando Perez]

On Mon, Apr 11, 2016 at 9:41 AM, Edward M Leonard Jr <iam...@gmail.com> wrote:

Amazing. Works like a charm. Thank you very kindly for taking the time to respond.

This is another thread that would be great to have in our docs if distilled... Edward, would you be willing to make a small contribution to the docs with this information, so that it could help others in the future more easily?

The docs live here:

https://github.com/jupyter/jupyterhub/tree/master/docs/source

and Carol might be able to provide better guidance...

Cheers,

f

--

Fernando Perez (@fperez_org; http://fperez.org)
fperez.net-at-gmail: mailing lists only (I ignore this when swamped!)
fernando.perez-at-berkeley: contact me here for any direct mail

[Carol Willing]

Thanks Fernando. I’m glad that Edward and Min’s discussion led to a
helpful solution.

I’ve opened an issue here (
https://github.com/jupyter/jupyterhub/issues/509 ) to add this to the
JupyterHub documentation.

Edward, if you would like to make a contribution to the docs that would
be wonderful. Any contribution, from quick notes on the issue to a pull
request, are welcome. Please feel free to leave comments or questions in
the issue.

Warmly,

Carol

P.S. If anyone that follows this list has a configuration that they
believe would be helpful to others, please open a new issue on the
JupyterHub repo https://github.com/jupyter/jupyterhub/issues

> --
> You received this message because you are subscribed to the Google
> Groups "Project Jupyter" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to jupyter+u...@googlegroups.com.
> To post to this group, send email to jup...@googlegroups.com.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/jupyter/CAHAreOqNY37iAyuPvyQ8-h0Bh8EDT_sU0_UnSJpgh_2Nk5wWFQ%40mail.gmail.com.

> For more options, visit https://groups.google.com/d/optout.

Carol Willing
Research Software Engineer, Project Jupyter @ Cal Poly
Director, Python Software Foundation

[Quang Minh DAO]

Dear all, 

After searching for a long time on the Internet, I found this topic which is closiest to my problem. 

I have the same logs with Edward Leonard but there is only different thing here is I am using apache 2.4 for proxy manager. 

Here are my configuration with Apache 

Listen 443

<VirtualHost my_domain:443>
    ServerName                      my_domain

    SSLEngine                       on    
    SSLCertificateFile              /etc/pki/tls/certs/server.crt
    SSLCertificateKeyFile           /etc/pki/tls/certs/server.key
    SSLProtocol                     All -SSLv2 -SSLv3
    SSLCipherSuite                  EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH
 
    RewriteEngine On
    RewriteCond %{REQUEST_URI} ^/api/v1/websocket [NC,OR]
    RewriteCond %{HTTP:UPGRADE} ^WebSocket$ [NC,OR]
    RewriteCond %{HTTP:CONNECTION} ^Upgrade$ [NC]
    RewriteRule /(.*) ws://127.0.0.1:8000/%{REQUEST_URI} [P,L]

    <Location "/">
        # preserve Host header to avoid cross-origin problems
        ProxyPreserveHost on
        # proxy to JupyterHub
        ProxyPass         http://127.0.0.1:8000/
        ProxyPassReverse  http://127.0.0.1:8000/
    </Location>
</VirtualHost>    

 and I already change c.JupyterHub.ip to 127.0.0.1 because I did successful with nginx on another server, but with apache, I tried many times and failed.

Now, I cannot login to JupyterHub although my jupyterhub-singleuser has already started (I checked with htop). My web browser returned "err_too_many_redirects".

Could you help me, pls?

Many thanks !!!

Best,

MinhDQ

PhD at ICAN | Institute of Cardiometabolism And Nutrition