How to use Ngnix as a reverse proxy to access OpenShift (OKD) 4.X?

[Eduardo Lúcio Amorim Costa]

I've tried hundreds of setups for the reverse proxy (Nginx) and they all fail with the error "Application is not available" when we access the oauth-openshift.apps.mbr.some.dm route.

NOTE: This problem does not occur if we access this route directly (without using Reverse Proxy). Perhaps some information necessary for the route to be resolved is not being sent.

This is the basic configuration template we are using...

    server {

        access_log /var/log/nginx/apps.mbr.some.dm-access.log;

        error_log /var/log/nginx/apps.mbr.some.dm-error.log;

        server_name ~^(?<subdomain>.+)\.apps\.mbr\.some\.dm$;

    

        location / {

            proxy_pass https://10.2.0.18:443;

            proxy_set_header Host $subdomain.apps.mbr.some.dm;

            proxy_set_header X-Forwarded-For https://$subdomain.apps.mbr.some.dm$request_uri;

        }

    

        listen 443;

        ssl_certificate /etc/letsencrypt/live/apps.mbr.some.dm/fullchain.pem;

        ssl_certificate_key /etc/letsencrypt/live/apps.mbr.some.dm/privkey.pem;

        include /etc/letsencrypt/options-ssl-nginx.conf;

        ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;

    }

We also tested these parameters and got some problems as you can see below...

    server {

        [...]

        location / {

            [...]

            proxy_ssl_certificate /etc/nginx/backend_ss_certs/apps.mbr.some.dm.crt;

            proxy_ssl_certificate_key /etc/nginx/backend_ss_certs/apps.mbr.some.dm.key;

            proxy_ssl_trusted_certificate /etc/nginx/backend_ss_certs/apps.mbr.some.dm.crt.key.pem;

            proxy_ssl_ciphers HIGH:!aNULL:!MD5;

            proxy_ssl_protocols TLSv1.2 TLSv1.3;

            proxy_ssl_server_name on;

            proxy_ssl_session_reuse on;

            proxy_ssl_verify on;

            [...]

        }

        [...]

    }

The certificates apps.mbr.some.dm.crt, apps.mbr.some.dm.key, apps.mbr.some.dm.crt.key.pem are the self-signed certificates used by OpenShift (OKD) to allow access to resources (HTTPS). However if we try to use these certificates with the reverse proxy (Nginx) the following error happens ("Bad Gateway")...

    2021/07/22 17:36:11 [error] 6999#6999: *1 upstream SSL certificate verify error: (21:unable to verify the first certificate) while SSL handshaking to upstream, client: 177.25.231.233, server: ~^(?<subdomain>.+)\.apps\.mbr\.brlight\.net$, request: "GET /favicon.ico HTTP/1.1", upstream: "https://10.2.0.18:443/favicon.ico", host: "oauth-openshift.apps.mbr.some.dm", referrer: "https://oauth-openshift.apps.mbr.some.dm/oauth/authorize?client_id=console&redirect_uri=https%3A%2F%2Fconsole-openshift-console.apps.mbr.some.dm%2Fauth%2Fcallback&response_type=code&scope=user%3Afull&state=ff6f3064"

NOTA: We tested the apps.mbr.some.dm.crt and apps.mbr.some.dm.crt.key.pem certificates using curl and both worked perfectly.

PLUS: We couldn't define a way to diagnose/observe (logs) about what goes wrong when the request arrives the route oauth-openshift.apps.mbr.some.dm . I think this would help us figure out what's going wrong.

ORIGINAL THREAD: https://github.com/openshift/okd/discussions/784

[Eduardo Lúcio Amorim Costa]

Well, from the lack of answers I can assume that OpenShift (OKD) is incapable to work behind a reverse proxy, right?

Thanks! =D

[Eduardo Lúcio Amorim Costa]

OKD behind haproxy or F5 is the standard

:peter pfläging

(gesendet von einem Gerät mit kleiner virtueller Tastatur und viel zu großen Fingern)

[...]
> Am 23.07.2021 um 19:24 schrieb Eduardo Lúcio Amorim Costa ***@***.***>:
>
> 

> Well, from the lack of answers I can assume that OpenShift (OKD) is incapable to work behind a reverse proxy, right?
>
> Thanks! =D
>

> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub, or unsubscribe.

[Eduardo Lúcio Amorim Costa]

Thank you Peter!

Do you have any clue as to why the error "Application is not available" occurs?
Would you know how I can find out (basically logs) what is going wrong in OpenShift (OKD) for the "oauth-openshift.apps.mbr.some.dm" route?

Thank you very much! =D

[Eduardo Lúcio Amorim Costa]

Community friends!

We currently have 463 members in this email group...

Has anyone here ever been able to use OpenShift (OKD) behind Nginx as a reverse proxy?

--------

I have a total of 70 applications/solutions published on 3 infrastructures accessible by 3 Nginx reverse proxies. Well... I can say that the only solution we can't fit into this infrastructure is OpenShift (OKD). Costs us to believe that OpenShift (OKD) is the only solution that doesn't work behind an Nginx reverse proxy... We contacted Red Hat through the customer-purchased support channel and they don't even know how to respond... Please, there is no need for anyone to be offended by what we are talking about here, but we need to say that the difficulties we are having with this solution are being unprecedented and unbelievable... =|

[Eduardo Lúcio Amorim Costa]

From Larry Brigman <larry...@gmail.com>

"Application is not available" occurs when the endpoint you are trying to reach doesn't exist or is down.

Any of the layers pod/service/route.  I would start at the route end and work inward.

A mis-spelling of the name and that would give you the same error.

[Eduardo Lúcio Amorim Costa]

Thank you so much for your help Larry!

About your considerations...

"

"Application is not available" occurs when the endpoint you are trying to reach doesn't exist or is down.

Any of the layers pod/service/route.  I would start at the route end and work inward.

A mis-spelling of the name and that would give you the same error.

"

... we can say that...

"

[...]

This problem does not occur if we access this route directly (without using Reverse Proxy).

[...]

"

... , conclusion, the route in the URL is correct.

We can also say that this problem only occurs in the oauth-openshift.apps.mbr.some.dm route. In the console-openshift-console.apps.mbr.some.dm route, for example, it does not occur.

We believe this problem may be related to...

"

[...]

Perhaps some information necessary for the route to be resolved is not being sent.

[...]

"

We can also append that...

"

[...]

We couldn't define a way to diagnose/see (logs from the OpenShift cluster) what goes wrong when the request arrives the route oauth-openshift.apps.mbr.some.dm . I think this would help us figure out what's going wrong.

[...]

"

Finally, if we can at least know the actual causes of the "Application is not available" error, then we can know what is going wrong...

[Eduardo Lúcio Amorim Costa]

People!

We confirm that the problem "Application is not available" only occurs in the route "oauth-openshift" ( oauth-openshift.apps.mbr.some.dm ) can any human being say why this problem is occurring specifically for this route?

NOTE: If we log into the "oauth-openshift" route using an SSH tunnel, for example, we can use all other resources via reverse proxy

[]'s

[Sri Ramanujam]

A possible cause could be that your auth pods have not deployed correctly for some reason. Check on the status of the pods in the "openshift-authentication" workspace as well as the authentication cluster operator (oc describe co authentication). Another thing to check: is the oauth-openshift service in the openshift-authentication namespace selecting the pods?