Redirect issues after upgrading to version 20.4 from 19.7

[Jon Barry]

Hi everyone, 

I am looking to see if I can get some help on an issue we are seeing after we have upgraded our GoCD Server from version 19.7 to 20.4.  Working on getting to the latest release....

Our GoCD server is sitting behind an F5 VIP.  The F5 up to and including version 19.7 has been forwarding the traffic to the GoCD server on port 8154.  After upgrading to version 20.4, and the SSL changes per https://github.com/gocd/gocd/issues/7872 we changed the member in the pool behind the VIP to go to port 8153 instead of 8154.

Since making that change we are seeing the client port being added into the redirects.  So for example after a successfull logon from https://our-gocd-server.com/go/auth/login, we get redirected to a url with a port number in it https://our-gocd-server.com:37680/go/pipelines which fails. 

If we manually remove the port from the url we get to the correct location https://our-gocd-server.com/go/pipelines

We are also seeing missing icons on some pages, as the URL to them also has the port in it.  e.g: http://our-gocd-server.com:37680/go/api/plugin_images/cd.go.authentication.ldap/9fdf150dbce01084106d76b18e3215af1f9c2beff6f3638e38083237f433662f

The Site URL and Secure Site URL are both set to https://our-gocd-server.com

Anyone done something similar or know what the issue may be?   I am not the person who manages the F5 so am at a bit of a loss there, but I expect something may be missing there.

Thanks in advance.

[Chad Wilson]

Hi Jon

Are you able to see if there are any differences in the HTTP headers of the proxied requests from the F5 before and after the change?

My educated guess (albeit possibly naive-to-GoCD-specifics) is that the URLs are being built from the incoming request, which inside GoCD is aware of headers such as Forwarded (RFC7239) and earlier de-facto standards such as X-Forwarded-Port X-Forwarded-Host X-Forwarded-Proto etc which are generally used to communicate the "real host/port/proto" that the client is using.

Done canonically, the left-most value of these headers (when multi-value) is taken as the outermost value and used to build redirects etc.

I suspect X-Forwarded-Port might be set to 37680 rather than unset, or 443. Are you able to capture traffic and take a look? If you find those headers there on an incoming request, then maybe you can figure out which piece of your env is the misbehaving proxy wrt headers.

-Chad

--
You received this message because you are subscribed to the Google Groups "go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email to go-cd+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/966f0230-cf31-446e-809a-ec47281d4f2bn%40googlegroups.com.

[Jon Barry]

Thanks Chad for the response.

My suspicion is that it is we are missing something on the F5.  The port I referred to earlier is a random port that changes on every request.  If I do the following curl the port gets added after the first request through the F5.

$ curl http://<server IP>:8153/ -H 'Host: ci.example.com' -H 'X-Forwarded-For: 5.6.7.8' -H 'X-Forwarded-Proto: https' --location -v

* About to connect() to <server IP> port 8153 (#0)
*   Trying <server IP>...
* Connected to <server IP> (<server IP>) port 8153 (#0)
> GET / HTTP/1.1
> User-Agent: curl/7.29.0
> Accept: */*
> Host: ci.example.com
> X-Forwarded-For: 5.6.7.8
> X-Forwarded-Proto: https
>
< HTTP/1.1 302 Found
< Date: Thu, 19 May 2022 13:31:06 GMT
< Cache-Control: no-cache, must-revalidate, no-store
< X-XSS-Protection: 1; mode=block
< X-Content-Type-Options: nosniff
< X-Frame-Options: SAMEORIGIN
< X-UA-Compatible: chrome=1
< Location: https://ci.example.com/go/pipelines
< Content-Length: 0
<
* Connection #0 to host <server IP> left intact

* Issue another request to this URL: 'https://ci.example.com/go/pipelines'
* About to connect() to ci.example.com port 443 (#1)
*   Trying <F5 IP>...
* Connected to ci.example.com (<F5 IP>) port 443 (#1)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate:
*     subject: <***> <the LB's certificate>
*     start date: <***>
*     expire date: <***>
*     common name: ci.example.com
*     issuer: <***>
> GET /go/pipelines HTTP/1.1
> User-Agent: curl/7.29.0
> Host: ci.example.com
> Accept: */*
> X-Forwarded-For: 5.6.7.8
> X-Forwarded-Proto: https
>
< HTTP/1.1 302 Found
< Date: Thu, 19 May 2022 13:31:06 GMT
< X-XSS-Protection: 1; mode=block
< X-Content-Type-Options: nosniff
< X-Frame-Options: SAMEORIGIN
< X-UA-Compatible: chrome=1
< Set-Cookie: JSESSIONID=<***>; Path=/go; Expires=Thu, 02-Jun-2022 13:31:06 GMT; Max-Age=1209600; Secure; HttpOnly
< Expires: Thu, 01 Jan 1970 00:00:00 GMT
< Location: https://ci.example.com:35499/go/auth/login
< Content-Length: 0
< Set-Cookie: LB_GO=<***>; path=/; Httponly; Secure
< Strict-Transport-Security: max-age=31536000; includeSubDomains
<
* Connection #1 to host ci.example.com left intact

* Issue another request to this URL: 'https://ci.example.com:35499/go/auth/login'
* Found bundle for host ci.example.com: 0x64a290
* About to connect() to ci.example.com port 35499 (#2)
*   Trying <F5 IP>...
* Connection timed out
* Failed connect to ci.example.com:35499; Connection timed out
* Closing connection 2
curl: (7) Failed connect to ci.example.com:35499; Connection timed out

[Chad Wilson]

Hmm, assuming there is no other reverse proxy/WAF/API gateway "in the mix" other than F5 I guess you might need to ask your team if something else changed other than the target server port (from HTTPS on 8154 to HTTP on 8153).

It'll be clearest as to which header(s) might be the culprit if you're able to capture the request as it arrives at GoCD itself (capturing network traffic, possibly, or could try the stuff here on GoCD side - not sure of how up-to-date it is)

I'm not really super familiar with use of F5 like this, however I imagine there is some kind of rule/scripting required to do the standard proxypass stuff Apache and NGINX do - possibly when editing it for your target port change (server_port in F5 terminology, I think?) something was screwed up on the X-Forwarded-Port rule (using remote_port instead of client_port or something like that)? In this theory, the "random" port is possibly the TCP source port of your browser/curl/whatever rather than the target port the client is connecting to on the LB (443 for a standard https connection).

If there are other network load balancers in the mix, other possibilities can arise where original ports are lost, but since there is no mention of any change in networking setup, I am assuming it's just F5 for now.

-Chad

To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/a59907e8-bd37-45a8-9681-6cdd88f94555n%40googlegroups.com.