[JIRA] (JENKINS-60199) jenkins 2.205 behind reverse proxy redirects to 127.0.0.1 after login

112 views
Skip to first unread message

ohauer@gmx.de (JIRA)

unread,
Nov 18, 2019, 7:30:05 AM11/18/19
to jenkinsc...@googlegroups.com
olli hauer created an issue
 
Jenkins / Bug JENKINS-60199
jenkins 2.205 behind reverse proxy redirects to 127.0.0.1 after login
Issue Type: Bug Bug
Assignee: Unassigned
Components: core
Created: 2019-11-18 12:29
Environment: Jenkins: 2.205
Apache 2.4.41 as reverse proxy
Priority: Minor Minor
Reporter: olli hauer

After the upgrade from 2.204 to 2.205 jenkins redirects to http(s)://127.0.0.1/ after login.
Add Comment Add Comment
 
This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)
Atlassian logo

mark.earl.waite@gmail.com (JIRA)

unread,
Nov 19, 2019, 1:07:03 PM11/19/19
to jenkinsc...@googlegroups.com

olamy@apache.org (JIRA)

unread,
Nov 19, 2019, 6:24:02 PM11/19/19
to jenkinsc...@googlegroups.com

Is it possible to have the reverse proxy configuration?

Mark Waite yes maybe...

olamy@apache.org (JIRA)

unread,
Nov 19, 2019, 7:04:03 PM11/19/19
to jenkinsc...@googlegroups.com
Olivier Lamy edited a comment on Bug JENKINS-60199
[~ohauer] Is it possible to have more details about the reverse proxy configuration ? (version, configuration entry)

[~markewaite] yes maybe...

mark.earl.waite@gmail.com (JIRA)

unread,
Nov 19, 2019, 7:38:03 PM11/19/19
to jenkinsc...@googlegroups.com

Olivier Lamy I successfully created an Apache reverse proxy configuration on Debian 10 installing the apache package for Debian and the jenkins 2.205 package for Debian. Then I configured the Apache installation per the directions at https://wiki.jenkins.io/display/JENKINS/Running+Jenkins+behind+Apache . I can't duplicate the problem.

olli hauer can you provide configuration information that will allow another person to see the same issue?

ohauer@gmx.de (JIRA)

unread,
Nov 20, 2019, 2:58:03 AM11/20/19
to jenkinsc...@googlegroups.com

Sure, here are the parts from my Jenkins config:

Jenkins parameters:
--httpPort=8180 --httpListenAddress=127.0.0.1 --prefix=/jenkins -Dhudson.DNSMultiCast.disabled=true -Djava.net.preferIPv6Addresses=false

Jenkins (tab) configure:
Jenkins Location

Relevant part from apache 2.4.41:

ProxyRequests Off
AllowEncodedSlashes NoDecode

RewriteRule ^(/jenkins)$ $1/ [R=301,L]

<Location /jenkins>
ProxyPass http://127.0.0.1:8180/jenkins nocanon
ProxyPassReverse http://127.0.0.1:8180/jenkins
RequestHeader set X-Forwarded-Proto "https"
RequestHeader set X-Forwarded-Port "443"
RequestHeader set X-Forwarded-Host "fqdn.host.name"
</Location>

 

ohauer@gmx.de (JIRA)

unread,
Nov 20, 2019, 5:09:04 AM11/20/19
to jenkinsc...@googlegroups.com

Perhaps a hint, the Apache is TLS only.

I've tried to analyze what happens the time I press the sign in button with chrome see trace below.

 

2.205 (broken)
j_acegi_security_check:

General:
Request URL: https://fqdn.host.name/jenkins/j_acegi_security_check
Request Method: POST
Status Code: 302 Found
Remote Address: IP.AD.DR.ES:443
Referrer Policy: no-referrer-when-downgrade

Response Header:
Connection: Keep-Alive
Content-Length: 0
Date: Wed, 20 Nov 2019 09:33:02 GMT
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Keep-Alive: timeout=5, max=100
*Location: https://127.0.0.1/jenkins/*
Server: Jetty(9.4.22.v20191022)
Set-Cookie: JSESSIONID.10db730a=node0173jclauyomxaxlnl8x11eypy2.node0; Path=/jenkins; Secure; HttpOnly
X-Content-Type-Options: nosniff

Request Header:
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,/;q=0.8,application/signed-exchange;v=b3
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9,de;q=0.8
Cache-Control: max-age=0
Connection: keep-alive
Content-Length: 71
Content-Type: application/x-www-form-urlencoded
Cookie: JSESSIONID.c0515192=node09fh3q2bodtxf6wgyy85sirkm7.node0; JSESSIONID.10db730a=node01pf20r2evzrjn7h09d2m3bk201.node0; screenResolution=1920x1200
DNT: 1
Host: fqdn.host.name
Origin: https://fqdn.host.name
Referer: https://fqdn.host.name/jenkins/login?from=%2Fjenkins%2F
Sec-Fetch-Mode: navigate
Sec-Fetch-Site: same-origin
Sec-Fetch-User: ?1
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 ...
j_username: username
j_password: password
from: /jenkins/
Submit: Sign in

 

and with 2.204 (OK)
j_acegi_security_check:

General:
Request URL: https://fqdn.host.name/jenkins/j_acegi_security_check
Request Method: POST
Status Code: 302 Found
Remote Address: IP.AD.DR.ES:443
Referrer Policy: no-referrer-when-downgrade

Response Header:
Connection: Keep-Alive
Content-Length: 0
Date: Wed, 20 Nov 2019 09:47:27 GMT
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Keep-Alive: timeout=5, max=100
*Location: https://fqdn.host.name/jenkins/*
Server: Jetty(9.4.z-SNAPSHOT)
Set-Cookie: JSESSIONID.e3d4d209=node01eapoart737kjzv2n9r5jqf6n1.node0;Path=/jenkins;Secure;HttpOnly
X-Content-Type-Options: nosniff

Request Header:
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,/;q=0.8,application/signed-exchange;v=b3
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9,de;q=0.8
Cache-Control: max-age=0
Connection: keep-alive
Content-Length: 71
Content-Type: application/x-www-form-urlencoded
Cookie: JSESSIONID.e3d4d209=node01deci271w2led1hqp8smatfa2j0.node0; screenResolution=1920x1200
DNT: 1
Host: fqdn.host.name
Origin: https://fqdn.host.name
Referer: https://fqdn.host.name/jenkins/login?from=%2Fjenkins%2F
Sec-Fetch-Mode: navigate
Sec-Fetch-Site: same-origin
Sec-Fetch-User: ?1
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 ...
j_username: username
j_password: password
from: /jenkins/
Submit: Sign in

 

ohauer@gmx.de (JIRA)

unread,
Nov 20, 2019, 5:11:02 AM11/20/19
to jenkinsc...@googlegroups.com
olli hauer edited a comment on Bug JENKINS-60199
Perhaps a hint, the Apache is TLS only.

I've tried to analyze what happens the time I press the sign in button with chrome see trace below.

 

*2.205 (broken)*
*j_acegi_security_check:*

*General:*
Request URL: [ https://fqdn.host.name/jenkins/j_acegi_security_check ]
Request Method: POST
Status Code: 302 Found
Remote Address: IP.AD.DR.ES:443
Referrer Policy: no-referrer-when-downgrade

*Response Header:*

Connection: Keep-Alive
Content-Length: 0
Date: Wed, 20 Nov 2019 09:33:02 GMT
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Keep-Alive: timeout=5, max=100
{color:#de350b} {color:#de350b} *Location: * {color}[{color:#de350b}* https://127.0.0.1/jenkins/*{color} |https://127.0.0.1/jenkins/*]{color}
{color:#de350b}*Server: Jetty(9.4.22.v20191022)*{color}

Set-Cookie: JSESSIONID.10db730a=node0173jclauyomxaxlnl8x11eypy2.node0; Path=/jenkins; Secure; HttpOnly
X-Content-Type-Options: nosniff

*Request Header:*
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3

Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9,de;q=0.8
Cache-Control: max-age=0
Connection: keep-alive
Content-Length: 71
Content-Type: application/x-www-form-urlencoded
Cookie: JSESSIONID.c0515192=node09fh3q2bodtxf6wgyy85sirkm7.node0; JSESSIONID.10db730a=node01pf20r2evzrjn7h09d2m3bk201.node0; screenResolution=1920x1200
DNT: 1
Host: fqdn.host.name
Sec-Fetch-Mode: navigate
Sec-Fetch-Site: same-origin
Sec-Fetch-User: ?1
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 ...
j_username: username
j_password: password
from: /jenkins/
Submit: Sign in

 

*and with 2.204 (OK)*
*j_acegi_security_check:*

*General:*
Request URL: [ https://fqdn.host.name/jenkins/j_acegi_security_check ]
Request Method: POST
Status Code: 302 Found
Remote Address: IP.AD.DR.ES:443
Referrer Policy: no-referrer-when-downgrade

*Response Header:*

Connection: Keep-Alive
Content-Length: 0
Date: Wed, 20 Nov 2019 09:47:27 GMT
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Keep-Alive: timeout=5, max=100
{color:#de350b} {color:#de350b} *Location: * {color}[{color:#de350b}* https://fqdn.host.name/jenkins/*{color} |https://fqdn.host.name/jenkins/*]{color}
{color:#de350b}*Server: Jetty(9.4.z-SNAPSHOT)*{color}

Set-Cookie: JSESSIONID.e3d4d209=node01eapoart737kjzv2n9r5jqf6n1.node0;Path=/jenkins;Secure;HttpOnly
X-Content-Type-Options: nosniff

*Request Header:*
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3

Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9,de;q=0.8
Cache-Control: max-age=0
Connection: keep-alive
Content-Length: 71
Content-Type: application/x-www-form-urlencoded
Cookie: JSESSIONID.e3d4d209=node01deci271w2led1hqp8smatfa2j0.node0; screenResolution=1920x1200
DNT: 1
Host: fqdn.host.name
Sec-Fetch-Mode: navigate
Sec-Fetch-Site: same-origin
Sec-Fetch-User: ?1
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 ...
j_username: username
j_password: password
from: /jenkins/
Submit: Sign in

 

ohauer@gmx.de (JIRA)

unread,
Nov 20, 2019, 5:12:03 AM11/20/19
to jenkinsc...@googlegroups.com
olli hauer edited a comment on Bug JENKINS-60199
Perhaps a hint, the Apache is TLS only.

I've tried to analyze what happens the time I press the sign in button with chrome see trace below.

 

*2.205 (broken)*
*j_acegi_security_check:*

*General:*
Request URL: [https://fqdn.host.name/jenkins/j_acegi_security_check]
Request Method: POST
Status Code: 302 Found
Remote Address: IP.AD.DR.ES:443
Referrer Policy: no-referrer-when-downgrade

*Response Header:*
Connection: Keep-Alive
Content-Length: 0
Date: Wed, 20 Nov 2019 09:33:02 GMT
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Keep-Alive: timeout=5, max=100
{color:#de350b} {color:#de350b} *Location:* {color} [ {color:#de350b} * [ https://127.0.0.1/jenkins/ *{color} |https://127.0.0.1/jenkins/*]{color}
* {color:#de350b} * Server {color} : Jetty(9.4.z-SNAPSHOT)* {color}

Rob.Munten@gxsoftware.com (JIRA)

unread,
Nov 20, 2019, 6:41:02 AM11/20/19
to jenkinsc...@googlegroups.com

Hi olli hauer,

Add the following line to your apache conf.

ProxyPreserveHost On

This fixed the issue in my case.

o.v.nenashev@gmail.com (JIRA)

unread,
Nov 20, 2019, 8:12:02 AM11/20/19
to jenkinsc...@googlegroups.com

Mark Waite  olli hauer looks like we need to add upgrade guidelines for this regression to the weekly changelog (summary banner for reverse proxy users?). AFAICT it also discourages selecting 2.205 s the new LTS baseline, needs more soak testing for other possible regressions

dbeck@cloudbees.com (JIRA)

unread,
Nov 20, 2019, 8:18:03 AM11/20/19
to jenkinsc...@googlegroups.com

I wonder whether this would be an instance that previously showed an administrative warnings about a bad reverse proxy configuration on /manage.

If so, wouldn't care that much.

jglick@cloudbees.com (JIRA)

unread,
Nov 20, 2019, 9:50:03 AM11/20/19
to jenkinsc...@googlegroups.com

Would really need to see what headers were sent to/from Jenkins by Apache to see if Jetty if behaving improperly here. Is there a simple way to reproduce from scratch, for example using this?

bcooksley@kde.org (JIRA)

unread,
Nov 20, 2019, 12:47:04 PM11/20/19
to jenkinsc...@googlegroups.com

I've found that for our instances (at build.kde.org and binary-factory.kde.org) that we've needed to use the following Apache snippets to make it work properly:

```AllowEncodedSlashes NoDecode
ProxyRequests Off
ProxyPass / http://127.0.0.1:8200/ nocanon retry=0
ProxyPassReverse / https://127.0.0.1/```

Given that ProxyPass and ProxyPassReverse should normally mirror each other, it would appear that whatever code is responsible for creating redirects in Jenkins/Jetty is:
a) Forcing HTTPS to On
b) Disregarding the port that Jenkins itself is listening on
c) Disregarding the Canonical URL preference set in Manage Jenkins > Configure in favour of the host it received the request under

jglick@cloudbees.com (JIRA)

unread,
Nov 20, 2019, 12:49:02 PM11/20/19
to jenkinsc...@googlegroups.com

I am guessing the issue only affects /j_acegi_security_check, the typical login form, and not other redirects in Jenkins?

bcooksley@kde.org (JIRA)

unread,
Nov 20, 2019, 1:34:02 PM11/20/19
to jenkinsc...@googlegroups.com

Other endpoints, including those used to access build logs are also impacted.

Anywhere where a redirect is used seems to be affected.

jglick@cloudbees.com (JIRA)

unread,
Nov 20, 2019, 2:35:00 PM11/20/19
to jenkinsc...@googlegroups.com

So I tried

  • docker run --rm -p 127.0.0.1:80:80 --network=host jglick/jenkins-demo-reverse-proxy
  • JENKINS_HOME=… java -jar ~/.m2/repository/org/jenkins-ci/main/jenkins-war/2.205/jenkins-war-2.205.war --prefix=/jenkins --httpListenAddress=127.0.0.1
  • navigating to http://whatever.127.0.0.1.beesdns.com/jenkins/
  • going through setup wizard (no plugins, confirming the above root URL, using default admin user)
  • logging out, logging in
  • running builds, deleting builds, etc., stuff that definitely makes use of 302s (confirmed in the Network tab of Chromium’s F12)

I did not notice any issues; the host name part of URLs remained constant. Also tried the same with HTTPS, using https://whatever.127.0.0.1.beesdns.com/jenkins/configure to adjust the Jenkins URL (accepting the lack of a certificate), and again had no issues with 302s.

The image mentioned above does set ProxyPreserveHost On, which has been our recommendation for Apache users for years.

My provisional evaluation is that some behavioral change in the past couple years of Jetty updates introduced a new symptom for Jenkins users who had always had a somewhat misconfigured reverse proxy.

Again, for those experiencing an issue, please try to describe a minimal, self-contained test case which would allow developers to reproduce the problem, determine what exactly is at fault, and decide if code changes are warranted.

jglick@cloudbees.com (JIRA)

unread,
Nov 20, 2019, 2:54:04 PM11/20/19
to jenkinsc...@googlegroups.com

As an aside, I also checked CloudBees Core built from Jenkins 2.205 running in EKS using nginx-ingress and it was fine, so it is possible the issue is limited to Apache.

bcooksley@kde.org (JIRA)

unread,
Nov 21, 2019, 12:21:05 AM11/21/19
to jenkinsc...@googlegroups.com

Sorry, but setting ProxyPreserveHost to On doesn't appear to be a recommendation currently as it isn't mentioned in the default HTTP configuration in your own documentation for this type of setup. It appears under HTTPS, but there is no justification given for why it is there when it isn't for HTTP.

See https://wiki.jenkins.io/display/JENKINS/Running+Jenkins+behind+Apache

In terms of a minimal reproduction usecase, the following should do the trick:

 

<VirtualHost *:443>
    ServerAdmin  webm...@example.org
    ServerName   jenkins.example.org
    DocumentRoot /srv/www/jenkins.example.org/

    ErrorLog ${APACHE_LOG_DIR}/jenkins.example.org-error.log
    CustomLog ${APACHE_LOG_DIR}/jenkins.example.org.log combined

    AllowEncodedSlashes NoDecode
    ProxyRequests Off
    ProxyPass / http://127.0.0.1:8200/ nocanon retry=0
    ProxyPassReverse  /  http://127.0.0.1:8200/
    
    RequestHeader set X-Forwarded-Proto "https"
    RequestHeader set X-Forwarded-Port "443"    # Enable SSL
</VirtualHost>

 

You will need to change ProxyPassReverse to the very incorrect following line to make it work (as ProxyPass / ProxyPassReverse should be mirrors of each other):

ProxyPassReverse / https://127.0.0.1/

This indicates that Jenkins isn't respecting protocols and ports correctly - and setting ProxyPreserveHost just masks the issue (as most people don't run Jenkins without a web server in front of it, so users won't be connecting to it over a non-standard port)

jglick@cloudbees.com (JIRA)

unread,
Nov 21, 2019, 9:39:04 AM11/21/19
to jenkinsc...@googlegroups.com

setting ProxyPreserveHost to On doesn't appear to be a recommendation currently as it isn't mentioned in the default HTTP configuration in your own documentation for this type of setup. It appears under HTTPS

ProxyPreserveHost On is suggested in the initial section, before any mention of HTTPS. Beware however that wiki.jenkins.io is just a collection of random edits made over many years with little oversight, not formal documentation. The demo container is what I actually verified five years ago (tracking down some more subtle issues with AllowEncodedSlashes), but ultimately the Jenkins project does not document and test a specific reverse proxy configuration—this is left either to individual administrators, or to a third-party packaging such as a Helm chart.

At any rate, I retried my test with ProxyPreserveHost On commented out, using HTTPS, and redirects still worked. This is from a local development environment, which necessarily differs from a production network in various ways, so it may be more fruitful to discuss a particular HTTP response from Jenkins that could be reproduced without actually running a reverse proxy: given a specific minimal set of request headers (mixture of browser-provided and proxy-provided/amended), whether a specific response header (Location I suppose) has one value in Jenkins 2.204 and a different value in 2.205.

bcooksley@kde.org (JIRA)

unread,
Nov 21, 2019, 12:21:06 PM11/21/19
to jenkinsc...@googlegroups.com

Could you provide the ProxyPass / ProxyPassReverse statements from your test image?
If they mirror the fix I mentioned above already, then this would explain why it works for you.

jglick@cloudbees.com (JIRA)

unread,
Nov 21, 2019, 12:25:05 PM11/21/19
to jenkinsc...@googlegroups.com

These directives look like they are properly mirrored to me. I also tried the same without the context path (/jenkins → /) and with nocanon replaced with retry=0 as in your example.

bcooksley@kde.org (JIRA)

unread,
Nov 21, 2019, 1:00:02 PM11/21/19
to jenkinsc...@googlegroups.com

I wonder if this is due to the non-standard port for Jenkins itself - that configuration uses 8080, whereas my setup uses 8100/8200 (for each of the two impacted instances respectively), and Olli above uses 8180.

jglick@cloudbees.com (JIRA)

unread,
Nov 21, 2019, 2:28:03 PM11/21/19
to jenkinsc...@googlegroups.com

My second test was using local port 8090 so I doubt it.

pierre-yves.bigourdan@bbc.co.uk (JIRA)

unread,
Dec 3, 2019, 11:35:04 AM12/3/19
to jenkinsc...@googlegroups.com

Adding ProxyPreserveHost On to our httpd configuration did solve the problem for us as well. Thanks for the tip!

katzdm@gmail.com (JIRA)

unread,
Mar 3, 2020, 9:03:02 PM3/3/20
to jenkinsc...@googlegroups.com

I believe the 2.204.x LTS branch is now affected by this as well - I started seeing the issue yesterday, after upgrading to 2.204.3 (which lends support for the Jetty theory).
This message was sent by Atlassian Jira (v7.13.12#713012-sha1:6e07c38)
Atlassian logo

katzdm@gmail.com (JIRA)

unread,
Mar 3, 2020, 10:59:03 PM3/3/20
to jenkinsc...@googlegroups.com
Daniel Katz updated an issue
 
Jenkins / Bug JENKINS-60199
Change By: Daniel Katz
Environment: Jenkins: 2.205 , 2.201.3, 2.201.4
Apache 2.4.41 as reverse proxy

olamy@apache.org (JIRA)

unread,
Mar 3, 2020, 11:32:04 PM3/3/20
to jenkinsc...@googlegroups.com

mark.earl.waite@gmail.com (JIRA)

unread,
Mar 3, 2020, 11:43:03 PM3/3/20
to jenkinsc...@googlegroups.com

Daniel Katz you'll need to provide more details about your configuration or even better, the greater precision of answers that Jesse Glick was seeking. He was unable to duplicate the problem in his attempts. If the issue is a result of the jetty upgrade, then it is likely that you'll need to apply the suggested configuration changes.

foreverbonfy@gmail.com (JIRA)

unread,
Mar 4, 2020, 12:31:03 AM3/4/20
to jenkinsc...@googlegroups.com
bon ch commented on Bug JENKINS-60199

I create example repo to this issue: https://github.com/bonfy/JENKINS-61317

Jenkinsfile in master branch Last commit : 64dd9920
library in jenkins-lib branch Last commit: a8c235b1

and when echo GIT_COMMIT, it is overwitten by lib's commit

[Pipeline] echo
13:22:18 stage 1
[Pipeline] echo
13:22:18 GIT COMMIT : a8c235b10c55e2f065293b1c1ad9b3bffd45d9d9
[Pipeline] echo
13:22:18 Hello world
[Pipeline] }

foreverbonfy@gmail.com (JIRA)

unread,
Mar 4, 2020, 12:36:03 AM3/4/20
to jenkinsc...@googlegroups.com
bon ch updated an issue
 
Change By: bon ch
Comment:
I create example repo to this issue: https://github.com/bonfy/JENKINS-61317

Jenkinsfile in master branch      Last commit : 64dd9920
library in  jenkins-lib branch    Last commit: a8c235b1

and when echo GIT_COMMIT, it is overwitten by lib's commit

[Pipeline] echo
13:22:18  stage 1
[Pipeline] echo
13:22:18  GIT COMMIT : a8c235b10c55e2f065293b1c1ad9b3bffd45d9d9
[Pipeline] echo
13:22:18  Hello world
[Pipeline] }

dbeck@cloudbees.com (JIRA)

unread,
Mar 4, 2020, 2:32:04 AM3/4/20
to jenkinsc...@googlegroups.com

katzdm@gmail.com (JIRA)

unread,
Mar 4, 2020, 11:12:04 AM3/4/20
to jenkinsc...@googlegroups.com

Hi Mark Waite, I've been able to isolate the problem. Consider the two following commands, which differ only in the ordering of the X-Forwarded-Port and X-Forwarded-Host headers.

  1. {{curl -L -w "%{url_effective}" "http://localhost:33000/whoAmI" \
        -H "Host: jenkins.example.com" \
        -H "X-Forwarded-Port: 80" \
    }}    -H "X-Forwarded-Host: localhost" -o/dev/null 2>/dev/null; echo
  2. {{curl -L -w "%{url_effective}" "http://localhost:33000/whoAmI" \
        -H "Host: jenkins.example.com" \
        -H "X-Forwarded-Host: localhost" \
        -H "X-Forwarded-Port: 80" -o/dev/null 2>/dev/null; echo}}

I ran these commands within docker containers based on various different Jenkins base-images, and observed the following behaviors:

So if I'm interpreting this right, it seems that the host used for Jenkins redirections is dependent on the ordering of the X-Forwarded-Port and X-Forwarded-Host headers, beginning in Jenkins 2.205+ and Jenkins 2.204.3+ for LTS. If I had to guess, this is probably due to changes made to the ForwardedRequestCustomizer class in Jetty, which were picked up in the move to Winstone 5.7. Maybe a bug in handlePort()?

To answer your previous question regarding "ProxyPreserveHost On", I am not behind an Apache reverse proxy (we use https://github.com/buzzfeed/sso), so this isn't directly relevant to me.

katzdm@gmail.com (JIRA)

unread,
Mar 4, 2020, 11:13:03 AM3/4/20
to jenkinsc...@googlegroups.com
Daniel Katz edited a comment on Bug JENKINS-60199
Hi [~markewaite], I've been able to isolate the problem. Consider the two following commands, which differ only in the ordering of the {{X-Forwarded-Port}} and {{X-Forwarded-Host}} headers.
# {{curl -L -w "%\{url_effective}" "http://localhost:33000/whoAmI" \ }}
{{     -H "Host: jenkins.example.com" \ }}
{{     -H "X-Forwarded-Port: 80" \
}}
{{ {{     -H "X-Forwarded-Host: localhost" -o/dev/null \}}}}
{{{{    2>/dev/null; echo}} }}

# {{curl -L -w "%\{url_effective}" "http://localhost:33000/whoAmI" \ }}
{{     -H "Host: jenkins.example.com" \ }}
{{     -H "X-Forwarded-Host: localhost" \ }}
{{     -H "X-Forwarded-Port: 80" -o/dev/null \}}
{{   
2>/dev/null; echo}}


I ran these commands within docker containers based on various different Jenkins base-images, and observed the following behaviors:
* *{{jenkins/jenkins:2.204.2-slim}}*:
Command 1: [{{ [ http://localhost/whoAmI/ ] }}
Command 2: |http://localhost/whoAmI/]{{[http://localhost/whoAmI/]}}
* *{{jenkins/jenkins:2.204.3-slim}}*:
Command 1: {{[http://jenkins.example.com/whoAmI/]}}[
Command 2: |http://localhost/whoAmI/]{{[http://localhost/whoAmI/]}}
* *{{jenkins/jenkins:slim}}*: (2.223; latest)
So if I'm interpreting this right, it seems that the host used for Jenkins redirections is dependent on the ordering of the {{X-Forwarded-Port}} and {{X-Forwarded-Host}} headers, beginning in Jenkins 2.205+ and Jenkins 2.204.3+ for LTS. If I had to guess, this is probably due to changes made to the {{ForwardedRequestCustomizer}} class in Jetty, which were picked up in the move to Winstone 5.7. Maybe a bug in [handlePort()| [https://github.com/eclipse/jetty.project/blob/jetty-9.4.24.v20191120/jetty-server/src/main/java/org/eclipse/jetty/server/ForwardedRequestCustomizer.java #L628]]?

To answer your previous question regarding "{{ProxyPreserveHost On}}", I am not behind an Apache reverse proxy (we use [ https://github.com/buzzfeed/sso ] ), so this isn't directly relevant to me.

katzdm@gmail.com (JIRA)

unread,
Mar 4, 2020, 11:14:10 AM3/4/20
to jenkinsc...@googlegroups.com
Daniel Katz edited a comment on Bug JENKINS-60199
Hi [~markewaite], I've been able to isolate the problem. Consider the two following commands, which differ only in the ordering of the {{X-Forwarded-Port}} and {{X-Forwarded-Host}} headers.
# curl -L -w "%\{url_effective}" "http://localhost:33000/whoAmI" \
    -H "Host: jenkins.example.com" \
    -H "X-Forwarded-Port: 80" \
    -H "X-Forwarded-Host: localhost" -o/dev/null \
 {{   }}  
2>/dev/null; echo

# curl -L -w "%\{url_effective}" "http://localhost:33000/whoAmI" \
    -H "Host: jenkins.example.com" \
    -H "X-Forwarded-Host: localhost" \
    -H "X-Forwarded-Port: 80" -o/dev/null \
    2>/dev/null; echo

I ran these commands within docker containers based on various different Jenkins base-images, and observed the following behaviors:
* *{{jenkins/jenkins:2.204.2-slim}}*:
Command 1: [{{[http://localhost/whoAmI/]}}
Command 2: |http://localhost/whoAmI/]{{[http://localhost/whoAmI/]}}
* *{{jenkins/jenkins:2.204.3-slim}}*:
Command 1: {{[http://jenkins.example.com/whoAmI/]}}[
Command 2: |http://localhost/whoAmI/]{{[http://localhost/whoAmI/]}}
* *{{jenkins/jenkins:slim}}*: (2.223; latest)
Command 1: {{[http://jenkins.example.com/whoAmI/]}}[
Command 2: |http://localhost/whoAmI/]{{[http://localhost/whoAmI/]}}

So if I'm interpreting this right, it seems that the host used for Jenkins redirections is dependent on the ordering of the {{X-Forwarded-Port}} and {{X-Forwarded-Host}} headers, beginning in Jenkins 2.205+ and Jenkins 2.204.3+ for LTS. If I had to guess, this is probably due to changes made to the {{ForwardedRequestCustomizer}} class in Jetty, which were picked up in the move to Winstone 5.7. Maybe a bug in [handlePort()|#L628]]?


To answer your previous question regarding "{{ProxyPreserveHost On}}", I am not behind an Apache reverse proxy (we use [https://github.com/buzzfeed/sso]), so this isn't directly relevant to me.

katzdm@gmail.com (JIRA)

unread,
Mar 4, 2020, 11:14:11 AM3/4/20
to jenkinsc...@googlegroups.com

katzdm@gmail.com (JIRA)

unread,
Mar 4, 2020, 11:14:12 AM3/4/20
to jenkinsc...@googlegroups.com

katzdm@gmail.com (JIRA)

unread,
Mar 4, 2020, 11:15:03 AM3/4/20
to jenkinsc...@googlegroups.com

mark.earl.waite@gmail.com (JIRA)

unread,
Mar 4, 2020, 11:34:03 AM3/4/20
to jenkinsc...@googlegroups.com

Thanks very much Daniel Katz!

katzdm@gmail.com (JIRA)

unread,
Mar 4, 2020, 11:40:03 AM3/4/20
to jenkinsc...@googlegroups.com
Daniel Katz updated an issue
 
Change By: Daniel Katz
Environment: Jenkins: 2.205, 2. 201 204 .3, 2. 201 204 .4

Apache 2.4.41 as reverse proxy

katzdm@gmail.com (JIRA)

unread,
Mar 4, 2020, 11:41:03 AM3/4/20
to jenkinsc...@googlegroups.com
Daniel Katz edited a comment on Bug JENKINS-60199
Hi [~markewaite], I've been able to isolate the problem. Consider the two following commands, which differ only in the ordering of the {{X-Forwarded-Port}} and {{X-Forwarded-Host}} headers.
# curl -L -w "%\{url_effective}" "http://localhost:33000/whoAmI" \
    -H "Host: jenkins.example.com" \
    -H "X-Forwarded-Port: 80" \
    -H "X-Forwarded-Host: localhost" -o/dev/null \
    2>/dev/null; echo
# curl -L -w "%\{url_effective}" "http://localhost:33000/whoAmI" \
    -H "Host: jenkins.example.com" \
    -H "X-Forwarded-Host: localhost" \
    -H "X-Forwarded-Port: 80" -o/dev/null \
    2>/dev/null; echo

I ran these commands within docker containers based on various different Jenkins base-images, and observed the following behaviors:
* *{{jenkins/jenkins:2.204.2-slim}}*:
Command 1: [ {{[http://localhost/whoAmI/]}}
Command 2: |http://localhost/whoAmI/] {{[http://localhost/whoAmI/]}}
* *{{jenkins/jenkins:2.204.3-slim}}*:
Command 1: {{[http://jenkins.example.com/whoAmI/]}}
Command 2: |http://localhost/whoAmI/] {{[http://localhost/whoAmI/]}}
* *{{jenkins/jenkins:slim}}*: (2.223; latest)
Command 1: {{[http://jenkins.example.com/whoAmI/]}}
Command 2: |http://localhost/whoAmI/]
{{[http://localhost/whoAmI/]}}

So if I'm interpreting this right, it seems that the host used for Jenkins redirections is dependent on the ordering of the {{X-Forwarded-Port}} and {{X-Forwarded-Host}} headers, beginning in Jenkins 2.205+ and Jenkins 2.204.3+ for LTS. If I had to guess, this is probably due to changes made to the {{ForwardedRequestCustomizer}} class in Jetty, which were picked up in the move to Winstone 5.7. Maybe a bug in handlePort()?


To answer your previous question regarding "{{ProxyPreserveHost On}}", I am not behind an Apache reverse proxy (we use [https://github.com/buzzfeed/sso]), so this isn't directly relevant to me.

katzdm@gmail.com (JIRA)

unread,
Mar 4, 2020, 12:27:03 PM3/4/20
to jenkinsc...@googlegroups.com

I found the bug in Jetty, and it seems the root cause is corrected in 9.4.27: https://github.com/eclipse/jetty.project/issues/4573

katzdm@gmail.com (JIRA)

unread,
Mar 4, 2020, 12:53:03 PM3/4/20
to jenkinsc...@googlegroups.com

There's currently a dependabot PR open to Winstone, which will pick up the latest Jetty release: https://github.com/jenkinsci/winstone/pull/94

I've commented there with a reference to this bug, with hope that it can get merged + released.

o.v.nenashev@gmail.com (JIRA)

unread,
Mar 4, 2020, 1:00:09 PM3/4/20
to jenkinsc...@googlegroups.com
Oleg Nenashev assigned an issue to Oleg Nenashev
 
Change By: Oleg Nenashev
Assignee: Oleg Nenashev

o.v.nenashev@gmail.com (JIRA)

unread,
Mar 4, 2020, 1:00:11 PM3/4/20
to jenkinsc...@googlegroups.com
Oleg Nenashev started work on Bug JENKINS-60199
 
Change By: Oleg Nenashev
Status: Open In Progress

o.v.nenashev@gmail.com (JIRA)

unread,
Mar 4, 2020, 1:00:13 PM3/4/20
to jenkinsc...@googlegroups.com
Oleg Nenashev updated an issue
Change By: Oleg Nenashev
Component/s: winstone-jetty

o.v.nenashev@gmail.com (JIRA)

unread,
Mar 4, 2020, 1:00:15 PM3/4/20
to jenkinsc...@googlegroups.com
Change By: Oleg Nenashev
Status: In Progress Review

o.v.nenashev@gmail.com (JIRA)

unread,
Mar 4, 2020, 1:00:15 PM3/4/20
to jenkinsc...@googlegroups.com
Oleg Nenashev updated an issue
Change By: Oleg Nenashev
Labels: lts-candidate regression

o.v.nenashev@gmail.com (JIRA)

unread,
Mar 4, 2020, 1:03:04 PM3/4/20
to jenkinsc...@googlegroups.com
Oleg Nenashev commented on Bug JENKINS-60199
 
Re: jenkins 2.205 behind reverse proxy redirects to 127.0.0.1 after login

Assigned it to myself so that I do not forget about the merge/release towards the next weekly. We need it to be backported to the 2.220.x LTS baseline.

At the same time backporting Jetty dependency bump is risky, so needs risk assessment. CC Olivier Lamy who is also a Jetty maintainer

olamy@apache.org (JIRA)

unread,
Mar 5, 2020, 12:57:05 AM3/5/20
to jenkinsc...@googlegroups.com
Change By: Olivier Lamy
Status: In Review Resolved
Resolution: Fixed

o.v.nenashev@gmail.com (JIRA)

unread,
Mar 5, 2020, 3:19:05 AM3/5/20
to jenkinsc...@googlegroups.com

o.v.nenashev@gmail.com (JIRA)

unread,
Mar 5, 2020, 3:19:10 AM3/5/20
to jenkinsc...@googlegroups.com
Oleg Nenashev reopened an issue
 
Change By: Oleg Nenashev
Resolution: Fixed
Status: Resolved Reopened

o.v.nenashev@gmail.com (JIRA)

unread,
Mar 5, 2020, 3:43:06 AM3/5/20
to jenkinsc...@googlegroups.com
Oleg Nenashev updated an issue
Change By: Oleg Nenashev
Environment: Jenkins: 2.205, Jenkins LTS 2.204.3 , and 2.204.4
 Winstone 5.5 to 5.8
Apache 2.4.41 as reverse proxy

o.v.nenashev@gmail.com (JIRA)

unread,
Mar 5, 2020, 3:47:07 AM3/5/20
to jenkinsc...@googlegroups.com
Oleg Nenashev updated an issue
After the upgrade from 2.204 to 2.205 jenkins redirects to http(s)://127.0.0.1/ after login.

 

**Jenkins LTS Notice**: Jenkins LTS 2.204.3 and 2.204.4 are also affected due to the Winstone upgrade which was backported as a part of the JENKINS-57888 fix backporting. Please see [https://groups.google.com/forum/#!topic/jenkinsci-dev/M_RtDuDXtbU] for the discussion and retrospective

o.v.nenashev@gmail.com (JIRA)

unread,
Mar 5, 2020, 3:50:06 AM3/5/20
to jenkinsc...@googlegroups.com
Oleg Nenashev updated an issue
After the upgrade from 2.204 to 2.205 jenkins redirects to http(s)://127.0.0.1/ after login.

 

**Jenkins LTS Notice**: Jenkins LTS 2.204.3 and 2.204.4 are also affected due to the Winstone upgrade which was backported introduced as a part of the JENKINS-57888 fix backporting. Please see [https://groups.google.com/forum/#!topic/jenkinsci-dev/M_RtDuDXtbU] for the discussion and retrospective

o.v.nenashev@gmail.com (JIRA)

unread,
Mar 5, 2020, 4:04:05 AM3/5/20
to jenkinsc...@googlegroups.com
Oleg Nenashev updated an issue
After the upgrade from 2.204 to 2.205 jenkins redirects to http(s)://127.0.0.1/ after login.

  *Workaround*: For Apache: set "ProxyPreserveHost On" as documented in [https://wiki.jenkins.io/display/JENKINS/Running+Jenkins+behind+Apache]

**Jenkins LTS Notice**: Jenkins LTS 2.204.3 and 2.204.4 are also affected due to the Winstone upgrade which was introduced as a part of the JENKINS-57888 fix backporting. Please see [https://groups.google.com/forum/#!topic/jenkinsci-dev/M_RtDuDXtbU] for the discussion and retrospective

mark.earl.waite@gmail.com (JIRA)

unread,
Mar 5, 2020, 10:25:03 AM3/5/20
to jenkinsc...@googlegroups.com
Mark Waite commented on Bug JENKINS-60199
 
Re: jenkins 2.205 behind reverse proxy redirects to 127.0.0.1 after login

Do we also need a message applied to the 2.204.3 changelog and the 2.204.4 changelog to alert users that they may be affected by it?

o.v.nenashev@gmail.com (JIRA)

unread,
Mar 5, 2020, 10:48:05 AM3/5/20
to jenkinsc...@googlegroups.com

Mark Waite it would be nice, I have not got to it yet due to LTS backports

jglick@cloudbees.com (JIRA)

unread,
Mar 5, 2020, 11:24:06 AM3/5/20
to jenkinsc...@googlegroups.com
Change By: Jesse Glick
Status: Reopened Open

jglick@cloudbees.com (JIRA)

unread,
Mar 5, 2020, 11:24:08 AM3/5/20
to jenkinsc...@googlegroups.com
Jesse Glick started work on Bug JENKINS-60199
 
Change By: Jesse Glick
Status: Open In Progress

jglick@cloudbees.com (JIRA)

unread,
Mar 5, 2020, 11:24:09 AM3/5/20
to jenkinsc...@googlegroups.com
Change By: Jesse Glick
Status: In Progress Review

o.v.nenashev@gmail.com (JIRA)

unread,
Mar 5, 2020, 4:28:03 PM3/5/20
to jenkinsc...@googlegroups.com
Oleg Nenashev updated an issue

I have just released an alternate Winstone 5.4.1 release for 2.204.x LTS. This patch reverts Jetty to older versions https://github.com/jenkinsci/winstone/releases/tag/winstone-5.4.1 but keeps other regression fixes. It should be more stable than upgrade to Winstone 5.9 with just another Jetty upgrade and a risk of new regressions. 

Pull request with the 2.204.x baseline update: https://github.com/jenkinsci/jenkins/pull/4545
Change By: Oleg Nenashev
Released As: Winstone 5.4.1, Winstone 5.9

o.v.nenashev@gmail.com (JIRA)

unread,
Mar 5, 2020, 4:47:06 PM3/5/20
to jenkinsc...@googlegroups.com
Oleg Nenashev updated an issue
Change By: Oleg Nenashev
Priority: Minor Major

jglick@cloudbees.com (JIRA)

unread,
Mar 6, 2020, 3:41:06 PM3/6/20
to jenkinsc...@googlegroups.com
Change By: Jesse Glick
Status: In Review Fixed but Unreleased
Resolution: Fixed

dbeck@cloudbees.com (JIRA)

unread,
Mar 7, 2020, 9:13:08 PM3/7/20
to jenkinsc...@googlegroups.com
Daniel Beck updated an issue
Change By: Daniel Beck
Labels: 2.204.5-fixed lts-candidate regression

o.v.nenashev@gmail.com (JIRA)

unread,
Mar 8, 2020, 8:14:04 AM3/8/20
to jenkinsc...@googlegroups.com
Oleg Nenashev updated an issue
Change By: Oleg Nenashev
Released As: Jenkins 2.204.5, Jenkins 2.224, Winstone 5.4. 1 3 , Winstone 5.9

o.v.nenashev@gmail.com (JIRA)

unread,
Mar 8, 2020, 8:14:04 AM3/8/20
to jenkinsc...@googlegroups.com
 
Re: jenkins 2.205 behind reverse proxy redirects to 127.0.0.1 after login

o.v.nenashev@gmail.com (JIRA)

unread,
Mar 9, 2020, 8:46:10 AM3/9/20
to jenkinsc...@googlegroups.com
Change By: Oleg Nenashev
Status: Fixed but Unreleased Resolved

ogondza@gmail.com (JIRA)

unread,
Mar 10, 2020, 9:46:04 AM3/10/20
to jenkinsc...@googlegroups.com
Oliver Gondža updated an issue
Change By: Oliver Gondža
Labels: 2.204.5-fixed lts 2.222.1 - candidate fixed regression
Reply all
Reply to author
Forward
0 new messages