Jenkins IIS Reverse Proxy

[Terry Lacy]

Hi,

 

Sorry if this has been answered before.

We're having a very difficult time trying to get an IIS reverse proxy working with Jenkins. No matter what we try, we still get the "It appears that your reverse proxy set up is broken" message on the Manage Jenkins page.

We've seen a few links describing how to do it, but none of them seem to get rid of the "broken" message, and there doesn't appear to be any "official" Jenkins documentation on the subject.

I'd be happy to write a HOWTO on the wiki if we could get some help getting it working.

Details:

Windows Server 2012 R2

Jenkins 2.46.1/2.46.2

Tomcat 8.5.14

Java 1.8.0_131

IIS 10 (.msi installer/Jetty) and IIS 8.5 with the .war on Tomcat

IIS tried creating standard reverse proxy rule 

with allow double escaping

maximum URL length 4096

also tried with server variables allowed and added to rule config

        RequestHeader set X-Forwarded-Proto https

        RequestHeader set X-Forwarded-Port 443 

tried with Outbound rule - rewrite the domain names and links with server variable 

        HTTP_ACCEPT_ENCODING 

We tried IIS with ISAPI redirector to tomcat

We also tried an IIS server farm setup.

We tried it with and without SSL.

Basically, we've tried all of the things that normally work fine with many, many other IIS-Tomcat installations we've set up. They've worked fine on older versions of Jenkins. We're aware that the reverse proxy check is more strict now, but we don't really know what it's complaining about.

Thanks,

 

Terry Lacy
Sr. Software Design Engineer

Pediatric Research Enterprise​
University of Utah

[Terry Lacy]

Is there other information I could supply that would help?

Alternatively, has anyone successfully set up Jenkins behind IIS with a setup similar to that described below *without* getting the bad proxy message? Would you be willing to share your setup?

So far, things seem to work fine despite the message, except for the "More info" button next to the "It appears that your reverse proxy set up is broken." message. That returns a 404 error.

Also, I neglected to mention that we're authenticating with LDAP against an AD server in the Jenkins Global Security configuration.

Thanks,

Terry

[Stephen Connolly]

What about X-Forewarded-Host or is IIS on the same port.

Also Jenkins must be on the same context path as it is running as (iow do not have the http proxy rewrite the context path)

tried with Outbound rule - rewrite the domain names and links with server variable 

        HTTP_ACCEPT_ENCODING 

We tried IIS with ISAPI redirector to tomcat

We also tried an IIS server farm setup.

We tried it with and without SSL.

Basically, we've tried all of the things that normally work fine with many, many other IIS-Tomcat installations we've set up. They've worked fine on older versions of Jenkins. We're aware that the reverse proxy check is more strict now, but we don't really know what it's complaining about.

Thanks,

 

Terry Lacy
Sr. Software Design Engineer

Pediatric Research Enterprise​
University of Utah

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/a183d447-5937-4a13-9bfb-0207a2f4f14a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Sent from my phone

[Terry Lacy]

Hi, Stephen. Thanks for your reply.

I'm not sure about X-Forwarded-Host. I think we tried that, but I'll have to check with our SA, and he's not here right now. I'll get back here as soon as I find out.

We did try a couple of iterations where we tried to redirect https://our.server.name/ to http://localhost:8080/jenkins, but we gave up on that. Both IIS and Jenkins are now using the same context root (just "/").

Our desired configuration is IIS on port 443 (https) talking to Jenkins on port 8080 (http) on the same server. That's how we've got it right now.

Terry

[Terry Lacy]

I'm not sure if this is useful information, but we see this in the jenkins-stderr logs when browsing to the Manage Jenkins page:

01-May-2017 12:10:39.038 WARNING [Handling GET /administrativeMonitor/hudson.diagnosis.ReverseProxySetupMonitor/testForReverseProxySetup/https:/our.server.name/manage/ from 0:0:0:0:0:0:0:1 : http-nio-8080-exec-2] hudson.diagnosis.ReverseProxySetupMonitor.getTestForReverseProxySetup https://our.server.name/manage vs. https:

Terry

On Mon, May 1, 2017 at 11:46 AM, Terry Lacy <terry...@gmail.com> wrote:

Hi, Stephen. Thanks for your reply.

I'm not sure about X-Forwarded-Host. I think we tried that, but I'll have to check with our SA, and he's not here right now. I'll get back here as soon as I find out.

We did try a couple of iterations where we tried to redirect https://our.server.name/ to http://localhost:8080/jenkins, but we gave up on that. Both IIS and Jenkins are now using the same context root (just "/").

Our desired configuration is IIS on port 443 (https) talking to Jenkins on port 8080 (http) on the same server. That's how we've got it right now.

Terry

[Terry Lacy]

Here's our current web.config file for IIS:

<?xml version="1.0" encoding="UTF-8"?>

<configuration>

    <system.webServer>

        <security>

            <requestFiltering allowDoubleEscaping="true" />

        </security>

        <rewrite>

            <rules>

                <rule name="ReverseProxyInboundRule1" stopProcessing="true">

                    <match url="(.*)" />

                    <action type="Rewrite" url="http://localhost:8080/{R:1}" />

                    <serverVariables>

                        <set name="HTTP_X_FORWARDED_HOST" value="{HTTP_HOST}" />

                        <set name="HTTP_X_FORWARDED_SCHEMA" value="https" />

                        <set name="HTTP_X_FORWARDED_PROTO" value="https" />

                    </serverVariables>

                </rule>

            </rules>

        </rewrite>        

    </system.webServer>

</configuration>

[Terry Lacy]

Summarizing this and adding a little information. We still don't have a solution. 

We're still getting "It appears that your reverse proxy set up is broken." on the Manage Jenkins page.

 

Jenkins 2.46.2 .war file 

Tomcat 8.5.14.

Windows Server 2012 R2 Datacenter 64-bit

IIS 8.5

Java 1.8.0_131 64-bit

 

IIS and Jenkins are on the same server.

 

We want to present an external URL of "https://our.server.name

Tomcat connector on port 8080.

We installed Jenkins as "ROOT.war" so that Jenkins is accessible from the localhost at "http://localhost:8080"

 

We configured the Jenkins Location/Jenkins URL in Jenkins' "Configure System" as "https://our.server.name/" (Jenkins insists on appending the trailing "/" when we save the configuration).

 

Here's our IIS web.config file:

 

<?xml version="1.0" encoding="UTF-8"?>

<configuration>

    <system.webServer>

        <security>

            <requestFiltering allowDoubleEscaping="true" />

        </security>

        <rewrite>

            <rules>

                <rule name="ReverseProxyInboundRule1" stopProcessing="true">

                    <match url="(.*)" />

                    <action type="Rewrite" url="http://localhost:8080/{R:1}" />

                    <serverVariables>

                        <set name="HTTP_X_FORWARDED_HOST" value="{HTTP_HOST}" />

                        <set name="HTTP_X_FORWARDED_SCHEMA" value="https" />

                        <set name="HTTP_X_FORWARDED_PROTO" value="https" />

                    </serverVariables>

                </rule>

            </rules>

        </rewrite>        

    </system.webServer>

</configuration>

 

Here's the connector configuration Tomcat's server.xml:

 

<Connector port="8080" protocol="HTTP/1.1"

  connectionTimeout="20000"

  URIEncoding="UTF-8"

  redirectPort="8443"

  proxyName="our.server.name" scheme="https" proxyPort="443" />

 

Again, I'm happy to write this up in the Jenkins wiki if we can find a working solution.

 

Does *anyone* have this working? Would you be willing to share your setup?

 

We've done this many times with other Tomcat applications on Windows without problems. We can't figure out what the issue is here.

 

There is just one related(?) warning in jenkins-stderr.2017-05-03.log:

 

03-May-2017 09:14:34.521 WARNING [Handling GET /administrativeMonitor/hudson.diagnosis.ReverseProxySetupMonitor/testForReverseProxySetup/https:/our.server.name/manage/ from 0:0:0:0:0:0:0:1 : http-nio-8080-exec-3] hudson.diagnosis.ReverseProxySetupMonitor.getTestForReverseProxySetup https://our.server.name/manage vs. https:

 

Here’s the contents of Tomcat's localhost_access_log.2017-05-03.txt around the reverse proxy test:

0:0:0:0:0:0:0:1 - - [03/May/2017:09:14:34 -0600] "GET /administrativeMonitor/jenkins.diagnostics.URICheckEncodingMonitor/checkURIEncoding?value=%E5%9F%B7%E4%BA%8B HTTP/1.1" 200 6

0:0:0:0:0:0:0:1 - - [03/May/2017:09:14:34 -0600] "POST /administrativeMonitor/hudson.diagnosis.ReverseProxySetupMonitor/test HTTP/1.1" 302 -

0:0:0:0:0:0:0:1 - - [03/May/2017:09:14:34 -0600] "GET /static/82168ce8/images/16x16/error.png HTTP/1.1" 200 543

0:0:0:0:0:0:0:1 - - [03/May/2017:09:14:34 -0600] "GET /administrativeMonitor/hudson.diagnosis.ReverseProxySetupMonitor/testForReverseProxySetup/https:/our.server.name/manage/ HTTP/1.1" 404 1116

 

Note the 404 response on that last entry.  Could that be an URL encoding issue? Anybody know how to fix that with IIS?

Thanks,

Terry Lacy

Sr. Software Design Engineer

Pediatric Research Enterprise​

[Stephen Connolly]

On 3 May 2017 at 16:30, Terry Lacy <terry...@gmail.com> wrote:

/administrativeMonitor/hudson.diagnosis.ReverseProxySetupMonitor/testForReverseProxySetup/https:/our.server.name/manage/

That looks to be the malformed url. you should want to see why it is converting the URL to "https:/our.server.name" and not "https://our.server.name" which would be resulting in the URL being treated as relative rather than absolute

[Terry Lacy]

Hi, Stephen,

Thanks for your reply. I don't have any ideas on why it would be converting the URL that way.  Could be an URL encoding issue, but I have no idea how to fix it.

Terry

[niristotle okram]

Is there an ip v6 enabled as well? 

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CABCG7bcx3589mXsME6on989%3D98OHbUu_7va%3D17sScfqmbJWNrg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Sent from mobile device, excuse typos if any.

[Terry Lacy]

For whatever it's worth, adding another "/" to that URL still results in a 404.

Terry

[niristotle okram]

Does this means that your reverse proxy is not working? 

And not just the error on the jenkins page 

--

You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/afcd559b-0d82-47b4-8603-9f20e56171a5%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

[Terry Lacy]

I wondered about that too. I'll get with our SA and see.  I wonder whether it'd be better to use "127.0.0.1" in the configuration instead of "localhost"

[Terry Lacy]

By all appearances, the reverse proxy is working. However, we haven't done really extensive testing yet. I've just done a couple of builds and tested with one slave.

So, yeah, so far, it's just the message on the Jenkins Manage Jenkins page.  However, I'm assuming the message is there for a reason. I just don't understand what it's complaining about.

Terry

[Terry Lacy]

Okay, we verified that ipv6 was disabled on the network interface. We also changed the IIS reverse proxy to point to "127.0.0.1:8080" instead of "localhost:8080". That didn't make any difference.

[Terry Lacy]

Since there doesn't seem to be a solution to this, would y'all advise that I post a bug report?

Terry

[Terry Lacy]

I posted a bug (https://issues.jenkins-ci.org/browse/JENKINS-44239), and it was resolved as "not a defect", so I'm back here, still looking for a solution.

Does anyone here run Jenkins behind IIS? Are you using a recent version? Are you able to get rid of the "It appears that your reverse proxy set up is broken" error?

Thanks,

Terry

[Terry Lacy]

Here's some extra information I haven't posted before.

The wiki page linked from the "More Info" button (https://wiki.jenkins-ci.org/display/JENKINS/Jenkins+says+my+reverse+proxy+setup+is+broken) says this curl command will provide more info:

curl -iL -e http://your.reverse.proxy/jenkins/manage \
   http://your.reverse.proxy/jenkins/administrativeMonitor/hudson.diagnosis.ReverseProxySetupMonitor/test

I ran that from one of our linux servers. This was the result:

$ curl -u 'XXXX:XXXXXXXX' -iL -e https://our.server.name/manage https://our.server.name/administrativeMonitor/hudson.diagnosis.ReverseProxySetupMonitor/test


HTTP/1.1 302
Cache-Control: private
Content-Length: 0
Expires: Wed, 31 Dec 1969 17:00:00 MST
Location: https://our.server.name/administrativeMonitor/hudson.diagnosis.ReverseProxySetupMonitor/testForReverseProxySetup/https%3A%2F%2Four.server.name%2Fmanage/
X-Content-Type-Options: nosniff
X-Powered-By: ARR/3.0
Date: Thu, 25 May 2017 19:30:37 GMT


HTTP/1.1 404
Cache-Control: private
Content-Length: 1116
Content-Type: text/html;charset=utf-8
Content-Language: en
Expires: Wed, 31 Dec 1969 17:00:00 MST
X-Content-Type-Options: nosniff
X-Powered-By: ARR/3.0
Date: Thu, 25 May 2017 19:30:40 GMT


<!doctype html><html lang="en"><head><title>HTTP Status 404 – Not Found</title><style type="text/css">h1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} h2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} h3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} body {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} b {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} p {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;} a {color:black;} a.name {color:black;} .line {height:1px;background-color:#525D76;border:none;}</style></head><body><h1>HTTP Status 404 – Not Found</h1><hr class="line" /><p><b>Type</b> Status Report</p><p><b>Message</b> https://our.server.name/manage vs. https:</p><p><b>Description</b> The origin server did not find a current representation for the target resource or is not willing to disclose that one exists.</p><hr class="line" /><h3>Apache Tomcat/8.5.14</h3></body></html>

Any helpful information there?

Terry

[Stephen Connolly]

https://our.server.name/administrativeMonitor/hudson.diagnosis.ReverseProxySetupMonitor/testForReverseProxySetup/https%3A%2F%2Four.server.name%2Fmanage/

Looks like the URL rewrite rule is setup wrong and is double prefix with https://...

--

You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/e9d7f7ee-2494-40a9-b0af-1f8ee57ec037%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Terry Lacy]

Thanks for your reply, Stephen.

Here's the IIS web.config file. Does anyone know how the rewrite rule should be configured for Jenkins?

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <system.webServer>
        <security>
            <requestFiltering allowDoubleEscaping="true" />
        </security>
        <rewrite>
            <rules>
                <rule name="ReverseProxyInboundRule1" stopProcessing="true">
                    <match url="(.*)" />
                    <action type="Rewrite" url="http://localhost:8080/{R:1}" />
                    <serverVariables>
                        <set name="HTTP_X_FORWARDED_HOST" value="{HTTP_HOST}" />
                        <set name="HTTP_X_FORWARDED_SCHEMA" value="https" />
                        <set name="HTTP_X_FORWARDED_PROTO" value="https" />
                    </serverVariables>
                </rule>
            </rules>
        </rewrite>        
    </system.webServer>
</configuration>

?

Thanks,

Terry

[Terry Lacy]

After pursuing this in this forum for over a month, posting a bug and having it promptly closed as "not a defect", and no working solution, I'm left to conclude that it is not possible to get rid of the "your reverse proxy set up is broken" with recent versions of Jenkins and IIS.

Or, nobody here uses IIS as a reverse proxy.

Or, folks who use IIS as a reverse proxy ignore the error message.

Terry

[Stephen Connolly]

I do not have easy access to a windows machine, so I cannot help with configuration of that

I should be possible to configure it correctly, if the warning is showing then you will find some features don't work, so I recommend trying to fix it

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/b838616b-e50d-422b-bc93-c700aed859dc%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Terry Lacy]

Thanks, Stephen.

Yes, I would certainly like to fix it. I'm sure it will break something down the road. But I can't find a working configuration anywhere. 

Any. Where.

Terry

[Chad Ruppert]

Terry, did you ever get this resolved?

[Terry Lacy]

Yes, see this:

https://wiki.jenkins.io/display/JENKINS/Running+Jenkins+behind+IIS

That procedure works for us.

Terry