[JIRA] (JENKINS-58094) mTLS client certificates broken in 2.176.1+ by changes in jetty

11 views
Skip to first unread message

joeym@joeym.net (JIRA)

unread,
Jun 19, 2019, 9:49:03 AM6/19/19
to jenkinsc...@googlegroups.com
joe miller created an issue
 
Jenkins / Bug JENKINS-58094
mTLS client certificates broken in 2.176.1+ by changes in jetty
Issue Type: Bug Bug
Assignee: Unassigned
Components: winstone-jetty
Created: 2019-06-19 13:48
Priority: Major Major
Reporter: joe miller

Client certificate support is broken in 2.176.1+ by changes in Jetty.

When running jenkins with winstone/jetty configured to require client cert auth most (all?) client certs will fail validation with an error such as:

"java.security.cert.CertificateException: No subject alternative names matching IP address x.x.x.x.x found"

This error message is only visible if debug logging is enabled: JAVA_OPTS= -Djavax.net.debug=ssl:handshake

We have tracked this down to the upgrade to Winstone 5.2 in 2.176.1 (https://github.com/jenkins-infra/jenkins.io/blob/master/content/_data/changelogs/lts.yml#L2518) which in turn was caused by a change in jetty 9.4.15+

The summary of the issue is that Jetty changed the default behavior of their SslContextFactory to improve cert verification for their HttpClient implementation. Since both the Http server and client functions in jetty use the same SslContextFactory this broke the https server use case.

There are several bug/issue reports against the Jetty project for this:

Jetty decided to address this bug by creating SslContextFactory.Server and .Client subclasses. But the broken behavior remains in the parent SslContextFactory which is what winstone is using: https://github.com/eclipse/jetty.project/pull/3480

It appears that there are two possible fixes in winstone:

  1. Switch from using SslContextFactory to SslContextFactory.Server
  2. or, set endpointIdentificationAlgorithm to "null" in the call to SslContextFactory

I have been unable to find a workaround for this such as setting the "endpointIdentificationAlgorithm" property to 'null' using JAVA_OPTS. The only fix we've found is to pin our use of jenkins to v2.164.3

Here is an example from the apache/felix project of how they fixed this bug by switching to the new .Server factory subclass introduced by jetty #3480 (v9.4.18+): https://github.com/apache/felix/pull/199/files

 
Add Comment Add Comment
 
This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)

joeym@joeym.net (JIRA)

unread,
Jun 19, 2019, 9:52:02 AM6/19/19
to jenkinsc...@googlegroups.com
joe miller updated an issue
Change By: joe miller
Client certificate support is broken in 2.176.1+ by changes in Jetty.

When running jenkins with winstone/jetty configured to require client cert auth most (all?) client certs will fail validation with an error such as:
{noformat}
" java.security.cert.CertificateException: No subject alternative names matching IP address x.x.x.x.x found " {noformat}

This error message is only visible if debug logging is enabled: JAVA_OPTS= -Djavax.net.debug=ssl:handshake

We have tracked this down to the upgrade to Winstone 5.2 upgrade in 2.176.1 ([https://github.com/jenkins-infra/jenkins.io/blob/master/content/_data/changelogs/lts.yml#L2518]) which in turn was caused by a change in jetty 9.4.15+


The summary of the issue is that Jetty changed the default behavior of their SslContextFactory to improve cert verification for their HttpClient implementation. Since both the Http server and client functions in jetty use the same SslContextFactory this broke the https server use case.

There are several bug/issue reports against the Jetty project for this:
* [https://github.com/eclipse/jetty.project/issues/3454]
* [https://github.com/eclipse/jetty.project/issues/3554]

Jetty decided to address this bug by creating SslContextFactory.Server and .Client subclasses in v9 . 4.18+. But the broken behavior remains in the parent SslContextFactory which is what winstone is using: [https://github.com/eclipse/jetty.project/pull/3480]


It appears that there are two possible fixes in winstone:
# Switch from using SslContextFactory to -> SslContextFactory.Server
# or, set endpointIdentificationAlgorithm to "null" in the call to SslContextFactory

I have been unable to find a workaround for this such as setting the "endpointIdentificationAlgorithm" property to 'null' using JAVA_OPTS. The only fix we've found is to pin our existing use of jenkins to v2.164.3 SslContextFactory

Here is an example from the apache/felix project of how they fixed this bug by switching to the new .Server factory subclass introduced by jetty #3480 (v9.4.18+): [https://github.com/apache/felix/pull/199/files]

I have been unable to find a workaround for this such as setting the "endpointIdentificationAlgorithm" property to 'null' using JAVA_OPTS. The only fix we've found is to pin our use of jenkins to v2.164.3

  

 

joeym@joeym.net (JIRA)

unread,
Jun 19, 2019, 12:46:02 PM6/19/19
to jenkinsc...@googlegroups.com
joe miller updated an issue
Client certificate support is broken in 2.176.1+ by changes in Jetty.

When running jenkins with winstone/jetty configured to require client cert auth most (all?) client certs will fail validation with an error such as:
{noformat}
java.security.cert.CertificateException: No subject alternative names matching IP address x.x.x.x.x found{noformat}
This error message is only visible if debug logging is enabled: JAVA_OPTS= -Djavax.net.debug=ssl:handshake

We have tracked this down to the Winstone 5.2 upgrade in 2.176.1 ([https://github.com/jenkins-infra/jenkins.io/blob/master/content/_data/changelogs/lts.yml#L2518]) which in turn was caused by a change in jetty 9.4.15+


The summary of the issue is that Jetty changed the default behavior of their SslContextFactory to improve cert verification for their HttpClient implementation. Since both the Http server and client functions in jetty use the same SslContextFactory this broke the https server use case.

There are several bug/issue reports against the Jetty
project for this:
Jetty decided to address this bug by creating SslContextFactory.Server and .Client subclasses in v9.4.18+. But the broken behavior remains in the parent SslContextFactory which is what winstone is using: [https://github.com/eclipse/jetty.project/pull/3480]

There are several bug/issue reports against the Jetty project for this:
* [https://github.com/eclipse/jetty.project/issues/3454]
* [https://github.com/eclipse/jetty.project/issues/3554]
It appears that there are two possible fixes in winstone:
# Switch from using SslContextFactory -> SslContextFactory.Server
# or, set endpointIdentificationAlgorithm to "null" in the existing use of SslContextFactory

Here is an example from the apache/felix project of how they fixed this bug by switching to the new . Server factory subclass introduced by jetty #3480 (v9.4.18+): [https://github.com/apache/felix/pull/199/files]


I have been unable to find a workaround for this such as setting the "endpointIdentificationAlgorithm" property to 'null' using JAVA_OPTS. The only fix we've found is to pin our use of jenkins to v2.164.3

 

 

o.v.nenashev@gmail.com (JIRA)

unread,
Jul 2, 2019, 8:47:02 PM7/2/19
to jenkinsc...@googlegroups.com

mtdeguzis@geisigner.edu (JIRA)

unread,
Jul 5, 2019, 3:44:02 PM7/5/19
to jenkinsc...@googlegroups.com

I wonder if this is also affecting our failure to bind to active directory using the JDK Trustore option of Jenkins > Configure Global Security > Security Ream > Active Directory > TLS Configuration.

simple bind failed: LDAP_HOST:636
sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)
	at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)
	at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280)
	at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:392)
Caused: sun.security.validator.ValidatorException: PKIX path building failed
	at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:397)
	at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:302)
	at sun.security.validator.Validator.validate(Validator.java:262)
	at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324)
	at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:229)
	at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:124)
	at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1621)
Caused: javax.net.ssl.SSLHandshakeException
	at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
	at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1946)
	at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:316)
	at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:310)
	at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1639)
	at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:223)
	at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1037)
	at sun.security.ssl.Handshaker.process_record(Handshaker.java:965)
	at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1064)
	at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1367)
	at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:750)
	at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:123)
	at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
	at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
	at com.sun.jndi.ldap.Connection.writeRequest(Connection.java:443)
	at com.sun.jndi.ldap.Connection.writeRequest(Connection.java:416)
	at com.sun.jndi.ldap.LdapClient.ldapBind(LdapClient.java:359)
	at com.sun.jndi.ldap.LdapClient.authenticate(LdapClient.java:214

As soon as I downgrade to 2.164.3, things work fine again. I setup a QA/Test master with only the Active Directory plugin.

Thoughts?

olamy@apache.org (JIRA)

unread,
Jul 8, 2019, 3:15:02 AM7/8/19
to jenkinsc...@googlegroups.com
Olivier Lamy assigned an issue to Olivier Lamy
 
Change By: Olivier Lamy
Assignee: Olivier Lamy

dbeck@cloudbees.com (JIRA)

unread,
Jul 8, 2019, 4:47:02 AM7/8/19
to jenkinsc...@googlegroups.com
Daniel Beck updated an issue
Change By: Daniel Beck
Labels: regression

mtdeguzis@geisigner.edu (JIRA)

unread,
Aug 19, 2019, 12:09:03 PM8/19/19
to jenkinsc...@googlegroups.com
 
Re: mTLS client certificates broken in 2.176.1+ by changes in jetty

I found an answer for my issue, and I'm still not sure if it's related. For whatever reason, beyond Jenkins 2.176.1, the JVM acts as if it no longer is looking at the default trust store at /etc/ssl/certs/java/cacerts. There was no need to import the public cert from our AD LDAP server into the default truststore.

export JAVA_OPTS="-Dhudson.plugins.active_directory.ActiveDirectorySecurityRealm.forceLdaps=true \
    -Dorg.apache.commons.jelly.tags.fmt.timeZone=America/New_York \
    -Djavax.net.ssl.trustStore=/etc/ssl/certs/java/cacerts \
    -Djavax.net.ssl.trustStorePassword=changeit"

Full docker run:

+ sudo docker run -d --name jenkins-server-test --network=jenkins-network --volume jenkins-test-data:/var/jenkins_home --publish 8555:8443 --env 'JAVA_OPTS=-Dhudson.plugins.active_directory.ActiveDirectorySecurityRealm.forceLdaps=true     -Dorg.apache.commons.jelly.tags.fmt.timeZone=America/New_York     -Djavax.net.ssl.trustStore=/etc/ssl/certs/java/cacerts     -Djavax.net.ssl.trustStorePassword=changeit' geisinger-jenkins-test:2.176.2 --sessionTimeout=15 --httpPort=-15 --httpsPort=8443 --httpsKeyStore=/var/jenkins_home/.keystore/host.domain.com.all.jks --httpsKeyStorePassword=*****************

mtdeguzis@geisigner.edu (JIRA)

unread,
Aug 19, 2019, 1:46:03 PM8/19/19
to jenkinsc...@googlegroups.com
Michael DeGuzis edited a comment on Bug JENKINS-58094
I found an answer for my issue, and I'm still not sure if it's related.  For whatever reason, beyond Jenkins 2. 176 164 . 1 3 , the JVM acts as if it no longer is looking at the default trust store at /etc/ssl/certs/java/cacerts. There was no need to import the public cert from our AD LDAP server into the default truststore.

{code}

export JAVA_OPTS="-Dhudson.plugins.active_directory.ActiveDirectorySecurityRealm.forceLdaps=true \
    -Dorg.apache.commons.jelly.tags.fmt.timeZone=America/New_York \
    -Djavax.net.ssl.trustStore=/etc/ssl/certs/java/cacerts \
    -Djavax.net.ssl.trustStorePassword=changeit"
{code}

Full docker run:
{code}

+ sudo docker run -d --name jenkins-server-test --network=jenkins-network --volume jenkins-test-data:/var/jenkins_home --publish 8555:8443 --env 'JAVA_OPTS=-Dhudson.plugins.active_directory.ActiveDirectorySecurityRealm.forceLdaps=true     -Dorg.apache.commons.jelly.tags.fmt.timeZone=America/New_York     -Djavax.net.ssl.trustStore=/etc/ssl/certs/java/cacerts     -Djavax.net.ssl.trustStorePassword=changeit' geisinger-jenkins-test:2.176.2 --sessionTimeout=15 --httpPort=-15 --httpsPort=8443 --httpsKeyStore=/var/jenkins_home/.keystore/host.domain.com.all.jks --httpsKeyStorePassword=*****************
{code}
Reply all
Reply to author
Forward
0 new messages