Error updating plugins from Jenkins

[Tamás Németh]

Hello,

When I try updating plugins, the very first plugin gets downloaded successfully, but the subsequent ones fail to download.

According to Details the SSL handshake fails due to a certificate error:

[...]

Caused: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490) at java.base/sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1963) at java.base/sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1958) at java.base/java.security.AccessController.doPrivileged(Native Method) at java.base/sun.net.www.protocol.http.HttpURLConnection.getChainedException(HttpURLConnection.java:1957) at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1525) at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1509) at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:245) at hudson.model.UpdateCenter$UpdateCenterConfiguration.download(UpdateCenter.java:1277) Caused: java.io.IOException: Failed to load https://updates.jenkins.io/download/plugins/antisamy-markup-formatter/2.5/antisamy-markup-formatter.hpi to /home/jenkins/Jenkins/DATA/plugins/antisamy-markup-formatter.jpi.tmp at hudson.model.UpdateCenter$UpdateCenterConfiguration.download(UpdateCenter.java:1284) Caused: java.io.IOException: Failed to download from https://updates.jenkins.io/download/plugins/antisamy-markup-formatter/2.5/antisamy-markup-formatter.hpi (redirected to: https://get.jenkins.io/plugins/antisamy-markup-formatter/2.5/antisamy-markup-formatter.hpi)

[...]

I don't really see why do I get a certificate error for the other downloads if the first one was successful.

I turned on ssl handshake logging at java level (-Djavax.net.debug=ssl:handshake), and the log shows that the download request for the first plugin was sent to

- updates.jenkins.io having a  proper certificate chain and the request get redirected to 

- get.jenkins.io also having a proper certificate chain, and redirecting the request to

- mirror site mirror.gruenehoelle.nl and the download succeeded.

However I was not able to track down the requests for the subsequent plugins, all I could find is that the ssl handshake failed on this certificate:

javax.net.ssl|DEBUG|4C|Update center installer thread [#1]|2021-11-19 17:27:28.618 CET|CertificateMessage.java:1148|Consuming server Certificate handshake message (
"Certificate": {
 "certificate_request_context": "",
 "certificate_list": [   
 {
   "certificate" : {
     "version"            : "v3",
     "serial number"      : "23 20 37 D2 97 B4 6A DB E3 CA 51 43 0D F9 9E F3",
     "signature algorithm": "SHA256withRSA",
     "issuer"             : "CN=Kubernetes Ingress Controller Fake Certificate, O=Acme Co",
     "not before"         : "2021-11-18 16:23:38.000 CET",
     "not  after"         : "2022-11-18 16:23:38.000 CET",
     "subject"            : "CN=Kubernetes Ingress Controller Fake Certificate, O=Acme Co",
     "subject public key" : "RSA",
     "extensions"         : [
       {
         ObjectId: 2.5.29.19 Criticality=true
         BasicConstraints:[
           CA:false
           PathLen: undefined
         ]
       },
       {
         ObjectId: 2.5.29.37 Criticality=false
         ExtendedKeyUsages [
           serverAuth
         ]
       },
       {
         ObjectId: 2.5.29.15 Criticality=true
         KeyUsage [
           DigitalSignature
           Key_Encipherment
         ]
       },
       {
         ObjectId: 2.5.29.17 Criticality=false
         SubjectAlternativeName [
           DNSName: ingress.local
         ]
       }
     ]}
   "extensions": {
     <no extension>
   }
 },
]
}
)
javax.net.ssl|DEBUG|4C|Update center installer thread [#1]|2021-11-19 17:27:28.618 CET|SSLExtensions.java:148|Ignore unavailable extension: status_request
javax.net.ssl|ERROR|4C|Update center installer thread [#1]|2021-11-19 17:27:28.630 CET|TransportContext.java:313|Fatal (CERTIFICATE_UNKNOWN): PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target ("throwable" : { sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
       at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidato .java:385)
       at java.base/sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:290)

       at java.base/sun.security.validator.Validator.validate(Validator.java:264)

       at java.base/sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:321)
       at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:221)
       at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:129)
       at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1
313)
       at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.ja
va:1204)
       at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1151)
       at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392)
       at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:444)
       at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:421)
       at java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:178)
       at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:164)
       at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1152)
       at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1063)
       at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:402)
       at java.base/sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:567)
       at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnect
ion.java:185)
       at java.base/sun.net.www.protocol.http.HttpURLConnection.followRedirect0(HttpURLConnection.java:2768)
       at java.base/sun.net.www.protocol.http.HttpURLConnection.followRedirect(HttpURLConnection.java:2680)
       at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1843)
       at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1509)
       at java.base/sun.net.www.protocol.http.HttpURLConnection.getHeaderField(HttpURLConnection.java:3084)
       at java.base/java.net.URLConnection.getHeaderFieldLong(URLConnection.java:636)
       at java.base/java.net.URLConnection.getContentLengthLong(URLConnection.java:508)
       at java.base/java.net.URLConnection.getContentLength(URLConnection.java:492)
       at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getContentLength(HttpsURLConnectionImpl.java:389)
       at hudson.model.UpdateCenter$UpdateCenterConfiguration.download(UpdateCenter.java:1261)
       at hudson.model.UpdateCenter$DownloadJob._run(UpdateCenter.java:1872)
       at hudson.model.UpdateCenter$InstallationJob._run(UpdateCenter.java:2167)
       at hudson.model.UpdateCenter$DownloadJob.run(UpdateCenter.java:1846)
       at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
       at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
       at hudson.remoting.AtmostOneThreadExecutor$Worker.run(AtmostOneThreadExecutor.java:121)
       at java.base/java.lang.Thread.run(Thread.java:834)
 Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to reques
ted target
       at java.base/sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)
       at java.base/sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)
       at java.base/java.security.cert.CertPathBuilder.build(CertPathBuilder.java:297)
       at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:380)
       ... 35 more}

Does anyone have any idea what this might be?

This issue is really annoying, I can upgrade plugins only one by one, restarting Jenkins after each plugin upgrade.

My Jenkins version is 2.303.3 (LTS).

java.runtime.version: 11.0.1+13-LTS (but I've also tried it on latest openjdk-11 with the same result).

I've also created a standalone java application downloading the same plugins from updates.jenkins.io using simple HttpsURLConnection calls. All plugins were successfully downloaded. There were always a redirect to get.jenkins.io and then to a mirror site, and the certificates were always correct. It could not reproduce the issue with this test application using the exact same jre as used for the Jenkins, indicating the jre  has the correct CA certificates in its keystore.

 

According to google, Kubernetes Ingress is a load balancing/networking tool, and the certificate above is its factory default certificate which intended to be replaced. I'm pretty sure we do not use Kubernetes Ingress on our servers.

Any idea about how to solve this issue would be appreciated.

-- Tamas

[Mark Waite]

I think that the output of https://updates.jenkins.io/latest/antisamy-markup-formatter.hpi?mirrorlist will show that your location may be served by multiple Jenkins mirrors.

You can then check each of the mirrors to identify if one of them is responding with an incorrect SSL certificate.

It could also be that the JDK on your Jenkins controller or the ca-certificates package on your Jenkins controller are too old to recognize the September 2021 updates to Let's Encrypt root certificates.  I suspect that Java 11.0.1+13-LTS on the controller likely indicates that the ca-certificates package is also similarly out of date (assuming you're running Linux).  Update the packages on your controller so that you have the latest security fixes for Java and for the operating system.

Mark Waite

[Tamás Németh]

Mark,

Thank you for your response.

Yes I'm running Linux. (sorry for missing this information)

I suspected the Let's Encrypt certificates earlier, so I've already added ISRG Root X1 (the issuer of Let's Encrypt) to the cacerts file.

Nevertheless I just replaced my cacerts file with the one in the most recent java 8 release, restarted jenkins but did not help. The first plugin was downloaded successfully, the second failed as before.

Actually what I don't understand is how downloading the very first plugin succeeds. It's consistently the first plugin which succeeds and the others fail.

If downloading the other plugins fail due to a missing/expired Let's Encrypt certificate, the first one should also fail. It should not be able to connect to updates.jenkins.io at all.

Also, I have a standalone application which I use to check the HTTP redirects  and certificates, and this never fail to download the plugins (the URLs are taken from Jenkins log), even though I use the very same jre instance on the very same machine.

My application use HttpsUrlConnection to download the files. I don't know whether Jenkins use the same or some framework.

The other thing I don't see is how the "CN=Kubernetes Ingress Controller Fake Certificate, O=Acme Co" comes into the picture. According to the logs (see my original e-mail) the SSL handshake fails on this certificate, but I don't really see which server this certificate comes from.

I've also checked the mirrorlist as you suggested. All mirrors listed in the response look good, that is have a valid certificate chain according to my cacerts file.

-- Tamas

[Mark Waite]

On Friday, November 19, 2021 at 12:35:36 PM UTC-7 you wrote:

Mark,

Thank you for your response.

Yes I'm running Linux. (sorry for missing this information)

I suspected the Let's Encrypt certificates earlier, so I've already added ISRG Root X1 (the issuer of Let's Encrypt) to the cacerts file.

Nevertheless I just replaced my cacerts file with the one in the most recent java 8 release, restarted jenkins but did not help. The first plugin was downloaded successfully, the second failed as before.

Actually what I don't understand is how downloading the very first plugin succeeds. It's consistently the first plugin which succeeds and the others fail.

If downloading the other plugins fail due to a missing/expired Let's Encrypt certificate, the first one should also fail. It should not be able to connect to updates.jenkins.io at all.

If there are multiple Jenkins mirrors that are roughly the same distance to you, then the mirrorbits response from https://get.jenkins.io might redirect the second request to a different HTTP server than the first request.  However, if you confirmed that they are all working, that seems unlikely to be the cause.

 

Also, I have a standalone application which I use to check the HTTP redirects  and certificates, and this never fail to download the plugins (the URLs are taken from Jenkins log), even though I use the very same jre instance on the very same machine.

I assume that curl or wget from that machine will successfully retrieve https://updates.jenkins.io/latest/antisamy-markup-formatter.hpi and https://updates.jenkins.io/latest/ant.hpi and https://updates.jenkins.io/latest/adoptopenjdk.hpi.  If not, then the issue is related to something outside of Java (like the ca-certs file).

 

My application use HttpsUrlConnection to download the files. I don't know whether Jenkins use the same or some framework.

The other thing I don't see is how the "CN=Kubernetes Ingress Controller Fake Certificate, O=Acme Co" comes into the picture. According to the logs (see my original e-mail) the SSL handshake fails on this certificate, but I don't really see which server this certificate comes from.

I thought that certificate could be reported when a request is made to the IP address of an HTTP server hosted in Kubernetes.  The Kubernetes server needs the hostname in order to use the correct SSL certificate (or something like that).

 

I've also checked the mirrorlist as you suggested. All mirrors listed in the response look good, that is have a valid certificate chain according to my cacerts file.

Unfortunately, I'm out of ideas if one of those do not resolve the issue.

Mark Waite 

[Daniel Beck]

> On 19. Nov 2021, at 20:52, Mark Waite <mark.ea...@gmail.com> wrote:
>
> I thought that certificate could be reported when a request is made to the IP address of an HTTP server hosted in Kubernetes. The Kubernetes server needs the hostname in order to use the correct SSL certificate (or something like that).
>

https://en.wikipedia.org/wiki/Server_Name_Indication

The Kubernetes cert in the first message is the one for get.jenkins.io without SNI.

[Tamás Németh]

Mark, Daniel,

According to the logs I can confirm that I got the certificate error when there was no server_name sent in the ClientHello message.

Actually the log shows that the SNI was tried to be set, but failed.

javax.net.ssl|DEBUG|49|Update center installer thread [#1]|2021-11-22 09:15:12.165 CET|Utilities.java:73|the previous server name in SNI (type=host_name (0), value=get.jenkins.io) was replaced with (type=host_name (0), value=get.jenkins.io)

javax.net.ssl|WARNING|49|Update center installer thread [#1]|2021-11-22 09:15:12.166 CET|ServerNameExtension.java:255|Unable to indicate server name

I've found this Java bug https://bugs.openjdk.java.net/browse/JDK-8220564, which seems to be related and fixed since then.

I managed to upgrade my java and it seems it solved the issue.

Thank you both.

-- Tamas