[JIRA] (JENKINS-60979) Installation of Plugin fails on LibertyProfile due using the wrong SocketFactory

0 views
Skip to first unread message

bmarwell@gmail.com (JIRA)

unread,
Feb 5, 2020, 7:42:03 AM2/5/20
to jenkinsc...@googlegroups.com
Ben M created an issue
 
Jenkins / Bug JENKINS-60979
Installation of Plugin fails on LibertyProfile due using the wrong SocketFactory
Issue Type: Bug Bug
Assignee: Natasha Stopa
Components: plugin-installation-manager-tool
Created: 2020-02-05 12:41
Environment: - Jenkins any version (broken for me for a few weeks now)
- Liberty Profile any version
- Java: adopt-...@1.11.0-6 or ad...@1.11.0-6
Priority: Major Major
Reporter: Ben M

When trying to install plugins, Jenkins fails with two errors.

 

First of all, the connectivity test will throw this exception, which is ignored:

Connectivity checkConnectivity check java.lang.RuntimeException: com.ibm.ws.ssl.protocol.LibertySSLSocketFactory at com.ibm.ws.kernel.boot.security.SSLSocketFactoryProxy.<init>(SSLSocketFactoryProxy.java:38) at java.base/java.lang.J9VMInternals.newInstanceImpl(Native Method) at java.base/java.lang.Class.newInstance(Class.java:2084) at java.base/javax.net.ssl.SSLSocketFactory.getDefault(SSLSocketFactory.java:110) at java.base/javax.net.ssl.HttpsURLConnection.getDefaultSSLSocketFactory(HttpsURLConnection.java:335) at java.base/javax.net.ssl.HttpsURLConnection.<init>(HttpsURLConnection.java:292) at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.<init>(HttpsURLConnectionImpl.java:100) at java.base/sun.net.www.protocol.https.Handler.openConnection(Handler.java:62) at java.base/java.net.URL.openConnection(URL.java:1123) at hudson.ProxyConfiguration.open(ProxyConfiguration.java:260) at hudson.FilePath.installIfNecessaryFrom(FilePath.java:857) at hudson.FilePath.installIfNecessaryFrom(FilePath.java:848) at hudson.tools.ZipExtractionInstaller.performInstallation(ZipExtractionInstaller.java:83) at hudson.tools.InstallerTranslator.getToolHome(InstallerTranslator.java:69) at hudson.tools.ToolLocationNodeProperty.getToolHome(ToolLocationNodeProperty.java:109) at hudson.tools.ToolInstallation.translateFor(ToolInstallation.java:206) at hudson.model.JDK.forNode(JDK.java:147) at hudson.model.AbstractProject.getEnvironment(AbstractProject.java:341) at hudson.scm.SubversionSCM.compareRemoteRevisionWith(SubversionSCM.java:1419) at hudson.scm.SCM.compareRemoteRevisionWith(SCM.java:400) at hudson.scm.SCM.poll(SCM.java:417) at hudson.model.AbstractProject._poll(AbstractProject.java:1390) at hudson.model.AbstractProject.poll(AbstractProject.java:1293) at jenkins.triggers.SCMTriggerItem$SCMTriggerItems$Bridge.poll(SCMTriggerItem.java:143) at hudson.triggers.SCMTrigger$Runner.runPolling(SCMTrigger.java:603) at hudson.triggers.SCMTrigger$Runner.run(SCMTrigger.java:649) at hudson.util.SequentialExecutionQueue$QueueEntry.run(SequentialExecutionQueue.java:119) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)Caused: java.net.SocketException at java.base/javax.net.ssl.DefaultSSLSocketFactory.throwException(SSLSocketFactory.java:263) at java.base/javax.net.ssl.DefaultSSLSocketFactory.createSocket(SSLSocketFactory.java:285) at java.base/sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:444) at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185) at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1587) at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1515) at java.base/java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:527) at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:334) at hudson.model.UpdateCenter$UpdateCenterConfiguration.testConnection(UpdateCenter.java:1302) at hudson.model.UpdateCenter$UpdateCenterConfiguration.checkUpdateCenter(UpdateCenter.java:1085) at hudson.model.UpdateCenter$ConnectionCheckJob.run(UpdateCenter.java:1534) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at hudson.remoting.AtmostOneThreadExecutor$Worker.run(AtmostOneThreadExecutor.java:112) at java.base/java.lang.Thread.run(Thread.java:834)

 

After this, plugins should get downloaded. This will also fail.

java.net.SocketTimeoutException: Read timed outjava.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(SocketInputStream.java:115) at java.base/java.net.SocketInputStream.read(SocketInputStream.java:168) at java.base/java.net.SocketInputStream.read(SocketInputStream.java:140) at java.base/java.io.BufferedInputStream.fill(BufferedInputStream.java:252) at java.base/java.io.BufferedInputStream.read1(BufferedInputStream.java:292) at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:351) at java.base/sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:754) at java.base/sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:689) at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1610) at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1515) at java.base/sun.net.www.protocol.http.HttpURLConnection.getHeaderField(HttpURLConnection.java:3094) at java.base/java.net.URLConnection.getHeaderFieldLong(URLConnection.java:636) at java.base/java.net.URLConnection.getContentLengthLong(URLConnection.java:508) at java.base/java.net.URLConnection.getContentLength(URLConnection.java:492) at hudson.model.UpdateCenter$UpdateCenterConfiguration.download(UpdateCenter.java:1161)Caused: java.net.SocketTimeoutException: Read timed out at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490) at java.base/sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1969) at java.base/sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1964) at java.base/java.security.AccessController.doPrivileged(AccessController.java:734) at java.base/sun.net.www.protocol.http.HttpURLConnection.getChainedException(HttpURLConnection.java:1963) at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1531) at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1515) at hudson.model.UpdateCenter$UpdateCenterConfiguration.download(UpdateCenter.java:1177)Caused: java.io.IOException: Failed to load http://updates.jenkins-ci.org/download/plugins/warnings/5.0.1/warnings.hpi to $JENKINS_HOME/plugins/warnings.jpi.tmp at hudson.model.UpdateCenter$UpdateCenterConfiguration.download(UpdateCenter.java:1184)Caused: java.io.IOException: Failed to download from http://updates.jenkins-ci.org/download/plugins/warnings/5.0.1/warnings.hpi (redirected to: http://mirror.serverion.com/jenkins/plugins/warnings/5.0.1/warnings.hpi) at hudson.model.UpdateCenter$UpdateCenterConfiguration.download(UpdateCenter.java:1218) at hudson.model.UpdateCenter$DownloadJob._run(UpdateCenter.java:1766) at hudson.model.UpdateCenter$InstallationJob._run(UpdateCenter.java:2037) at hudson.model.UpdateCenter$DownloadJob.run(UpdateCenter.java:1740) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at hudson.remoting.AtmostOneThreadExecutor$Worker.run(AtmostOneThreadExecutor.java:112) at java.base/java.lang.Thread.run(Thread.java:834)

 

 

Using curl on the same system with the same settings (URL, proxy, etc.) without liberty and java, the download works just fine.

 

The [documentation for Liberty Profile|https://wiki.jenkins.io/display/JENKINS/Liberty+profile] does not mention such a problem. Please try your test setup with a proxy.

Add Comment Add Comment
 
This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)
Atlassian logo

bmarwell@gmail.com (JIRA)

unread,
Feb 5, 2020, 8:05:06 AM2/5/20
to jenkinsc...@googlegroups.com
Ben M commented on Bug JENKINS-60979
 
Re: Installation of Plugin fails on LibertyProfile due using the wrong SocketFactory

This specific stack trace is from Jenkins v2.190.2.

The offending line for the actual download is: InputStream in = con.getInputStream();.

The offending lines for the connection tests are (line "int responseCode = … "):

        private void testConnection(URL url) throws IOException {
            try {
                URLConnection connection = ProxyConfiguration.open(url);                if(connection instanceof HttpURLConnection) {
                    int responseCode = ((HttpURLConnection)connection).getResponseCode();
                    if(HttpURLConnection.HTTP_OK != responseCode) {
                        throw new HttpRetryException("Invalid response code (" + responseCode + ") from URL: " + url, responseCode);
                    }
                } else {
                    try (InputStream is = connection.getInputStream()) {
                        IOUtils.copy(is, new NullOutputStream());
                    }
                }
            } catch (SSLHandshakeException e) {
                if (e.getMessage().contains("PKIX path building failed"))
                   // fix up this crappy error message from JDK
                    throw new IOException("Failed to validate the SSL certificate of "+url,e);
            }
        }

bmarwell@gmail.com (JIRA)

unread,
Feb 5, 2020, 9:53:04 AM2/5/20
to jenkinsc...@googlegroups.com
Ben M commented on Bug JENKINS-60979

This may be related to JDK11 adopt-openj9 and running on liberty profile.

bmarwell@gmail.com (JIRA)

unread,
Feb 5, 2020, 10:02:02 AM2/5/20
to jenkinsc...@googlegroups.com
Ben M edited a comment on Bug JENKINS-60979
This may be related to JDK11 adopt-openj9 and running on liberty profile when enabling the feature "transportsecurity-1 . 0".

 

If I turn this feature off, the error from the connectivity check disappears.

 

The second error seems to disappear when setting very high connectivity values.
{code:java}
-Dhudson.model.UpdateCenter.pluginDownloadReadTimeoutSeconds=240
-Dhudson.ProxyConfiguration.DEFAULT_CONNECT_TIMEOUT_MILLIS=1200000
{code}

kwhetstone@cloudbees.com (JIRA)

unread,
Feb 12, 2020, 10:33:02 AM2/12/20
to jenkinsc...@googlegroups.com

Were you trying to download the plugin using the plugin installation manger tool or is this happening when you're launching Jenkins? If the later, we should change the component to be against core since the `plugin-installation-manager-tool` is a separate thing and under core this would get more attention.

bmarwell@gmail.com (JIRA)

unread,
Feb 12, 2020, 11:11:02 AM2/12/20
to jenkinsc...@googlegroups.com
Ben M commented on Bug JENKINS-60979

Kristin Whetstone it happens when using the plugin installation manager.

This does not happen when either not using the liberty feature "transportsecurity" or when not using java >= 9.

I think this issue can be closed here, I will open one at openliberty on github instead.

bmarwell@gmail.com (JIRA)

unread,
Mar 24, 2020, 3:30:03 AM3/24/20
to jenkinsc...@googlegroups.com
Ben M commented on Bug JENKINS-60979

Hello Kristin Whetstone

IBM found out that the underlying issue is https://issues.jenkins-ci.org/browse/JENKINS-49206. A new Thread is created using a different class loader, overwriting the container's class loader.

I asked IBM if I can cite their findings to let you know more about the technical detail. I consider this confirmed now, it is broken by design and never should have worked on Java 8 either.

This message was sent by Atlassian Jira (v7.13.12#713012-sha1:6e07c38)
Atlassian logo

bmarwell@gmail.com (JIRA)

unread,
Mar 25, 2020, 3:20:03 AM3/25/20
to jenkinsc...@googlegroups.com
Ben M commented on Bug JENKINS-60979

IBM has granted me to cite their findings - thanks!

 

We have traced the modified thread context class loader to a Jenkins class that is explicitly setting a thread context class loader for some of its internal threads, but doing so to a class loader that's inappropriate for thread context operations in Liberty.

The class in question is ClassLoaderSanityThreadFactory, which appears to have been added to Jenkins 2.105 as a fix to the following issue:https://issues.jenkins-ci.org/browse/JENKINS-49206

 […]

The Jenkins thread factory mentioned above does an explicit set of the thread context class loader to its own class loader when it creates a new thread, as follows:

 
@OverridepublicThreadnewThread(Runnabler) {
{{    Thread t = delegate.newThread(r); }}
    t.setContextClassLoader(ClassLoaderSanityThreadFactory.class.getClassLoader());
    return t;
}
Liberty's thread context management, however, does not intend that application class loaders (the class loaders that would load these classes) be used as a thread context class loader - rather, it has its own ThreadContextClassLoader class that is granted extra visibility to server-level classes intended to be available to thread context class loader operations. Class loads initiated by other class loader types (such as the AppClassLoader) are not able to access packages exported with threadContext visibility, and as in this case, the load fails.
  
 The solution would be for Jenkins to either provide a switchable implementation (providing a configuration setting or system property to avoid this thread context class loader swap) or to utilize more robust logic in deciding whether to change the thread context class loader - for example, rather than immediately switching the thread context class loader to the current class' loader, it could first check to see if the existing thread context class loader can successfully load it (if it can, then that loader is adequate).

 

I traced the underlying issue back to the solution of another issue I had, where IBM JVM engineers found a difference between the CommonThreadPool-Implementations of Java 8 and Java 11. I shortened it a bit as well. I created threads using CompletableFuture.supplyAsync(), but I think the behaviour is the very same.

By specifying the ManagedExecutorService, you are using Libertys EE DefaultContextService, which will propagate the application classloader context

We (WAS L3: EEConcurrency) can confirm that ManagedExecutorService by default propagates the thread context class loader of the submitter thread to the task/action that is submitted to it. With CompletableFuture.supplyAsync(supplier, executor), the submitter is actually the JDK's CompletableFuture.supplyAsync implementation, which is understood to delegate to executor.execute from within the supplyAsync method, meaning that if the application invoked supplyAsync, its thread context class loader is put onto the thread that runs the Supplier action. This seems like a reasonable way to ensure the application's thread context class loader is used. When CompletableFuture.supplyAsync(supplier) is used without specifying any executor parameter, the supplier is documented, see https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/CompletableFuture.html#supplyAsync-java.util.function.Supplier-to run on ForkJoinPool.commonPool() which is provided by the JDK, which is outside the control of EE Concurrency, and outside the control of Liberty as well.  Questions as to what sort of context the JDK's common ForkJoinPool uses, including differences in their implementation between JDK levels, must be directed to the Java team. We cannot answer that for them."
  
 So this narrowed the problem down further and our java L3 have now been able to recreate the issue and also point out where and why the
 change in behaviour was made and why SDK 8 behaves differently to sdk 11 -
  
 "Ok, so the issue that's being reported is that with Java 8, the task threads in the common ForkJoinPool inherit the context class loader from the thread which calls ForkJoinPool.execute(). However, with Java 11 the context class loader is not inherited.

I then looked at the changelog for java.util.concurrent.ForkJoinWorkerThread in the OpenJDK mercurial repos, which led me to this change:
  
     https://bugs.openjdk.java.net/browse/JDK-8172726

 So this is an intentional change in Java 11 to avoid memory leaks caused by threads in the common ForkJoinPool retaining references to custom class loaders. However, the change has not been backported to Java 8. The net of this is that both Java 8 and Java 11 are working as intended.

So, it looks like […] we have been able to narrow down the problem to a deliberate change of behaviour  in the java versions and this is considered now working as designed for this reason."

Takeaway: Do not create threads unless you can set the ThreadContext using a container-provided way.

Kudos to IBM for their findings and in-depth analysis!

bmarwell@gmail.com (JIRA)

unread,
Mar 25, 2020, 3:21:02 AM3/25/20
to jenkinsc...@googlegroups.com
Ben M edited a comment on Bug JENKINS-60979
IBM has granted me to cite their findings - thanks!

 
{quote}We have traced the modified thread context class loader to a Jenkins class that is explicitly setting a thread context class loader for some of its internal threads, but doing so to a class loader that's inappropriate for thread context operations in Liberty.


The class in question is ClassLoaderSanityThreadFactory, which appears to have been added to Jenkins 2.105 as a fix to the following issue:https://issues.jenkins-ci.org/browse/JENKINS-49206

 […]

The Jenkins thread factory mentioned above does an explicit set of the thread context class loader to its own class loader when it creates a new thread, as follows:

 
{ { code:java}
@OverridepublicThreadnewThread(Runnabler) { }}
{{    Thread t = delegate.newThread(r); }}
{{     t.setContextClassLoader(ClassLoaderSanityThreadFactory.class.getClassLoader()); }}
{{    return t; }}
} { { code } }}

Liberty's thread context management, however, does not intend that application class loaders (the class loaders that would load these classes) be used as a thread context class loader - rather, it has its own ThreadContextClassLoader class that is granted extra visibility to server-level classes intended to be available to thread context class loader operations. Class loads initiated by other class loader types (such as the AppClassLoader) are not able to access packages exported with threadContext visibility, and as in this case, the load fails.
  
 The solution would be for Jenkins to either provide a switchable implementation (providing a configuration setting or system property to avoid this thread context class loader swap) or to utilize more robust logic in deciding whether to change the thread context class loader - for example, rather than immediately switching the thread context class loader to the current class' loader, it could first check to see if the existing thread context class loader can successfully load it (if it can, then that loader is adequate).
{quote}
 

I traced the underlying issue back to the solution of another issue I had, where IBM JVM engineers found a difference between the CommonThreadPool-Implementations of Java 8 and Java 11. I shortened it a bit as well. I created threads using {{CompletableFuture.supplyAsync()}}, but I think the behaviour is the very same.
{quote}By specifying the ManagedExecutorService, you are using Libertys EE DefaultContextService, which will propagate the application classloader context

We (WAS L3: EEConcurrency) can confirm that ManagedExecutorService by default propagates the thread context class loader of the submitter thread to the task/action that is submitted to it. With CompletableFuture.supplyAsync(supplier, executor), the submitter is actually the JDK's CompletableFuture.supplyAsync implementation, which is understood to delegate to executor.execute from within the supplyAsync method, meaning that if the application invoked supplyAsync, its thread context class loader is put onto the thread that runs the Supplier action. This seems like a reasonable way to ensure the application's thread context class loader is used. When CompletableFuture.supplyAsync(supplier) is used without specifying any executor parameter, the supplier is documented, see [https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/CompletableFuture.html#supplyAsync-java.util.function.Supplier-]to run on ForkJoinPool.commonPool() which is provided by the JDK, which is outside the control of EE Concurrency, and outside the control of Liberty as well.  Questions as to what sort of context the JDK's common ForkJoinPool uses, including differences in their implementation between JDK levels, must be directed to the Java team. We cannot answer that for them."

  
 So this narrowed the problem down further and our java L3 have now been able to recreate the issue and also point out where and why the
 change in behaviour was made and why SDK 8 behaves differently to sdk 11 -
  
 "Ok, so the issue that's being reported is that with Java 8, the task threads in the common ForkJoinPool inherit the context class loader from the thread which calls ForkJoinPool.execute(). However, with Java 11 the context class loader is not inherited.

I then looked at the changelog for java.util.concurrent.ForkJoinWorkerThread in the OpenJDK mercurial repos, which led me to this change:
  
     [https://bugs.openjdk.java.net/browse/JDK-8172726]

 So this is an intentional change in Java 11 to avoid memory leaks caused by threads in the common ForkJoinPool retaining references to custom class loaders. However, the change has not been backported to Java 8. The net of this is that both Java 8 and Java 11 are working as intended.

So, it looks like […] we have been able to narrow down the problem to a deliberate change of behaviour  in the java versions and this is considered now working as designed for this reason."
{quote}

Takeaway: Do not create threads unless you can set the ThreadContext using a container-provided way.

Kudos to IBM for their findings and in-depth analysis!

bmarwell@gmail.com (JIRA)

unread,
Mar 25, 2020, 3:22:02 AM3/25/20
to jenkinsc...@googlegroups.com
Ben M edited a comment on Bug JENKINS-60979
IBM has granted me to cite their findings - thanks!

 
{quote}We have traced the modified thread context class loader to a Jenkins class that is explicitly setting a thread context class loader for some of its internal threads, but doing so to a class loader that's inappropriate for thread context operations in Liberty.

The class in question is ClassLoaderSanityThreadFactory, which appears to have been added to Jenkins 2.105 as a fix to the following issue:https://issues.jenkins-ci.org/browse/JENKINS-49206

 […]

The Jenkins thread factory mentioned above does an explicit set of the thread context class loader to its own class loader when it creates a new thread, as follows:
{code:java}
@ OverridepublicThreadnewThread Override
public Thread newThread
(Runnabler) {

  Thread t = delegate.newThread(r); }}
 t.setContextClassLoader(ClassLoaderSanityThreadFactory.class.getClassLoader());
  return t;
}{code}

Liberty's thread context management, however, does not intend that application class loaders (the class loaders that would load these classes) be used as a thread context class loader - rather, it has its own ThreadContextClassLoader class that is granted extra visibility to server-level classes intended to be available to thread context class loader operations. Class loads initiated by other class loader types (such as the AppClassLoader) are not able to access packages exported with threadContext visibility, and as in this case, the load fails.
  
 The solution would be for Jenkins to either provide a switchable implementation (providing a configuration setting or system property to avoid this thread context class loader swap) or to utilize more robust logic in deciding whether to change the thread context class loader - for example, rather than immediately switching the thread context class loader to the current class' loader, it could first check to see if the existing thread context class loader can successfully load it (if it can, then that loader is adequate).
{quote}
 

I traced the underlying issue back to the solution of another issue I had, where IBM JVM engineers found a difference between the CommonThreadPool-Implementations of Java 8 and Java 11. I shortened it a bit as well. I created threads using {{CompletableFuture.supplyAsync()}}, but I think the behaviour is the very same.
{quote}By specifying the ManagedExecutorService, you are using Libertys EE DefaultContextService, which will propagate the application classloader context

We (WAS L3: EEConcurrency) can confirm that ManagedExecutorService by default propagates the thread context class loader of the submitter thread to the task/action that is submitted to it. With CompletableFuture.supplyAsync(supplier, executor), the submitter is actually the JDK's CompletableFuture.supplyAsync implementation, which is understood to delegate to executor.execute from within the supplyAsync method, meaning that if the application invoked supplyAsync, its thread context class loader is put onto the thread that runs the Supplier action. This seems like a reasonable way to ensure the application's thread context class loader is used. When CompletableFuture.supplyAsync(supplier) is used without specifying any executor parameter, the supplier is documented, see [https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/CompletableFuture.html#supplyAsync-java.util.function.Supplier-]to run on ForkJoinPool.commonPool() which is provided by the JDK, which is outside the control of EE Concurrency, and outside the control of Liberty as well.  Questions as to what sort of context the JDK's common ForkJoinPool uses, including differences in their implementation between JDK levels, must be directed to the Java team. We cannot answer that for them."
  
 So this narrowed the problem down further and our java L3 have now been able to recreate the issue and also point out where and why the
 change in behaviour was made and why SDK 8 behaves differently to sdk 11 -
  
 "Ok, so the issue that's being reported is that with Java 8, the task threads in the common ForkJoinPool inherit the context class loader from the thread which calls ForkJoinPool.execute(). However, with Java 11 the context class loader is not inherited.

I then looked at the changelog for java.util.concurrent.ForkJoinWorkerThread in the OpenJDK mercurial repos, which led me to this change:
  
     [https://bugs.openjdk.java.net/browse/JDK-8172726]

 So this is an intentional change in Java 11 to avoid memory leaks caused by threads in the common ForkJoinPool retaining references to custom class loaders. However, the change has not been backported to Java 8. The net of this is that both Java 8 and Java 11 are working as intended.

So, it looks like […] we have been able to narrow down the problem to a deliberate change of behaviour  in the java versions and this is considered now working as designed for this reason."
{quote}
Takeaway: Do not create threads unless you can set the ThreadContext using a container-provided way.

Kudos to IBM for their findings and in-depth analysis!

bmarwell@gmail.com (JIRA)

unread,
Mar 26, 2020, 5:36:03 AM3/26/20
to jenkinsc...@googlegroups.com
Ben M updated an issue
 
Change By: Ben M
Comment: [~kwhetstone] it happens when using the plugin installation manager.


This does not happen when either not using the liberty feature "transportsecurity" or when not using java >= 9.

I think this issue can be closed here, I will open one at openliberty on github instead.

bmarwell@gmail.com (JIRA)

unread,
Mar 27, 2020, 9:21:02 AM3/27/20
to jenkinsc...@googlegroups.com
Ben M commented on Bug JENKINS-60979
 
Re: Installation of Plugin fails on LibertyProfile due using the wrong SocketFactory

Natasha Stopa I think there is now everything you need. Can we start working on this issue at some point? Other users will also  be affected.

kwhetstone@cloudbees.com (JIRA)

unread,
Apr 7, 2020, 12:32:04 PM4/7/20
to jenkinsc...@googlegroups.com

kwhetstone@cloudbees.com (JIRA)

unread,
Apr 7, 2020, 12:36:03 PM4/7/20
to jenkinsc...@googlegroups.com
Kristin Whetstone commented on Bug JENKINS-60979
 
Re: Installation of Plugin fails on LibertyProfile due using the wrong SocketFactory

Unassigning from Natasha since she was working on the plugin-installation-manager-tool and this appears to be the actual PluginManager in Jenkins proper. I've also updated the component to reflect the change. Ben, if you have a solution, feel free to make a PR!

kwhetstone@cloudbees.com (JIRA)

unread,
Apr 7, 2020, 12:36:03 PM4/7/20
to jenkinsc...@googlegroups.com
Kristin Whetstone updated an issue
 
Change By: Kristin Whetstone
Component/s: core
Component/s: plugin-installation-manager-tool
Reply all
Reply to author
Forward
0 new messages