IBM has granted me to cite their findings - thanks!
We have traced the modified thread context class loader to a Jenkins class that is explicitly setting a thread context class loader for some of its internal threads, but doing so to a class loader that's inappropriate for thread context operations in Liberty.
The class in question is ClassLoaderSanityThreadFactory, which appears to have been added to Jenkins 2.105 as a fix to the following issue:https://issues.jenkins-ci.org/browse/JENKINS-49206
[…]
The Jenkins thread factory mentioned above does an explicit set of the thread context class loader to its own class loader when it creates a new thread, as follows:
@OverridepublicThreadnewThread(Runnabler) { {{ Thread t = delegate.newThread(r); }} t.setContextClassLoader(ClassLoaderSanityThreadFactory.class.getClassLoader()); return t; } Liberty's thread context management, however, does not intend that application class loaders (the class loaders that would load these classes) be used as a thread context class loader - rather, it has its own ThreadContextClassLoader class that is granted extra visibility to server-level classes intended to be available to thread context class loader operations. Class loads initiated by other class loader types (such as the AppClassLoader) are not able to access packages exported with threadContext visibility, and as in this case, the load fails. The solution would be for Jenkins to either provide a switchable implementation (providing a configuration setting or system property to avoid this thread context class loader swap) or to utilize more robust logic in deciding whether to change the thread context class loader - for example, rather than immediately switching the thread context class loader to the current class' loader, it could first check to see if the existing thread context class loader can successfully load it (if it can, then that loader is adequate).
I traced the underlying issue back to the solution of another issue I had, where IBM JVM engineers found a difference between the CommonThreadPool-Implementations of Java 8 and Java 11. I shortened it a bit as well. I created threads using CompletableFuture.supplyAsync(), but I think the behaviour is the very same.
By specifying the ManagedExecutorService, you are using Libertys EE DefaultContextService, which will propagate the application classloader context
We (WAS L3: EEConcurrency) can confirm that ManagedExecutorService by default propagates the thread context class loader of the submitter thread to the task/action that is submitted to it. With CompletableFuture.supplyAsync(supplier, executor), the submitter is actually the JDK's CompletableFuture.supplyAsync implementation, which is understood to delegate to executor.execute from within the supplyAsync method, meaning that if the application invoked supplyAsync, its thread context class loader is put onto the thread that runs the Supplier action. This seems like a reasonable way to ensure the application's thread context class loader is used. When CompletableFuture.supplyAsync(supplier) is used without specifying any executor parameter, the supplier is documented, see https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/CompletableFuture.html#supplyAsync-java.util.function.Supplier-to run on ForkJoinPool.commonPool() which is provided by the JDK, which is outside the control of EE Concurrency, and outside the control of Liberty as well. Questions as to what sort of context the JDK's common ForkJoinPool uses, including differences in their implementation between JDK levels, must be directed to the Java team. We cannot answer that for them." So this narrowed the problem down further and our java L3 have now been able to recreate the issue and also point out where and why the change in behaviour was made and why SDK 8 behaves differently to sdk 11 - "Ok, so the issue that's being reported is that with Java 8, the task threads in the common ForkJoinPool inherit the context class loader from the thread which calls ForkJoinPool.execute(). However, with Java 11 the context class loader is not inherited.
I then looked at the changelog for java.util.concurrent.ForkJoinWorkerThread in the OpenJDK mercurial repos, which led me to this change: https://bugs.openjdk.java.net/browse/JDK-8172726
So this is an intentional change in Java 11 to avoid memory leaks caused by threads in the common ForkJoinPool retaining references to custom class loaders. However, the change has not been backported to Java 8. The net of this is that both Java 8 and Java 11 are working as intended.
So, it looks like […] we have been able to narrow down the problem to a deliberate change of behaviour in the java versions and this is considered now working as designed for this reason."
Takeaway: Do not create threads unless you can set the ThreadContext using a container-provided way. Kudos to IBM for their findings and in-depth analysis! |