Third party dependencies, APIs, and Jenkins: Proposals
51 views
Skip to first unread message
Matt Sicker
Apr 10, 2019, 10:48:51 AM4/10/19
to jenkin...@googlegroups.com
I've been struggling for a while now on how to best approach
dependency management in Jenkins. One of our key pain points has been
in shared libraries like Jackson, HttpComponents, AsyncHTTPClient, and
various other commonly used libraries. As we've already found, simply
including these common dependencies in Jenkins itself leads to
problems such as never being able to upgrade that library again if it
breaks backward compatibility in the slightest.
In order to address these types of issues, I have four different proposals:
1. Shade in third party dependencies of Jenkins core into Jenkins with
a package rename. This will allow core to use the dependencies, but
plugins will still need to include the dependencies explicitly. This
could potentially be combined with a compatibility shim for plugins
that try to load a class that doesn't exist. This would allow for the
core dependencies to upgrade more easily as well. This is certainly a
hacky way of approaching it, but it's fairly common for low level
libraries to package rename their dependencies to avoid JAR hell.
2. Create facade APIs for common third party libraries such as a JSON
API, an HTTP client, etc. This approach could at least work the most
Jenkins-native way, though it seems like a rather large task.
3. Adopt JakartaEE (formerly known as JavaEE) tech via microprofile
<https://microprofile.io/> or similar. This is fairly similar to #2,
but instead, we delegate the design of these facade APIs to the
experts. I like this idea myself, though I'm not sure how well this
works with Jetty and Tomcat.
4. Utilize OSGi in the plugin system. We can dynamically add OSGi
metadata via bndtools at load time, and the hpi plugin could be
updated to include those metadata for faster startup. Hell, I'm pretty
sure the access modifier annotations could even be implemented via
OSGi bundle wiring. Based on my previous experience with OSGi, I
believe we would only be able to utilize up to and including the
module/bundle layer of OSGi. Higher level APIs from OSGi tend to have
duplicate functionality in Jenkins, so that's not necessary. The
advantages to this approach would be the ability to run multiple
versions of a library in the same JVM if necessary, the potential
ability to install and update plugins without restarting Jenkins (!),
and a proper shared library system we could use instead of needing to
package API plugins.
4.alt. I don't know enough about JPMS, but perhaps it can be used to
help enforce something similar? It would only be available in Jenkins
running on Java 9 or higher, though.
--
Matt Sicker
Senior Software Engineer, Jenkins Security, CloudBees
dependency management in Jenkins. One of our key pain points has been
in shared libraries like Jackson, HttpComponents, AsyncHTTPClient, and
various other commonly used libraries. As we've already found, simply
including these common dependencies in Jenkins itself leads to
problems such as never being able to upgrade that library again if it
breaks backward compatibility in the slightest.
In order to address these types of issues, I have four different proposals:
1. Shade in third party dependencies of Jenkins core into Jenkins with
a package rename. This will allow core to use the dependencies, but
plugins will still need to include the dependencies explicitly. This
could potentially be combined with a compatibility shim for plugins
that try to load a class that doesn't exist. This would allow for the
core dependencies to upgrade more easily as well. This is certainly a
hacky way of approaching it, but it's fairly common for low level
libraries to package rename their dependencies to avoid JAR hell.
2. Create facade APIs for common third party libraries such as a JSON
API, an HTTP client, etc. This approach could at least work the most
Jenkins-native way, though it seems like a rather large task.
3. Adopt JakartaEE (formerly known as JavaEE) tech via microprofile
<https://microprofile.io/> or similar. This is fairly similar to #2,
but instead, we delegate the design of these facade APIs to the
experts. I like this idea myself, though I'm not sure how well this
works with Jetty and Tomcat.
4. Utilize OSGi in the plugin system. We can dynamically add OSGi
metadata via bndtools at load time, and the hpi plugin could be
updated to include those metadata for faster startup. Hell, I'm pretty
sure the access modifier annotations could even be implemented via
OSGi bundle wiring. Based on my previous experience with OSGi, I
believe we would only be able to utilize up to and including the
module/bundle layer of OSGi. Higher level APIs from OSGi tend to have
duplicate functionality in Jenkins, so that's not necessary. The
advantages to this approach would be the ability to run multiple
versions of a library in the same JVM if necessary, the potential
ability to install and update plugins without restarting Jenkins (!),
and a proper shared library system we could use instead of needing to
package API plugins.
4.alt. I don't know enough about JPMS, but perhaps it can be used to
help enforce something similar? It would only be available in Jenkins
running on Java 9 or higher, though.
--
Matt Sicker
Senior Software Engineer, Jenkins Security, CloudBees
Jesse Glick
Apr 10, 2019, 4:55:56 PM4/10/19
to Jenkins Dev
On Wed, Apr 10, 2019 at 10:48 AM Matt Sicker <msi...@cloudbees.com> wrote:
> 1. Shade in third party dependencies of Jenkins core into Jenkins with
> a package rename. This will allow core to use the dependencies, but
> plugins will still need to include the dependencies explicitly. This
> could potentially be combined with a compatibility shim for plugins
> that try to load a class that doesn't exist.
You do not necessarily need to rename packages. It suffices for the
> 1. Shade in third party dependencies of Jenkins core into Jenkins with
> a package rename. This will allow core to use the dependencies, but
> plugins will still need to include the dependencies explicitly. This
> could potentially be combined with a compatibility shim for plugins
> that try to load a class that doesn't exist.
plugin class loader defined in Jenkins core to decline to forward
class loading requests for these libraries to the parent. (The
implementation already exists to some extent, in the
`pluginFirstClassLoader` option, but this is too extreme because it
applies to _all_ class loads whereas we only want to decline certain
names.) Combined with the existing detached plugin system for backward
compatibility, you have a complete runtime solution, and rather easily
at that.
The problems are in the developer tooling. `compiler:compile` and
`JenkinsRule` via `surefire:test` are going to pick up transitive
dependencies of `jenkins-core` in a flat classpath. Thus if a plugin
built against a new core baseline (one after the change) _does_
declare an explicit dependency on the new library wrapper plugin,
Maven will pick one or the other version to build & test against
(depending on POM details of where this dependency appears in the tree
relative to `jenkins-core`); if it does _not_, Maven will silently
build & test against the library as bundled in core, even though at
runtime this would be a `NoClassDefFoundError`. These problems make me
think that shading is more practical.
> 4.alt. I don't know enough about JPMS, but perhaps it can be used to
> help enforce something similar?
Jenkins core, so it would be hard to migrate to. (Requiring Java 11+
would be the least of our worries.)
Daniel Beck
Apr 10, 2019, 5:01:12 PM4/10/19
to jenkin...@googlegroups.com
> On 10. Apr 2019, at 22:55, Jesse Glick <jgl...@cloudbees.com> wrote:
>
> You do not necessarily need to rename packages. It suffices for the
> plugin class loader defined in Jenkins core to decline to forward
> class loading requests for these libraries to the parent.
> The problems are in the developer tooling. `compiler:compile` and
> `JenkinsRule` via `surefire:test` are going to pick up transitive
> dependencies of `jenkins-core` in a flat classpath. Thus if a plugin
> built against a new core baseline (one after the change) _does_
> declare an explicit dependency on the new library wrapper plugin,
> Maven will pick one or the other version to build & test against
> (depending on POM details of where this dependency appears in the tree
> relative to `jenkins-core`); if it does _not_, Maven will silently
> build & test against the library as bundled in core, even though at
> runtime this would be a `NoClassDefFoundError`. These problems make me
> think that shading is more practical.
Stephen Connolly
Apr 10, 2019, 6:44:42 PM4/10/19
to jenkin...@googlegroups.com
If the “classboxed” dependencies have scope=provided then they shouldn’t be transitive and you could leave them unshaded.
The JenkinsRule would just need to setup the classloaders from the war and all would be fine
> 4.alt. I don't know enough about JPMS, but perhaps it can be used to
> help enforce something similar?
Last I checked it was much less flexible than what we already have in
Jenkins core, so it would be hard to migrate to. (Requiring Java 11+
would be the least of our worries.)
--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr285%2Bc7FnDwmes-H5T04kNX%2BhP5z_8JkY2heHYH_VXhzw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Sent from my phone
Ullrich Hafner
Apr 11, 2019, 4:10:08 AM4/11/19
to Jenkins Developers
Am 10.04.2019 um 22:55 schrieb Jesse Glick <jgl...@cloudbees.com>:On Wed, Apr 10, 2019 at 10:48 AM Matt Sicker <msi...@cloudbees.com> wrote:1. Shade in third party dependencies of Jenkins core into Jenkins with
a package rename. This will allow core to use the dependencies, but
plugins will still need to include the dependencies explicitly. This
could potentially be combined with a compatibility shim for plugins
that try to load a class that doesn't exist.
As far as I understand shading it does not prevent developers to use the shaded classes, they are still there, just with another package name. And since every IDE has automatic code completion nobody actually looks at the package names anymore. (We can add an architecture test or check style rule to prevent those imports though.)
The problems are in the developer tooling. `compiler:compile` and
`JenkinsRule` via `surefire:test` are going to pick up transitive
dependencies of `jenkins-core` in a flat classpath.
Maybe we can use an enforcer rule like https://github.com/Scout24/illegal-transitive-dependency-check that checks that no transitive dependencies are used at all? Then developers need to provide explicit dependencies. The only problem I see here is if classes of the dependency are part of the plugin API. Then it is unclear to me which class will be picked by Jenkins class loader. (But as far as I know this problem will occur in OSGi as well)
Thus if a plugin
built against a new core baseline (one after the change) _does_
declare an explicit dependency on the new library wrapper plugin,
Maven will pick one or the other version to build & test against
(depending on POM details of where this dependency appears in the tree
relative to `jenkins-core`); if it does _not_, Maven will silently
build & test against the library as bundled in core, even though at
runtime this would be a `NoClassDefFoundError`. These problems make me
think that shading is more practical.4.alt. I don't know enough about JPMS, but perhaps it can be used to
help enforce something similar?
Last I checked it was much less flexible than what we already have in
Jenkins core, so it would be hard to migrate to. (Requiring Java 11+
would be the least of our worries.)
Jesse Glick
Apr 11, 2019, 2:55:10 PM4/11/19
to Jenkins Dev
On Thu, Apr 11, 2019 at 4:10 AM Ullrich Hafner <ullrich...@gmail.com> wrote:
> [shading does not] prevent developers to use the shaded classes, they are still there, just with another package name
True, but the Jenkins plugin class loader can refuse to serve such
classes, so any such mistake would immediately fail at runtime.
> We can add an architecture test or check style rule to prevent those imports though.
Yes, you can use the Enforcer plugin to check this sort of thing at build time.
> Maybe we can use an enforcer rule […] that checks that no transitive dependencies are used at all? Then developers need to provide explicit dependencies.
We may want to do this anyway. (The Maven plugin for NetBeans modules
enforces such a rule.)
> The only problem I see here is if classes of the dependency are part of the plugin API.
You can use explicit dependencies when an API includes other APIs in
its signatures. It just means that clients need to declare
dependencies on both, or they will not be able to compile.
> [shading does not] prevent developers to use the shaded classes, they are still there, just with another package name
True, but the Jenkins plugin class loader can refuse to serve such
classes, so any such mistake would immediately fail at runtime.
> We can add an architecture test or check style rule to prevent those imports though.
> Maybe we can use an enforcer rule […] that checks that no transitive dependencies are used at all? Then developers need to provide explicit dependencies.
We may want to do this anyway. (The Maven plugin for NetBeans modules
enforces such a rule.)
> The only problem I see here is if classes of the dependency are part of the plugin API.
its signatures. It just means that clients need to declare
dependencies on both, or they will not be able to compile.
Jesse Glick
Apr 11, 2019, 2:57:34 PM4/11/19
to Jenkins Dev
On Wed, Apr 10, 2019 at 6:44 PM Stephen Connolly
<stephen.al...@gmail.com> wrote:
> If the “classboxed” dependencies have scope=provided then they shouldn’t be transitive and you could leave them unshaded.
True. `war/pom.xml` would need to be massaged to bundle them.
<stephen.al...@gmail.com> wrote:
> If the “classboxed” dependencies have scope=provided then they shouldn’t be transitive and you could leave them unshaded.
> The JenkinsRule would just need to setup the classloaders from the war and all would be fine
feasible—I have done so in the past (see link in JIRA)—but migrating
existing functional tests to run in a new harness may not be
straightforward.
Reply all
Reply to author
Forward
0 new messages