Feedback on some analysis I'm doing

[rsomas...@netflix.com]

Hi!

The rush to patch log4j2 in plugins got me thinking if we can do something to reduce the risks of vulnerabilities from dependencies in plugins.

The log4j2 dependency in particular seems like one that shouldn't have been there at all.

Back in the days of J2EE Application servers, the recommendation was - use a provided dependency for the APIs of things like logging, LDAP, persistence, etc. when building an application. The server will provide the implementation of those APIs.

When developing a library or framework, the recommendation historically was to use an API for logging - like commons-logging, slf4j-api. This would let the consumer of the library decide on the implementation. Typically, this meant the user would configure a CLI or a webserver like tomcat with a logging implementation - log4j (v1), log4j-core (v2), or logback.

With apps that run as jars - like jenkins, that distinction is a little harder to make. I personally, run jenkins within tomcat, and am familiar with configuring logging at the tomcat level.

I wrote this tool - https://github.com/rahulsom/analyze-jenkins-plugins. Right now it can generate a CSV listing plugins and the jars that they bundle. It does as much as possible to clearly identify the jars going in. When the jar contains one (or more) pom.properties it lists those out. For jars that do not contain that, it uses the SHA1 to identify the jar. That means it can sufficiently identify jars when they get shadowed or renamed. There are still cases where it fails, but those failures are few enough. This is an example of the CSV it generates - https://gist.github.com/rahulsom/4f992c5c16afc91ee4113ec5da61c21b

This is all libraries bundling log4j-core. It's the same libraries bundling log4j-api. One of them is shadow-jar-ing.

❯ csvsql "SELECT pluginName, pluginVersion, jarName, jarGroupId, jarArtifactId, jarVersion, jarGAVSource FROM plugin-jars.csv WHERE jarArtifactId = 'log4j-core'" pluginName pluginVersion jarName jarGroupId jarArtifactId jarVersion jarGAVSource audit-log 1.2 WEB-INF/lib/log4j-core-2.13.3.jar org.apache.logging.log4j log4j-core 2.13.3 POM bootstraped-multi-test-results-report 2.1.3 WEB-INF/lib/log4j-core-2.6.2.jar org.apache.logging.log4j log4j-core 2.6.2 POM checkmarx 2021.4.2 WEB-INF/lib/log4j-core-2.13.3.jar org.apache.logging.log4j log4j-core 2.13.3 POM hp-application-automation-tools-plugin 7.1 WEB-INF/lib/log4j-core-2.13.3.jar org.apache.logging.log4j log4j-core 2.13.3 POM lambdatest-automation 1.19.4 WEB-INF/lib/log4j-core-2.11.2.jar org.apache.logging.log4j log4j-core 2.11.2 POM peass-ci 2.0.0-540.v244012ecda48 WEB-INF/lib/log4j-core-2.15.0.jar org.apache.logging.log4j log4j-core 2.15.0 POM pipeline-huaweicloud-plugin 0.0.1 WEB-INF/lib/log4j-core-2.8.2.jar org.apache.logging.log4j log4j-core 2.8.2 POM reliza-integration 0.1.13 WEB-INF/lib/log4j-core-2.14.0.jar org.apache.logging.log4j log4j-core 2.14.0 POM talend 1.3-rc42.f3ec422d618b WEB-INF/lib/log4j-core-2.15.0.jar org.apache.logging.log4j log4j-core 2.15.0 POM testdroid-run-in-cloud 2.116.0 WEB-INF/lib/log4j-core-2.13.3.jar org.apache.logging.log4j log4j-core 2.13.3 POM thundra-foresight 11.vbc9483778bb3 WEB-INF/lib/thundra-agent-maven-test-instrumentation-0.0.6.jar org.apache.logging.log4j log4j-core 2.14.1 POM thundra-foresight 11.vbc9483778bb3 WEB-INF/lib/log4j-core-2.14.1.jar org.apache.logging.log4j log4j-core 2.14.1 POM venafi-vcert 2.0.0 WEB-INF/lib/log4j-core-2.13.3.jar org.apache.logging.log4j log4j-core 2.13.3 POM xray-connector 2.5.1 WEB-INF/lib/log4j-core-2.13.3.jar org.apache.logging.log4j log4j-core 2.13.3 POM

This is the first 20 libraries bundling slf4j-api.

❯ csvsql "SELECT pluginName, pluginVersion, jarName, jarGroupId, jarArtifactId, jarVersion, jarGAVSource FROM plugin-jars.csv WHERE jarArtifactId = 'slf4j-api' LIMIT 20" pluginName pluginVersion jarName jarGroupId jarArtifactId jarVersion jarGAVSource GatekeeperPlugin 3.0.5 WEB-INF/lib/slf4j-api-1.6.4.jar org.slf4j slf4j-api 1.6.4 POM NegotiateSSO 1.4 WEB-INF/lib/slf4j-api-2.0.0-alpha0.jar org.slf4j slf4j-api 2.0.0-alpha0 POM acunetix-360-scan 2.0.1 WEB-INF/lib/slf4j-api-1.7.30.jar org.slf4j slf4j-api 1.7.30 POM adobe-cloud-manager 43.v4560e7fa191c WEB-INF/lib/slf4j-api-1.7.30.jar org.slf4j slf4j-api 1.7.30 POM analysis-model-api 10.8.1 WEB-INF/lib/slf4j-api-1.7.30.jar org.slf4j slf4j-api 1.7.30 POM appcenter 0.11.1 WEB-INF/lib/slf4j-api-1.7.30.jar org.slf4j slf4j-api 1.7.30 POM aqua-security-scanner 3.1.2 WEB-INF/lib/slf4j-api-1.7.26.jar org.slf4j slf4j-api 1.7.26 POM artifactory 3.14.2 WEB-INF/lib/slf4j-api-1.7.25.jar org.slf4j slf4j-api 1.7.25 POM atlassian-jira-software-cloud 1.4.4 WEB-INF/lib/slf4j-api-1.7.30.jar org.slf4j slf4j-api 1.7.30 POM autocomplete-parameter 1.1 WEB-INF/lib/slf4j-api-1.7.31.jar org.slf4j slf4j-api 1.7.31 POM autograding 3.3.1 WEB-INF/lib/slf4j-api-1.7.30.jar org.slf4j slf4j-api 1.7.30 POM aws-kinesis-consumer 1.0.5 WEB-INF/lib/slf4j-api-1.7.32.jar org.slf4j slf4j-api 1.7.32 POM aws-lambda 0.5.10 WEB-INF/lib/slf4j-api-1.7.10.jar org.slf4j slf4j-api 1.7.10 POM aws-yum-parameter 1.5 WEB-INF/lib/slf4j-api-1.7.7.jar org.slf4j slf4j-api 1.7.7 POM azure-credentials-ext 1.0 WEB-INF/lib/slf4j-api-1.7.26.jar org.slf4j slf4j-api 1.7.26 POM bitbucket-kubernetes-credentials 0.0.4 WEB-INF/lib/slf4j-api-1.7.26.jar org.slf4j slf4j-api 1.7.26 POM blackduck-detect 7.0.0 WEB-INF/lib/slf4j-api-1.7.30.jar org.slf4j slf4j-api 1.7.30 POM build-failure-analyzer 2.1.0 WEB-INF/lib/slf4j-api-1.7.30.jar org.slf4j slf4j-api 1.7.30 POM build-time-blame 64.vd8f4018a2bbe WEB-INF/lib/slf4j-api-1.7.26.jar org.slf4j slf4j-api 1.7.26 POM ca-mat-performance-benchmarking-by-broadcom 1.5 WEB-INF/lib/slf4j-api-1.7.30.jar org.slf4j slf4j-api 1.7.30 POM

Looking at jenkins-core's POM - https://mvnrepository.com/artifact/org.jenkins-ci.main/jenkins-core/2.324 it looks like it has a top level dependency on these two libraries

org.slf4j:jcl-over-slf4j:1.7.32 org.slf4j:log4j-over-slf4j:1.7.32

They in turn have a dependency on slf4j-api

org.slf4j:slf4j-api:1.7.32

That suggests that Jenkins Core would prefer plugins to use slf4j-api. If that's the case, all plugins that depend on jenkins-core should use slf4j-api and remove the dependency on any other logging library.

Is my analysis of the plugins reasonable?

If it is, we have a few points to influence plugins:

1. The parent-pom used by plugins uses enforcer to block some libraries being in the dependency tree. We could add most logging libraries to that list.
2. The maven-hpi-plugin could do some linting of the plugins to ensure they only use slf4j-api and even then, only get it as a transitive from the provided dependency on jenkins-core.
3. I could send PRs to the plugins that bundle logging libraries.

I think there is more analysis we can do for other problems - there are a lot of plugins bundling okhttp jars instead of depending on the okhttp-api plugin. Plugins could lose a lot of weight by cleaning up their builds to bundle only libraries that are unique to their needs. I suspect that would also improve jenkins startup time for users who depend on a lot of these plugins.

Thanks!

Rahul

[Basil Crow]

Might be interesting to look into adding something like OWASP Dependency-Check to the parent POM and plugin parent POM, with suppressions for existing false positives. We could start by adding warnings to the build and then later upgrade those warnings to errors once we feel confident that most false positives have been suppressed.

[rsomas...@netflix.com]

While sometimes (like log4j-core) it is about security and owasp can help.

Other times, it's mostly about reducing redundant libraries - like slf4j-api or log4j-api.

Other common libraries are

❯ csvsql "SELECT jarGroupId, jarArtifactId, jarVersion, count(*) as CT
FROM plugin-jars.csv
GROUP BY jarGroupId, jarArtifactId, jarVersion
ORDER BY CT DESC
LIMIT 20
"
jarGroupId                    jarArtifactId         jarVersion  CT
344
commons-codec                 commons-codec         1.9         75
com.google.code.findbugs      jsr305                3.0.2       62
org.apache.httpcomponents     httpcore              4.4.13      58
com.github.stephenc.findbugs  findbugs-annotations  1.3.9-1     56
org.slf4j                     slf4j-api             1.7.30      48
com.google.code.gson          gson                  2.8.5       45
org.apache.httpcomponents     httpclient            4.5.13      44
org.apache.httpcomponents     httpclient            4.5.2       42
com.google.code.findbugs      jsr305                1.3.9       41
org.apache.httpcomponents     httpcore              4.4.4       41
commons-codec                 commons-codec         1.10        40
commons-codec                 commons-codec         1.11        38
com.google.code.gson          gson                  2.8.6       37
commons-io                    commons-io            2.4         35
commons-lang                  commons-lang          2.6         35
org.apache.commons            commons-lang3         3.7         34
commons-httpclient            commons-httpclient    3.1         33
org.apache.commons            commons-lang3         3.4         33
com.fasterxml.jackson.core    jackson-annotations   2.9.0       31

Some of these are already part of jenkins-core. E.g. commons-codec, commons-io, commons-httpclient, etc.

[Basil Crow]

On Mon, Dec 13, 2021 at 11:22 AM 'rsomas...@netflix.com' via Jenkins
Developers <jenkin...@googlegroups.com> wrote:
>
> While sometimes […] it is about security […]
> Other times, it's mostly about reducing redundant libraries […]

Indeed.

> Other common libraries are
> com.google.code.gson gson 2.8.5 45

Ought to be made into an API plugin I think.

> org.apache.commons commons-lang3 3.4 33

Ditto.

On Mon, Dec 13, 2021 at 10:35 AM 'rsomas...@netflix.com' via Jenkins
Developers <jenkin...@googlegroups.com> wrote:
>
> The maven-hpi-plugin could do some linting of the plugins

Would certainly be nice to issue warnings to plugin developers
directing them to use API plugins when possible. Another common
mistake I have seen is mistakenly bundling test libraries in
production, which in some cases results in memory leaks that can take
down a controller (cf. JENKINS-65650 and JENKINS-65771).

> only get it as a transitive from the provided dependency on jenkins-core

Might be interesting to look into avoiding the bundling of
dependencies that are provided by Jenkins core. These are unused at
runtime: https://github.com/jenkinsci/jenkins/blob/165d559469c7a58af581931bd1b89e5b9ed4a9af/core/src/main/java/hudson/ClassicPluginStrategy.java#L247

[Matt Sicker]

Note that some plugins (like audit-log) use log4j-api due to slf4j-api
not supporting non-String messages (e.g., audit log structured data
messages, syslog metadata, etc.). Promoting logging APIs to an API
plugin may require some care in implementation. Also, Jenkins' UI for
recording and viewing logs is currently tied to the java.util.logging
API, not SLF4J or Log4j2, so if logging is further consolidated, that
may also need to be updated.

> --
> You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAFwNDjoGO%3DrHhqttnUJ_--eLq_BE2qq%3DXNnvi8FK_fkJfXgFEg%40mail.gmail.com.

[Jesse Glick]

On Mon, Dec 13, 2021 at 1:35 PM 'rsomas...@netflix.com' via Jenkins Developers <jenkin...@googlegroups.com> wrote:

That suggests that Jenkins Core would prefer plugins to use slf4j-api. If that's the case, all plugins that depend on jenkins-core should use slf4j-api

No, Jenkins core uses java.util.logging, and the great majority of plugins do the same. slf4j-api is just there for a handful of third-party components that require it.

[rsomas...@netflix.com]

Thanks Jesse & boa...@!

I get it now. Does it make sense to get plugins to stop using libraries already in jenkins-core's dependency tree? i.e. are those libraries part of the contract?

If they can be considered guaranteed by jenkins-core, we can just remove the dependencies.

If they can't, then we can create api-plugins for each of the popular ones, and get the plugins to depend on them.

- Rahul

[Jesse Glick]

Please refer to https://www.jenkins.io/doc/developer/plugin-development/dependencies-and-class-loading/ for background.

[Conrad T. Pino]

Hi Rahul,

 

I can't comment much on Jenkins internals but my professional work includes full migration to SLF4J with LogBack implementation while still using 3rd party libraries depending upon other logging frameworks.  While evaluating our Log4j exposure I revisited SLF4J pages:

• Bridging legacy APIs
• Coding new plugins to SLF4J API is IMO a very smart move but not strictly necessary.
• Recoding old plugins to SLF4J API is optional; IMO substituting API bridge for legacy implementation will do.
• Legacy API to SLF4J API bridge JAR files are available for
• Jakarta Commons Logging
• Apache Log4j
• java.util.logging
• Comments on the CVE-2021-44228 vulnerability - TL;DR SLF4J and LogBack implementation are not vulnerable

Best regards,

Conrad T. Pino