Jenkins SDLC Security: LFX Security/Snyk adoption status updates

[Oleg Nenashev]

Hi all,

TL;DR: I ask for permission to start an official pilot project together with LFX Security and Snyk + Call for contributors and maintainers who want to participate in the pilot program.

At the previous Jenkins Contributor Summit we discussed  Securing Jenkins Delivery Pipelines and, specifically, developer tools we would like to adopt to discover issues with dependencies, licenses and the codebase (meeting notes). LFX Security, a Snyk-based service provided by the Linux Foundation, is one of the tools we discussed and added to our wishlist. LFX Security is a result of the partnership between the Linux Foundation and Snyk.

Discovery status. After the summit I spent some time getting access to the Jenkins profile on LFX Security. It was created for us automatically as a CDF member project. As you may see in the screenshots below, the results are quite funny. There is a HUGE number of false positives which IMO is a blocker for further adoption. 

Root cause: Unique Jenkins plugin packaging. We use JAR dependencies to declare both library and plugin requirements. Then Maven HPI Plugin (or Gradle JPI Plugin) analyzes the dependency tree, properly packages Jenkins plugin files with libraries, and generates metadata used by Jenkins to resolve plugin dependencies. This engine is unique to Jenkins and not supported by other tools, they consider all dependencies on the Jenkins core and plugins as real dependencies to be bundled. Hence a huge number of false positives due to dependencies on old Jenkins core/plugin versions containing vulnerabilities published by the Jenkins project (we are a CNA, thanks to Daniel Beck and the team).

I have started exploring LFX Security, and apparently we had no way of resolving the issue.due to the lack of the admin interface and APIs. Only the per-prepo configuration as code with CVE ignore lists is available, but it does not look feasible with 2500+ repositories used by Jenkins. Semi-manual management is also a no go as you may guess.

Meeting with Snyk and the LFX Security Team. After discussing with the Linux Foundation Team, we reached out to Snyk and scheduled a call on April 28th. From the Jenkins side we had me and Olivier Verning there, Daniel Beck, Mark Waite and Wadeck Follonier were also invited. You can find the summary and the recording/meeting notes links below.I did not send an invite in public, because of the confidentiality status of the discussion. But now we can share it in public

• Wendy Swank from Snyk presented the LFX Security 2.0 platform - a new generation of LFX Security. It is under internal testing at the moment, and it is expected to address our needs so that LFX Security becomes useful for projects of Jenkins scale.  There was a short demo, see the recording link below.
• The timeline for the pilot is TBD, there is no time commitment for the LFX Security 2.0 release
  
• The Jenkins project is invited to participate in the LFX Security 2.0 pilot program after they finish internal testing.
• As a part of this offer, we will get direct assistance from the Snyk and LFX teams while setting up our scanning pipelines
• If the project is successful, Jenkins would do a joint announcement with Snyk and LFX Security of LFX Security 2.0. We will be also referenced through the LF and Snyk marketing channels and events.

The proposal is to start the pilot from a small list of the repositories controlled by the pilot project participants: Jenkins core, its libraries, and some plugins from maintainers who are interested to participate in the pilot project.

Call for feedback:

• Would you approve doing an official pilot project together with Snyk and LFX Security?
• Would you like to participate as a contributor?
• Would you like your plugin to participate in the pilot project?

Thanks for any feedback,

Oleg Nenashev

Links. Disclaimer: Many of the topics represent future state or great feature ideas not in product (on both LFX + Snyk side) at this time

• Meeting notes from

• Slides: “Snyk + LFX Security”by Wendy Swank (current state, LFX Security 2 plans and roadmap)

• Recording (passcode: %C7wv4mt)

LFX Security stats for Jenkins:

[Daniel Beck]

Thanks again for driving this, Oleg!

> On 3. May 2021, at 19:14, Oleg Nenashev <o.v.ne...@gmail.com> wrote:
>
> The proposal is to start the pilot from a small list of the repositories controlled by the pilot project participants: Jenkins core, its libraries, and some plugins from maintainers who are interested to participate in the pilot project.
>
> Call for feedback:
> • Would you approve doing an official pilot project together with Snyk and LFX Security?

Yes, definitely.

> • Would you like to participate as a contributor?

What does this entail?

> • Would you like your plugin to participate in the pilot project?

Yes (although my plugins tend not to have interesting dependencies so it's probably not that interesting).


Re private PRs and security process (minute 25-28 in the transcripts, again around 0:51) -- I don't really see a need to handle dependency updates in private in most cases, as all information that is based on is usually public anyway (CVEs in dependencies as well as dependency declarations). Additionally, vulnerabilities in plugins don't necessarily mean we're vulnerable (or that the metadata is correct to begin with), and how to exploit it isn't often obvious either. So I don't feel strongly about keeping such content private. What we prepare in jenkinsci-cert in private is almost exclusively fixes for exploitable vulnerabilities originating in Jenkins code.

Re CVE ignore list and such -- would like to see a plan (perhaps with the custom graph API that pre-filters plugin dependencies) how to deal with transitive plugin dependencies: Plugin X depends on plugin Y which bundles library Z. The maintainer of X doesn't really care about Z being outdated. (The same applies to "core Y" -- no need for a tool to tell plugin maintainers about core dependencies being outdated.) While we could dump all Jenkins CVEs into a giant ignore list, that wouldn't take care of this problem.

Re licenses -- Is this something we're actively looking at as well, or rather secondary. FWIW I can think of these use cases: That it's actually open source; that it's compatible (no MIT plugin bundling GPL dependencies or similar weirdness); that it's all OSI approved per governance document. Anything else?

[Oleg Nenashev]

Thanks for the interest Daniel!

>> Would you like to participate as a contributor?
> What does this entail?

That's a good question, to be seen. As a part of the pilot project we will need:

• Try out LFX Security 2.0 and configure it for some of our projects
• Explore options for filtering out false positives, find a solution for the Jenkins project taking its scale and needs
• Try out other features like license analysis
• Document the implementation for other maintainers
• Keep evaluation notes and share feedback with Snyk/LFX Security. If we experience blockers, multiple iterations might be required

Note from the discussion: It is unlikely that we will be able to use the standard Snyk's GitHub integration via GitHub App. We will likely need to integrate scanning submissions into our Jenkins pipelines (there is a Snyk plugin FTR) or to use GitHub actions. Reason - GitHub Integration cannot handle custom Bills of Materials which will be supported by the LFX Security 2.0 API (actually, by the Snyk backend).

> Re licenses -- Is this something we're actively looking at as well, or rather secondary. FWIW I can think of these use cases: That it's actually open source; that it's compatible (no MIT plugin bundling GPL dependencies or similar weirdness); that it's all OSI approved per governance document. Anything else?

Yes, looks good to me. I would rather prefer to avoid some licenses where compatibility is disputed, e.g. traditional AGPL concerns.

[Olblak]

Hi Oleg,

Thanks for driving this.

Once we are allowed to have a second Github Organization, I would be very interested to experiment with it for the jenkins-infra organization.

[Daniel Beck]

On Mon, May 3, 2021 at 10:24 PM Oleg Nenashev <o.v.ne...@gmail.com> wrote:

>> Would you like to participate as a contributor?

> What does this entail?

That's a good question, to be seen. As a part of the pilot project we will need:

• Try out LFX Security 2.0 and configure it for some of our projects
• Explore options for filtering out false positives, find a solution for the Jenkins project taking its scale and needs
• Try out other features like license analysis
• Document the implementation for other maintainers
• Keep evaluation notes and share feedback with Snyk/LFX Security. If we experience blockers, multiple iterations might be required

Note from the discussion: It is unlikely that we will be able to use the standard Snyk's GitHub integration via GitHub App. We will likely need to integrate scanning submissions into our Jenkins pipelines (there is a Snyk plugin FTR) or to use GitHub actions. Reason - GitHub Integration cannot handle custom Bills of Materials which will be supported by the LFX Security 2.0 API (actually, by the Snyk backend).

Right, that's what I mentioned further down. I don't see CVE ignore lists having satisfactory results for plugin maintainers as soon as plugin/core dependencies are scanned.

I'd like to contribute to this effort, unfortunately I don't know yet how much time I can actually commit to this.

[Oleg Nenashev]

Hi Olivier,

For experiments, it might make sense to register Jenkins Infrastructure as the second organization. I do not anticipate major overlap between configs taking different technology stacks.

Once Snyk adds support for multiple GitHub orgs, we can explore our options. M<aybe it won't be even a problem since we can unlikely use the standard GitHub App integration anyway (see my response to Daniel)

If you want to start with the Jenkins Infra org, I think you can do it even now on LFX Security 1.x && standard GitHUb integration. Then it can be migrated to 2x.

Let the LFX Security team know if you want one to be created.

Best regards,

Oleg

--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-dev/lIeGyefi1Y4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/496a4b3a-0489-4783-b06b-1f27fd369bcan%40googlegroups.com.

[Oleg Nenashev]

Hi all,

Just a quick update, we will be setting up a shared Slack channel in the CDF workspace. Once it is ready, we can use it for conversations.

Pending ticket to the CDF: https://github.com/cdfoundation/foundation/issues/330

BR, Oleg

[Oleg Nenashev]

UPD: The Jenkins Governance meeting approved official participation in the Pilot project.

Approved by: Ulli Hafner, Gavin Mogan, Ewelina Wilkosz, Daniel Beck, Baptiste Mathus, Mark Waite (+ Olivier I'd guess)

Thanks everyone for the feedback!

[Oleg Nenashev]

FTR we created the new #lfx-security-2-pilot-with-snyk channel in the CDF Slack workspace so that all sides can sync-up in a chat if needed.

You can join the workspace by following the guidelines in https://www.jenkins.io/chat/#continuous-delivery-foundation

All interested contributors and maintainers are welcome!