Bill of Materials

127 views
Skip to first unread message

Carlos Sanchez

unread,
Apr 20, 2018, 11:56:33 AM4/20/18
to Jenkins Developers
Hi there, 

I have filed a new JEP to address the concept of Bill of Materials (BoM).


The BoM idea came up after different conversations with Tyler/KK/Oleg and several more people - thanks for the feedback! - talking about how can we define a Jenkins package ("classic" jenkins, essentials, custom) to ensure that latest master of core and plugins can be easily used and tested, as well as the too typical use case of PRs in progress across multiple modules.

The main goal is to increase velocity by providing developers automatic artifacts and the result of tests against those artifacts before these combinations of PRs are merged. In order to create some safety nets that encourage contributions and facilitate reviews.

Please share your feedback

Thanks!

R. Tyler Croy

unread,
Apr 27, 2018, 12:55:07 PM4/27/18
to jenkin...@googlegroups.com
(replies inline)

On Fri, 20 Apr 2018, Carlos Sanchez wrote:

> Hi there,
>
> I have filed a new JEP to address the concept of Bill of Materials (BoM).
>
> *https://github.com/jenkinsci/jep/pull/92
> <https://github.com/jenkinsci/jep/pull/92>*
>
> The BoM idea came up after different conversations with Tyler/KK/Oleg and
> several more people - thanks for the feedback! - talking about how can we
> define a Jenkins package ("classic" jenkins, essentials, custom) to ensure
> that latest master of core and plugins can be easily used and tested, as
> well as the too typical use case of PRs in progress across multiple modules.
>
> The main goal is to increase velocity by providing developers automatic
> artifacts and the result of tests against those artifacts before these
> combinations of PRs are merged. In order to create some safety nets that
> encourage contributions and facilitate reviews.


There have been a lot of comments on that pull request, but I wanted to bring
one aspect of the discussion back to the development mailing list.

As I have been working this past week with jglick on the incrementals
publishing, I'm recognizing that the yaml we sketched out previously requires
some logic to be re-implemented in a few different places.

Take the following snippet for example:

plugins:
- groupId: org.jenkins-ci.plugins
artifactId: git
ref: PR-1663 # or how are pull requests handled here?


Whether it's the Jenkins Essentials tooling, custom-war-packager, or any
test/maven tooling, something is going to have to translate those GAV
coordinates into an actual artifact which is either downloaded, or locally
referenced.

In an incrementals build, a pull-request built Git plugin artifact would actually be
located:
https://repo.jenkins-ci.org/incrementals/org/jenkins-ci/plugins/git/3.8.1-rc1663.454c4a5e4d30/git-3.8.1-rc1663.454c4a5e4d30.hpi

I'm not sure how I would build the tooling to use the above YAML to resolve to
the URL, which makes me wonder: why not just have the full artifact URL listed
in the Bill of Materials? For example:

plugins:
- groupId: org.jenkins-ci.plugins
artifactId: git
url: https://repo.jenkins-ci.org/incrementals/org/jenkins-ci/plugins/git/3.8.1-rc1663.454c4a5e4d30/git-3.8.1-rc1663.454c4a5e4d30.hpi


From the Jenkins Essentials perspective, all I really need is an artifact ID
and URL for the artifact, but I'm not sure if that's all that's needed for the
other use-cases here.


Just thought I would raise the question since I don't feel like I understand
how to go from the Bill of Materials to binaries right now.



Cheers
-R Tyler Croy




signature.asc

Jesse Glick

unread,
Apr 30, 2018, 1:07:00 PM4/30/18
to Jenkins Dev
On Fri, Apr 27, 2018 at 12:54 PM, R. Tyler Croy <ty...@monkeypox.org> wrote:
> why not just have the full artifact URL listed
> in the Bill of Materials? For example:
>
> plugins:
> - groupId: org.jenkins-ci.plugins
> artifactId: git
> url: https://repo.jenkins-ci.org/incrementals/org/jenkins-ci/plugins/git/3.8.1-rc1663.454c4a5e4d30/git-3.8.1-rc1663.454c4a5e4d30.hpi
>
> From the Jenkins Essentials perspective, all I really need is an artifact ID
> and URL for the artifact, but I'm not sure if that's all that's needed for the
> other use-cases here.

You could do that, though a plain `version` attribute would suffice,
so long as the tool pulling binaries includes the Incrementals repo in
its list of locations. _If_ you wish to permit branch references or
local disk paths or other options in this format, then you would
anyway need some alternative ways of referring to binaries.

The more interesting question to my mind is how you make edits to the
manifest like the above. After you have more than a handful of plugins
with versions like these, manually copying and pasting such
URLs/versions would be cumbersome. I think JENKINS-50953 can be used
to track this. For example, I might run something like

$ update-incrementals configuration/essentials.yaml

or

$ update-incrementals configuration/essentials.yaml git workflow-api

which for each selected component (or all) would check the current
version; gather a list of all deployed versions (from
`maven_metadata.xml`) that compare as strictly “newer” than that;
filter those by commits which are ancestors of current `master`; pick
the newest of the resulting versions; and edit the file to select that
version.

You could then commit the result and push to a pull request, which
would get validated by a PR builder somehow TBD, and if all automated
checks pass it would be eligible for merging, meaning deployment to
production.

WDYT? Admittedly the above is a very rough sketch, which I need to
flesh out quite a bit.

R. Tyler Croy

unread,
Apr 30, 2018, 4:47:59 PM4/30/18
to jenkin...@googlegroups.com
(replies inline)
> `maven_metadata.xml`) that compare as strictly ???newer??? than that;
> filter those by commits which are ancestors of current `master`; pick
> the newest of the resulting versions; and edit the file to select that
> version.
>
> You could then commit the result and push to a pull request, which
> would get validated by a PR builder somehow TBD, and if all automated
> checks pass it would be eligible for merging, meaning deployment to
> production.
>
> WDYT? Admittedly the above is a very rough sketch, which I need to
> flesh out quite a bit.



If an update-incrementals tool existed, then yes, that would address my
concerns here. So long as something can automatically update my yaml file, i'm
happy to use it, I more wanted to make sure that we weren't going to have two
or three implementations of what `update-incrementals` effectively should be
providing.


signature.asc

Jesse Glick

unread,
May 1, 2018, 9:17:44 AM5/1/18
to Jenkins Dev
On Mon, Apr 30, 2018 at 4:47 PM, R. Tyler Croy <ty...@monkeypox.org> wrote:
> If an update-incrementals tool existed, then yes, that would address my
> concerns here.

OK, good to know I am not completely off base here.

> I more wanted to make sure that we weren't going to have two
> or three implementations of what `update-incrementals` effectively should be
> providing.

There might be two or three scripts to deal with different syntaxes;
for example, it would be useful to have an equivalent to `mvn
versions:use-latest-releases` which grokked Incrementals and updated a
`pom.xml`. But yes they should share an underlying library of some
kind.

Oleg Nenashev

unread,
May 11, 2018, 6:43:57 AM5/11/18
to Jenkins Developers
Hi all,

Just to provide an update on this topic, I have added support of Bill of Materials (in the current draft version) to Custom WAR Packager and used it in several automation flows. E.g. see Integration Testing for Artifact Manager S3. I can also rework Remoting (PR) and Stapler (PR) flows to use BOM there if it is needed for a reference implementation.

Do I gravely need BOM in these pull requests? The answer is "No" as Jesse pointed out. Custom WAR packager uses its own YAML format which can be integrated into essentials.yml format created by Raul Arabaolaza for integration tests. This format is generally more flexible for packaging WARs for testing purposes (build settings, relative FS paths, support of embedding System properties and Groovy scripts, etc.). In this PR and JEP comments I have pointed out several weaknesses of the current BOM proposal:
  • The provided justification for BOM is true, but actually it is already achieved by Custom WAR Packager.
    • BOM does not make much difference there as described above
    • "We want to unify the description format across projects within Jenkins (e.g. Jenkins Essentials, Jenkins X, Docker Packaging, etc.)." is the real thing we should focus on IMHO, but it needs to be expanded (see proposals below)
  • BOM format is not specified. The JEP proposal just provides an example without specifying fields. It refers the K8s notation, but that's it. There is no link to K8s specification and the version, and this specification may drift over time. It leads to all my further comments here
  • "Metadata/labels" are not specified. What is a "must", what is "optional"?
    • E.g. what is field 'name' and how should it be used?
  • Purpose of "Metadata/annotations" section is not clear.
    • There is neither examples nor documentation
    • I would expect to be able to put any custom data there. E.g. for Custom WAR packager I would expect to put build and packaging settings there. OTOH for Jenkins Essentials I would expect this section to have some deployment rules and so
    • We had a private discussion with Carlos, and his explanation was that "annotations" in K8s notation is just a key-value mapping without object support. It does not allow passing complex configurations via them and hence does not allow using BOM as a sole input for test automation flows (like ATH which is used for justification)
  • "Core" section needs specification
    • I'd guess it defines version of WAR file, but it's not clear from documentation. Note that version of WAR and version of the bundled Jenkins Core may differ (timestamped releases, custom WAR builds, etc.)
    • "Core" does not define "artifactId" or "groupId". It means that only standard Jenkins WAR "org.jenkins-ci.main:jenkins-war" may be used. What if I want to define BOM for a custom product based on Jenkins? It is a real use-case for me
  • Vague "components" section. It is not clear what can be defined there and how it should be handled.
    • Is it only JARs? What about Jenkins Modules, Resources, System Groovy Scripts, etc.
    • There is no "type" field or whatever which would allow to distinguish it
  • Plugins and Components support only "version" and "ref" fields
    • In test automation flows we need to test unreleased components in repositories, and it would be really useful to just reference components by path to sources and by path to HPI/JAR like it's done in custom WAR packager
  • It is not clear how "environment" should be used.
    • Can we define multiple environments? Or just a single one?
    • Can environments override plugins/components? Or do they just append new ones?
  • "status" section requires all components/plugins to have artifactId/groupId/versions
    • What if I add a "Groovy init gook" script from a repository as a component? This component has no real version since it's not released separately. Am I expected to mock them somehow (e.g. commitId)?
    • Does BOM expect all listed components to be available in Maven repo?

IMHO BOM specification should be created and documented properly, maybe using one of standard YAML specification languages.


Even with the comments above, I think that the BOM is actually useful and should be finalized. Why?

  • We really need an inter-exchange format. As Jesse said somewhere, Maven POM/BOM is not a silver-bullet in this area since it does not allow passing extra metadata easily
  • Examples of inter-exchange:
    • Custom Packager generates a specification of bundled components which is then used by runPCT/runATH steps to define defaults for running steps (INFRA-1623)
    • WAR bundles ship BOMs internally so that tools like PCT can easily determine what's inside without parsing files
    • BOM can be used as a source for repackaging third-party WARs. E.g. somebody may be building a custom Docker image on the top of jenkins/blueocean, and BOM could help to resolve dependencies there.
    • Jenkins Essentials could be shipping BOM so that the same set of "Verified plugins" could be used by users who are not ready to switch to AWS but still want to get extra insurance about the plugin cross-compatibility

So I am +1 regarding finalizing BOM as JEP.


Bes regards,

Oleg

Carlos Sanchez

unread,
May 11, 2018, 10:22:21 AM5/11/18
to Jenkins Developers
Thanks for the input. Yes there are missing bits and that was the point of circulating the definition. The format is the least important of things, more need to focus on the interoperability and common definition for war packager, essentials and any future usage
 


Bes regards,

Oleg


On Tuesday, May 1, 2018 at 3:17:44 PM UTC+2, Jesse Glick wrote:
On Mon, Apr 30, 2018 at 4:47 PM, R. Tyler Croy <ty...@monkeypox.org> wrote:
> If an update-incrementals tool existed, then yes, that would address my
> concerns here.

OK, good to know I am not completely off base here.

> I more wanted to make sure that we weren't going to have two
> or three implementations of what `update-incrementals` effectively should be
> providing.

There might be two or three scripts to deal with different syntaxes;
for example, it would be useful to have an equivalent to `mvn
versions:use-latest-releases` which grokked Incrementals and updated a
`pom.xml`. But yes they should share an underlying library of some
kind.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/542d0afb-86bd-4f00-a714-96a2bec639b4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Jesse Glick

unread,
May 11, 2018, 12:20:36 PM5/11/18
to Jenkins Dev
On Fri, May 11, 2018 at 6:43 AM, Oleg Nenashev <o.v.ne...@gmail.com> wrote:
> We really need an inter-exchange format. As Jesse said somewhere, Maven
> POM/BOM is not a silver-bullet in this area since it does not allow passing
> extra metadata easily

Indeed the v4 POM format does not permit custom metadata inline in
`<dependency>`s. On the other hand you can always use a POM to define
component versions and then mix it with other metadata for particular
purposes. This is certainly friendlier to existing developer workflows
than attempting to generate or update POMs from another source. We
anyway need a Maven BOM (i.e., `pom.xml` with
`<dependencyManagement>`) just to slim down per-plugin POMs and make
it easy to develop against the “latest” versions of other stuff.

I think the trouble here is that we have a bunch of different tools
and workflows with different needs, and it is not that clear to me how
much overlap there really is, or what problems we are trying to solve
with a unified format. For example, for Evergreen deployment, we just
need a list of components with versions (all assumed to come from
`master`), which could easily fit in a POM if you also differentiate
“environments” like AWS somewhere. For `buildPlugin` integration
testing, it makes the most sense to pick up versions of other
components from the Maven `test` scope, which is already used for
functional tests. (You also need an optional branch override for
`incrementals:update`, but this can easily be handled in mojo
configuration.)

> Custom Packager generates a specification of bundled components which is
> then used by runPCT/runATH steps to define defaults for running steps

Details of these test suites are configured via the `essentials.yaml`.
I think you do not also need a BOM, if you are picking up `test`
scope.

> WAR bundles ship BOMs internally so that tools like PCT can easily determine
> what's inside without parsing files

The code to look up component versions in PCT already exists. Are
there some subtleties not captured by the traditional logic?

> BOM can be used as a source for repackaging third-party WARs.

This is Custom WAR Packager IIUC?

> Jenkins Essentials could be shipping BOM

Well, Evergreen will be “shipping” a flat list of core & plugin URLs,
if I gather Tyler’s intention correctly—the minimum information needed
by the client. It is an open question what build process generates
this list and from what source.

Liam Newman

unread,
Jun 15, 2018, 8:00:22 PM6/15/18
to Jenkins Developers
I've approved JEP-309 as Draft.  

Oleg Nenashev

unread,
Aug 6, 2018, 6:11:24 AM8/6/18
to Jenkins Developers
Hi all,

Status update: By now Custom War Packager has been released in 1.0, and there are also many updates in Evergreen. IMHO it is a good time to get this story over the fence.

Last week I have submitted a patch to the design, which does the following:
  • Full format specification has been added to the JEP
  • New "version" field has been added to support changing the format in the future. It addresses my comment about extensibility in the future
  • "components" section has been documented, so it is no longer an empty blob
  • "type" has been added to "components". It addresses my comment about component type packaging
  • Dependency resolution rules have been documented explicitly
  • Reference implementation links are updated
  • Tooling has been explicitly excluded out of the scope (though Custom War Packager offers a lib)
With the current changes, I believe that the JEP can be reviewed by the BDFL Delegate. The only outstanding comment is "YAGNI" from Jesse, but I believe that the reference implementations justify it a bit. I do not expect this Bill of Materials to be widely used, but accepting this JEP at least allows to stabilize and specify the current implementations which we already use in some cases.

Best regards,
Oleg

Jesse Glick

unread,
Aug 6, 2018, 11:12:17 AM8/6/18
to Jenkins Dev
On Mon, Aug 6, 2018 at 6:11 AM Oleg Nenashev <o.v.ne...@gmail.com> wrote:
> The only outstanding comment is "YAGNI" from Jesse, but I believe that the reference implementations justify it a bit.

So I see that

https://github.com/jenkinsci/jep/blob/master/jep/309/README.adoc#prototype-implementation

lists Evergreen as a consumer, yet

https://github.com/jenkins-infra/evergreen/blob/master/services/essentials.yaml

is not really ready for use; together with the `ingest-update-center`
script that processes it, it looks like a throwaway implementation,
and as I found in PR #144, it does not support JEP-305 adequately and
does not properly manage transitive dependencies. I have proposed ways
to fix `essentials.yaml` so that it would actually serve the needs of
Evergreen, but it is not clear the resulting format would still match
whatever is being proposed in JEP-309.

As to `essentialsTest`, IIUC this is a usage of `custom-war-packager`,
rather than a separate client. So if Evergreen is indeed fixed at some
point by reworking the format of this YAML file, that leaves CWP as
the only direct consumer of the BOM, making it not be an intetrchange
format. Even if Evergreen can productively use the same format as the
JEP currently specifies, it is not apparent to me why we care: there
seems to be no overlap in use case between the YAML file stored in the
Evergreen repository, to define what is pushed to production, and the
YAML files used now in a few plugins to request integration tests
during PR builds.

Oleg Nenashev

unread,
Aug 6, 2018, 11:25:24 AM8/6/18
to JenkinsCI Developers
I would prefer to get JEP-309 accepted, maybe as "Bill of Materials 1.0" specification. We define cross-dependencies between CWP formats and Jenkins Infrastructure (essentialsTest() on ci.jenkins.io), so I would prefer to have this format accepted even if it is the only reference implementation.

I am fine with renaming JEP if we want to narrow down the scope.

BR, Oleg



--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-dev/pR2ZQMj95Zc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jenkinsci-dev+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr09_dhT%3D6ataL2iMF%3D7sGex49a%3DR6hbNn50-x3wGCrJ%2BA%40mail.gmail.com.

Oleg Nenashev

unread,
Aug 6, 2018, 11:26:52 AM8/6/18
to JenkinsCI Developers
Anyway, let's get some feedback from Jenkins Essentials folks before we proceed.

Jesse Glick

unread,
Aug 6, 2018, 12:10:50 PM8/6/18
to Jenkins Dev
On Mon, Aug 6, 2018 at 11:25 AM Oleg Nenashev <o.v.ne...@gmail.com> wrote:
> We define cross-dependencies between CWP formats and Jenkins Infrastructure (essentialsTest() on ci.jenkins.io)

Yes, this is a cross-dependency, since the `pipeline-library` method
calls do not seem to specify a version of CWP.

Where is this method being used BTW?

https://github.com/search?ref=simplesearch&type=Code&q=user%3Ajenkinsci+essentialstest

does not turn up any hits. I searched for

https://github.com/search?q=user%3Ajenkinsci+user%3Ajenkins-infra+filename%3Aessentials.yml+filename%3Aessentials.yaml&type=Code

(note the extension mismatch!) and saw that only Jenkins core and the
Git plugin were currently using this. Neither `Jenkinsfile` calls the
library function; they seem to use `runATH` and `runPCT` directly.
Jenkins core runs only ATH, no PCT (even though it would be very
useful to check compatibility against some plugins like `workflow-job`
which could easily be broken by subtle core changes). The Git plugin
runs some ATH tests and also claims to run PCT but only on itself,
which seems to make no sense (it should be running tests of _other_
plugins which would likely be broken by mistakes in `git-plugin`). Or
rather it _would_ be running ATH & PCT—currently this is all disabled.

So not a very large sample size here. Are there some plans to clean
this up and get `essentialsTest` widely used?

Oleg Nenashev

unread,
Aug 6, 2018, 12:23:57 PM8/6/18
to JenkinsCI Developers
Hi Jesse,

Yes, essentialsTest() didn't get as much progress as I would have expected. Not my decision, the story has been handed over to other contributors. But there is a number of stories related to it, e.g. are pending patches for essentialsTest():
These stories will be completed by somebody at some point (sorry for being sooo specific), and that's why I would rather prefer to have JEP-309 finalized instead of staying in flux. essentialTest() will also eventually use BOM as consumer in order to automatically determine defaults for PCT and ATH. So far it just produces BOM and publishes it as an artifact.

Hopefully it answers the question,
Oleg



--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-dev/pR2ZQMj95Zc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jenkinsci-dev+unsubscribe@googlegroups.com.

Jesse Glick

unread,
Aug 6, 2018, 1:05:24 PM8/6/18
to Jenkins Dev
On Mon, Aug 6, 2018 at 12:23 PM Oleg Nenashev <o.v.ne...@gmail.com> wrote:
> there is a number of stories related to it, e.g. are pending patches for essentialsTest()

Interesting, I will take a look at these.

R. Tyler Croy

unread,
Aug 6, 2018, 6:50:02 PM8/6/18
to jenkin...@googlegroups.com
(replies inline)

On Mon, 06 Aug 2018, Oleg Nenashev wrote:

> Hi all,
>
> Status update: By now Custom War Packager has been released in 1.0, and
> there are also many updates in Evergreen. IMHO it is a good time to get
> this story over the fence.


Thanks for working to drive this to completion Oleg! Overall I think the
document looks great, and as of last Friday I had actually gotten the
implementation prepared for the Evergreen backend system (see:
https://github.com/jenkins-infra/evergreen/pull/169) which processes the
essentials.yaml and prepares the update records needed.


I believe the only think which needs to be resolved which is likely just an
obsolete part of the example YAML. The root `status` key in the YAML for a
"realized" BOM I don't believe we've ever actually used and is worth removing.

I wanted to sanity check that with you first though.


If you remove that, I think overall this JEP looks good and is just about ready
to go.


Cheers
signature.asc

Oleg Nenashev

unread,
Aug 6, 2018, 7:59:25 PM8/6/18
to JenkinsCI Developers
Hi Tyler,

Thanks for the feedback!
 
I believe the only think which needs to be resolved which is likely just an
obsolete part of the example YAML.  The root `status` key in the YAML for a
"realized" BOM I don't believe we've ever actually used and is worth removing.

Actually I use it in some cases in order to implement custom packaging Pipelines after customWARPackager().
  • BOM's specification lists explicit dependencies
  • BOM's specification does not require all dependencies to be explicit
    • Some dependencies may have "dir" references
    • Some dependencies may be transitive. JEP-309 permits that though does not recommend for production use (dependency resolution in the spec)
  • "status" key returns the full list of resolved dependencies
    • In addition to transitive deps, CWP uses "status" to squash the "environment" definitions into a single list in order to show what was actually packaged into the WAR file
I would rather prefer the "status" section to stay in the specification. It is helpful for CWP at least (though it may be possible to just generate a new output BOM). If we do that, it would be nice to get feedback from Raul who is also experimenting with processing of BOMs.

In order to address your comment, we could explicitly say that the "status" section is optional so that you do not need to implement it in Evergreen if not needed. WDYT?

BR, Oleg


--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-dev/pR2ZQMj95Zc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jenkinsci-dev+unsubscribe@googlegroups.com.

R. Tyler Croy

unread,
Aug 14, 2018, 11:38:35 AM8/14/18
to jenkin...@googlegroups.com
(replies inline)

On Tue, 07 Aug 2018, Oleg Nenashev wrote:

> Hi Tyler,
>
> Thanks for the feedback!
>
>
> > I believe the only think which needs to be resolved which is likely just an
> > obsolete part of the example YAML. The root `status` key in the YAML for a
> > "realized" BOM I don't believe we've ever actually used and is worth
> > removing.
>
>
> Actually I use it in some cases in order to implement custom packaging
> Pipelines after customWARPackager()
> <https://github.com/jenkins-infra/pipeline-library/blob/master/vars/customWARPackager.groovy>.
>
>
> - BOM's specification lists explicit dependencies
> - BOM's specification does not require all dependencies to be explicit
> - Some dependencies may have "dir" references
> - Some dependencies may be transitive. JEP-309 permits that though
> does not recommend for production use (dependency resolution
> <https://github.com/jenkinsci/jep/tree/master/jep/309#dependency-resolution>
> in the spec)
> - "status" key returns the full list of resolved dependencies
> - In addition to transitive deps, CWP uses "status" to squash the
> "environment" definitions into a single list in order to show what was
> actually packaged into the WAR file
>
> I would rather prefer the "status" section to stay in the specification. It
> is helpful for CWP at least (though it may be possible to just generate a
> new output BOM). If we do that, it would be nice to get feedback from Raul
> who is also experimenting with processing of BOMs.
>
> In order to address your comment, we could explicitly say that the "status"
> section is optional so that you do not need to implement it in Evergreen if
> not needed. WDYT?



I mentioned in a video call with Oleg this morning that I've gone ahead and
implemented the `status` section for the Bill of Materials being used in the
jenkins-infra/evergreen repository.


Overall I'm quite happy with this work by Oleg and Carlos, and I will be
submitting a PR (with my BDFL-Delegate hat on) to mark JEP-309 as 'Accepted'
later today.


Thanks for the hard work everybody!
signature.asc

Oleg Nenashev

unread,
Aug 14, 2018, 11:42:53 AM8/14/18
to JenkinsCI Developers
Thanks for the update, Tyler!


--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-dev/pR2ZQMj95Zc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/20180814153826.GH17800%40grape.lasagna.io.

Carlos Sanchez

unread,
Aug 15, 2018, 5:19:40 AM8/15/18
to jenkin...@googlegroups.com
Thanks!

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/20180814153826.GH17800%40grape.lasagna.io.

Jesse Glick

unread,
Aug 22, 2018, 6:36:57 AM8/22/18
to Jenkins Dev
On Tue, Aug 14, 2018 at 11:38 AM R. Tyler Croy <ty...@monkeypox.org> wrote:
> I've gone ahead and
> implemented the `status` section for the Bill of Materials being used in the
> jenkins-infra/evergreen repository.

From what I can tell, this is not working so well. There is a lot of
duplication between `spec` and `status`; lots of transitive
dependencies are much too old; there are unwanted transitive
dependencies and no clear way to track them.

I would suggest dropping `status` and having all plugins which are to
be included explicitly listed in the main and environment sections
(with a mandatory alphabetical order). Produce a build-time error
(`make -C services check` perhaps) if a plugin expresses a dependency
which is too old, or nonoptional and missing—the same checks the
Jenkins plugin manager would do at runtime. The equivalent of `mvn
incrementals:update` should be easy to write which would go through
the list and offer updates of every entry to the latest version
available in Artifactory, including in the case of incremental
versions the latest deployed `master` commit.

Then for each plugin, add a mandatory boolean `transitive` attribute.
If a plugin is marked `transitive: true`, a build-time error would
result if there is no dependency chain to this plugin from a plugin
which is marked `transitive: false`. That would make it clear in the
text that library plugins (`docker-commons`, `git-client`,
`bouncycastle-api`, …) are “along for the ride” as opposed to
independent features. More importantly, when we see unwanted plugins
in the list (`conditional-buildstep`, `icon-shim`, `matrix-project`,
`maven-plugin`, …) with `transitive: true`, we can simply try deleting
them and seeing if everything is still OK, or if not which bogus
dependency(ies) must be broken. And if you update a `transitive:
false` plugin to a version which no longer requires some library
plugin, and that was the last usage of the library (“refcount goes to
zero”), you will immediately get a build error if you forgot to remove
the now-obsolete library.

Also it is unclear how well `environments` will work when we have a
bunch. Currently `aws-global-configuration` & `artifact-manager-s3` &
`ec2` are specified in the main section as well as `aws-ec2-cloud`,
which seems like a mistake—presumably they should be listed only in
`aws-ec2-cloud`. But then when there are common library plugins used
in several environments but _not_ in all, we will have duplicated
`version`s (a likely source of developer error) unless some equivalent
to Maven’s `<dependencyManagement>` is introduced.

Oleg Nenashev

unread,
Aug 22, 2018, 9:47:44 AM8/22/18
to Jenkins Developers
Hi Jesse,

You are a bit late to the party, JEP-309 has been already accepted. It would not be possible to fully address your feedback without introducing a 2.0 BOM version.
Also note that JEP-309 defines a BOM definition format. It does NOT define strict guidelines how BOM packager implementations should package the specification section.
It may address some of your concerns, see the responses below.

I would suggest dropping `status` and having all plugins which are to be included explicitly listed in the main and environment sections
 I hope that reasoning explains why it was decided to support transitive dependencies.
In your BOM packager implementation you have an option to refuse unexplicit dependencies as it is defined in the Dependency Management section.

Removing "status" would require a breaking change in the format.

Then for each plugin, add a mandatory boolean `transitive` attribute.
Not a compatible change
 
Also it is unclear how well `environments` will work when we have a
bunch. Currently `aws-global-configuration` & `artifact-manager-s3` &
`ec2` are specified in the main section as well as `aws-ec2-cloud`,
which seems like a mistake—presumably they should be listed only in
`aws-ec2-cloud`. But then when there are common library plugins used
in several environments but _not_ in all, we will have duplicated
`version`s (a likely source of developer error) unless some equivalent
to Maven’s `<dependencyManagement>` is introduced.

Although it is not mentioned explicitly, JEP-309 does not prohibit combining environments if needed.
IMO it is up to the BOM packager implementation
 
Best regards,
Oleg

Jesse Glick

unread,
Aug 22, 2018, 2:39:52 PM8/22/18
to Jenkins Dev
On Wed, Aug 22, 2018 at 9:47 AM Oleg Nenashev <o.v.ne...@gmail.com> wrote:
> JEP-309 has been already accepted. It would not be possible to fully address your feedback without introducing a 2.0 BOM version.

Which is why I said on Aug 06 that the JEP seemed premature while the
basic problems in how developers publish updates to Evergreen were
still being worked out.

> I hope that reasoning explains why it was decided to support transitive dependencies.

You mean in the Specification (not Reasoning)?

https://github.com/jenkinsci/jep/tree/master/jep/309#dependency-resolution

does not specify anything about semantics of transitive dependencies
the `spec` section.

> Removing "status" would require a breaking change in the format.

Yes…breaking to what, exactly? `essentials.yaml` is stored in one repo
and processed by one script in the same directory.

>> Then for each plugin, add a mandatory boolean `transitive` attribute.
>
> Not a compatible change

Sure, but what concretely is affected by changing the format of
`essentials.yaml`?

> Although it is not mentioned explicitly, JEP-309 does not prohibit combining environments if needed.
> IMO it is up to the BOM packager implementation

My discussion is specifically about what _Evergreen_ does.
Reply all
Reply to author
Forward
0 new messages