Upcoming JEP for usage of Incrementals repository

104 views
Skip to first unread message

Jesse Glick

unread,
Apr 12, 2018, 3:19:06 PM4/12/18
to Jenkins Dev
After IEP-9¹ there was a need for an actual system for producing and
consuming these quasi-release “incremental” artifacts from Jenkins
projects (typically based on Maven). This is just a heads-up that I am
working on a preparing a JEP, probably for the 3xx “Essentials”
series; and a reference implementation, namely some Maven tooling.
This is being tracked under JENKINS-50686². Anyone with deep
understanding of Maven (like Stephen…) is encouraged to take a peek at
my ongoing work.

¹ https://github.com/jenkins-infra/iep/blob/master/iep-009/README.adoc
² https://issues.jenkins-ci.org/browse/JENKINS-50686

Jesse Glick

unread,
Apr 13, 2018, 6:26:59 PM4/13/18
to Jenkins Dev

R. Tyler Croy

unread,
Apr 13, 2018, 8:21:21 PM4/13/18
to jenkin...@googlegroups.com
(replies inline)

On Fri, 13 Apr 2018, Jesse Glick wrote:

> Filed a draft request:
>
> https://github.com/jenkinsci/jep/pull/84


Jesse this is a very thorough and well-thought out document! Thank you so much
for putting the effort into it.

I understand that you have prototyped this work with strictly plugins at this
point, and the document makes allusions to core and 'modules' potentially being
able to use this same approach and tooling.

The only question I had was whether you think it's important to verify that
core and modules function with this model before rolling out this type of a
design? From the IEP-9 standpoint, I don't really care what tooling is required
to get incremental builds into the maven repository, but I'm not sure if you
consider applicability to core part of "acceptance criteria" for this design.

I hope the question makes sense, put more shortly: does core need to be figured
out for this to come into effect?




Have a good weekend :D

signature.asc

Jesse Glick

unread,
Apr 14, 2018, 7:50:56 AM4/14/18
to Jenkins Dev
On Fri, Apr 13, 2018 at 8:21 PM, R. Tyler Croy <ty...@monkeypox.org> wrote:
> [Do] you think it's important to verify that
> core and modules function with this model before rolling out this type of a
> design?

Yes I would consider that a prerequisite for this JEP to be Accepted
and the system to be rolled out for general use:

https://issues.jenkins-ci.org/browse/JENKINS-50692

I am not sure if there is some convention for indicating in the JEP
itself, or in JIRA, which stories in an Epic like JENKINS-50686 should
be considered “mandatory” vs. “optional” for various stages (Draft,
Accepted, Final).

Baptiste Mathus

unread,
Apr 19, 2018, 7:12:58 AM4/19/18
to Jenkins Developers
Hello,

Finally took some time to review the JEP 305, thanks a lot again Jesse for that big step forward, here's some feedback:


I guess we should then add also RequireMavenVersion 3.5+ shouldn't we?

* I love the part https://github.com/jenkinsci/jep/blob/master/jep/305/README.adoc#usage-from-the-update-center and the idea that we do GitOps and get better traceability/revertability ("Emergency rollbacks are as simple as git revert." indeed). This is easily controllable manually, and can be automated/supported via additional tooling progressively.


I agree with the global reasoning, small concern about "borked releases". When one is using MRP, a burned release is /just/ running MRP again.
If a given commit does map to a given release naming with JEP-305, then if something goes wrong during "mvn deploy", like say a failure in the middle of the physical upload, then I cannot (rightly) retry to override that borked binary.

In that case, do I have to create some kind of fake commit to retry, or even git commit -m "fake commit to retry release deploy" --allow-empty ?




--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr3r7VDqUYBQ1PKf4jdiSmGmOYneaqi%3Dy3S-zm3T6%3D9wXA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Baptiste Mathus

unread,
Apr 19, 2018, 7:37:30 AM4/19/18
to Jenkins Developers
2018-04-19 13:12 GMT+02:00 Baptiste Mathus <m...@batmat.net>:
Hello,

Finally took some time to review the JEP 305, thanks a lot again Jesse for that big step forward, here's some feedback:


Jesse Glick

unread,
Apr 19, 2018, 9:27:55 AM4/19/18
to Jenkins Dev
On Thu, Apr 19, 2018 at 7:12 AM, Baptiste Mathus <m...@batmat.net> wrote:
> When one is using MRP, a burned release is /just/ running MRP again.
> If a given commit does map to a given release naming with JEP-305, then if
> something goes wrong during "mvn deploy", like say a failure in the middle
> of the physical upload, then I cannot (rightly) retry to override that
> borked binary.
>
> In that case, do I have to create some kind of fake commit to retry, or even
> git commit -m "fake commit to retry release deploy" --allow-empty ?

You could I guess, but it should not be necessary because unlike our
current process for MRP, I am working on server-side deployment.
Rather than `mvn deploy` (which is indeed flaky), it is using
Artifactory REST with `X-Explode-Archive-Atomic`¹:

https://github.com/jglick/incrementals-downstream-publisher/blob/e80393721af3fad30ff0c6c7af4a1c56ef1b0ae2/sample-downstream.groovy#L9

¹BTW Google Image Search gets this all wrong.

Jesse Glick

unread,
Apr 27, 2018, 9:09:39 PM4/27/18
to Jenkins Dev
A little status update here.

On Thu, Apr 19, 2018 at 9:27 AM, Jesse Glick <jgl...@cloudbees.com> wrote:
> I am working on server-side deployment.

Thanks to the boundless patience of Tyler and others, this is live!

https://github.com/jenkinsci/pom/blob/master/incrementals.md#usage-in-plugin-poms

describes how to get started in a plugin repository;

https://github.com/jenkinsci/jenkins/pull/3394

would make it work in core.

To deploy your patch, just make sure the repository is configured as
above, then file a pull request from a repository fork. If build &
tests pass, and the PR is up to date with the target branch, the
artifacts should be deployed for you, and you will see a commit status
check to that effect in the PR.

At that point you can use the artifacts as a dependency in any other
POM, or download them from Artifactory, or whatever. If you do not
care to wait for the CI builder, you can also build equivalent
artifacts locally, as described in the guide above.

Note: currently commits on branches in the origin repository do not
get deployed. This should be fixed once a new Git plugin release is
cut and ci.jenkins.io updates to it.

James Nord

unread,
May 16, 2018, 7:14:50 AM5/16/18
to Jenkins Developers
So the objection I have is that as it stands in JEP-305 the versions are releases and these are expected to be garbage collected.

Anyone also wanting to consume incrementals via a proxy would need to write tooling to garbage collect releases and there is also the Maven principal that you never garbage collect releases, let alone the impact this will have on other pipelines that do canary builds by picking up the latest *release* of a artifacts/plugins in order to do bleeding edge testing in our pipelines. And that is also discounting the ability once it is in our mirror for things to accidentally depend on these releases.

So, in analysing the proposal we believe that you can accomplish this with SNAPSHOTS.
You do not need to use timestamped snapshots (which is the main objection in JEP-305) against SNAPSHOTS - you just add -SNAPSHOT on to the end of the version you already have (such that 1.7-rc1652.cd45427eb4e2 becomes {{1.7-rc1652.cd45427eb4e2-SNAPSHOT}

e.g. the equivalent command to generate the changelist setting becomes:

-Pproduce-incrementals -Dchangelist=-rc$(git rev-list --first-parent --count HEAD).$(git rev-parse --short=12 HEAD)-SNAPSHOT

then in the dependencies you depend on the xxx-SNAPSHOT and not the timestamped snapshot version which allows the version to be used from the local repo cache and not to have to download it from the hosted repo (artifaactory etc).

Because the changelist produced starts with the ("-rc" git rev list) it becomes monotonically increasing and ends with "-SNAPSHOT" we can garbage collect without any additional tooling. Because it ends in -SNAPSHOT it will not be picked up by canary builds or have the possibility of being included in releases, which will then allow us to include it in the mirror by default so there would be no need to change any settings.xml.

If you have a separate artifactory repo to have incremental snapshots then you can just scan this repo to find the latest versions for an update center. It may be possible in artifactory to also change the "snapshots" repo into a group and use routing rules to efficiently serve these whilst still maintaining compatability for users deploying snapshots with existing settings.


The change in the JEP of the version being $(git rev-parse --short=12 HEAD) to being $(version.removeEnd('-SNAPSHOT')}-rc$(git rev-list --first-parent --count HEAD).$(git rev-parse --short=12 HEAD) removed a lot of my spider-sense objections... but I think we need to have the deployed version include -SNAPSHOT to get something that aligns with the semantics of the incrementals repo from a Maven perspective



Regards

/James

Stephen Connolly

unread,
May 16, 2018, 9:04:41 AM5/16/18
to jenkin...@googlegroups.com
To be clear, this is a joint response from myself and James


--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/315af002-b836-4480-b053-beed9d05eacf%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.
--
Sent from my phone

Daniel Beck

unread,
May 16, 2018, 10:04:54 AM5/16/18
to jenkin...@googlegroups.com

> On 16. May 2018, at 13:14, James Nord <jn...@cloudbees.com> wrote:
>
> Anyone also wanting to consume incrementals via a proxy would need to write tooling to garbage collect releases and there is also the Maven principal that you never garbage collect releases, let alone the impact this will have on other pipelines that do canary builds by picking up the latest *release* of a artifacts/plugins in order to do bleeding edge testing in our pipelines. And that is also discounting the ability once it is in our mirror for things to accidentally depend on these releases.

I don't understand these concerns.

While Maven treats these artifacts as releases, that's about the extent of the problem. If you don't use repo.j.o/incrementals as a repository containing regular releases, none of that should be relevant. In fact, repo.j.o/incrementals is not part of the repo.j.o/public virtual repository typically referenced in settings.xml and POMs. Why would there be consumers of incremental releases that are also unaware of the retention policy?

And don't dependencies on snapshots that get garbage collected break the same as dependencies on non-snapshots, if there's no matching release?

Jesse Glick

unread,
May 16, 2018, 12:16:56 PM5/16/18
to Jenkins Dev
My response would mostly follow Daniel’s but I will add more
background—something I can write up into a subsection of

https://github.com/jenkinsci/jep/blob/master/jep/305/README.adoc#other-version-schemes

once the dust settles.

James Nord & Stephen Connolly wrote:
> as it stands in JEP-305 the versions are
> releases and these are expected to be garbage collected.

Technically this is from IEP-9, but as that attracted little notice
(indeed I only stumbled across it after it was finalized and
implemented), it makes sense to discuss in the context of JEP-305.

> Anyone also wanting to consume incrementals via a proxy would need to write
> tooling to garbage collect releases

I think GC is the crux of the question. For the record, I would have
recommended a rather more conservative collection schedule in IEP-9
(basically keep everything until storage costs start to raise
eyebrows); I am not actually sure whether the collection scheme was
even implemented, or if so, whether this is custom tooling or some
stock feature of Artifactory. Daniel could speak to that.

There are several things to consider about GC, and none have anything
to do with whether the version number happens to end in `-SNAPSHOT` or
not, so adding that suffix solves no practical problem.

First, how likely is it that a newly run build of a downstream
component will refer to an incremental version that has been expired
from Artifactory? This is hard to know until JEP-305 gets broader
usage, but my expectation would be that new builds will usually refer
to pretty recent dependencies, especially if something like
JENKINS-47498 is adopted so that merely picking the latest parent POM
for a component will switch `<dependencyManagement>` to recent
`master` commits (or MRP-tagged releases). There will always be some
cases when an old line of development is suddenly resurrected and the
developer declines to start by refreshing dependencies:

mvn incrementals:update

Second, what is the impact of expiration? For builds on ci.jenkins.io,
possibly none: even though local repositories will typically start
empty, the build will download incremental releases from

https://repo.azure.jenkins.io/incrementals/

which (Olivier correct me if I am mistaken) uses an LRU cache.
Therefore even an old dependency will remain in the cache indefinitely
if it continues to be used actively in builds. You could still get a
broken build in cases like that mentioned above, where the old
dependency is used by a hitherto-dormant branch of some downstream
component that has not gotten a refresh.

Third, can the release be reconstructed? For use from local developer
builds, pretty easily: just check out the specified commit of the
upstream component and

mvn -DskipTests -Dset.changelist clean install

For use from CI builds, currently there is no mechanism in place to
re`deploy` old incremental releases on demand, such as jitpack.io
provides. One idea however is that we could copy back artifacts still
present in the Azure mirror, or alternately decline to GC artifacts
known to be in that mirror. We could also scan for incremental release
versions textually mentioned in reference heads of various
repositories (akin to “conservative” GC in some programming
languages).

Either way, incremental releases are if anything _safer_ to recreate
from sources than MRP-tagged releases: a tag is easily overwritten, or
just lost during an incomplete `git-clone`, whereas a 12-digit commit
hash prefix is forgeable only with deliberate malice. Note that
incremental releases always deploy a flattened POM containing an
`<scm>` section with a valid `<url>` and `<tag>` (actually full commit
hash, not technically a “tag”, but then neither is `HEAD`).

> the impact this will have
> on other pipelines that do canary builds by picking up the latest *release*
> of a artifacts/plugins

Do not run `mvn versions:use-latest-releases` if your build
configuration refers to the official Incrementals repository or a
mirror thereof. Use `incrementals:update` if you wish to get
incremental updates (this is more nuanced—pays attention to Git
topology), or just omit this repository if you do not. (Currently
governed by the presence of `-Pconsume-incrementals` in
`.mvn/maven.config`.)

> in order to do bleeding edge testing in our
> pipelines

Of course a reasonable interpretation of “bleeding-edge testing” is
that you _do_ want incremental versions!

> the ability once it is in our mirror
> for things to accidentally depend on these releases.

JENKINS-50804 if you are concerned. The most obvious harm would be if
a tagged release of a downstream plugin were to express a dependency
on an incremental release of an upstream plugin, since the resulting
update center would then be internally inconsistent if it were built
from the Nexus index of the traditional release repository. Of course
this can already happen quite easily with upstream release versions
marked `-alpha` or `-beta`, or in cases where the upstream plugin is
excluded from the UC. The best solution is for the UC generation code
to actually perform a consistency check before publishing
anything—that seems like the very least we could do to prevent
accidents.

> we believe that you can accomplish this with
> SNAPSHOTS.
> You do not need to use timestamped snapshots (which is the main objection in
> JEP-305 against SNAPSHOTS)

Timestamped snapshots are indeed especially problematic, but snapshots
generally do not match the use case for Incrementals. Besides the
practical annoyance of Maven checking daily for updates which should
not be there (we forbid redeployment), philosophically it does not
follow what I think should be the distinction between snapshots and
releases.

A release is something which is reconstructible from a specific
commit. It is a globally identifiable version of that component which
you may in turn refer to in other commits and be assured that other
developers or CI systems will not encounter ambiguity. You can create
permalinks to its sources.

A snapshot might be built from a commit, or a commit with some local
modifications, or even from an unversioned source tree—we do not know
or care. You may refer to a `*-SNAPSHOT` version temporarily in the
course of testing some changes locally, but it should never be
deployed to a public server, nor should such a version ever be
mentioned in a committed POM/manifest/whatever. (Just like you should
never refer to `LATEST`/`RELEASE` or a branch name.)

An incremental release is just a kind of release that is friendlier to
CI/CD. It is marked as a “release candidate” since every commit is
indeed a candidate for deployment. The selection of which versions are
actually used in production after running a gantlet of tests is just
left to a different layer of tooling, such as a versioned bill of
materials, rather than the awkward dance of MRP and (worse) staging
repositories.

Baptiste Mathus

unread,
May 17, 2018, 5:09:11 AM5/17/18
to Jenkins Developers
Le mer. 16 mai 2018 à 13:14, James Nord <jn...@cloudbees.com> a écrit :
So the objection I have is that as it stands in JEP-305 the versions are releases and these are expected to be garbage collected.

Anyone also wanting to consume incrementals via a proxy would need to write tooling to garbage collect releases and there is also the Maven principal that you never garbage collect releases, let alone the impact this will have on other pipelines that do canary builds by picking up the latest *release* of a artifacts/plugins in order to do bleeding edge testing in our pipelines. And that is also discounting the ability once it is in our mirror for things to accidentally depend on these releases.

I must confess I had missed reading related IEP-9 when reading JEP-305. I do agree releases should normally be kept basically forever.
Now nowadays Nexus, for one, had added a dedicated Scheduled Task to clean up also releases "Remove Releases From Repository" IIUC (I never used it myself, but did use a lot the other "Remove Snapshots From Repository"). 

So yes, this shows IMO a pretty usage of Maven, but it shows at least people using Nexus (and I suspect Artifactory can do the same) would normally *not* have to handle that themselves.

Now, about the whole artifact collection idea that I had missed:

* I think this part in IEP-9 could be just ditched overall, for now at least. As Jesse nicely phrases it below: "until storage costs start to raise eyebrows". IIUC what Daniel told me once, as it stands currently: anyway we never ever remove *any* artifacts from Artifactory, SNAPSHOTs are already kept forever. So I guess the eyebrow would have raised long ago if this was an issue. OK, this could grow possibly faster once Incrementals starts to get used more and more, we /might/ want to revisit then.
** (A possible way to revisit could for instance to better differentiate PRs from merged commits, and maybe more aggressively actually clean up artifacts coming from PRs).
* Anyway, people using artifacts for in-house purposes would (should) have put an MRM in place. So they wouldn't even see any issue if an artifact is ever deleted upstream.
* We must indeed clearly document that people using Incrementals releases and repository must understand they're adopting a special pattern, and I guess as this is indeed not exposed automatically behing /public or /releases, this is an explicit action that one must not do lightly.
  So BTW I think https://github.com/jenkins-infra/iep/blob/master/iep-009/README.adoc#existing-maven-repository ought to be completed with a precision saying we *really* must not ever expose http://repo.jenkins-ci.org/incrementals behind http://repo.jenkins-ci.org/public or some other virtual group repository. Keep it separated so that it requires an explicit thinking/action to get used.

-- Baptiste

James Nord

unread,
May 17, 2018, 5:45:50 AM5/17/18
to Jenkins Developers


On Wednesday, May 16, 2018 at 5:16:56 PM UTC+1, Jesse Glick wrote:
My response would mostly follow Daniel’s but I will add more
background—something I can write up into a subsection of

https://github.com/jenkinsci/jep/blob/master/jep/305/README.adoc#other-version-schemes

once the dust settles.

James Nord & Stephen Connolly wrote:
> as it stands in JEP-305 the versions are
> releases and these are expected to be garbage collected.

Technically this is from IEP-9, but as that attracted little notice
(indeed I only stumbled across it after it was finalized and
implemented), it makes sense to discuss in the context of JEP-305.


So this is the big sticking point and what Stephen and I are mostly concerned about.
If that little "gem" was to be removed then I think all other issues (except for disk usage!) would just fall by the wayside (and you can skip the rest of this message and get some time back to <insert favourite passtime here>)

 
> Anyone also wanting to consume incrementals via a proxy would need to write
> tooling to garbage collect releases

I think GC is the crux of the question.

Yes it is.

 
For the record, I would have
recommended a rather more conservative collection schedule in IEP-9
(basically keep everything until storage costs start to raise
eyebrows); I am not actually sure whether the collection scheme was
even implemented, or if so, whether this is custom tooling or some
stock feature of Artifactory. Daniel could speak to that.

There are several things to consider about GC, and none have anything
to do with whether the version number happens to end in `-SNAPSHOT` or
not, so adding that suffix solves no practical problem.

Actually it does, as SNAPSHOTs are expected be be garbage collected, releases are not, and that is fundamental to the Maven philosophy.
 

First, how likely is it that a newly run build of a downstream
component will refer to an incremental version that has been expired
from Artifactory? This is hard to know until JEP-305 gets broader
usage, but my expectation would be that new builds will usually refer
to pretty recent dependencies, especially if something like
JENKINS-47498 is adopted so that merely picking the latest parent POM
for a component will switch `<dependencyManagement>` to recent
`master` commits (or MRP-tagged releases). There will always be some
cases when an old line of development is suddenly resurrected and the
developer declines to start by refreshing dependencies:

mvn incrementals:update

Second, what is the impact of expiration? For builds on ci.jenkins.io,
possibly none: even though local repositories will typically start
empty, the build will download incremental releases from

https://repo.azure.jenkins.io/incrementals/

which (Olivier correct me if I am mistaken) uses an LRU cache.
Therefore even an old dependency will remain in the cache indefinitely
if it continues to be used actively in builds. You could still get a
broken build in cases like that mentioned above, where the old
dependency is used by a hitherto-dormant branch of some downstream
component that has not gotten a refresh.

Ahh so now you have the debuging environments when things behave a little differently.  Ie a unit test fails on CI, and you can not even build the thing locally anymore.  This is not a good thing for a CI server, and you have now replaced "it works on my machine" with "well it works on CI"


Third, can the release be reconstructed? For use from local developer
builds, pretty easily: just check out the specified commit of the
upstream component and

mvn -DskipTests -Dset.changelist clean install

Alas I am Skeptical about this.  For exapmple say I depended on BlueOcean.
If I tried to check that out and build it I am left with something that is conmpletely dead on arival as it produces garbage on windows.And the argument that the CI also does a build on windows doesn't mean that that build is any good, BO builds on windows, and the unit tests pass, but the result is garbage, so unless you run the ATH on windows and Linux (which you don't)

I actually hit a bug on a closed source plugin yesterday that the build failed on windows (that dreaded npm..)

You may also at points be able to replace Windows with Mac or some Linux (or even a specific version of the JDK - which has also caused builds to break in the past, or even maven).  incrementals does not fix the environment so reproducability on a different environment is never guaranteed (and hence why you should not mess with releases because whilst you hope the build is reproducable you are probably not recording everything you need to know about the env in order to guarantee it)

 
For use from CI builds, currently there is no mechanism in place to
re`deploy` old incremental releases on demand, such as jitpack.io
provides. One idea however is that we could copy back artifacts still
present in the Azure mirror, or alternately decline to GC artifacts
known to be in that mirror. We could also scan for incremental release
versions textually mentioned in reference heads of various
repositories (akin to “conservative” GC in some programming
languages).

or just not delete them... 
 
Either way, incremental releases are if anything _safer_ to recreate
from sources than MRP-tagged releases: a tag is easily overwritten, or
just lost during an incomplete `git-clone`, whereas a 12-digit commit
hash prefix is forgeable only with deliberate malice.

Is the CI the only place that can deploy them? (I don't recall but it may well be).
If building an incremental from a PR then the hash is likely not enough (as that may/will not be pushed back upstream)

 
Note that
incremental releases always deploy a flattened POM containing an
`<scm>` section with a valid `<url>` and `<tag>` (actually full commit
hash, not technically a “tag”, but then neither is `HEAD`).

> the impact this will have
> on other pipelines that do canary builds by picking up the latest *release*
> of a artifacts/plugins

Do not run `mvn versions:use-latest-releases` if your build
configuration refers to the official Incrementals repository or a
mirror thereof.

The problem is when it is consumed in mirrors downstream.  Many people will just have a corporate single URL and many people (maven repo admins) when asked to add a mirror of a upstream release will just add it in to the main mirror - which will as you say likely break builds that are already using tooling.

 
Use `incrementals:update` if you wish to get
incremental updates (this is more nuanced—pays attention to Git
topology), or just omit this repository if you do not. (Currently
governed by the presence of `-Pconsume-incrementals` in
`.mvn/maven.config`.)

> in order to do bleeding edge testing in our
> pipelines

Of course a reasonable interpretation of “bleeding-edge testing” is
that you _do_ want incremental versions!

for some definitions of bleeding edge...  
think of a slightly less bleeding edge if you will.

 
> the ability once it is in our mirror
> for things to accidentally depend on these releases.

JENKINS-50804 if you are concerned. The most obvious harm would be if
a tagged release of a downstream plugin were to express a dependency
on an incremental release of an upstream plugin, since the resulting
update center would then be internally inconsistent if it were built
from the Nexus index of the traditional release repository. Of course
this can already happen quite easily with upstream release versions
marked `-alpha` or `-beta`, or in cases where the upstream plugin is
excluded from the UC. The best solution is for the UC generation code
to actually perform a consistency check before publishing
anything—that seems like the very least we could do to prevent
accidents.

> we believe that you can accomplish this with
> SNAPSHOTS.
> You do not need to use timestamped snapshots (which is the main objection in
> JEP-305 against SNAPSHOTS)

Timestamped snapshots are indeed especially problematic, but snapshots
generally do not match the use case for Incrementals. Besides the
practical annoyance of Maven checking daily for updates which should
not be there (we forbid redeployment), philosophically it does not
follow what I think should be the distinction between snapshots and
releases.

You can configure maven to check as you want on a per repo basis.
ie in the repo definition in you pom use never (same as releases) so only new versions are looked for.

in a mirror nexus (I would also guess artifactory etc) you can also set the check time (and again set it to have an infinite not found cache.

 
A release is something which is reconstructible from a specific
commit. It is a globally identifiable version of that component which
you may in turn refer to in other commits and be assured that other
developers or CI systems will not encounter ambiguity. You can create
permalinks to its sources.

Nah.  this is FUD.  
I can create a release can call it whatever I want and deploy it (subject to access control on the repo server).

if you make it such that only the CI can deploy these so you can ensure that your above statement is true, then you can by inference also do it for SNAPSHOTs.


A snapshot might be built from a commit, or a commit with some local
modifications, or even from an unversioned source tree—we do not know
or care. You may refer to a `*-SNAPSHOT` version temporarily in the
course of testing some changes locally, but it should never be
deployed to a public server, nor should such a version ever be
mentioned in a committed POM/manifest/whatever. (Just like you should
never refer to `LATEST`/`RELEASE` or a branch name.)

An incremental release is just a kind of release that is friendlier to
CI/CD. It is marked as a “release candidate” since every commit is
indeed a candidate for deployment. The selection of which versions are
actually used in production after running a gantlet of tests is just
left to a different layer of tooling, such as a versioned bill of
materials, rather than the awkward dance of MRP and (worse) staging
repositories.

Erm, so if you are making these release canditates pass the full suite, you are then using them in production via promotion, but the version in them is still a release candidate?  the above is just confusing?



 

R. Tyler Croy

unread,
May 17, 2018, 10:18:18 AM5/17/18
to jenkin...@googlegroups.com
(replies inline)

On Thu, 17 May 2018, Baptiste Mathus wrote:

> Le mer. 16 mai 2018 à 13:14, James Nord <jn...@cloudbees.com> a écrit :
>
> > So the objection I have is that as it stands in JEP-305 the versions are
> > releases and these are expected to be garbage collected.
> >
> > Anyone also wanting to consume incrementals via a proxy would need to
> > write tooling to garbage collect releases and there is also the Maven
> > principal that you never garbage collect releases, let alone the impact
> > this will have on other pipelines that do canary builds by picking up the
> > latest **release** of a artifacts/plugins in order to do bleeding edge
> > testing in our pipelines. And that is also discounting the ability once it
> > is in our mirror for things to accidentally depend on these releases.
> >
> I must confess I had missed reading related IEP-9 when reading JEP-305. I
> do agree releases should normally be kept basically forever.
> Now nowadays Nexus, for one, had added a dedicated Scheduled Task
> <https://help.sonatype.com/repomanager2/configuration/managing-scheduled-tasks>
> to
> clean up also releases "Remove Releases From Repository" IIUC (I never used
> it myself, but did use a lot the other "Remove Snapshots From Repository").
>
> So yes, this shows IMO a pretty usage of Maven, but it shows at least
> people using Nexus (and I suspect Artifactory can do the same) would
> normally *not* have to handle that themselves.
>
> Now, about the whole artifact collection idea that I had missed:
>
> * I think this part in IEP-9 could be just ditched overall, for now at
> least. As Jesse nicely phrases it below: "until storage costs start to
> raise eyebrows". IIUC what Daniel told me once, as it stands currently:
> anyway we never ever remove *any* artifacts from Artifactory, SNAPSHOTs are
> already kept forever. So I guess the eyebrow would have raised long ago if
> this was an issue. OK, this could grow possibly faster once Incrementals
> starts to get used more and more, we /might/ want to revisit then.
> ** (A possible way to revisit could for instance to better differentiate
> PRs from merged commits, and maybe more aggressively actually clean up
> artifacts coming from PRs).


There is no garbage collection currently implemneted with IEP-9.

There is simply no comparable usage today of our Artifactory environment in
terms of capacity planning. While we may not remove SNAPSHOTs today,
ci.jenkins.io isn't publishing those. And in fact, nothing compares to
publishing artifacts for every single pull request of every plugin (400+),
libraries (a few dozen), and core in ci.jenkins.io.

We're looking at many hundreds of megabytes of data per day uploading into
Artifactory, a hosted solution with somewhat unclear limtis (we basically will
fill it up until JFrog says "hey wtf m8"").


So basically, I'm not concerned with what "Maven best practices" prescribe, I
do not want the expectation to be that incremental built releases are going to
be kept indefinitely. Only so much disk space on the planet :)


Cheers

signature.asc

Jesse Glick

unread,
May 17, 2018, 1:32:30 PM5/17/18
to Jenkins Dev
On Thu, May 17, 2018 at 5:45 AM, James Nord <jn...@cloudbees.com> wrote:
> say I depended on BlueOcean.
> If I tried to check that out and build it I am left with something that is
> conmpletely dead on arival as it produces garbage on windows.

Not sure what that is about, but I guess someone should be fixing it?

> Is the CI the only place that can deploy them?

Daniel would know the actual authorization settings for the
repository, but CI (or, technically, an Azure function triggered by
CI) is the only thing that is _expected_ to deploy these artifacts.

> If building an incremental from a PR then the hash is likely not enough (as
> that may/will not be pushed back upstream)

You can check out a commit hash from a fork. GitHub does not really care.

> many people (maven repo admins) when
> asked to add a mirror of a upstream release will just add it in to the main
> mirror

Well, tell them not to?

> I can create a release can call it whatever I want and deploy it (subject to
> access control on the repo server).

Of course that is physically _possible_, just as deleting release
artifacts is physically possible. :-/ Maven does not enforce these
things.

> so if you are making these release canditates pass the full suite, you
> are then using them in production via promotion, but the version in them is
> still a release candidate? the above is just confusing?

There needs to be a consistent version number the whole way through
the Pipeline, so “RC” is as good as any. It would be more confusing
(and far harder) to retroactively relabel the same artifact with
another version.

Daniel Beck

unread,
May 21, 2018, 5:41:48 PM5/21/18
to jenkin...@googlegroups.com


> On May 16, 2018, at 6:16 PM, Jesse Glick <jgl...@cloudbees.com> wrote:
>
> I am not actually sure whether the collection scheme was
> even implemented, or if so, whether this is custom tooling or some
> stock feature of Artifactory. Daniel could speak to that.

No such feature I’m aware of. There’s configurable snapshot retention but not for releases because of that Maven philosophy :-) And we’ve not needed the feature for snapshots since we don’t use those all that often.

That said, Artifactory has a mostly nice API, and also tracks # of downloads and date of last download of any given artifact, so we can rather easily implement GC taking that into account — the simplest one, just nuking whatever is >N days old and has never been downloaded, might be pretty safe and very effective, if my expectation that most incrementals builds will never be downloaded is correct.

> Of course
> this can already happen quite easily with upstream release versions
> marked `-alpha` or `-beta`, or in cases where the upstream plugin is
> excluded from the UC. The best solution is for the UC generation code
> to actually perform a consistency check before publishing
> anything—that seems like the very least we could do to prevent
> accidents.

We’ve already had that when plugins were blacklisted for security reasons, and forgot to check its dependencies. Not to mention unsatisfiable dependencies on tiered update sites (a dependency needs a newer core than the current tier), etc.

It should be noted however that the runtime of update center generation is very easy to increase by orders of magnitude unless implemented carefully, so a real solution ensuring consistency of the created JSON (in a manner not just failing the build if inconsistent) probably requires something approaching a rewrite.


Jesse Glick

unread,
May 21, 2018, 5:57:21 PM5/21/18
to Jenkins Dev
On Mon, May 21, 2018 at 5:41 PM, Daniel Beck <m...@beckweb.net> wrote:
> we can rather easily implement GC taking that into account — the simplest one, just nuking whatever is >N days old and has never been downloaded

Might work well. Again you need to keep in mind that the Azure mirror
can mask the real download count and date. But if your only criterion
is whether the artifact has _ever_ been downloaded, that is not a
problem.

> if my expectation that most incrementals builds will never be downloaded

Another thought I had was to turn off the automatic publishing
(`infra.maybePublishIncrementals()`) and implement some kind of bot
that would just listen for comments on a PR. That would not work to
publish `master` commits, FWIW.

Jesse Glick

unread,
Jun 1, 2018, 8:51:32 AM6/1/18
to Jenkins Dev
Another thing that might help—turn off unconditional deployment of PR
builds, and enable it only when a comment of a particular format is
added to the PR, to be applied to the chronologically last commit if
that indeed gets built and the build passes. I am not sure exactly how
that would be implemented—I suppose an Azure function triggered by
GitHub webhook events? (Comment added, if the build is already
complete; or build status set, if the comment was added earlier.)

That leaves open the question of deployment of non-PR builds, though,
such as the commit after a merge to `master`. One possible answer is
that we do not care about those anyway—after all, `mvn
incrementals:update` will select a topologically latest commit that is
an ancestor of (by default) `master`, so if the PR was merged normally
(_not_ rebase or squash!), the PR commit will be a candidate. (As will
traditional releases, from MRP.) Whether we need to worry about the
heads from two merged PRs being tied for latest will depend on whether
or not we have protected branches blocking a merge of a PR which is
not up to date. So if you had PR #1 merged and then PR #2 without PR
#2 first being brought up to date

/-PR#1(1)-\
M1 M2 M3
\-PR#2(1)-----/

then one of PR #1 (1) and PR #2 (1) will be considered “latest”,
arbitrarily, since none of M1/M2/M3 have been deployed. However if we
require PR #2 to be brought up to date before merging


/-PR#1(1)-\
M1 M2 M3
\-PR#2(1)----\-PR#2(2)-/

then PR #2 (2) will be latest, as expected.

Another possible answer is that we continue to unconditionally deploy
all non-PR (i.e., origin branch) builds, but this then reduces the
benefit of the change—you will still get a lot of builds deployed
whether anyone wanted them or not.

James Nord

unread,
Jun 7, 2018, 9:37:36 AM6/7/18
to Jenkins Developers
Sorry for the late reply I have been busy on other things and didn't notice this.


On Thursday, May 17, 2018 at 6:32:30 PM UTC+1, Jesse Glick wrote:
On Thu, May 17, 2018 at 5:45 AM, James Nord <jn...@cloudbees.com> wrote:
> say I depended on BlueOcean.
> If I tried to check that out and build it I am left with something that is
> conmpletely dead on arival as it produces garbage on windows.

Not sure what that is about, but I guess someone should be fixing it?


It was your comment about well you can just rebuilt X as you have the SHA of it.
Basically I am saying that your statement above can not be relied upon.  there are plugins that require certain OSes to build (in the case of blue ocean) (heck there may be some that are windows only!)
whilst it may be a bug that plugin X only builds on Y, it may also be completely legit, or too much effort to fix.
 
> Is the CI the only place that can deploy them?

Daniel would know the actual authorization settings for the
repository, but CI (or, technically, an Azure function triggered by
CI) is the only thing that is _expected_ to deploy these artifacts.

> If building an incremental from a PR then the hash is likely not enough (as
> that may/will not be pushed back upstream)

You can check out a commit hash from a fork. GitHub does not really care.

But when we build PRs we likely build it merged with the origin target branch thus github has no reference of said hash.  Just because you have a hash does not mean it is available anywhere but locally.
and even if you pushed it, without a ref it could be garbage collected.
 
> many people (maven repo admins) when
> asked to add a mirror of a upstream release will just add it in to the main
> mirror

Well, tell them not to?

I did - hence what focussed myself and Stephen on this Thread :-)
but only because we went looking (I do not expect everyone to be that diligent :-) )
 
> I can create a release can call it whatever I want and deploy it (subject to
> access control on the repo server).

Of course that is physically _possible_, just as deleting release
artifacts is physically possible. :-/ Maven does not enforce these
things.

> so if you are making these release canditates pass the full suite, you
> are then using them in production via promotion, but the version in them is
> still a release candidate?  the above is just confusing?

There needs to be a consistent version number the whole way through
the Pipeline, so “RC” is as good as any. It would be more confusing
(and far harder) to retroactively relabel the same artifact with
another version.

What I was trying to say is that given you may well end up running this version in production (that's your end goal AFAIU) then why not drop the "rc" from the version suffix? 
or to take it to the extreeme - eventually in production I think you are expecting everything to be an rc version (jenkins, plugins the whole works).

That is don't relabel.  I guess there was probably a reason that I have forgotten (the fact that the rc makes maven think it is older than a non rc version)

Jesse Glick

unread,
Jun 7, 2018, 2:21:44 PM6/7/18
to Jenkins Dev
On Thu, Jun 7, 2018 at 9:37 AM, James Nord <jn...@cloudbees.com> wrote:
> whilst it may be a bug that plugin X only builds on Y, it may also be
> completely legit, or too much effort to fix.

I would argue it is never legitimate, though it may be too much effort
to fix. If someone is blocked from building a given plugin (especially
in Essentials) because of operating system issues, then we should
either be investing in containerized tools, or moving some of the
complexity out of the build process and into runtime code. That would
presumably apply to Blue Ocean. If you are talking about a relatively
obscure plugin with one maintainer and a thousand installs, build
reproducibility is indeed liable to be poor.

>> You can check out a commit hash from a fork. GitHub does not really care.
>
> But when we build PRs we likely build it merged with the origin target
> branch thus github has no reference of said hash.

Ah but such builds do not deploy to Incrementals anyway—only when the
PR is fully synchronized with the base branch. (The Jenkins build will
produce artifacts even from an ephemeral merge commit, but the
deployment “function” will reject them and exit early.)

Anyway no one would be referring to an ephemeral merge commit via
version number even if they _were_ deployed, since the ways you might
pick an incremental release version to add to a downstream POM would
be:

· Because you just committed to the upstream repository and expect to
push it and have it be deployed.
· Because you ran `mvn incrementals:update`, which verifies Git(Hub)
DAG ancestry of candidate versions.

> given you may well end up running this
> version in production […] then why not drop the
> "rc" from the version suffix?

Well there is a practical consideration of version number sorting.

1.0 < 1.1-rcNNN.XXXXXXXXXXXX < 1.1 < 1.1.NNN.XXXXXXXXXXXX

so it is deliberate that `-Dset.changelist` switches `1.1-SNAPSHOT` to
something which sorts in the same interval between `1.0` and `1.1`. I
am unsure exactly where actual snapshot versions should sit; if you
know, please advise me in

https://github.com/jenkinsci/lib-version-number/pull/6

since the sorting in Jenkins core is a bit of a mess.

> eventually in production I think you are
> expecting everything to be an rc version (jenkins, plugins the whole works).

TBD. Essentials could define a policy such as: if a plugin commit is
pushed to production (via incremental version) and then some time
period (a month?) elapses with no further changes, it is assumed that
no significant issues were detected by telemetry that required
rollback or fix-forward, so a traditional (MRP) release should be cut
so that even users of the traditional update center will see those
bits.

For Jenkins core, we have traditionally done a weekly release
regardless of what is in it, so if we keep that up, then sometimes the
weekly MRP version would be deployed also to Essentials (since no one
has bothered to push anything since Sunday). For core components like
Remoting, I suppose we would want an MRP version cut prior to the core
weekly if an incremental version were present in the core `master`
POMs, though the process has not yet been nailed down.
Reply all
Reply to author
Forward
0 new messages