--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr3r7VDqUYBQ1PKf4jdiSmGmOYneaqi%3Dy3S-zm3T6%3D9wXA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Hello,Finally took some time to review the JEP 305, thanks a lot again Jesse for that big step forward, here's some feedback:* About using Maven CI Friendly Versions:
Anyone also wanting to consume incrementals via a proxy would need to write tooling to garbage collect releases and there is also the Maven principal that you never garbage collect releases, let alone the impact this will have on other pipelines that do canary builds by picking up the latest *release* of a artifacts/plugins in order to do bleeding edge testing in our pipelines. And that is also discounting the ability once it is in our mirror for things to accidentally depend on these releases.
So, in analysing the proposal we believe that you can accomplish this with SNAPSHOTS.
You do not need to use timestamped snapshots (which is the main objection in JEP-305) against SNAPSHOTS - you just add -SNAPSHOT on to the end of the version you already have (such that 1.7-rc1652.cd45427eb4e2 becomes {{1.7-rc1652.cd45427eb4e2-SNAPSHOT}
e.g. the equivalent command to generate the changelist setting becomes:
-Pproduce-incrementals -Dchangelist=-rc$(git rev-list --first-parent --count HEAD).$(git rev-parse --short=12 HEAD)-SNAPSHOT
then in the dependencies you depend on the xxx-SNAPSHOT and not the timestamped snapshot version which allows the version to be used from the local repo cache and not to have to download it from the hosted repo (artifaactory etc).
Because the changelist produced starts with the ("-rc" git rev list) it becomes monotonically increasing and ends with "-SNAPSHOT" we can garbage collect without any additional tooling. Because it ends in -SNAPSHOT it will not be picked up by canary builds or have the possibility of being included in releases, which will then allow us to include it in the mirror by default so there would be no need to change any settings.xml.
If you have a separate artifactory repo to have incremental snapshots then you can just scan this repo to find the latest versions for an update center. It may be possible in artifactory to also change the "snapshots" repo into a group and use routing rules to efficiently serve these whilst still maintaining compatability for users deploying snapshots with existing settings.
The change in the JEP of the version being $(git rev-parse --short=12 HEAD) to being $(version.removeEnd('-SNAPSHOT')}-rc$(git rev-list --first-parent --count HEAD).$(git rev-parse --short=12 HEAD) removed a lot of my spider-sense objections... but I think we need to have the deployed version include -SNAPSHOT to get something that aligns with the semantics of the incrementals repo from a Maven perspective
--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/315af002-b836-4480-b053-beed9d05eacf%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
So the objection I have is that as it stands in JEP-305 the versions are releases and these are expected to be garbage collected.Anyone also wanting to consume incrementals via a proxy would need to write tooling to garbage collect releases and there is also the Maven principal that you never garbage collect releases, let alone the impact this will have on other pipelines that do canary builds by picking up the latest *release* of a artifacts/plugins in order to do bleeding edge testing in our pipelines. And that is also discounting the ability once it is in our mirror for things to accidentally depend on these releases.
My response would mostly follow Daniel’s but I will add more
background—something I can write up into a subsection of
https://github.com/jenkinsci/jep/blob/master/jep/305/README.adoc#other-version-schemes
once the dust settles.
James Nord & Stephen Connolly wrote:
> as it stands in JEP-305 the versions are
> releases and these are expected to be garbage collected.
Technically this is from IEP-9, but as that attracted little notice
(indeed I only stumbled across it after it was finalized and
implemented), it makes sense to discuss in the context of JEP-305.
> Anyone also wanting to consume incrementals via a proxy would need to write
> tooling to garbage collect releases
I think GC is the crux of the question.
For the record, I would have
recommended a rather more conservative collection schedule in IEP-9
(basically keep everything until storage costs start to raise
eyebrows); I am not actually sure whether the collection scheme was
even implemented, or if so, whether this is custom tooling or some
stock feature of Artifactory. Daniel could speak to that.
There are several things to consider about GC, and none have anything
to do with whether the version number happens to end in `-SNAPSHOT` or
not, so adding that suffix solves no practical problem.
First, how likely is it that a newly run build of a downstream
component will refer to an incremental version that has been expired
from Artifactory? This is hard to know until JEP-305 gets broader
usage, but my expectation would be that new builds will usually refer
to pretty recent dependencies, especially if something like
JENKINS-47498 is adopted so that merely picking the latest parent POM
for a component will switch `<dependencyManagement>` to recent
`master` commits (or MRP-tagged releases). There will always be some
cases when an old line of development is suddenly resurrected and the
developer declines to start by refreshing dependencies:
mvn incrementals:update
Second, what is the impact of expiration? For builds on ci.jenkins.io,
possibly none: even though local repositories will typically start
empty, the build will download incremental releases from
https://repo.azure.jenkins.io/incrementals/
which (Olivier correct me if I am mistaken) uses an LRU cache.
Therefore even an old dependency will remain in the cache indefinitely
if it continues to be used actively in builds. You could still get a
broken build in cases like that mentioned above, where the old
dependency is used by a hitherto-dormant branch of some downstream
component that has not gotten a refresh.
Third, can the release be reconstructed? For use from local developer
builds, pretty easily: just check out the specified commit of the
upstream component and
mvn -DskipTests -Dset.changelist clean install
For use from CI builds, currently there is no mechanism in place to
re`deploy` old incremental releases on demand, such as jitpack.io
provides. One idea however is that we could copy back artifacts still
present in the Azure mirror, or alternately decline to GC artifacts
known to be in that mirror. We could also scan for incremental release
versions textually mentioned in reference heads of various
repositories (akin to “conservative” GC in some programming
languages).
Either way, incremental releases are if anything _safer_ to recreate
from sources than MRP-tagged releases: a tag is easily overwritten, or
just lost during an incomplete `git-clone`, whereas a 12-digit commit
hash prefix is forgeable only with deliberate malice.
Note that
incremental releases always deploy a flattened POM containing an
`<scm>` section with a valid `<url>` and `<tag>` (actually full commit
hash, not technically a “tag”, but then neither is `HEAD`).
> the impact this will have
> on other pipelines that do canary builds by picking up the latest *release*
> of a artifacts/plugins
Do not run `mvn versions:use-latest-releases` if your build
configuration refers to the official Incrementals repository or a
mirror thereof.
Use `incrementals:update` if you wish to get
incremental updates (this is more nuanced—pays attention to Git
topology), or just omit this repository if you do not. (Currently
governed by the presence of `-Pconsume-incrementals` in
`.mvn/maven.config`.)
> in order to do bleeding edge testing in our
> pipelines
Of course a reasonable interpretation of “bleeding-edge testing” is
that you _do_ want incremental versions!
> the ability once it is in our mirror
> for things to accidentally depend on these releases.
JENKINS-50804 if you are concerned. The most obvious harm would be if
a tagged release of a downstream plugin were to express a dependency
on an incremental release of an upstream plugin, since the resulting
update center would then be internally inconsistent if it were built
from the Nexus index of the traditional release repository. Of course
this can already happen quite easily with upstream release versions
marked `-alpha` or `-beta`, or in cases where the upstream plugin is
excluded from the UC. The best solution is for the UC generation code
to actually perform a consistency check before publishing
anything—that seems like the very least we could do to prevent
accidents.
> we believe that you can accomplish this with
> SNAPSHOTS.
> You do not need to use timestamped snapshots (which is the main objection in
> JEP-305 against SNAPSHOTS)
Timestamped snapshots are indeed especially problematic, but snapshots
generally do not match the use case for Incrementals. Besides the
practical annoyance of Maven checking daily for updates which should
not be there (we forbid redeployment), philosophically it does not
follow what I think should be the distinction between snapshots and
releases.
A release is something which is reconstructible from a specific
commit. It is a globally identifiable version of that component which
you may in turn refer to in other commits and be assured that other
developers or CI systems will not encounter ambiguity. You can create
permalinks to its sources.
A snapshot might be built from a commit, or a commit with some local
modifications, or even from an unversioned source tree—we do not know
or care. You may refer to a `*-SNAPSHOT` version temporarily in the
course of testing some changes locally, but it should never be
deployed to a public server, nor should such a version ever be
mentioned in a committed POM/manifest/whatever. (Just like you should
never refer to `LATEST`/`RELEASE` or a branch name.)
An incremental release is just a kind of release that is friendlier to
CI/CD. It is marked as a “release candidate” since every commit is
indeed a candidate for deployment. The selection of which versions are
actually used in production after running a gantlet of tests is just
left to a different layer of tooling, such as a versioned bill of
materials, rather than the awkward dance of MRP and (worse) staging
repositories.
On Thu, May 17, 2018 at 5:45 AM, James Nord <jn...@cloudbees.com> wrote:
> say I depended on BlueOcean.
> If I tried to check that out and build it I am left with something that is
> conmpletely dead on arival as it produces garbage on windows.
Not sure what that is about, but I guess someone should be fixing it?
> Is the CI the only place that can deploy them?
Daniel would know the actual authorization settings for the
repository, but CI (or, technically, an Azure function triggered by
CI) is the only thing that is _expected_ to deploy these artifacts.
> If building an incremental from a PR then the hash is likely not enough (as
> that may/will not be pushed back upstream)
You can check out a commit hash from a fork. GitHub does not really care.
> many people (maven repo admins) when
> asked to add a mirror of a upstream release will just add it in to the main
> mirror
Well, tell them not to?
> I can create a release can call it whatever I want and deploy it (subject to
> access control on the repo server).
Of course that is physically _possible_, just as deleting release
artifacts is physically possible. :-/ Maven does not enforce these
things.
> so if you are making these release canditates pass the full suite, you
> are then using them in production via promotion, but the version in them is
> still a release candidate? the above is just confusing?
There needs to be a consistent version number the whole way through
the Pipeline, so “RC” is as good as any. It would be more confusing
(and far harder) to retroactively relabel the same artifact with
another version.