Support for experimental (beta) releases on JEP-229

[José Lamas Ríos]

Hi,

I've switched to the CD implemented on JEP-229 for genexus-plugin (https://github.com/jenkinsci/genexus-plugin) and couldn't find a way to publish experimental releases, as I used to do based on what's described at https://www.jenkins.io/doc/developer/publishing/releasing-experimental-updates/

First tried including "-beta" in the changelist property in pom.xml

<changelist>999999-beta-7-SNAPSHOT</changelist>

The first release was successful

( https://github.com/jenkinsci/genexus-plugin/releases/tag/250.v1b0d8b7f9a4a ) but got published as version 250.v1b0d8b7f9a4a , that is, not including the "-beta" qualifier and thus became available at the standard update center ( https://updates.jenkins.io/update-center.json )  instead of the intended experimental one ( https://updates.jenkins.io/experimental/update-center.json )

Then I tried moving the "-beta" qualifier to the <version> itself (see https://github.com/jenkinsci/genexus-plugin/commit/fbb5777ce84f5c8a4dfe51ddd5dc3166fc85e06e#diff-9c5fb3d1b7e3b0f54bc5c4182965c4fe1f9023d449017cece3005d3f90e8e4d8 )

As you can see on the build log at https://ci.jenkins.io/blue/rest/organizations/jenkins/pipelines/Plugins/pipelines/genexus-plugin/branches/master/runs/115/log/?start=0 the -beta qualifier was present when installing the genexus.hpi file:

[2021-11-05T22:08:42.395Z] [INFO] --- maven-install-plugin:3.0.0-M1:install (default-install) @ genexus ---

[2021-11-05T22:08:42.395Z] [INFO] Installing C:\Jenkins\workspace\Plugins_genexus-plugin_master\target\genexus.hpi to C:\Jenkins\workspace\Plugins_genexus-plugin_master@tmp\m2repo\org\jenkins-ci\plugins\genexus\251.vfbb5777ce84f-beta\genexus-251.vfbb5777ce84f-beta.hpi

[2021-11-05T22:08:42.395Z] [INFO] Installing C:\Jenkins\workspace\Plugins_genexus-plugin_master\target\genexus-251.vfbb5777ce84f-beta.pom to C:\Jenkins\workspace\Plugins_genexus-plugin_master@tmp\m2repo\org\jenkins-ci\plugins\genexus\251.vfbb5777ce84f-beta\genexus-251.vfbb5777ce84f-beta.pom

[2021-11-05T22:08:42.395Z] [INFO] Installing C:\Jenkins\workspace\Plugins_genexus-plugin_master\target\genexus.jar to C:\Jenkins\workspace\Plugins_genexus-plugin_master@tmp\m2repo\org\jenkins-ci\plugins\genexus\251.vfbb5777ce84f-beta\genexus-251.vfbb5777ce84f-beta.jar

[2021-11-05T22:08:42.395Z] [INFO] Installing C:\Jenkins\workspace\Plugins_genexus-plugin_master\target\genexus-sources.jar to C:\Jenkins\workspace\Plugins_genexus-plugin_master@tmp\m2repo\org\jenkins-ci\plugins\genexus\251.vfbb5777ce84f-beta\genexus-251.vfbb5777ce84f-beta-sources.jar

[2021-11-05T22:08:42.395Z] [INFO] Installing C:\Jenkins\workspace\Plugins_genexus-plugin_master\target\genexus-javadoc.jar to C:\Jenkins\workspace\Plugins_genexus-plugin_master@tmp\m2repo\org\jenkins-ci\plugins\genexus\251.vfbb5777ce84f-beta\genexus-251.vfbb5777ce84f-beta-javadoc.jar

But then later, when trying to archive it as artifact, it failed to find it because it didn't expect it to contain that "-beta" qualifier:

[2021-11-05T22:09:38.560Z] Archiving artifacts [2021-11-05T22:09:39.220Z] java.lang.InterruptedException: no matches found within 10000

[...]

[2021-11-05T22:09:39.220Z] No artifacts found that match the file pattern "**/*251.vfbb5777ce84f/*251.vfbb5777ce84f*". Configuration error?

Is publising experimental releases supported? I'm not sure I'm doing it the way it's supposed to be done...

That "**/*${changelist}/*${changelist}*" pattern does include a trailing wildcard after the filename, so I guess the problem might be fixed adding a wildcard after the folder too (that is, changing the pattern to ""**/*${changelist}*/*${changelist}*"") ?

Thanks in advance for any advice,

Regards,

-- 

jlr

 

[José Lamas Ríos]

The pattern is at https://github.com/jenkins-infra/pipeline-library/blob/67b51a985f803bf2ed6bab67fa6764e4d7dfc956/vars/buildPlugin.groovy#L233

archiveArtifacts artifacts: "**/*$changelist/*$changelist*",

Should it be changed to this?

archiveArtifacts artifacts: "**/*$version/*$version*",

For instance, I can see that at https://github.com/jenkins-infra/pipeline-library/blob/67b51a985f803bf2ed6bab67fa6764e4d7dfc956/vars/infra.groovy#L306 it does use $version instead of $changelist

Given that the install is done based on $version it does seem that the archive should use $version too (rather than assuming $version == $changelist), but then again I'm not sure wether it's ok to change the version definition the way I'm doing it, or how experimental releases should be done.

[Jesse Glick]

The system is not designed to support marking releases as beta. I am not sure such a requirement is even conceptually compatible with the style of automatic deployment. If such a requirement is common, I think it would need to be done differently, as a true promotion: the same version number and binary artifact would initially be published to the experimental update center and then subsequently marked somehow as eligible for the main update center.

Consider skipping the experimental update center and gating new features behind a flag that users can opt into. If only a select few users actually run the experimental releases to begin with, they can also obtain them from the incrementals repository—you can turn off automatic releasing of the default branch and trigger releases only with the manual workflow, while every successful trunk build will be deployed to the incrementals repository where it can be downloaded on demand for beta testing.

[José Lamas Ríos]

On Sunday, November 7, 2021 at 8:07:46 PM UTC-3 Jesse Glick wrote:

The system is not designed to support marking releases as beta. I am not sure such a requirement is even conceptually compatible with the style of automatic deployment.

Regarding 'conceptually compatible' I'm not sure whether you refer to automatic deployment in general or to the way it's implemented in this case, using incremental versions in particular.

 

If such a requirement is common,

Personally I find it practical to have a separate release channel for beta releases and its corresponding discovery and update simplicity for end users: just change the URL to be able to see possible updates and apply them through the UI), but not sure if it may be considered as common. 

The update channels currently list 1874 plugins and comparing the experimental at https://updates.jenkins.io/experimental/update-center.json with the standard  at https://updates.jenkins.io/current/update-center.json there are right now differences for around 30 plugins.

 

I think it would need to be done differently, as a true promotion: the same version number and binary artifact would initially be published to the experimental update center and then subsequently marked somehow as eligible for the main update center.

FWICS, the update center currently checks the version to exclude artifacts containing "alpha" or "beta" from the regular update center:

https://github.com/jenkins-infra/update-center2/blob/1690b80d6a92489c1be9ba51105c47ba4f3e5d9d/src/main/java/io/jenkins/update_center/MavenArtifact.java#L88

It also checks for "SNAPSHOT" and "JENKINS" to ignore those:

https://github.com/jenkins-infra/update-center2/blob/e72cebfa594356c94f0c40d310c956058439e576/src/main/java/io/jenkins/update_center/BaseMavenRepository.java#L56

But if keep adding qualifiers to versions is a problem, then yes, it would require moving those marks (and the update center code) somewhere else. 

 

Consider skipping the experimental update center and gating new features behind a flag that users can opt into.

Yes, feature flags may help on some particular scenarios but not for many beta processes.

 

If only a select few users actually run the experimental releases to begin with, they can also obtain them from the incrementals repository—you can turn off automatic releasing of the default branch and trigger releases only with the manual workflow, while every successful trunk build will be deployed to the incrementals repository where it can be downloaded on demand for beta testing.

Agree, that would be a work-around, but it does penalize both the CD experience for the developer and the discover and update experience for beta testers.

 

[Jesse Glick]

On Tue, Nov 9, 2021 at 3:06 PM José Lamas Ríos <j...@genexus.com> wrote:

Regarding 'conceptually compatible' I'm not sure whether you refer to automatic deployment in general or to the way it's implemented in this case

Well, any scheme by which the update center pattern-matches on plugin version numbers.

I think the most natural way to support JEP-229 in the experimental update center would be:

• Give `jenkins-maven-cd-action` an option to leave releases in draft after deployment.
• Patch `isAlphaOrBeta` in the UC code to look for a GitHub release matching the version number of the plugin. If it exists, and it is currently in draft, treat the plugin like it is beta.

Then if you enabled CD on your plugin and enabled the new option, your workflow would be

• Merge a PR with an appropriate label.
• Within the hour, it should appear on the experimental, but not regular, UC.
• Collect feedback from beta testers, pushing fixes or unrelated changes as you like.
• Whenever you are satisfied that a particular version (might have to be latest?) is safe, mark its release as ready in GitHub.
• It should then soon appear on the regular UC. (Anyone who already has it will not see an upgrade—the bits are unmodified.)

The first patch I could do pretty easily, if someone wanted to pick up the second patch. Looks like there is some GraphQL that could probably be extended, assuming Releases is supported.

[José Lamas Ríos]

On Tuesday, November 9, 2021 at 10:47:29 PM UTC-3 Jesse Glick wrote:

On Tue, Nov 9, 2021 at 3:06 PM José Lamas Ríos wrote:

Regarding 'conceptually compatible' I'm not sure whether you refer to automatic deployment in general or to the way it's implemented in this case

Well, any scheme by which the update center pattern-matches on plugin version numbers.

I think the most natural way to support JEP-229 in the experimental update center would be:

• Give `jenkins-maven-cd-action` an option to leave releases in draft after deployment.
• Patch `isAlphaOrBeta` in the UC code to look for a GitHub release matching the version number of the plugin. If it exists, and it is currently in draft, treat the plugin like it is beta.

I think that although this might work, it would be confusing. An experimental (alpha or beta) release is not the same thing as a release in draft state. I think 'draft' refers to the release process (we are preparing a release, and although we have a draft we haven't completed it yet), while 'experimental' refers to the content of the release (we did release it, but its content and functionality may not be fully validated).

Making the UC code be aware of the GitHub releases and consider their draft or final states also seems to be an undesirable coupling of so far independent mechanisms.

GitHub does allow marking releases as in draft state and also, as a separate concept, optionally marking them as 'pre-release', which in their words is a way "to notify users that the release is not ready for production and may be unstable".

https://docs.github.com/en/repositories/releasing-projects-on-github/managing-releases-in-a-repository

I may be wrong, but still fail to see why the use of JEP-229 may not be compatible with the publication of experimental releases, that can be a) exposed in a separate update center as usual and b) marked as "Pre-release" when published on GitHub.

In my previous experiment I had tried adding "-beta" to the <version> in the pom file and that failed when archiving the artifacts because it was using <changelist> instead <version> when looking for them. I think that changing that code to use <version> (ie: not taking as granted that version == changelist) would be more appropriate, but as an alternative I tried this:

• Reverted the pom to declare <version>${changelist}</version>
• Added instead the beta qualifier to the changelist.format in maven.config (ie: -Dchangelist.format=%d.%s-beta )
• Forced a new release.

The release was successful. As expected, it's not listed in the regular update center (https://updates.jenkins.io/update-center.json) but it does appear on the experimental one (https://updates.jenkins.io/experimental/update-center.json) as version 252.356d312df76f-beta

It also got published as a GitHub release using the same version identifier. However, since it wasn't declared as "pre-release", it showed there as "Latest", as a regular release.

I later manually fixed that through the web UI, but it would be nice that the jenkins-maven-cd-action also recognized 'beta' or 'alpha' qualifiers in the version identifier to take care of that, the same way it's done for the update center.

I think this would be easy to achieve by modifying its run.sh as follows:

gh api -X PATCH -F draft=false -F name=$version -F tag_name=$version /repos/$GITHUB_REPOSITORY/releases/$release

[[ $version = *alpha* || $version = *beta* ]] && isAlphaOrBeta=true || isAlphaOrBeta=false

gh api -X PATCH -F draft=false -F name=$version -F tag_name=$version -F prerelease=$isAlphaOrBeta /repos/$GITHUB_REPOSITORY/releases/$release

Does it make sense? It does seem simple and good enough for me, but I may be missing something.

[Jesse Glick]

On Wed, Nov 10, 2021 at 7:26 PM José Lamas Ríos <j...@genexus.com> wrote:

GitHub does allow marking releases as in draft state and also, as a separate concept, optionally marking them as 'pre-release'

You are right, pre-release status is what I was thinking of, not draft. I have not used this feature before but I presume you can retroactively remove that status by editing a release. Thus `jenkins-maven-cd-action` should have an option to mark the new release as pre-release (while still taking it out of draft status as today); and the UC generator should automatically limit a plugin version to the experimental UC when it can find an associated GH release and that release is marked as pre-release. The latter enhancement would also be a much friendlier way for users of `maven-release-plugin` to temporarily hold back wide distribution of a new version, without needing to run MRP all over again with a new version number.

In my previous experiment I had tried adding "-beta" to the <version> in the pom file…

It might be possible to make something like that work in the current scheme but it would certainly be a workaround. The intention of JEP-229 is that you should simply be able to merge PRs to functional code and have the resulting binaries be available, without ever touching the Maven `version` field after initial setup.