Incrementals and ATH tests (inside a plugin CI)

13 views
Skip to first unread message

Ullrich Hafner

unread,
Aug 27, 2020, 3:07:09 PM8/27/20
to Jenkins Developers
My warnings plugin is using in our Jenkins CI build also UI tests (using the ATH container). Is there a way to start these UI tests (using the ATH container) so that depending plugins that have an incremental version are also resolved? Incrementals are resolved fine in integration tests but in UI tests the official update center is used which does not contain the incremental versions.


java.io.IOException: Failed to load: Warnings Next Generation Plugin (8.5.0-SNAPSHOT (private-2e85864e-ubuntu))
master49529| - Plugin is missing: forensics-api (0.8.0-rc587.65fc61fe9be2)

Oliver Gondža

unread,
Aug 27, 2020, 3:48:49 PM8/27/20
to jenkin...@googlegroups.com
I am not aware that any work have been done to support this.
> --
> You received this message because you are subscribed to the Google
> Groups "Jenkins Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to jenkinsci-de...@googlegroups.com
> <mailto:jenkinsci-de...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/jenkinsci-dev/10362FBD-68D4-4ADD-BEA9-267CB38E6EED%40gmail.com
> <https://groups.google.com/d/msgid/jenkinsci-dev/10362FBD-68D4-4ADD-BEA9-267CB38E6EED%40gmail.com?utm_medium=email&utm_source=footer>.


--
oliver

Jesse Glick

unread,
Aug 27, 2020, 4:46:33 PM8/27/20
to Jenkins Dev
On Thu, Aug 27, 2020 at 3:07 PM Ullrich Hafner <ullrich...@gmail.com> wrote:
> Incrementals are resolved fine in integration tests but in UI tests the official update center is used

I suspect this is just the most visible special case of a general
problem that the ATH tests are using the latest release at the time of
the test run, regardless of what versions your POM asks to test
against. This is a bad thing IMO: as one example, it means tests can
start failing in a PR which did not change sources, leading to
confusion. Verifying compatibility with recent upstream changes is of
course valuable but this can be accomplished more deterministically
with Dependabot.

Ullrich Hafner

unread,
Aug 27, 2020, 5:22:43 PM8/27/20
to Jenkins Developers
>
> I suspect this is just the most visible special case of a general
> problem that the ATH tests are using the latest release at the time of
> the test run, regardless of what versions your POM asks to test
> against. This is a bad thing IMO: as one example, it means tests can
> start failing in a PR which did not change sources, leading to
> confusion.

Yes, I had that problem a couple of weeks ago. One of my dependencies upgraded its Jenkins version to a higher version than the version I was using in my UI tests: that broke my build.

A possible workaround would be to use our plugin-installation-manager to download all dependencies in the correct version before the ATH container starts the Jenkins instance (the ATH already can load a set of pre-configured plugins). That would also have the benefit, that the startup time when running several ATH tests will be reduced as well.

Do we have something existing that can extract the list of dependencies for a plugin so that the plugin-installation-manager can be feeded with that input?

> Verifying compatibility with recent upstream changes is of
> course valuable but this can be accomplished more deterministically
> with Dependabot.
>
> --
> You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr3qetGKx9g1Rz6xLWL%2BB9SSeWeZm-8SR79HgGMXTyUJ-w%40mail.gmail.com.

Daniel Beck

unread,
Aug 28, 2020, 1:54:11 AM8/28/20
to JenkinsCI Developers
On Thu, Aug 27, 2020 at 11:22 PM Ullrich Hafner <ullrich...@gmail.com> wrote:

Yes, I had that problem a couple of weeks ago. One of my dependencies upgraded its Jenkins version to a higher version than the version I was using in my UI tests: that broke my build.

Do you still have the failing build log, or know when exactly this happened?

There was a bug with the update center tiers I introduced a few months ago that made the tiers not work quite correctly ( https://github.com/jenkins-infra/update-center2/pull/406 ) but I fixed that four weeks ago, so for that time it should have been working without problems.

Otherwise, the ATH (WithPlugins) should download plugins using the plugin manager, and that should only install compatible dependencies.

Ullrich Hafner

unread,
Aug 28, 2020, 4:22:30 AM8/28/20
to Jenkins Developers
Am 28.08.2020 um 07:52 schrieb Daniel Beck <db...@cloudbees.com>:


On Thu, Aug 27, 2020 at 11:22 PM Ullrich Hafner <ullrich...@gmail.com> wrote:

Yes, I had that problem a couple of weeks ago. One of my dependencies upgraded its Jenkins version to a higher version than the version I was using in my UI tests: that broke my build.

Do you still have the failing build log, or know when exactly this happened?

There was a bug with the update center tiers I introduced a few months ago that made the tiers not work quite correctly ( https://github.com/jenkins-infra/update-center2/pull/406 ) but I fixed that four weeks ago, so for that time it should have been working without problems.


Hmm, I don’t know the exact time and those builds are already deleted. But it was around 4 weeks ago as well. So maybe this was actually caused by the bug you are describing… 

Otherwise, the ATH (WithPlugins) should download plugins using the plugin manager, and that should only install compatible dependencies.

Since no builds fail anymore now I think ATH picks the right versions anyway. 

Jesse Glick

unread,
Aug 28, 2020, 11:21:32 AM8/28/20
to Jenkins Dev
On Thu, Aug 27, 2020 at 5:22 PM Ullrich Hafner <ullrich...@gmail.com> wrote:
> Do we have something existing that can extract the list of dependencies for a plugin…

I think it would suffice to inject a new implementation of
`UpdateCenterMetadataDecorator` using
`PluginMetadata.LocalOverride.create` based on
`target/{,test-classes/test-dependencies/}*.hpi`. Or simply use the
existing implementation and define the environment variable
`LOCAL_JARS` in your launcher script.

Plugins not mentioned in this list will still be retrieved according
to versions in the online UC (though by default using Maven to
retrieve local copies of the actual binaries). You could inject an
alternate `UpdateCenterMetadataProvider` to completely override that,
so that only dependencies in the test classpath are offered; this
would match the behavior of `JenkinsRule` (other than when annotated
by `@WithPlugin`).

Ullrich Hafner

unread,
Aug 28, 2020, 4:49:34 PM8/28/20
to Jenkins Developers
Am 28.08.2020 um 17:21 schrieb Jesse Glick <jgl...@cloudbees.com>:

On Thu, Aug 27, 2020 at 5:22 PM Ullrich Hafner <ullrich...@gmail.com> wrote:
Do we have something existing that can extract the list of dependencies for a plugin…

I think it would suffice to inject a new implementation of
`UpdateCenterMetadataDecorator` using
`PluginMetadata.LocalOverride.create` based on
`target/{,test-classes/test-dependencies/}*.hpi`. Or simply use the
existing implementation and define the environment variable
`LOCAL_JARS` in your launcher script.


This is a beautiful solution! Thank you for the tip: I now use the following properties:
<environmentVariables>
  <PLUGINS_DIR>../plugin/target/test-classes/test-dependencies</PLUGINS_DIR>
  <JENKINS_VERSION>${jenkins.version}</JENKINS_VERSION>
  <BROWSER>firefox-container</BROWSER>
</environmentVariables>
(And before the UI tests start it is required to copy the hpi under test into the test-dependencies folder as well) 

Now everything is green (or blue) again! This also improves the runtime of the tests (now about 2min per test):

Old variant (about 4min per test):


Plugins not mentioned in this list will still be retrieved according
to versions in the online UC (though by default using Maven to
retrieve local copies of the actual binaries). You could inject an
alternate `UpdateCenterMetadataProvider` to completely override that,
so that only dependencies in the test classpath are offered; this
would match the behavior of `JenkinsRule` (other than when annotated
by `@WithPlugin`).

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages