Re: Corner case with Acceptance Test Harness

[oliver gondža]

On Mon, 14 Sep 2015 16:59:11 +0200, Scott Hebert <sco...@gmail.com> wrote:

> In the case of a bundled plugin that has a test in the ATH, it seems that
> the test will exercise the bundled version and never the latest version
> that is available.
>
> Take matrix-auth for example:
>
> The exercised-plugins.properties file shows:
>
> plugins.MatrixAuthPluginTest.helloWorld$matrix-auth = 1.1
> plugins.MatrixAuthPluginTest.projectMatrixAuth$matrix-auth = 1.1
>
> while the latest version available is 1.2
>
> We have an in-house tool that packages plugins based on versions that are
> tested in the ATH and for bundled plugins, we have the case where we want
> to package 1.2 but wind up taking 1.1 since we base ourselves on what was
> tested.

There is a envvar NEVER_REPLACE_EXISTING_PLUGINS (false by default) that
controls what should happen if plugin does not satisfy `WithPlugins`
version constraint (which is not you case). Perhaps it can be extended to
use ternary logic: never replace, update iff outdated, always try to
update (what you want, not implemented for now).

IIUC correctly, you use what ATH end up running against to deploy to
actual Jenkins. I control plugins the other way around: prepare plugin
collection as a set of HPI files and run ATH agains that using
`PLUGINS_DIR=... NEVER_REPLACE_EXISTING_PLUGINS=true`.

HTH
--
oliver

[Jesse Glick]

Of course if we kill bundled plugins (as in another thread) this
problem goes away.