Today the Jenkins functional test harness, containing `JenkinsRule`
and its ilk¹, is built from the Jenkins core repository, in
`test/src/main/`. That is fine for functional tests of core itself
(`test/src/test/`). But this scheme is undesirable for functional
tests in plugins, for two reasons.
One, whenever a fix is made to the test harness itself, such as my
recent PR 1967², it does not become available to plugins until they
switch their core baseline to the version including the fix. That
could be months away, even if the fix is backported to an LTS branch
as is occasionally done. Sometimes it is possible to work around the
lack of the fix in plugin tests³, but even when possible this is
awkward.
Second, incompatible changes to the test harness are sometimes
necessary or desirable. We try hard to avoid incompatible changes to
core itself—these could break user installations—but incompatibilities
of tests is a price paid only by plugin developers, usually
manageable. In particular, PR 1774⁴ changed some details of HtmlUnit
usage. Now if you upgrade your plugin’s core baseline to a version
after that point (generally 1.625.x), you update your tests to
match—fine. The problem comes as we try to expand usage of
`plugin-compat-tester`⁵ to hunt for core regressions. To verify that a
plugin’s tests still pass against a newer core, this tool updates the
baseline in a working copy of the POM. But if the tests are still
written against the old forked HtmlUnit, they may no longer compile;
and a single version of a test cannot always be made to work against
either version of HtmlUnit. You can see an example of this dilemma in
the `copyartifact` plugin⁶.
My proposal is to start treating the harness as an independently
versioned test library, in its own GitHub repository.
`jenkinsci/jenkins/test/src/main/` would be extracted there (with
history intact). `test/src/test/` core functional tests would pick it
up as a binary dependency, as plugin tests already do. The advantage
would be that we could iterate quickly on new features or fixes to the
harness, cutting a Maven release at any time. Core and each plugin
would decide which version to pick up; whenever you wish to update to
the latest version, you can edit your tests in the same commit to
reflect any required changes, or to take advantage of new features.
`plugin-compat-tester` would leave this version untouched.
In principle there could be a change to core which requires a
synchronized matching change to the test harness, which would become
more difficult to handle in such a system, but I am not aware of any
examples.⁷
After a split it should be possible to do further refactorings that
would make for smaller artifacts and better modularity. Currently most
of the size of the test harness artifact is consumed by bundling four
(!) copies of Maven, plus one copy of Ant, which are only relevant to
plugins actually testing builds using these tools. The harness also
has fixed dependencies on a number of plugins which were long since
split out of core—`subversion`, `maven-plugin`, `matrix-project`,
etc.—which thickens the `test`-scope dependency tree for every plugin
and can cause weird version resolution problems that complicate test
development and maintenance.
I am sending this proposal verbally rather than in a PR since
cross-repository refactorings like this are not easily handled with
PRs, especially since a new artifact release is required to get
everything working. There are some details that will only fall out of
an actual attempt to do it, but I wanted to get general agreement that
this is a good direction before starting on things like the slow Git
command to extract a subdirectory with history.
For context, there is also some work being prepared to create a new
plugin POM which, unlike the current one coversioned with core⁸, would
be in its own repository and independently versioned, with the core
version simply being a Maven property. Stephen Connolly and I have
long discussed the possibility—CloudBees proprietary plugins do
exactly this, and it is much more comfortable to work with as a plugin
developer and from `plugin-compat-tester`—but there is finally some
movement. The idea would be that with a refactored test harness, your
plugin POM would specify both the versions of core and the test
harness independently.
WDYT?
¹
https://github.com/jenkinsci/jenkins/blob/6306952779acc311719c175201e736685349e507/test/src/main/java/org/jvnet/hudson/test/JenkinsRule.java#L225-L234
²
https://github.com/jenkinsci/jenkins/pull/1967
³
https://github.com/jenkinsci/workflow-plugin/pull/277
⁴
https://github.com/jenkinsci/jenkins/pull/1774
⁵
https://wiki.jenkins-ci.org/display/JENKINS/Plugin+Compatibility+Tester
⁶
https://github.com/jenkinsci/copyartifact-plugin/pull/76
⁷
https://github.com/jenkinsci/jenkins/commits/6306952779acc311719c175201e736685349e507/test/src/main
includes
https://github.com/jenkinsci/jenkins/commit/6aabbbd282083716d631be7f2fa07ce93b93ee23#diff-8e0ebb909d001d948b257dbbc1450cfd
that would imply a newer core dep; this part of the patch was very low
priority.
https://github.com/jenkinsci/jenkins/commit/b6ae4f5b798068fad48d292dee860fe97bdc7d55
and
https://github.com/jenkinsci/jenkins/commit/fe1d4472015d978f45f219369fc7123bbdd558dd
handle core incompatibilities but are harmless to use with older
cores.
⁸
https://github.com/jenkinsci/jenkins/blob/6306952779acc311719c175201e736685349e507/plugins/pom.xml#L15