[PROPOSAL] Split of `jenkins-test-harness` out of `jenkinsci/jenkins`
90 views
Skip to first unread message
Jesse Glick
Jan 8, 2016, 12:16:37 PM1/8/16
to Jenkins Dev
Today the Jenkins functional test harness, containing `JenkinsRule`
and its ilk¹, is built from the Jenkins core repository, in
`test/src/main/`. That is fine for functional tests of core itself
(`test/src/test/`). But this scheme is undesirable for functional
tests in plugins, for two reasons.
One, whenever a fix is made to the test harness itself, such as my
recent PR 1967², it does not become available to plugins until they
switch their core baseline to the version including the fix. That
could be months away, even if the fix is backported to an LTS branch
as is occasionally done. Sometimes it is possible to work around the
lack of the fix in plugin tests³, but even when possible this is
awkward.
Second, incompatible changes to the test harness are sometimes
necessary or desirable. We try hard to avoid incompatible changes to
core itself—these could break user installations—but incompatibilities
of tests is a price paid only by plugin developers, usually
manageable. In particular, PR 1774⁴ changed some details of HtmlUnit
usage. Now if you upgrade your plugin’s core baseline to a version
after that point (generally 1.625.x), you update your tests to
match—fine. The problem comes as we try to expand usage of
`plugin-compat-tester`⁵ to hunt for core regressions. To verify that a
plugin’s tests still pass against a newer core, this tool updates the
baseline in a working copy of the POM. But if the tests are still
written against the old forked HtmlUnit, they may no longer compile;
and a single version of a test cannot always be made to work against
either version of HtmlUnit. You can see an example of this dilemma in
the `copyartifact` plugin⁶.
My proposal is to start treating the harness as an independently
versioned test library, in its own GitHub repository.
`jenkinsci/jenkins/test/src/main/` would be extracted there (with
history intact). `test/src/test/` core functional tests would pick it
up as a binary dependency, as plugin tests already do. The advantage
would be that we could iterate quickly on new features or fixes to the
harness, cutting a Maven release at any time. Core and each plugin
would decide which version to pick up; whenever you wish to update to
the latest version, you can edit your tests in the same commit to
reflect any required changes, or to take advantage of new features.
`plugin-compat-tester` would leave this version untouched.
In principle there could be a change to core which requires a
synchronized matching change to the test harness, which would become
more difficult to handle in such a system, but I am not aware of any
examples.⁷
After a split it should be possible to do further refactorings that
would make for smaller artifacts and better modularity. Currently most
of the size of the test harness artifact is consumed by bundling four
(!) copies of Maven, plus one copy of Ant, which are only relevant to
plugins actually testing builds using these tools. The harness also
has fixed dependencies on a number of plugins which were long since
split out of core—`subversion`, `maven-plugin`, `matrix-project`,
etc.—which thickens the `test`-scope dependency tree for every plugin
and can cause weird version resolution problems that complicate test
development and maintenance.
I am sending this proposal verbally rather than in a PR since
cross-repository refactorings like this are not easily handled with
PRs, especially since a new artifact release is required to get
everything working. There are some details that will only fall out of
an actual attempt to do it, but I wanted to get general agreement that
this is a good direction before starting on things like the slow Git
command to extract a subdirectory with history.
For context, there is also some work being prepared to create a new
plugin POM which, unlike the current one coversioned with core⁸, would
be in its own repository and independently versioned, with the core
version simply being a Maven property. Stephen Connolly and I have
long discussed the possibility—CloudBees proprietary plugins do
exactly this, and it is much more comfortable to work with as a plugin
developer and from `plugin-compat-tester`—but there is finally some
movement. The idea would be that with a refactored test harness, your
plugin POM would specify both the versions of core and the test
harness independently.
WDYT?
¹ https://github.com/jenkinsci/jenkins/blob/6306952779acc311719c175201e736685349e507/test/src/main/java/org/jvnet/hudson/test/JenkinsRule.java#L225-L234
² https://github.com/jenkinsci/jenkins/pull/1967
³ https://github.com/jenkinsci/workflow-plugin/pull/277
⁴ https://github.com/jenkinsci/jenkins/pull/1774
⁵ https://wiki.jenkins-ci.org/display/JENKINS/Plugin+Compatibility+Tester
⁶ https://github.com/jenkinsci/copyartifact-plugin/pull/76
⁷ https://github.com/jenkinsci/jenkins/commits/6306952779acc311719c175201e736685349e507/test/src/main
includes https://github.com/jenkinsci/jenkins/commit/6aabbbd282083716d631be7f2fa07ce93b93ee23#diff-8e0ebb909d001d948b257dbbc1450cfd
that would imply a newer core dep; this part of the patch was very low
priority. https://github.com/jenkinsci/jenkins/commit/b6ae4f5b798068fad48d292dee860fe97bdc7d55
and https://github.com/jenkinsci/jenkins/commit/fe1d4472015d978f45f219369fc7123bbdd558dd
handle core incompatibilities but are harmless to use with older
cores.
⁸ https://github.com/jenkinsci/jenkins/blob/6306952779acc311719c175201e736685349e507/plugins/pom.xml#L15
and its ilk¹, is built from the Jenkins core repository, in
`test/src/main/`. That is fine for functional tests of core itself
(`test/src/test/`). But this scheme is undesirable for functional
tests in plugins, for two reasons.
One, whenever a fix is made to the test harness itself, such as my
recent PR 1967², it does not become available to plugins until they
switch their core baseline to the version including the fix. That
could be months away, even if the fix is backported to an LTS branch
as is occasionally done. Sometimes it is possible to work around the
lack of the fix in plugin tests³, but even when possible this is
awkward.
Second, incompatible changes to the test harness are sometimes
necessary or desirable. We try hard to avoid incompatible changes to
core itself—these could break user installations—but incompatibilities
of tests is a price paid only by plugin developers, usually
manageable. In particular, PR 1774⁴ changed some details of HtmlUnit
usage. Now if you upgrade your plugin’s core baseline to a version
after that point (generally 1.625.x), you update your tests to
match—fine. The problem comes as we try to expand usage of
`plugin-compat-tester`⁵ to hunt for core regressions. To verify that a
plugin’s tests still pass against a newer core, this tool updates the
baseline in a working copy of the POM. But if the tests are still
written against the old forked HtmlUnit, they may no longer compile;
and a single version of a test cannot always be made to work against
either version of HtmlUnit. You can see an example of this dilemma in
the `copyartifact` plugin⁶.
My proposal is to start treating the harness as an independently
versioned test library, in its own GitHub repository.
`jenkinsci/jenkins/test/src/main/` would be extracted there (with
history intact). `test/src/test/` core functional tests would pick it
up as a binary dependency, as plugin tests already do. The advantage
would be that we could iterate quickly on new features or fixes to the
harness, cutting a Maven release at any time. Core and each plugin
would decide which version to pick up; whenever you wish to update to
the latest version, you can edit your tests in the same commit to
reflect any required changes, or to take advantage of new features.
`plugin-compat-tester` would leave this version untouched.
In principle there could be a change to core which requires a
synchronized matching change to the test harness, which would become
more difficult to handle in such a system, but I am not aware of any
examples.⁷
After a split it should be possible to do further refactorings that
would make for smaller artifacts and better modularity. Currently most
of the size of the test harness artifact is consumed by bundling four
(!) copies of Maven, plus one copy of Ant, which are only relevant to
plugins actually testing builds using these tools. The harness also
has fixed dependencies on a number of plugins which were long since
split out of core—`subversion`, `maven-plugin`, `matrix-project`,
etc.—which thickens the `test`-scope dependency tree for every plugin
and can cause weird version resolution problems that complicate test
development and maintenance.
I am sending this proposal verbally rather than in a PR since
cross-repository refactorings like this are not easily handled with
PRs, especially since a new artifact release is required to get
everything working. There are some details that will only fall out of
an actual attempt to do it, but I wanted to get general agreement that
this is a good direction before starting on things like the slow Git
command to extract a subdirectory with history.
For context, there is also some work being prepared to create a new
plugin POM which, unlike the current one coversioned with core⁸, would
be in its own repository and independently versioned, with the core
version simply being a Maven property. Stephen Connolly and I have
long discussed the possibility—CloudBees proprietary plugins do
exactly this, and it is much more comfortable to work with as a plugin
developer and from `plugin-compat-tester`—but there is finally some
movement. The idea would be that with a refactored test harness, your
plugin POM would specify both the versions of core and the test
harness independently.
WDYT?
¹ https://github.com/jenkinsci/jenkins/blob/6306952779acc311719c175201e736685349e507/test/src/main/java/org/jvnet/hudson/test/JenkinsRule.java#L225-L234
² https://github.com/jenkinsci/jenkins/pull/1967
³ https://github.com/jenkinsci/workflow-plugin/pull/277
⁴ https://github.com/jenkinsci/jenkins/pull/1774
⁵ https://wiki.jenkins-ci.org/display/JENKINS/Plugin+Compatibility+Tester
⁶ https://github.com/jenkinsci/copyartifact-plugin/pull/76
⁷ https://github.com/jenkinsci/jenkins/commits/6306952779acc311719c175201e736685349e507/test/src/main
includes https://github.com/jenkinsci/jenkins/commit/6aabbbd282083716d631be7f2fa07ce93b93ee23#diff-8e0ebb909d001d948b257dbbc1450cfd
that would imply a newer core dep; this part of the patch was very low
priority. https://github.com/jenkinsci/jenkins/commit/b6ae4f5b798068fad48d292dee860fe97bdc7d55
and https://github.com/jenkinsci/jenkins/commit/fe1d4472015d978f45f219369fc7123bbdd558dd
handle core incompatibilities but are harmless to use with older
cores.
⁸ https://github.com/jenkinsci/jenkins/blob/6306952779acc311719c175201e736685349e507/plugins/pom.xml#L15
Mark Waite
Jan 8, 2016, 2:01:43 PM1/8/16
to Jenkins Dev
I like the idea of moving JenkinsRule outside jenkins core very much. The git plugin and git client plugin intentionally support older Jenkins versions, but that means their test infrastructure is also limited to those older Jenkins versions. I'd much rather be able to use the latest Jenkins test infrastructure to create better tests.
That would also provide me more incentive to attempt fixes to test infrastructure. Today, if the test classes have a problem, there is very little chance that I'll investigate the problem in the test classes, because I can't access updates to those test classes anyway.
Mark Waite
--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr1w18BohqOEpcnf0r_2PYvV-wmHzDEjLj%2BNst6WNtaOPA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Andrew Bayer
Jan 8, 2016, 3:01:33 PM1/8/16
to jenkin...@googlegroups.com
I'd be a big +1 on this as well. It makes sense to have the harness as a separate project consumed by Jenkins core and plugins for their testing. A nice side effect is that we'd be able to actually move the core tests currently in jenkins.git/test/src/test/... into jenkins.git/core/src/test/..., I think, which would just make things feel much more organic.
A.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAO49JtGrzf5%3DtD3w8%2Bo7qSGjHBON5sG0cuxbRYwrVrrq9xwo2g%40mail.gmail.com.
Stephen Connolly
Jan 8, 2016, 3:32:54 PM1/8/16
to jenkin...@googlegroups.com
+1
--
Sent from my phone
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAPbPdOacnvj1wJBEKCkzt_bzr5U8arkjo7Kgvi_2-XuTbWdhLA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
--
Sent from my phone
Jesse Glick
Jan 8, 2016, 5:50:40 PM1/8/16
to Jenkins Dev
Moving core tests into the same module would be great, but I am not going to promise that it works until I try it.
Arnaud Héritier
Jan 8, 2016, 6:52:32 PM1/8/16
to jenkin...@googlegroups.com
+1 we did a similar on maven core and maven integration tests a long time ago
It is now very useful and help to apply all these tests to all maven core versions (with few conditions to exclude some tests for some versions)
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CA%2BnPnMy7haSLnQba1Y1dAzMPnpYQ7CPoB2i7S_kWs-HQw62XzA%40mail.gmail.com.
-----
Arnaud Héritier
Mail/GTalk: aheritier AT gmail DOT com
Twitter/Skype : aheritier
Jesse Glick
Jan 8, 2016, 8:06:24 PM1/8/16
to Jenkins Dev
Well I was not proposing to move the core tests themselves. Just the harness part. The tests are routinely edited in conjunction with code changes.
Oleg Nenashev
Jan 9, 2016, 4:59:57 AM1/9/16
to Jenkins Developers
+1 for decoupling.
I suppose we want to have multiple independent libs instead of a single multi-module Maven monster. BTW the test harness is simple enough to be kept as a single library.
суббота, 9 января 2016 г., 4:06:24 UTC+3 пользователь Jesse Glick написал:
I suppose we want to have multiple independent libs instead of a single multi-module Maven monster. BTW the test harness is simple enough to be kept as a single library.
суббота, 9 января 2016 г., 4:06:24 UTC+3 пользователь Jesse Glick написал:
Kanstantsin Shautsou
Jan 9, 2016, 5:42:41 AM1/9/16
to Jenkins Developers
Jesse, would it include issues that i described on IRC yesterday?
- plugin extraction happens only from hpi context. Allow using JenkinsRule from modules.
- plugin extraction happens only from hpi context. Allow using JenkinsRule from modules.
- unhardcode JenkinsRule from been localhost (have working test example with docker + CLI, but it re-implements parts of rule)
Jesse Glick
Jan 9, 2016, 9:17:12 AM1/9/16
to Jenkins Dev
No, my proposal was only to move the code, not change its behavior.
Robert Sandell
Jan 11, 2016, 5:41:11 AM1/11/16
to jenkin...@googlegroups.com
+1
On Sat, Jan 9, 2016 at 3:17 PM, Jesse Glick <jgl...@cloudbees.com> wrote:
No, my proposal was only to move the code, not change its behavior.
--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr1OmTwyhV%3DbmJKAewtcnPdiU1mNfDv%3DPbCOOin7JSGkZQ%40mail.gmail.com.
--
Robert Sandell
Software Engineer
CloudBees Inc.
Kohsuke Kawaguchi
Jan 11, 2016, 1:13:57 PM1/11/16
to jenkin...@googlegroups.com
+1, sounds good to me.
--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr1w18BohqOEpcnf0r_2PYvV-wmHzDEjLj%2BNst6WNtaOPA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Kohsuke Kawaguchi
Baptiste Mathus
Jan 11, 2016, 3:20:47 PM1/11/16
to jenkin...@googlegroups.com
+1 for splitting. I also like the idea of being able to switch Jenkins version quickly with a prop like you or Stephen described some time ago.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAN4CQ4yByQKcLjCeSbZoC1qs%3DdhxQf_-LUyvqqWvHU-kMjcK9w%40mail.gmail.com.
James Nord
Jan 12, 2016, 4:08:55 PM1/12/16
to Jenkins Developers
Sounds good to me
Manuel Jesús Recena Soto
Jan 12, 2016, 6:14:56 PM1/12/16
to Jenkins Developers
I always try to avoid big repositories. In this case, there are
several reason to split it.
+1
Maybe a good moment to review this page:
https://wiki.jenkins-ci.org/display/JENKINS/Unit+Test#UnitTest-Overview
Regards,
2016-01-12 22:08 GMT+01:00 James Nord <james...@gmail.com>:
> Sounds good to me
Manuel Recena Soto
* manuelrecena.com [/blog]
* linkedin.com/in/recena
several reason to split it.
+1
Maybe a good moment to review this page:
https://wiki.jenkins-ci.org/display/JENKINS/Unit+Test#UnitTest-Overview
Regards,
2016-01-12 22:08 GMT+01:00 James Nord <james...@gmail.com>:
> Sounds good to me
>
> --
> You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/b08be9cb-c39f-4794-ba35-420b12e0290a%40googlegroups.com.
> --
> You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
Manuel Recena Soto
* manuelrecena.com [/blog]
* linkedin.com/in/recena
Jesse Glick
Jan 13, 2016, 6:38:22 PM1/13/16
to Jenkins Dev
https://github.com/jenkinsci/jenkins/pull/1977 as a starting point.
Jesse Glick
Jan 15, 2016, 3:15:09 PM1/15/16
to Jenkins Dev
To continue: https://github.com/jenkinsci/jenkins/pull/1982
Jesse Glick
Jan 18, 2016, 12:01:38 PM1/18/16
to Jenkins Dev
Completed toward 1.645.
Reply all
Reply to author
Forward
0 new messages