Automate recommended plugin unittest run against latest Jenkins core

[Oliver Gondža]

Hi,

I created an issue[1] to verify if all recommended plugins has passing
unittest when run against latest core. I would like to use that as a GA
criteria for releases, LTS ones at least.

I see people are still updating plugin-compat-tester so I guess it is
not dead, though I did not manage to find any recent documentation or
running CI job. Can it be used to implement that? Is somebody working on
this?

[1] https://issues.jenkins-ci.org/browse/JENKINS-37145
--
oliver

[Mark Waite]

I don't understand why you would make that a GA criteria for releases, or for LTS releases.  Can you explain in more detail what you're trying to confirm with that check?

I had maintained a separate branch of the git client plugin which depended on the latest LTS release (called "ongoing/latest-jenkins-lts") on my fork of the repo.  I treated it as a long-lived branch that contained the master branch and any changes required to build against LTS.  API changes in the core between the plugin Jenkins core version and the LTS required changes to the plugin before it could be built based on that latest LTS, yet the plugin continued to work very well with the LTS.

If other plugins have the same conditions, then it seems that making this a GA criteria will block GA of releases.

I think the check is interesting and may be useful, but I don't think it should be a GA criteria for a Jenkins release.

Mark Waite

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/08d33372-c328-71d5-a2c6-870976e3b688%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

[ogondza]

Right, not the best wording. What I had in mind is: "It is something we would like to have confirmed before declaring the release is good to go".

Unfortunately, many maintainers do not put that much effort into making sure their plugins work with latests cores, myself including.

--
oliver

[Jesse Glick]

On Wed, Aug 3, 2016 at 9:31 AM, Oliver Gondža <ogo...@gmail.com> wrote:
> Is somebody working on this?

Maybe ask andresrc.

[Manuel Jesús Recena Soto]

Hello Oliver,

I like the idea.

General speaking, ATH and PCT are very important and we should look after its healthy.

IMHO, PCT results are a nice place to figure out bad symptoms on some plugins.

Please, find here two issues as result of PCT: JENKINS-36646 and JENKINS-36623

What we maybe need is make more usable and accessible this information.

Regards,

--
oliver

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/08d33372-c328-71d5-a2c6-870976e3b688%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

--

Manuel Recena Soto
* manuelrecena.com [/blog]
* linkedin.com/in/recena

[Mark Waite]

On Thu, Aug 4, 2016 at 12:46 PM Manuel Jesús Recena Soto <rec...@gmail.com> wrote:

Hello Oliver,

I like the idea.

General speaking, ATH and PCT are very important and we should look after its healthy.

IMHO, PCT results are a nice place to figure out bad symptoms on some plugins.

Can you describe the types of symptoms you envision detecting with PCT?

I am interested in running the plugin tests with a different jenkins.version value, after compiling with an unmodified jenkins.version value.  That seems like an interesting test alternative that might surface interesting differences in the test harnesses or the core code.

I'm not really interested in having some external automation tell me that the current code does not compile with the latest Jenkins core.  JENKINS-36646 seems like the same condition I see, failure to compile with the newer Jenkins version because code changes are required before the plugin can depend on the newer version.

I update the Jenkins core dependency rarely and with great care.  A bug report like JENKINS-36646 is a distraction rather than a help, since I will see the compilation failure as soon as I prepare the change for the plugin to support the new Jenkins core version.

For example, the git plugin 3.0.0-beta already has the necessary change to compile with more recent Jenkins core, but the automation probably won't know that it needs to evaluate a beta version on a different branch than master.

Mark Waite

 

Please, find here two issues as result of PCT: JENKINS-36646 and JENKINS-36623

What we maybe need is make more usable and accessible this information.

Regards,

2016-08-03 15:31 GMT+02:00 Oliver Gondža <ogo...@gmail.com>:

Hi,

I created an issue[1] to verify if all recommended plugins has passing unittest when run against latest core. I would like to use that as a GA criteria for releases, LTS ones at least.

I see people are still updating plugin-compat-tester so I guess it is not dead, though I did not manage to find any recent documentation or running CI job. Can it be used to implement that? Is somebody working on this?

[1] https://issues.jenkins-ci.org/browse/JENKINS-37145

--
oliver

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/08d33372-c328-71d5-a2c6-870976e3b688%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

--

Manuel Recena Soto
* manuelrecena.com [/blog]
* linkedin.com/in/recena

--

You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CABa-UocUg78nQkNas9iyKK0JLKZvv%3DuT9eDoZLn1f0YYfsTPSA%40mail.gmail.com.

[Slide]

With releases coming often, it is hard to test against the latest Jenkins when trying to deal with issues and so forth. Most maintainers just don't have the time to test against the latest release, the latest LTS and a few things in between.

--

You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/f373a327-d354-4446-87d8-25eb2472d733%40googlegroups.com.

[Manuel Jesús Recena Soto]

Hello Mark,

In advance, thanks for your feedback.

(replies inline)

2016-08-04 21:27 GMT+02:00 Mark Waite <mark.ea...@gmail.com>:
>
> On Thu, Aug 4, 2016 at 12:46 PM Manuel Jesús Recena Soto <rec...@gmail.com> wrote:
>>
>> Hello Oliver,
>>
>> I like the idea.
>>
>> General speaking, ATH and PCT are very important and we should look after its healthy.
>>
>> IMHO, PCT results are a nice place to figure out bad symptoms on some plugins.
>>
>
> Can you describe the types of symptoms you envision detecting with PCT?
>
> I am interested in running the plugin tests with a different jenkins.version value, after compiling with an unmodified jenkins.version value. That seems like an interesting test alternative that might surface interesting differences in the test harnesses or the core code.
>
> I'm not really interested in having some external automation tell me that the current code does not compile with the latest Jenkins core. JENKINS-36646 seems like the same condition I see, failure to compile with the newer Jenkins version because code changes are required before the plugin can depend on the newer version.
>

1. You can always ignore that information. There may be developers
interested on it.
2. I totally agree with you that JENKINS-36646 is not a bug, but
something that it should be investigate. I'd say: a task.

> I update the Jenkins core dependency rarely and with great care. A bug report like JENKINS-36646 is a distraction rather than a help, since I will see the compilation failure as soon as I prepare the change for the plugin to support the new Jenkins core version.
>
> For example, the git plugin 3.0.0-beta already has the necessary change to compile with more recent Jenkins core, but the automation probably won't know that it needs to evaluate a beta version on a different branch than master.
>

IMHO, when we find this situation: a plugin based on an old Jenkins
base line + not recent releases > bad symptom (a red flag in my
production environments).

PCT helps on these cases.

Regards,

> To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAO49JtFWnPvcmjCXE3ikhe9cFUSjNbnoLzuBq9AQU49gwB7zYw%40mail.gmail.com.

[Mark Waite]

Replies inline

On Fri, Aug 5, 2016 at 1:51 AM Manuel Jesús Recena Soto <rec...@gmail.com> wrote:

Hello Mark,

In advance, thanks for your feedback.

(replies inline)

2016-08-04 21:27 GMT+02:00 Mark Waite <mark.ea...@gmail.com>:
>
> On Thu, Aug 4, 2016 at 12:46 PM Manuel Jesús Recena Soto <rec...@gmail.com> wrote:
>>
>> Hello Oliver,
>>
>> I like the idea.
>>
>> General speaking, ATH and PCT are very important and we should look after its healthy.
>>
>> IMHO, PCT results are a nice place to figure out bad symptoms on some plugins.
>>
>
> Can you describe the types of symptoms you envision detecting with PCT?
>
> I am interested in running the plugin tests with a different jenkins.version value, after compiling with an unmodified jenkins.version value.  That seems like an interesting test alternative that might surface interesting differences in the test harnesses or the core code.
>
> I'm not really interested in having some external automation tell me that the current code does not compile with the latest Jenkins core.  JENKINS-36646 seems like the same condition I see, failure to compile with the newer Jenkins version because code changes are required before the plugin can depend on the newer version.
>

1. You can always ignore that information. There may be developers
interested on it.
2. I totally agree with you that JENKINS-36646 is not a bug, but
something that it should be investigate. I'd say: a task.

> I update the Jenkins core dependency rarely and with great care.  A bug report like JENKINS-36646 is a distraction rather than a help, since I will see the compilation failure as soon as I prepare the change for the plugin to support the new Jenkins core version.
>
> For example, the git plugin 3.0.0-beta already has the necessary change to compile with more recent Jenkins core, but the automation probably won't know that it needs to evaluate a beta version on a different branch than master.
>

IMHO, when we find this situation: a plugin based on an old Jenkins
base line + not recent releases > bad symptom (a red flag in my
production environments).

PCT helps on these cases.

Sorry that I'm not making myself clear.  The git client plugin (1.19.7 and it predecessors) matches the situation you describe (old Jenkins baseline, not recent release, does not compile without changes against current core).  It has intentionally been kept at that older release in order to not create unnecessary compatibility issues for users of its APIs.

When git client plugin 2.0 releases (planned by mid-September) it will update to depend on JDK 7, Jenkins 1.625, and JGit 4.  That will be an intentional major version of the plugin to switch from JDK 6 to JDK 7.  Even then, there is a risk that it may not compile against Jenkins 2.7.2.  Acceptance tests are run regularly against Jenkins 2.7.2, but it is not regularly checked that it compiles against 2.7.2 because I've not found that check especially helpful.

I really have no problem ignoring infrastructure that tells me irrelevant things are failing, but when irrelevant failures are reported to me, I develop the habit of ignoring more and more information from that source.  Sometimes, that means I miss a real problem because I've been ignoring the irrelevant.

Mark Waite

 

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CABa-UocPDEZ%3DzX1DZbbO3pOc0W86n3tbPA%2B%3DW7ShVqPQSMzMTg%40mail.gmail.com.

[Slide]

+1 to Mark's comments.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAO49JtH5mNWCVt9T3vwTOb%2Buv1wpizRjWg6dAjboZ98ioMnd0A%40mail.gmail.com.

[Jesse Glick]

On Fri, Aug 5, 2016 at 7:57 AM, Mark Waite <mark.ea...@gmail.com> wrote:
> The git client plugin (1.19.7 and
> it predecessors) matches the situation you describe (old Jenkins baseline,
> not recent release, does not compile without changes against current core).

Why exactly does it not compile against current core? If SECURITY-144,
just update to 1.580.1, which is quite old by now.

[Mark Waite]

The changes required to compile against current core include:

- change jenkins.MasterToSlaveFileCallable<GitClient> to FileCallable<GitClient>

- add checkRoles(RoleChecker) to HudsonTestCase

My recollection (when I did that investigation many months ago) is that those changes came in one or more of the 1.6xx versions.  I was not ready to move the git client plugin dependency forward to 1.6xx at that time without a major version number change.

If I were making a major version change, I wanted other benefits (like dropping support for Java 6, switching to JGit 4, using try with resources, etc.).  That's what is being prepared in the git client plugin 2.0.0-beta branch, and what I hope to release in time for Jenkins World in September.

Mark Waite

 

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr1JY-9OaYoUUhmNWiUMvOO5v_xkq_NW6Bkkd2KoT7REgg%40mail.gmail.com.

[Jesse Glick]

On Fri, Aug 5, 2016 at 4:52 PM, Mark Waite <mark.ea...@gmail.com> wrote:
> - change jenkins.MasterToSlaveFileCallable<GitClient> to
> FileCallable<GitClient>

Vice-versa I guess you mean.

> - add checkRoles(RoleChecker) to HudsonTestCase

Probably basically the same change: SECURITY-144.

> My recollection (when I did that investigation many months ago) is that
> those changes came in one or more of the 1.6xx versions.

No, 1.580.1. (from Oct 2014 FWIW)