Proposal - Adding JCasC tests into Jenkins core test suite

[Joseph P]

Hi All

In an effort to improve JCasC compatibility we would like to add the configuration-as-code-plugin into the Jenkins core test suite

This will help us improve test coverage of JCasC and we would avoid regressions that are related to configuration-as-code

We have several issues open that are related to Jenkins core and currently a few in progress. See dashboard (sort by component and check core)

This is based on Oleg's suggestion: https://github.com/jenkinsci/jenkins/pull/3994#issuecomment-498730322

Best regards

Joseph Petersen

[Jon Brohauge]

Sounds like a great idea to me.

My 2-cents

Jon

[Jesse Glick]

On Tue, Jun 4, 2019 at 1:23 PM Joseph P <jose...@gmail.com> wrote:
> we would avoid regressions that are related to configuration-as-code

I think that should be handled differently, for example by something
like JENKINS-45047, or `plugin-compat-tester`, or
`buildPlugin.recommendedConfigurations()`, or Dependabot. There are
plenty of critical plugins which are prone to breaking due to changes
in core, and we cannot possibly stuff the core test suite full of all
of them. (In fact I would prefer to be _removing_ some plugins from
the `test` POM.) Instead, plugins define tests and a `jenkins.version`
known to pass them, and we use various tooling to ensure that these
tests are also run against newer dependency versions so that
regressions are caught in a timely manner.

[Slide]

I agree with Jesse. The suite as is is so large already. The plugin-compat-test was made for purposes similar to this I believe, so why not use the tools that are available?

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr0ALHHvTcOeRRrmLV10B%3DaeODTGWsOcQ2yCWu4BKoT5dw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--

Website: http://earl-of-code.com

[Jon Brohauge]

If it makes sense to use the plugin-compat-test, we should go with this. I wasn't aware of its existence.

You received this message because you are subscribed to a topic in the Google Groups "Jenkins Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-dev/cgbYEB3c_Vc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAPiUgVduMQyrmKzkjyktOCHw8EhbzzOhMO6tAotZVSvO5cp8dw%40mail.gmail.com.

[Oleg Nenashev]

Yes, we could use PCT for Jenkins Configuration-as-Code tests.

But, if we want to run JCasC integration tests as a part of the core pipeline, we would need to integrate PCT directly into Jenkins Core pipeline.

I had a pull request for that (https://github.com/jenkinsci/jenkins/pull/3438), but it was stalled. Maybe we could revisit it.

BR, Oleg

On Tuesday, June 4, 2019 at 11:24:53 PM UTC+2, Jon Brohauge wrote:

If it makes sense to use the plugin-compat-test, we should go with this. I wasn't aware of its existence.

On Tue, 4 Jun 2019, 22.18 Slide, <slid...@gmail.com> wrote:

I agree with Jesse. The suite as is is so large already. The plugin-compat-test was made for purposes similar to this I believe, so why not use the tools that are available?

On Tue, Jun 4, 2019 at 11:29 AM Jesse Glick <jgl...@cloudbees.com> wrote:

On Tue, Jun 4, 2019 at 1:23 PM Joseph P <jose...@gmail.com> wrote:
> we would avoid regressions that are related to configuration-as-code

I think that should be handled differently, for example by something
like JENKINS-45047, or `plugin-compat-tester`, or
`buildPlugin.recommendedConfigurations()`, or Dependabot. There are
plenty of critical plugins which are prone to breaking due to changes
in core, and we cannot possibly stuff the core test suite full of all
of them. (In fact I would prefer to be _removing_ some plugins from
the `test` POM.) Instead, plugins define tests and a `jenkins.version`
known to pass them, and we use various tooling to ensure that these
tests are also run against newer dependency versions so that
regressions are caught in a timely manner.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to jenkin...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr0ALHHvTcOeRRrmLV10B%3DaeODTGWsOcQ2yCWu4BKoT5dw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--

Website: http://earl-of-code.com

--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-dev/cgbYEB3c_Vc/unsubscribe.

To unsubscribe from this group and all its topics, send an email to jenkin...@googlegroups.com.

[Jesse Glick]

On Wed, Jun 5, 2019 at 3:33 AM Oleg Nenashev <o.v.ne...@gmail.com> wrote:
> we would need to integrate PCT directly into Jenkins Core pipeline.
> I had a pull request for that (https://github.com/jenkinsci/jenkins/pull/3438), but it was stalled. Maybe we could revisit it.

+1, this would benefit core stability broadly.

[Oleg Nenashev]

Taking this discussion, what about including *optional* JCasC dependency to Jenkins Test Harness

Jenkins Performance Test Framework supports JCasC rules, and we want to make it accessible to users.

Her is the ongoing pull request: https://github.com/jenkinsci/jenkins-test-harness/pull/135/files#r290641674

I see two possible options:

• Keep the JCasC code in the PR as is
• Move the JCasC code to the Configuration-as-Code plugin codebase
• Taking ongoing activities like https://github.com/jenkinsci/configuration-as-code-plugin/pull/908 , and it might make sense to introduce a new module for JCasC test tools

WDYT?

On Wednesday, June 5, 2019 at 3:51:31 PM UTC+2, Jesse Glick wrote:

[Jesse Glick]

On Fri, Jun 7, 2019 at 3:23 AM Oleg Nenashev <o.v.ne...@gmail.com> wrote:
> Taking this discussion, what about including *optional* JCasC dependency to Jenkins Test Harness

Seems wrong to include any plugin as a dep of `jenkins-test-harness`.

> Jenkins Performance Test Framework supports JCasC rules

You can create a `tests`-classifier JAR from `configuration-as-code`
which offers specialized test utilities making use of that plugin’s
functionality.

> Taking ongoing activities like https://github.com/jenkinsci/configuration-as-code-plugin/pull/908 , and it might make sense to introduce a new module for JCasC test tools

A new module, or just a test JAR, does not make much difference.

[Oleg Nenashev]

You can create a `tests`-classifier JAR from `configuration-as-code`
which offers specialized test utilities making use of that plugin’s
functionality. 

 It is already there. I want to refactor it out in https://github.com/jenkinsci/configuration-as-code-plugin/issues/917

--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-dev/cgbYEB3c_Vc/unsubscribe.

To unsubscribe from this group and all its topics, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr0A0BwNZcCm2W-Mta9kJVqnbh39J4Jh%3D78K0zbwm-6rGg%40mail.gmail.com.