Test case contribution

[Romain SOMMERARD]

Hi guys,

We are a team of 3 Master students who want to contribute to Jenkins test case.
I come to you because I want to know if you have sections which have low test case coverage or need improvement?

Regards,
-- Romain

[Mark Waite]

The git client plugin and the git plugin both have low test coverage and need improvement.  See https://github.com/jenkinsci/git-client-plugin and https://github.com/jenkinsci/git-plugin .

Pull requests welcomed, especially of tests.  Pull requests without tests are discouraged.

Mark Waite

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/9ff27e72-2a2f-4203-9694-579ac05a2190%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Daniel Beck]

Hi Romain,

great to hear!

It's probably a good idea for you to look at the acceptance test harness[1] and improve its coverage of Jenkins features. There should be a lot of low hanging fruit in there, especially tests for plugins. Look to plugin popularity[2] if you don't know how else to prioritize.

Daniel

1: https://github.com/jenkinsci/acceptance-test-harness
2: http://stats.jenkins-ci.org/jenkins-stats/svg/201507-top-plugins1000.svg

[Kanstantsin Shautsou]

I'm trying design some "AIT (advanced integrations test)" :D for docker-plugin/crowd2-plugins that has no tests at all and doesn't fit into JenkinsRule/ATH, but it requires time and jenkins knowledge (plus suggestions from experienced devs) https://groups.google.com/forum/#!topic/jenkinsci-dev/R2_tHJDF0KI .

[Oleg Nenashev]

I would also recommend Acceptance Test Harness. 

Another getting-started option would be to take a simple plugin without test suites and to try to introduce full cycle of tests there. Git plugins (proposed by Mark) are also an option, but probably their internals are too complex.  

For example, https://github.com/jenkinsci/conditional-buildstep-plugin has a limited test coverage, which could be improved. 

понедельник, 14 сентября 2015 г., 14:17:01 UTC+3 пользователь Kanstantsin Shautsou написал:

[Ullrich Hafner]

> Am 16.09.2015 um 10:49 schrieb Oleg Nenashev <o.v.ne...@gmail.com>:
>
> […]

>
> For example, https://github.com/jenkinsci/conditional-buildstep-plugin has a limited test coverage, which could be improved.
>

BTW: Did someone already manage it to start the Jenkins under Test with coverage monitoring (emma, cobertura, jacoco, etc.)? Seems not so easy due to the Jenkins class loader.
So we could see the actual coverage of the ATH.

[Jesse Glick]

On Wed, Sep 16, 2015 at 8:04 AM, Ullrich Hafner
<ullrich...@gmail.com> wrote:
> Did someone already manage it to start the Jenkins under Test with coverage monitoring (emma, cobertura, jacoco, etc.)?

I have run JaCoCo during plugin tests.

> Seems not so easy due to the Jenkins class loader.

With `JenkinsRule`? Not an issue, all classes in the test root, plus
the plugin being tested and its stated dependencies, and core and its
modules and libraries, are in the Surefire classpath and so available
for processing by JaCoCo, just like in a unit test. The only classes
that might slip past are those in unrelated bundled plugins, or
plugins loaded via `@WithPlugins`, that would be loaded via a plugin
class loader, but surely you do not care about coverage of these.

[Mark Waite]

On Wed, Sep 16, 2015 at 2:49 AM Oleg Nenashev <o.v.ne...@gmail.com> wrote:

I would also recommend Acceptance Test Harness. 

Another getting-started option would be to take a simple plugin without test suites and to try to introduce full cycle of tests there. Git plugins (proposed by Mark) are also an option, but probably their internals are too complex.  

I think the git client plugin and the git plugin are great places to start because you could pick a bug report, create a test which shows that the bug is present, and submit a pull request to include that evidence of a bug.  In the process, you will have learned about automated testing, command line git, JGit, the git plugin, Jenkins, and more.

However, I'm biased...

Mark Waite

[Ullrich Hafner]

For unit tests and JenkinsRule integration tests I get the coverage working, too.

But not yet for the acceptance tests: here Jenkins is started and loading the plain plug-ins from JENKINS_HOME. It would be interesting to see the coverage of these tests, too. I used this approach for manual testing of some enterprise applications… With emma you can enhance the classes of a packaged application (jar) and then start testing. During the manual tests the coverage data is incrementally written. After testing, you simply can create a report of these tests. Would be nice to get this for the acceptance tests, too.

[Romain SOMMERARD]

Thanks for your feedback.
As you say, I think that we will go on git-plugin.

But I have a question about test contribution:
If we write a test which describe a bug, is it good to make a pull request for this test without fix ?
I mean, if pull request is accepted, automatic build will fail, is it good way to make that like you say@Mark ? (Maybe the pull request would stay in stage while a fix's coming ?)

I precise that we want write test cases and not fix bugs.

[Jesse Glick]

On Thu, Sep 17, 2015 at 2:58 PM, Romain SOMMERARD <rsomm...@gmail.com> wrote:
> If we write a test which describe a bug, is it good to make a pull request
> for this test without fix ?

IMO yes please: the most important contribution to a bug fix is the test.

> if pull request is accepted, automatic build will fail

Simply mark the test with:

@Ignore("TODO currently failing exactly as reported")
@Issue("JENKINS-12345")

If and when someone supplies a fix, they can simply remove the
`@Ignore` in the same commit.

[Mark Waite]

On Fri, Sep 18, 2015 at 8:08 AM Jesse Glick <jgl...@cloudbees.com> wrote:

On Thu, Sep 17, 2015 at 2:58 PM, Romain SOMMERARD <rsomm...@gmail.com> wrote:
> If we write a test which describe a bug, is it good to make a pull request
> for this test without fix ?

IMO yes please: the most important contribution to a bug fix is the test.

I agree with Jesse.  The test is the most critical part (for me) in considering a proposed fix.  If the test shows that the submitter understood the conditions which cause the bug, I'm much more willing to believe they understood the risks of the fix.

Mark Waite

[Romain SOMMERARD]

Hi,

I tried to checkout the project from github (master branch, 1.609 branch and others) but, without make any source editing, mvn test always fail.
Which branch need I to checkout to start ? Is it from Github or jenkins-ci (jenkins_main_trunk) ?

Thanks,
-- Romain

[Mark Waite]

If you want to contribute a test to the git plugin project, then you need to fork the github repository https://github.com/jenkinsci/git-plugin,git.  Clone the fork to your local machine, create a branch, make your changes, then push that branch back to your fork on github.  Then you open the github web page and use the button to create a pull request.

Similar pattern for the git client plugin, but the URL is https://github.com/jenkinsci/git-client-plugin.git

Mark Waite

--

You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/34d1d6de-74c0-4e20-a2dc-94d054b8358f%40googlegroups.com.

[Daniel Beck]

On 25.01.2016, at 15:15, Yassine Badache <yassine...@gmail.com> wrote:

> How can I know which classes the injected tests covers ?

Assume it covers nothing. It's really just a general test making sure nothing is too badly broken, e.g. that the plugin actually starts, that resource files contain no duplicate keys, that Jelly files look sane, etc. It covers no plugin specific functionality.

https://github.com/jenkinsci/maven-hpi-plugin/blob/5d26cc0cfc1830a30999b204e9702b8f33f243a2/src/main/java/org/jenkinsci/maven/plugins/hpi/TestInsertionMojo.java
https://github.com/jenkinsci/jenkins-test-harness/blob/542cf82e5efca6a42831caae88eb7e7446d10057/src/main/java/org/jvnet/hudson/test/PluginAutomaticTestBuilder.java
https://github.com/jenkinsci/jenkins-test-harness/blob/542cf82e5efca6a42831caae88eb7e7446d10057/src/main/java/org/jvnet/hudson/test/PropertiesTestSuite.java