Project ideas for student projects in the area of testing

[Ullrich Hafner]

In the summer semester I will again give a testing course here at the University of Applied Sciences in Munich.
Several master students have the chance to improve their testing skills on a real project: Jenkins.
The scope is integration and system testing (maybe also some unit testing if there is some work required, too).

Do you have some ideas what needs to be done in the area of testing in Jenkins? What are our pain points, etc.?

Currently I’m thinking about the following topics:
- Extend the existing acceptance test cases (ATH) for
a) core (i.e., create tests for Jira issues)
b) plugins (i.e., create tests for the most important and not yet covered plugins)
- Improve the speed of the ATH execution
- Make the ATH work on all operation systems (Windows, Mac, Linux)
- Set up some performance tests for Jenkins
- Participate in LTS testing

Are there any other things where we could help?

Ulli

[Victor Martinez]

Maybe it is a bit out of the scope and I don't know whether there is already a ongoing testing framework for the pipeline. But, is it something feasible? It might be a really good contribution to the community
Cheers

[Mark Waite]

I'd love to have students help create Jenkins jobs which illustrate specific bugs in the git plugin, for example.  Sample jobs which illustrate different bugs as Jenkins jobs are available from the lts-with-plugins branch of my docker repository.  I've found it very powerful to have a Jenkins job per bug report, either as a pipeline job (branch in a git repository like my jenkins-bugs repository) or as a freestyle job which shows the bug and marks itself unstable when the bug is detected.

I hope to propose a blog posting describing the technique of using one or more Jenkins job definitions for each bug.  I've found it very helpful to verify bugs remain fixed while detecting problems on multiple platforms.

Mark Waite

On Mon, Feb 13, 2017 at 11:44 AM Victor Martinez <victormar...@gmail.com> wrote:

Maybe it is a bit out of the scope and I don't know whether there is already a ongoing testing framework for the pipeline. But, is it something feasible? It might be a really good contribution to the community
Cheers

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/bf2dbfa0-c524-4673-bfb5-6e55c05de2d2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Oleg Nenashev]

Hi Ulli,

Do you consider aligning these activities with Google Summer of Code? Currently we are looking for project ideas, and actually test framework improvements + test improvements could be a good project.

I understand that GSoC implies a significant time dedication by students, which likely goes far beyond a single course, but it may it can be an option if the students are interested.

Thanks in advance,
Oleg

понедельник, 13 февраля 2017 г., 22:40:00 UTC+3 пользователь Mark Waite написал:

[Ullrich Hafner]

This looks somewhat too complicated for a course, would better fit for a thesis…

> Am 13.02.2017 um 19:44 schrieb Victor Martinez <victormar...@gmail.com>:
>
> Maybe it is a bit out of the scope and I don't know whether there is already a ongoing testing framework for the pipeline. But, is it something feasible? It might be a really good contribution to the community
> Cheers
>

[Ullrich Hafner]

Am 13.02.2017 um 20:39 schrieb Mark Waite <mark.ea...@gmail.com>:

I'd love to have students help create Jenkins jobs which illustrate specific bugs in the git plugin, for example.  Sample jobs which illustrate different bugs as Jenkins jobs are available from the lts-with-plugins branch of my docker repository.  I've found it very powerful to have a Jenkins job per bug report, either as a pipeline job (branch in a git repository like my jenkins-bugs repository) or as a freestyle job which shows the bug and marks itself unstable when the bug is detected.

I hope to propose a blog posting describing the technique of using one or more Jenkins job definitions for each bug.  I've found it very helpful to verify bugs remain fixed while detecting problems on multiple platforms.

That sounds interesting and would be a good basis. Do you think you can give some more pointers until the course starts (15. March)?

A job per bug report is also a good working unit so we can start with a simple one and then move to more difficult setups. If there would be an example to start with it would be even better.

What is not yet clear: these jobs you are referring to sound like a test setup for the ATH. So if you have such a setup it would be not much harder to add a verification step as an ATH test case. Then you will also have regression tests. Would that be also useful?

In which order should we pick up issues? Priority in Jira? Or Newest first?

Mark Waite

On Mon, Feb 13, 2017 at 11:44 AM Victor Martinez <victormar...@gmail.com> wrote:

Maybe it is a bit out of the scope and I don't know whether there is already a ongoing testing framework for the pipeline. But, is it something feasible? It might be a really good contribution to the community
Cheers

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/bf2dbfa0-c524-4673-bfb5-6e55c05de2d2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAO49JtFfBbcs16yghxjQHB58EF3%2B%3D3mUGOck1WK-EpQ_eaD17w%40mail.gmail.com.

[Jesse Glick]

I would also much rather see a genuine regression test. If some aspect of ATH is too weak to reproduce the bug properly, then that is what needs fixing.

[Mark Waite]

I'm think that a Jenkins job which asserts the conditions related to a bug report is a regression test.  I admit that the Jenkins jobs I create are not Selenium tests, and they do not require a web browser in order to run.  They are definitely not able to exercise JavaScript, and they do not evaluate behavior across multiple browsers.  If the goal is to check JavaScript or to check browser compatibility, then Jenkins jobs are a poor choice.

If the goal is to duplicate a bug report, and reuse the steps that duplicated the bug report for later regression testing, then I think a Jenkins job is simpler to create, easier to maintain, and easier to diagnose than a Selenium test.

Ulrich, if you are interested in some "test drive" bug reports from the git plugin or the git client plugin, you can review how I use the technique to report and duplicate bugs in:

• JENKINS-42204 - NPE due to recent GitSCMSource change
  
• JENKINS-41948 - NoSuchMethodError scanning multi-branch project
• JENKINS-41867 - IllegalStateException scanning multi-branch project
• JENKINS-41785 - Pipeline status icon no longer shows worst status

If your students would like to try to duplicate bugs that I think are interesting, I would suggest starting with:

• JENKINS-38860 - renamed submodules fail checkout
• JENKINS-41553 - submodule defs read from wrong location
• JENKINS-42004 - jobs triggered unnecessarily (scenario 1)
• JENKINS-41906 - jobs triggered unnecessarily (scenario 2)
• JENKINS-40733 - notifyCommit provides wrong parameters
• JENKINS-40352 - duplicate changesets in pipeline jobs (and other jobs)
• JENKINS-39621 - infinite loop of job rebuilds

Those bug reports are likely well suited to creating one or more jobs to show that the bug exists.  If they want to use the infrastructure already provided in the lts-with-plugins branch of my docker repository, they're welcome to do that.

If you decide that the technique works well enough for the students, then you can consider attempting other bugs from the git-plugin and git-client-plugin projects, most recent bugs first.  I suspect other SCM system bugs (Subversion, etc.) are reasonably well suited to the same technique.

I've also found that it is quite powerful to create a pipeline based branch per bug verification job so that Jenkins will automatically create the job and execute it.  You can see examples of that technique in my jenkins-bugs repository (which is referenced in several different ways from the docker instance).  That has the benefit that I don't have to commit anything to the docker repository, and the Jenkins pipeline code still creates and runs a job per bug.

I'm also OK if you decide it is better to submit pull requests to the acceptance test harness.  The acceptance test harness has the compelling benefit that there are many more people creating Selenium tests with the acceptance test harness, and there are many more locations where those tests can be executed.

In my case, I don't need the flexibility of a Selenium test, and I want the easier diagnose and trivial parallel execution that comes from running a Jenkins instance with multiple slaves and with tests defined as Jenkins jobs.

Thanks!

Mark Waite

P.S. I have pipeline utility functions assertLogContains and assertLogDoesNotContain to search for specific strings in the log files.  When one of those functions finds something unexpected, it marks the job as "Unstable".  I can then search for all failed jobs and all unstable jobs in my Jenkins instance to locate test failures.  Because it is a docker instance, I can spread the execution load trivially across multiple agents by adding or deleting agents.  I get interesting (and useful) test randomization by choosing a mix of operating systems (Windows, CentOS 6. CentOS 7, Debian 7, Debian 8, Ubuntu 14, Ubuntu 16, etc.) and git versions.

On Fri, Feb 17, 2017 at 5:47 AM Jesse Glick <jgl...@cloudbees.com> wrote:

I would also much rather see a genuine regression test. If some aspect of ATH is too weak to reproduce the bug properly, then that is what needs fixing.

--

You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr2eGwB-hbFXtJa%2BJE72HbTqE6t8iTqYQO_iRL-SBarbBw%40mail.gmail.com.

[Ullrich Hafner]

I’ll ask them, but from the past courses I can say that they will probably have not enough time: most of them are doing their master in parallel to their normal job. 

(I am already very happy if they can spend a day per week for the project at the least...)

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/75a29c62-c428-4ffe-8b59-a74053281829%40googlegroups.com.

[Jesse Glick]

If Selenium testing is overkill, then just reproduce the bug in `git-plugin/src/test/`. You can add a `docker-fixtures` test dependency if it is helpful.

[Mark Waite]

Thanks for the suggestion.  Any pointers to examples of cases which use docker-fixtures as a test dependency?

Mark Waite

On Tue, Feb 21, 2017 at 6:45 AM Jesse Glick <jgl...@cloudbees.com> wrote:

If Selenium testing is overkill, then just reproduce the bug in `git-plugin/src/test/`. You can add a `docker-fixtures` test dependency if it is helpful.

--

You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr0EL9okVMftzt_BSdZVntpcs9ZgkAcwpEaHibV1W8hXQg%40mail.gmail.com.

[Robert Sandell]

I just wrote one for my experimentation with the  ldap plugin here https://github.com/rsandell/ldap-plugin/commit/2c85eb0d3f258529562b111a75604e0af7a3486c 

I think Jesse added some in the mercurial plugin when he extracted docker-fixtures from the ATH.

And the ATH has plenty of examples as well as docker-fixtures itself that has some tests that I used for inspiration.

/B

On Tue, Feb 21, 2017 at 2:48 PM, Mark Waite <mark.ea...@gmail.com> wrote:

Thanks for the suggestion.  Any pointers to examples of cases which use docker-fixtures as a test dependency?

Mark Waite

On Tue, Feb 21, 2017 at 6:45 AM Jesse Glick <jgl...@cloudbees.com> wrote:

If Selenium testing is overkill, then just reproduce the bug in `git-plugin/src/test/`. You can add a `docker-fixtures` test dependency if it is helpful.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr0EL9okVMftzt_BSdZVntpcs9ZgkAcwpEaHibV1W8hXQg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAO49JtHvoThDzZdgAucJV1OG6jPjjsUFkEx2SGT33om-cf_b9A%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Robert Sandell

Software Engineer

CloudBees Inc.

[Jesse Glick]

On Tue, Feb 21, 2017 at 10:22 AM, Robert Sandell <rsan...@cloudbees.com> wrote:
> I think Jesse added some in the mercurial plugin when he extracted
> docker-fixtures from the ATH.

A work in progress but yes.

> And the ATH has plenty of examples as well as docker-fixtures itself

Right, this is where most of the examples would be.