The future of LTS certification

[Lucie Votypkova]

Hello,

I would like to start discussion about ATH selenium testing. After year of attempts to help with maintenance and LTS testing, we are dealing with problem that maintenance is becoming very time demanding and added value of it is not sufficient. The tests were created for replacing manual testing LTS with the most used plugins, showing that changes in Jenkins core does not break widely used plugins and of course testing of Jenkins UI. In current state is better then manual testing, but it is still time demanding to keep the step with UI changes in plugins and core and in the same time deal with browsers, driver, selenium issues and bugs in Jenkins (and its plugins). Due to a lot of constant failures (which are not connected with UI issues or integration of Jenkins with plugins in many cases), it can not be used for watching impact of every Jenkins release on current plugins and it does not save a lot of time in LTS testing (if we consider the time for checking constant failures plus time for maintenance) .  I collected the main problems I had to deal with. And a few questions which we should think about and suggestions what could help. Please feel free to share your thoughts.

Problems

• Maintenance is very time consuming.

• It is revealed only for very small amount of bugs in core or plugins (most failures are connected with changes in UI).

• The most of revealed plugin bugs in last year can be detected by functional testing.

• Some failures which are real bugs in plugins have a very low priority because it has a small impact on real user. But it sometime bloks whole testing and building workaround would not fit the main purpose - use jenkins as a real user do. This causes that there are real failures (sometime in the beginning of the tests) which will not be fixed in near future.

• There are a lot of tests failures because of changes in plugin UI. When it happens with widely used plugin which is a dependency of many other popular plugins (credentials, for example), the ATH tests are useless until this changes are fixed. For example: https://issues.jenkins-ci.org/browse/JENKINS-49026

• Vast majority of breakage caused by UI changes is mostly fixed by maintainers of ATH because plugins/core contributors are not generally aware of ATH existence, that they have broke some of the tests or even how to fix that.

• Reliability of ATH suite is now stuck in a cycle: ATH is not reliable because there are always quite a lot of failures and it never gives zero failures result. The most failures are caused by changes in plugin UI or low priority bugs in plugins which mainteners are not keen to fix. After a year attempts to fix ATH failures and achieve +- zero failures at least for LTS and the last versions of plugins, I realized that the issues appears faster than I am able to fix them. Now I am able to keep with them but not to decrease their amount.

• Due to many failures caused by UI changes, plugin bugs, browser upgrade, marionette/geckodriver bugs are not very useful tool for showing successful integration new core versions (and LTS) with the most widely used plugins. Currently to get such a measurement, it is necessary to verify about 30-50 failures in ATH (if they are still caused only by plugin bugs, UI changes….). Which is quite time consuming in comparing with situation when there is 5-10 failures.

• Important integration aspects has no coverage in automated testing?

• The testsuite takes ~20 hours to complete.

Questions:

• How to deal with tests that are broken (failing without detecting bug) for long time and there is noone to fix the test?

• How to deal with known bugs which breaks ATH test and do not seem to get fixed in near future?

• How to make maintainers of plugins to feel that it is beneficial to participate and help with maintenance of ATH, ideally having them actively engaged in making ATH compatible with their plugin changes?

Suggestions:

• Avoid ATH tests that duplicates JTH tests in core/plugin?

• Drawback: Would require PCT to be part of LTS certification.

• Use only subset of ATH for LTS certification?

• Drawback: even less attention to the second class citizens.

• Guideline of removing fragile tests.

• How to detect fragility?

• Guideline for including (or excluding) plugin tests into ATH based on the size of the userbase?

• Guideline for reporting broken plugin tests to plugin/test maintainer including test removal if not fixed in time?

• What would be the deadline?

• Avoid trivial tests in favor of meaningful use-cases

• Will cause collapsing/removal of many tests.

• Guideline for maximal number of tests / maximal runtime per plugin?

• Suggest perform ATH testing for pull requests of plugins which are stable for long time in ATH by changing Jenkinsfile (only tests for given plugin will be performed, not whole ATH)

• It will definitely make ATH more visible make people more cooperate, but!

• Will it be acceptable from time perspective (ATH is running for long time)?

• Will it complicate the pull request? ATH tests does not fail only because of bugs in given plugin, it does not have to be connected with given pull request.

• Can those tests be moved to plugin repositories altogether?

[Mark Waite]

My replies are inline.  I'm a plugin maintainer.  I am deeply attached to automated tests, especially for the plugins I maintain.

I have not yet had a positive experience with automated tests that use a web browser to perform those tests.  I've been in software development for a long time.  I've seen several attempts, but no personal positive experiences yet.

My replies are based on those biases.

No, it won't be acceptable from a time perspective (at least it seems unlikely to me).  PCT tests were added to the git plugin.  The cycle time for pull request evaluation increased beyond acceptable limits (from 15 minutes to an hour or more if I remember correctly).  That might have been acceptable except for the other issues (described later in this message)

 

• Will it complicate the pull request? ATH tests does not fail only because of bugs in given plugin, it does not have to be connected with given pull request.

PCT tests were added to the git plugin.  Random failures happened when there were no relevant code changes.  As a plugin maintainer, I had no interest in investigating the failure of a long-running test which fails for unrelated or transient reasons.  I didn't see the benefit of investigating those failures.  They were not telling me something especially useful about the problem they were reporting.  Time researching an unrelated or transient failure is time invested in something that rarely helps the plugin I maintain and rarely helps the Jenkins community.

I've shifted my "end to end" testing for the git plugin to use one multibranch pipeline job for each bug that I'm investigating (see jenkins-bugs).  I use the Jenkins Pipeline code and the build tools of my choice to perform checks that specific conditions of the specific bug are satisfied or not satisfied.  However, those end to end tests involve no web browser and tend to remain tightly focused on verifying a very specific bug, not an entire use case.

Use case verification still requires that I test interactively.

 

• Can those tests be moved to plugin repositories altogether?

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAMGCEQ1%2BQMESwByc_78p7r17b2cxxM9%3DqXz3zSBgE9ZNOMACGg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--

Thanks!

Mark Waite

[Ullrich Hafner]

Lucie, thank you for your excellent summary of the current situation!

Since only Mark responded so far this is an indication that actually only a few Jenkins developers are using ATH right now. I also think, that using ATH in the current way makes not much sense anymore. 

As you suggested, a valid option would be to split the two groups of tests (smoke tests and plugin specific tests) also physically. This would make much sense for me, too. Having the ATH tests for my plugins in my own plugin would improve my development cycle: The tests could be automatically started in our CI server as soon as I push some changes (or a PR has been submitted). Then I also would get feedback if I broke something in the UI (at least if I run these tests in travis). On the other hand, this would require that we must be more carefully when we refactor ATH core classes since now we won’t see compile errors (due to API changes) immediately. And if plugin authors do not really care about these test in their plugins then they can feel free to delete them. 

However, this approach makes LTS testing more difficult. At least in theory - as far as I understood your post, in the current state ATH is not helpful in ATH testing anymore.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAO49JtGmcygJqsvwmVgxEuTYtWfy8zuQ7LoAzAj%2BKy0AN%3DAApw%40mail.gmail.com.

[martinda]

Hello,

There is currently a Google Summer of Code (GSoC) proposal to improve the ATH.

Would anyone be able to lay out a plan to improve the ATH using that GSoC proposal? We are actually looking for mentors too.

The Jenkins Org will apply to GSoC 2019, and if we are accepted, improving the ATH might attract a student, which mean we'd start to see improvements (providing we also find a couple of mentors).

You can comment on the proposal, but if you feel like it should be re-written from scratch, simply create a new doc using the template. Working on the plan does not make you a mentor (but of course that would be nice).

You can find the gsoc special interest group on the gitter chat.

Martin d'Anjou

Jenkins GSoC 2019 Org Admin

On Saturday, December 22, 2018 at 8:22:32 AM UTC-5, Ullrich Hafner wrote:

[deleted]

[Oliver Gondža]

On 22/12/2018 14.22, Ullrich Hafner wrote:
> Lucie, thank you for your excellent summary of the current situation!
>
> Since only Mark responded so far this is an indication that actually
> only a few Jenkins developers are using ATH right now. I also think,
> that using ATH in the current way makes not much sense anymore.
>
> As you suggested, a valid option would be to split the two groups of
> tests (smoke tests and plugin specific tests) also physically. This
> would make much sense for me, too. Having the ATH tests for my plugins
> in my own plugin would improve my development cycle: The tests could be
> automatically started in our CI server as soon as I push some changes
> (or a PR has been submitted). Then I also would get feedback if I broke
> something in the UI (at least if I run these tests in travis). On the
> other hand, this would require that we must be more carefully when we
> refactor ATH core classes since now we won’t see compile errors (due to
> API changes) immediately. And if plugin authors do not really care about
> these test in their plugins then they can feel free to delete them.
>
> However, this approach makes LTS testing more difficult. At least in
> theory - as far as I understood your post, in the current state ATH is
> not helpful in ATH testing anymore.

Thanks for your insights, Ullrich. This is very useful as you are/ware
quite and active user and contributor. I hope the initiative we are
starting here with Lucie will get more traction after the winter break
so we will certainly get back to you.

--
oliver

[Sahil Khan]

Hello,

Last year in 2018 I applied for the similar GSoC Project for Improvement of ATH.Here is the link to my GSoC Proposal of 2018.

I would like to start contributing to the project from now as it is nearly impossible to complete the whole project in GSoC timeline.

I would like to get some suggestons on like how to start the project and if going for container technology then which one would be the best.

[Oleg Nenashev]

Thanks for the interest Sahil!

My advice would be to stick to Docker for now, unless you see special benefits of K8s and other systems.

We do not have a ready-to-fly Kubernetes setup inside the Jenkins infrastructure, and it's also quite complicated to do local development with it.

I would like to start contributing to the project from now as it is nearly impossible to complete the whole project in GSoC timeline.

Yes, likely. Note that we do not expect a GSoC student to complete the entire project of such scale. In your proposal you can focus on a particular area only so that you are not overcommitted.

Best regards,

Oleg

[Manuel Ramón León Jiménez]

Hello and happy new year.

Splitting the ATH tests is very interesting from my PoV. It will let plugin developers be more conscious of its existence and perhaps, it could help to have a more well maintained and useful set of tests.

Best regards.

--

You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/43fd6d65-fa15-457a-ba0a-78d15e0f3f49%40googlegroups.com.

[martinda]

Hello Manuel,

On Thursday, January 3, 2019 at 4:49:12 AM UTC-5, Manuel Ramón León Jiménez wrote:

Hello and happy new year.

Splitting the ATH tests is very interesting from my PoV. It will let plugin developers be more conscious of its existence and perhaps, it could help to have a more well maintained and useful set of tests.

 Would you be interested mentoring the ATH improvements proposal? We need at least one more mentor on that proposal.

Martin

[Manuel Ramón León Jiménez]

Hello Martin.

I don't have clear enough what this implies, but count on me if you need.

Best regards.

--

You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/df71c90a-ee16-4f2c-a22f-9290f6c6e443%40googlegroups.com.

[martinda]

On Friday, January 4, 2019 at 8:47:57 AM UTC-5, Manuel Ramón León Jiménez wrote:

Hello Martin.

I don't have clear enough what this implies, but count on me if you need

Thank you. Can you please add your gitter id to the mentor list for the ATH GSoC proposal?

I can try to clarify what GSoC mentoring implies.

Being a GSOC mentor has many benefits IMO. You get a student for full time for 4 months to work on your project (ATH in this case). You get to define and steer the project in ways that help you and the community. At the end of the program, the project has moved forward, and could even be entirely completed. The mentors becomes better at mentoring which can be seen as career development. Mentors do not work alone, they work as a team of mentors (3+) per project. Then there are org admins who guide and support mentors.

The mentor participates in the project definition up until about the first coding phase (see the timeline). Students officially start to apply in late March but to be honest, students are already asking questions in the gsoc gitter chat about the projects they find interesting. In any case, the mentor involvement is varies between 5 to 6 to 8 hours per week, depending on the phase of the project. At the start, time is invested in defining the project, answering students' questions, and working with students on a project design document and schedule. Once the coding starts, mentors meet students twice per week, and review their pull-requests on a daily basis, and continue to guide them. All communications are on public channels (no private email communications except for private matters).

Hope this brings some clarity to the role of a gsoc mentor,

Martin

[lvot...@redhat.com]

Thank you all for your replies and insights! 

Ullrich, I totally agree with splitting tests but putting them into plugins would increase work for maintainers and contributors (as was mentioned by Mark). So in that case I wanted start with plugins which maintainer is aware of ATH and does not mind to do it. So your proposal is great, we can start with analysis plugins - choose the one plugin and see how it works if you agree. I will create jira for it. If it goes well, we can move tests into more plugins. 

Oliver, do you wish to move all tests of plugins from ATH or just those which are not very interesting from integration point of view (for example let there 1-3 tests for the most used scenarios of plugin which have a lot of installation)? I would let in ATH a few interested scenarios for most used plugins. On the other hand it is not such a problem to run the selenium tests for the most used plugins separately. What do you thing about it?

Martin, it sounds great. You can count with me anytime you need any help with it.

Regards,

Lucie

[Oliver Gondža]

On 07/01/2019 14.28, lvot...@redhat.com wrote:
> Oliver, do you wish to move all tests of plugins from ATH or just those
> which are not very interesting from integration point of view (for
> example let there 1-3 tests for the most used scenarios of plugin which
> have a lot of installation)? I would let in ATH a few interested
> scenarios for most used plugins. On the other hand it is not such a
> problem to run the selenium tests for the most used plugins separately.
> What do you thing about it?

In the end, we still need a testsuite to certify the LTS (or other kind
of release) and a single repo with harness and tests or 2 split repos
would do. Though I suggest keeping the essential tests and harness
together so changes are easier to make and prove correct (otherwise any
meaningful contribution would span multiple repositories).

--
oliver

[Ullrich Hafner]

Am 07.01.2019 um 14:28 schrieb lvot...@redhat.com:

Thank you all for your replies and insights! 

Ullrich, I totally agree with splitting tests but putting them into plugins would increase work for maintainers and contributors (as was mentioned by Mark). So in that case I wanted start with plugins which maintainer is aware of ATH and does not mind to do it. So your proposal is great, we can start with analysis plugins - choose the one plugin and see how it works if you agree. I will create jira for it. If it goes well, we can move tests into more plugins. 

Ok. With a slightly changed topic: since the analysis plugins are end-of-life I would suggest to delete the tests all together. (Or keep some for LTS testing since the plugins will still be used for some time). I created a set of new tests for the warnings-ng plugin with students in the last summer (https://github.com/uhafner/acceptance-test-harness/commits/hm-edu-testing). I still need to polish them a little bit before we can continue. (Currently they are out-of-date since I changed some options in the 1.0 release.) I’m not sure if I get that done in January...

Oliver, do you wish to move all tests of plugins from ATH or just those which are not very interesting from integration point of view (for example let there 1-3 tests for the most used scenarios of plugin which have a lot of installation)? I would let in ATH a few interested scenarios for most used plugins. On the other hand it is not such a problem to run the selenium tests for the most used plugins separately. What do you thing about it?

Martin, it sounds great. You can count with me anytime you need any help with it.

Regards,

Lucie

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/c8f2b5aa-4259-4d75-91f9-d910ad5341b2%40googlegroups.com.

[Ullrich Hafner]

I’m almost done with the refactoring of the acceptance tests for the Warnings Next Generation Plugin.

I created a PR (WIP), so you can already have a look at the tests:

https://github.com/jenkinsci/acceptance-test-harness/pull/479

The tests work fine on Linux/Chrome and most times on macOS/Chrome and macOS/Firefox (sometimes I get timeouts).

I still need to investigate why they not work on Linux/Firefox.

One idea would be to move these tests to the Warnings Next Generation Plugin. Then I can use these tests to validate PRs to my plugin as well. 

[ogondza]

Thank you everyone for the feedback. We have put together an initial draft of what will become new process of ATH maintenance. I intend to collectively craft the final shape of the document in a couple of coming weeks and then make it official by adding it as a github repo contribution guidelines.

There are several long term goals we kept in mind:

- Tolerable running time over breadth of the suite.

- Reliable build status over keeping every test ever written.

Please, take a time to digest that and let us know what you think. Every question, suggestion or correction is welcome.

https://docs.google.com/document/d/1k9sDlssxs0URZOjbF3jmBhTf-LoA7JQKj5TLIONmuVA/edit#

Thanks

--

oliver

[ogondza]

There is a striving discussion in the docs. One thing I failed to realize we need soon enough is the list of "Recognized use-cases" of ATH (drafted in the doc) I would like to get feedback on from as many people as I can.

Another hot topic is how do deal with tests failing because of a defect in a component during the time it is being fixed. On one hand people do not want to be annoyed by known red tests but on the other we need ATH to be able to point out that defect to people interested in using the defective component.

Do not hesitate to join us!

Thanks!

--

oliver