Hello,
I would like to start discussion about ATH selenium testing. After year of attempts to help with maintenance and LTS testing, we are dealing with problem that maintenance is becoming very time demanding and added value of it is not sufficient. The tests were created for replacing manual testing LTS with the most used plugins, showing that changes in Jenkins core does not break widely used plugins and of course testing of Jenkins UI. In current state is better then manual testing, but it is still time demanding to keep the step with UI changes in plugins and core and in the same time deal with browsers, driver, selenium issues and bugs in Jenkins (and its plugins). Due to a lot of constant failures (which are not connected with UI issues or integration of Jenkins with plugins in many cases), it can not be used for watching impact of every Jenkins release on current plugins and it does not save a lot of time in LTS testing (if we consider the time for checking constant failures plus time for maintenance) . I collected the main problems I had to deal with. And a few questions which we should think about and suggestions what could help. Please feel free to share your thoughts.
Problems
Maintenance is very time consuming.
It is revealed only for very small amount of bugs in core or plugins (most failures are connected with changes in UI).
The most of revealed plugin bugs in last year can be detected by functional testing.
Some failures which are real bugs in plugins have a very low priority because it has a small impact on real user. But it sometime bloks whole testing and building workaround would not fit the main purpose - use jenkins as a real user do. This causes that there are real failures (sometime in the beginning of the tests) which will not be fixed in near future.
There are a lot of tests failures because of changes in plugin UI. When it happens with widely used plugin which is a dependency of many other popular plugins (credentials, for example), the ATH tests are useless until this changes are fixed. For example: https://issues.jenkins-ci.org/browse/JENKINS-49026
Vast majority of breakage caused by UI changes is mostly fixed by maintainers of ATH because plugins/core contributors are not generally aware of ATH existence, that they have broke some of the tests or even how to fix that.
Reliability of ATH suite is now stuck in a cycle: ATH is not reliable because there are always quite a lot of failures and it never gives zero failures result. The most failures are caused by changes in plugin UI or low priority bugs in plugins which mainteners are not keen to fix. After a year attempts to fix ATH failures and achieve +- zero failures at least for LTS and the last versions of plugins, I realized that the issues appears faster than I am able to fix them. Now I am able to keep with them but not to decrease their amount.
Due to many failures caused by UI changes, plugin bugs, browser upgrade, marionette/geckodriver bugs are not very useful tool for showing successful integration new core versions (and LTS) with the most widely used plugins. Currently to get such a measurement, it is necessary to verify about 30-50 failures in ATH (if they are still caused only by plugin bugs, UI changes….). Which is quite time consuming in comparing with situation when there is 5-10 failures.
Important integration aspects has no coverage in automated testing?
The testsuite takes ~20 hours to complete.
Questions:
How to deal with tests that are broken (failing without detecting bug) for long time and there is noone to fix the test?
How to deal with known bugs which breaks ATH test and do not seem to get fixed in near future?
How to make maintainers of plugins to feel that it is beneficial to participate and help with maintenance of ATH, ideally having them actively engaged in making ATH compatible with their plugin changes?
Suggestions:
Avoid ATH tests that duplicates JTH tests in core/plugin?
Drawback: Would require PCT to be part of LTS certification.
Use only subset of ATH for LTS certification?
Drawback: even less attention to the second class citizens.
Guideline of removing fragile tests.
How to detect fragility?
Guideline for including (or excluding) plugin tests into ATH based on the size of the userbase?
Guideline for reporting broken plugin tests to plugin/test maintainer including test removal if not fixed in time?
What would be the deadline?
Avoid trivial tests in favor of meaningful use-cases
Will cause collapsing/removal of many tests.
Guideline for maximal number of tests / maximal runtime per plugin?
Suggest perform ATH testing for pull requests of plugins which are stable for long time in ATH by changing Jenkinsfile (only tests for given plugin will be performed, not whole ATH)
It will definitely make ATH more visible make people more cooperate, but!
Will it be acceptable from time perspective (ATH is running for long time)?
Will it complicate the pull request? ATH tests does not fail only because of bugs in given plugin, it does not have to be connected with given pull request.
Can those tests be moved to plugin repositories altogether?
Will it complicate the pull request? ATH tests does not fail only because of bugs in given plugin, it does not have to be connected with given pull request.
Can those tests be moved to plugin repositories altogether?
--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAMGCEQ1%2BQMESwByc_78p7r17b2cxxM9%3DqXz3zSBgE9ZNOMACGg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAO49JtGmcygJqsvwmVgxEuTYtWfy8zuQ7LoAzAj%2BKy0AN%3DAApw%40mail.gmail.com.
[deleted]
I would like to start contributing to the project from now as it is nearly impossible to complete the whole project in GSoC timeline.
--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/43fd6d65-fa15-457a-ba0a-78d15e0f3f49%40googlegroups.com.
Hello and happy new year.Splitting the ATH tests is very interesting from my PoV. It will let plugin developers be more conscious of its existence and perhaps, it could help to have a more well maintained and useful set of tests.
--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/df71c90a-ee16-4f2c-a22f-9290f6c6e443%40googlegroups.com.
Hello Martin.I don't have clear enough what this implies, but count on me if you need
Am 07.01.2019 um 14:28 schrieb lvot...@redhat.com:Thank you all for your replies and insights!Ullrich, I totally agree with splitting tests but putting them into plugins would increase work for maintainers and contributors (as was mentioned by Mark). So in that case I wanted start with plugins which maintainer is aware of ATH and does not mind to do it. So your proposal is great, we can start with analysis plugins - choose the one plugin and see how it works if you agree. I will create jira for it. If it goes well, we can move tests into more plugins.
Oliver, do you wish to move all tests of plugins from ATH or just those which are not very interesting from integration point of view (for example let there 1-3 tests for the most used scenarios of plugin which have a lot of installation)? I would let in ATH a few interested scenarios for most used plugins. On the other hand it is not such a problem to run the selenium tests for the most used plugins separately. What do you thing about it?Martin, it sounds great. You can count with me anytime you need any help with it.Regards,Lucie
--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/c8f2b5aa-4259-4d75-91f9-d910ad5341b2%40googlegroups.com.