Request to host plug-in: Flaky Tests Handler

[Qingzhou Luo]

Hi,

I'd like to host our new plug-in.

Plugin name: flaky-test-handler

My github account: qingzhouluo/seriousamlqz (Please add both, thanks).

Description:

This plugin is designed to handle flaky tests. Currently it aims for Git and Maven.

1. We recently made some contribution to Maven Surefire to add the new option -rereunFailingTestsCount. It was already accepted, and may be released with 2.18 in the near future:

https://github.com/apache/maven-surefire/commit/fefaae7f0534a59f52c046a64c96987e8561dd48

It basically let users to choose to re-run failing tests up to N times, and if it passes any of those N times, it will be marked as a "Flake" and the build will be marked as successful. The generated test report will contain each re-run information.

So the first part of our plugin is to integrate with this new feature from Maven Surefire. It will parse the report and display flaky tests, output information of all the re-runs(stacktrace, output, etc) on test result page.

https://drive.google.com/file/d/0Bzfyw2NYCLFBV2lnMVlrWVhueGc

https://drive.google.com/file/d/0Bzfyw2NYCLFBZXFxSThlMWlaeVU

2. For each failing build we provide a "Deflake" action. It is different from rebuilding the project, as it will:

1) Checkout the exact revision of that failing build (GIT)

2) Only run all those failed tests.

So this will give developer a good idea about how many tests are flaky for this build.

https://drive.google.com/file/d/0Bzfyw2NYCLFBR3RDM3psQjhIRUk/

https://drive.google.com/file/d/0Bzfyw2NYCLFBeTFjQk5sQW5XR3M/

3. Aggregate information from both 1 and 2, and display them to the users.

For each git revision and each test, we count how many times it passed, how many times it failed. So if a test passed and failed for the both revision, we count it as a flake; if it always failed, then it's a fail, otherwise it's a pass.

https://drive.google.com/file/d/0Bzfyw2NYCLFBRlI2SHpubTltd28/

https://drive.google.com/file/d/0Bzfyw2NYCLFBVWJNWFAweEl1Y28/

So this help users to see what are my most flaky tests (they can compute some kind of flakiness), so they can fix them first. It basically serves as the test health page.

We have a repo on github, however it was empty right now. 

https://github.com/google/jenkins-flaky-test-report

Will push our code to it very soon. So here I want to get a repo from Jenkins so we can push our code to Jenkins too.

Thank you very much!

Regards,

Qingzhou

[Bruno Meneguello]

Amazing idea! In my company we have developed a RunConfiguration to do exactly that and implemented a plugin to do the same thing (but haven't opened it because it become a very specific implementation)

Thumbs up for the initiative!

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Qingzhou Luo]

Thanks for supporting, Bruno. Still waiting for approval..

[Vincent Latombe]

Hello Quingzhou,

I have just created https://github.com/jenkinsci/flaky-test-handler for you and granted commit access.

Welcome aboard!

Vincent

[Fabio Farronato]

Hi all,
  I'm in trouble with the Subversion plug-in, here the scenario:

My job have a parameter "SOURCE" to pass svn location in format "http://svnserv/repo/myproject/branches/7.20.0@506192".
In job configuration I defined the SCM location using a variable $SOURCE.
The first time the job ran out compiling successfully.
Because the build was a test, the branch has been removed and is no longer available in the latest revision.
Now if i try to require a build with the same parameter the job fail to check out:

Location 'http://svnserv/repo/myproject/branches/7.20.0@506192' does not exist
Cleaning local Directory .
Checking out http://svnserv/repo/myproject/branches/7.20.0@506192 at revision 506192
ERROR: Failed to check out http://svnserv/repo/myproject/branches/7.20.0@506192
org.tmatesoft.svn.core.SVNException: svn: E160013: '/repo/!svn/bc/521195/myproject/branches/7.20.0' path not found: 404 Not Found (http://svnserv)
	at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:64)
	at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:51)
	at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.getLocationsImpl(DAVRepository.java:1068)
	at org.tmatesoft.svn.core.io.SVNRepository.getLocations(SVNRepository.java:1088)
	at org.tmatesoft.svn.core.io.SVNRepository.getLocations(SVNRepository.java:1516)
	at org.tmatesoft.svn.core.internal.wc2.SvnRepositoryAccess.getLocations(SvnRepositoryAccess.java:178)
	at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgRepositoryAccess.createRepositoryFor(SvnNgRepositoryAccess.java:45)
	at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.checkout(SvnNgAbstractUpdate.java:706)
	at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgCheckout.run(SvnNgCheckout.java:14)
	at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgCheckout.run(SvnNgCheckout.java:9)
	at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgOperationRunner.run(SvnNgOperationRunner.java:20)
	at org.tmatesoft.svn.core.internal.wc2.SvnOperationRunner.run(SvnOperationRunner.java:20)
	at org.tmatesoft.svn.core.wc2.SvnOperationFactory.run(SvnOperationFactory.java:1238)
	at org.tmatesoft.svn.core.wc2.SvnOperation.run(SvnOperation.java:294)
	at org.tmatesoft.svn.core.wc.SVNUpdateClient.doCheckout(SVNUpdateClient.java:777)
	at hudson.scm.subversion.CheckoutUpdater$1.perform(CheckoutUpdater.java:99)
	at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:161)
	at hudson.scm.SubversionSCM$CheckOutTask.perform(SubversionSCM.java:1006)
	at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:987)
	at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:963)
	at hudson.FilePath.act(FilePath.java:922)
	at hudson.FilePath.act(FilePath.java:895)
	at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:912)
	at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:848)
	at hudson.scm.SCM.checkout(SCM.java:488)
	at hudson.model.AbstractProject.checkout(AbstractProject.java:1254)
	at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:624)
	at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:530)
	at hudson.model.Run.execute(Run.java:1732)
	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
	at hudson.model.ResourceController.execute(ResourceController.java:88)
	at hudson.model.Executor.run(Executor.java:234)
Caused by: svn: E160013: '/repo/!svn/bc/521195/myproject/branches/7.20.0' path not found: 404 Not Found (http://svnserv)
	at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:208)
	at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:189)
	at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:141)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPRequest.createDefaultErrorMessage(HTTPRequest.java:455)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPRequest.readError(HTTPRequest.java:290)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPRequest.dispatch(HTTPRequest.java:215)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection._request(HTTPConnection.java:480)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:382)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:373)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:361)
	at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.performHttpRequest(DAVConnection.java:707)
	at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.doReport(DAVConnection.java:334)
	at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.doReport(DAVConnection.java:324)
	at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.getLocationsImpl(DAVRepository.java:1062)
	... 30 more

I try to check out the URL with svn client from my desktop and it works.
I think the problem is in variable resolution and parsing svn location, but I don't know where looking for in subversion plugin source code.
Have you any idea?

In addition I tried to register at https://jenkins-ci.org/account/signup, for submit an issue, but the server answers me with:

HTTP ERROR 500

Problem accessing /account/doSignup. Reason:

challenge missing

Caused by:

java.lang.Error: challenge missing at org.jenkinsci.account.Application.doDoSignup(Application.java:108)
....


Can someone help me?
Thanks in advance

Fabio

[Jesse Glick]

On Tue, Aug 5, 2014 at 12:16 AM, Vincent Latombe
<vincent...@gmail.com> wrote:
> I have just created https://github.com/jenkinsci/flaky-test-handler for you

Do not use this. Plugin repositories should have a name ending with
‘-plugin’. (The artifactId should _not_.)

[Jesse Glick]

On Fri, Aug 1, 2014 at 5:44 PM, Qingzhou Luo <qingz...@google.com> wrote:
> We recently made some contribution to Maven Surefire to add the new
> option -rereunFailingTestsCount.

Sounds very valuable.

> the first part of our plugin is to integrate with this new feature from
> Maven Surefire. It will parse the report and display flaky tests, output
> information of all the re-runs(stacktrace, output, etc) on test result page.

Very nice. We could definitely use this on

https://jenkins.ci.cloudbees.com/job/core/job/jenkins-core/

for example. Currently we rely on JUnit-based means of suppressing flaky tests:

https://github.com/jenkinsci/jenkins/commit/d29a90150c839855d4ff0c084c3cd838b48fab5c
https://github.com/jenkinsci/jenkins/commit/7bbf36b7f0de43db908b23278f39ad8cc573186c
https://github.com/jenkinsci/jenkins/commit/6fd76ea2e882d7439d3d0fb50df79aee3c073155

which could result in hiding genuine regressions. Of course you could
do the rerunning logic in a custom JUnit runner but this would not
compose well, and displaying these notes in Jenkins is a good idea.

> 2. For each failing build we provide a "Deflake" action. It is different
> from rebuilding the project, as it will:
>
> 1) Checkout the exact revision of that failing build (GIT)
> 2) Only run all those failed tests.

Pity that this can only work in constrained circumstances: that you
are using Git, that you have a freestyle project, and that there are
no particular side effects of running the build (I presume with
$MAVEN_OPTS or something set to specify which tests to run).

> https://github.com/google/jenkins-flaky-test-report
>
> Will push our code to it very soon. So here I want to get a repo from
> Jenkins so we can push our code to Jenkins too.

By the way it is discouraged to have two repositories with the same
plugin—better to have the @jenkinsci repo host definitive sources.
Otherwise there is endless confusion when people want to contribute
pull requests and so on. Of course Git lets you reconcile changes
eventually but at the cost of delays and conflicts.

[Vincent Latombe]

Sorry for that, I realised it after the fact but couldn't perform the rename operation.

Vincent

[Jesse Glick]

On Thu, Aug 7, 2014 at 7:57 AM, Vincent Latombe
<vincent...@gmail.com> wrote:
> Sorry for that, I realised it after the fact but couldn't perform the rename
> operation.

OK, renamed: https://github.com/jenkinsci/flaky-test-handler-plugin

[Vincent Latombe]

Thanks a lot Jesse.

Vincent

[Qingzhou Luo]

Thank you very much, Jesse. Your suggestions are very valuable.

I'm not very familiar with Maven 2/3 build and it looks like as you said, most of the features only work for freestyle build. In a Maven 2/3 build it looks like JUnit and its publisher/result archiver is embedded inside the build process, so the plugin cannot find its own place.

Supporting SCMs other than GIT (for example, SVN) may not be that hard, but have all the features supported in Maven 2/3 project looks hard. I am still pretty new to Jenkins :)

Regards,

Qingzhou

--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-dev/VcXaSGprUTU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jenkinsci-de...@googlegroups.com.

[Luca Naldini]

I'm confused on why the plugin does not work in our environment.

I've created a freestyle project, I have set the -Dsurefire.rerunFailingTestsCount=10 in the maven goal options, I have added "publish JUnit test result reports" and "Publish JUnit flaky test reports" to the build steps. 

Also I've added the "Publish JUnit flaky test stats":

But after the run of the tests, all I get is:

If I go inside "Latest Test Result" I can see details of the ran tests, so I guess that the surefire reports are published and parsed correctly.

Any idea of what might cause this issue?

Thanks