Gerrit Trigger plugin - detect when number of failing tests has INCREASED

[Matthew...@diamond.ac.uk]

I've just started experimenting with the Gerrit Trigger Plugin. It's great, but I have one problem:

If the build job is UNSTABLE (i.e. if any tests fail) Jenkins will post this status back to Gerrit, and flag the Gerrit patchset as not passing. That strategy only works if the normal condition is "all tests pass".

In my case, there is a test case, unrelated to the patchset under test, that has been failing for some time. I'd like the Gerrit trigger to only mark the patchset being tested if introduces a NEW test failure. Is there a way to achieve this?

(Yes I do now that having failing tests that have not been fixed is poor practice, but in a large test suite, that is sometimes the case).

Thanks
Matthew


--
This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail.
Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd.
Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message.
Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

[James Nord]

On 12/01/2015 16:58, Matthew...@Diamond.ac.uk wrote:
> I've just started experimenting with the Gerrit Trigger Plugin. It's great, but I have one problem:
>
> If the build job is UNSTABLE (i.e. if any tests fail) Jenkins will post this status back to Gerrit, and flag the Gerrit patchset as not passing. That strategy only works if the normal condition is "all tests pass".

Your logic is the wrong way around.

You want to mark a build with the flaky test as stable - so this would
be almost the same for a normal job with the complication that Gerrit
patchsets are not linear...

> In my case, there is a test case, unrelated to the patchset under test, that has been failing for some time. I'd like the Gerrit trigger to only mark the patchset being tested if introduces a NEW test failure. Is there a way to achieve this?

Take a look at https://github.com/samsta/quarantine this may be the only
way of ignoring your bad test.
Note it forks the junit plugin and is probably missing features fixes in
that junit plugin.

> (Yes I do now that having failing tests that have not been fixed is poor practice, but in a large test suite, that is sometimes the case).

In this case I really would just recommend removing the test and raising
a defect for the team to fix it and tracking it that way - it will
reduce the headache (given you are not currently fixing it) and it gets
the team to accept that they can't push changes to the branch when the
code is broken.

/James