[JIRA] (JENKINS-61198) Thresholds not reliable

16 views
Skip to first unread message

Simon.Richter@hogyros.de (JIRA)

unread,
Feb 24, 2020, 8:49:05 AM2/24/20
to jenkinsc...@googlegroups.com
Simon Richter created an issue
 
Jenkins / Bug JENKINS-61198
Thresholds not reliable
Issue Type: Bug Bug
Assignee: Nikolas Falco
Components: xunit-plugin
Created: 2020-02-24 13:48
Priority: Minor Minor
Reporter: Simon Richter

We have a few jobs that use the xUnit plugin with multiple sources to collect results from multiple test frameworks. The build should be marked as unstable if there is a single test failure.

In several builds, the build is marked as successful with multiple test failures. My suspicion is that thresholds are checked only for one of the sources, not all of them cumulatively.

[Example build](https://jenkins.simonrichter.eu/job/linux-kicad-head/4211/)
Add Comment Add Comment
 
This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)
Atlassian logo

Simon.Richter@hogyros.de (JIRA)

unread,
Feb 24, 2020, 8:51:06 AM2/24/20
to jenkinsc...@googlegroups.com
Simon Richter updated an issue
Change By: Simon Richter
We have a few jobs that use the xUnit plugin with multiple sources to collect results from multiple test frameworks. The build should be marked as unstable if there is a single test failure.

In several builds, the build is marked as successful with multiple test failures. My suspicion is that thresholds are checked only for one of the sources, not all of them cumulatively.

Simon.Richter@hogyros.de (JIRA)

unread,
Feb 24, 2020, 8:52:02 AM2/24/20
to jenkinsc...@googlegroups.com
Simon Richter updated an issue
We have a few jobs that use the xUnit plugin with multiple sources to collect results from multiple test frameworks. The build should be marked as unstable if there is a single test failure.

In several builds, the build is marked as successful with multiple test failures. My suspicion is that thresholds are checked only for one of the sources, not all of them cumulatively.

Simon.Richter@hogyros.de (JIRA)

unread,
Feb 24, 2020, 8:53:06 AM2/24/20
to jenkinsc...@googlegroups.com
Simon Richter updated an issue
 
Jenkins / Bug JENKINS-61198
Thresholds not reliable
Change By: Simon Richter
We have a few jobs that use the xUnit plugin with multiple sources to collect results from multiple test frameworks. The build should be marked as unstable if there is a single test failure.

In several builds, the build is marked as successful with multiple test failures. My suspicion is that thresholds are checked only for one of the sources, not all of them cumulatively.

[Example build| [ https://jenkins.simonrichter.eu/job/linux-kicad-head/4211/]

nfalco79@hotmail.com (JIRA)

unread,
Feb 24, 2020, 8:57:07 AM2/24/20
to jenkinsc...@googlegroups.com
Nikolas Falco commented on Bug JENKINS-61198
 
Re: Thresholds not reliable

As you can see from the logs the threadshold check is performed at the end.

INFO: Starting to record.
INFO: Processing BoostTest-1.x (default)
INFO: [BoostTest-1.x (default)] - 7 test report file(s) were found with the pattern 'build/qa/**/*.boost-results.xml' relative to '/var/lib/jenkins-agent/workspace/linux-kicad-head' for the testing framework 'BoostTest-1.x (default)'.
INFO: Processing JUnit
INFO: [JUnit] - 6 test report file(s) were found with the pattern 'build/qa/*.xunit-results/*.xml' relative to '/var/lib/jenkins-agent/workspace/linux-kicad-head' for the testing framework 'JUnit'.
INFO: Processing CTest-Version 3.x (default)
INFO: [CTest-Version 3.x (default)] - 2 test report file(s) were found with the pattern 'build/Testing/**/*.xml' relative to '/var/lib/jenkins-agent/workspace/linux-kicad-head' for the testing framework 'CTest-Version 3.x (default)'.
INFO: Check 'Failed Tests' threshold.
INFO: Setting the build status to SUCCESS
INFO: Stopping recording.

How did you configure the threshold?

Simon.Richter@hogyros.de (JIRA)

unread,
Feb 24, 2020, 9:31:03 AM2/24/20
to jenkinsc...@googlegroups.com
Simon Richter updated an issue
Change By: Simon Richter
Attachment: warn.png

Simon.Richter@hogyros.de (JIRA)

unread,
Feb 24, 2020, 9:31:03 AM2/24/20
to jenkinsc...@googlegroups.com
Simon Richter commented on Bug JENKINS-61198
 
Re: Thresholds not reliable

Thresholds are "1", "1", empty, empty:

Simon.Richter@hogyros.de (JIRA)

unread,
Feb 24, 2020, 9:33:07 AM2/24/20
to jenkinsc...@googlegroups.com
Thresholds are "1", "1", empty, empty:

!warn.png|thumbnail!

With 1 failing test, I'd expect this to give me "unstable"

Simon.Richter@hogyros.de (JIRA)

unread,
Feb 24, 2020, 9:54:01 AM2/24/20
to jenkinsc...@googlegroups.com
Simon Richter commented on Bug JENKINS-61198
 
Re: Thresholds not reliable

Wait, am I being stupid?

Simon.Richter@hogyros.de (JIRA)

unread,
Feb 24, 2020, 10:07:02 AM2/24/20
to jenkinsc...@googlegroups.com
Change By: Simon Richter
Status: Open Fixed but Unreleased
Resolution: Not A Defect

Simon.Richter@hogyros.de (JIRA)

unread,
Feb 24, 2020, 10:07:02 AM2/24/20
to jenkinsc...@googlegroups.com
 
Re: Thresholds not reliable

Okay, that was me being stupid and interpreting "greater than" as "greater or equal".

Simon.Richter@hogyros.de (JIRA)

unread,
Feb 24, 2020, 10:08:06 AM2/24/20
to jenkinsc...@googlegroups.com
Change By: Simon Richter
Status: Fixed but Unreleased Closed
Reply all
Reply to author
Forward
0 new messages