Jenkins marking builds failed if any tests are skipped

[il...@datto.com]

Hello,

I have a Jenkins build that runs two very simple tests using Pytest. Pytest outputs a JUnit XML file for Jenkins to parse after the tests to determine the results.

One of the tests is hard-coded to pass, and the other one is hard-coded to be skipped. When both tests are run, the Jenkins build gets labeled FAILURE. If the skipping test is removed entirely (meaning only the passing test gets run), the build is marked SUCCESS.

Here is the Jenkins console output at the end of the run. The first two lines are Pytest stdout, the rest are all from Jenkins.

 generated xml file: /home/jenkins/workspace/debug-test/results/results.xml 
===================== 1 passed, 1 skipped in 2.54 seconds ======================
Recording test results
Archiving artifacts
[JUnitReportPublisher] Compiling JUnit Html Reports ...
[JUnit test report builder] Copying XML files from slave: /home/jenkins/workspace/debug-test to master reports directory: /var/lib/jenkins/jobs/debug-test/builds/19/junit-reports-with-handlebars
[JUnitReportPublisher] Found 1 xml files.
[JUnit test report builder] 0. Found a xml file: results/results.xml
[JUnit test report builder] Generating HTML reports
processing: /var/lib/jenkins/jobs/debug-test/builds/19/junit-reports-with-handlebars/xmlData/results/results.xml
Build step 'Publish JUnit reports generated with handlebars' changed build result to FAILURE
Finished: FAILURE

 And here is the results.xml file:

<?xml version="1.0" encoding="utf-8"?>
<testsuite errors="0" failures="0" name="pytest" skipped="1" tests="2" time="2.548">
  <testcase classname="debug_test" file="debug_test.py" line="7" name="test_jenkins2" time="0.001">
    <skipped message="SKIP" type="pytest.skip">debug_test.py:7: SKIP</skipped>
  </testcase>
  <testcase classname="debug_test" file="debug_test.py" line="3" name="test_jenkins" time="0.002"/>
</testsuite>

 

Is there a setting somewhere that tells either Jenkins, or the plugin that parses the XML file, not to mark builds as failed if a test is skipped?

Thank you.

[il...@datto.com]

This appears to be a problem with the JUnit plugin. I switched to the XUnit plugin and it does not mark the build as failed when a test is skipped.

[il...@datto.com]

It looks like this is ACTUALLY a problem with Pytest.

Using Pytest 4.2.0, the JUnit XML contains the following line (seen above):

<testsuite errors="0" failures="0" name="pytest" skipped="1" tests="2" time="2.548">

But with Pytest 3.6.2, this line is:

<testsuite errors="0" failures="0" name="pytest" skips="1" tests="2" time="2.271">

 

Notice the "skips" was changed to "skipped" at some point between Pytest 3.6.2 and 4.2.0.