Xunit and python xmlrunner
Jérôme Godbout
Hi,
I managed to make my python unit test run under my Jenkins scripted pipeline Jenkinsfile. I have multiple .xml files into my output folders. When I try to gather the result with xunit, it doesn’t work and I don’t know why, the xml seem like a legitimate Junit result:
My jenkinsfile call (I want an unstanble built when test doesn’t pass, error build is the test did not complete or scripts fail):
xunit(
thresholds: [
skipped(failureThreshold: '99999', failureNewThreshold: '99999', unstableThreshold: '0', unstableNewThreshold: '0'),
failed( failureThreshold: '99999', failureNewThreshold: '99999', unstableThreshold: '0', unstableNewThreshold: '0')
],
tools: [ JUnit(pattern: "TestResults/TEST-*.xml", skipNoTestFiles: false, failIfNotNew: true) ]
);
Here is the error:
13:47:04 [Pipeline] xunit
13:47:04 INFO: Starting to record.
13:47:04 INFO: Processing JUnit
13:47:04 INFO: [JUnit] - 5 test report file(s) were found with the pattern 'TestResults/TEST-*.xml' relative to 'c:\Jenkins\workspace\ZZZ\ZZZCoRegistration-UnitTests\CoRegistration\Prototype' for the testing framework 'JUnit'.
13:47:04 WARNING: The file 'c:\Jenkins\workspace\ZZZ\ZZZCoRegistration-UnitTests\CoRegistration\Prototype\TestResults\TEST-Tests.Algorithms.AlgorithmInterfaceTestCase-20200116134659.xml' is an invalid file.
13:47:04 WARNING: At line 3 of file:/c:/Jenkins/workspace/ZZZ/ZZZCoRegistration-UnitTests/CoRegistration/Prototype/TestResults/TEST-Tests.Algorithms.AlgorithmInterfaceTestCase-20200116134659.xml:cvc-complex-type.3.2.2: Attribute 'file' is not allowed to appear in element 'testcase'.
13:47:04 WARNING: At line 3 of file:/c:/Jenkins/workspace/ZZZ/ZZZCoRegistration-UnitTests/CoRegistration/Prototype/TestResults/TEST-Tests.Algorithms.AlgorithmInterfaceTestCase-20200116134659.xml:cvc-complex-type.3.2.2: Attribute 'line' is not allowed to appear in element 'testcase'.
13:47:04 WARNING: At line 3 of file:/c:/Jenkins/workspace/ZZZ/ZZZCoRegistration-UnitTests/CoRegistration/Prototype/TestResults/TEST-Tests.Algorithms.AlgorithmInterfaceTestCase-20200116134659.xml:cvc-complex-type.3.2.2: Attribute 'timestamp' is not allowed to appear in element 'testcase'.
The report look like this:
<?xml version="1.0" encoding="UTF-8"?>
<testsuite errors="0" failures="0" file="Tests/Algorithms.py" name="Tests.Algorithms.AlgorithmInterfaceTestCase-20200116134659" skipped="0" tests="5" time="0.005" timestamp="2020-01-16T13:46:59">
<testcase classname="Tests.Algorithms.AlgorithmInterfaceTestCase" file="Tests\Algorithms\__init__.py" line="51" name="test_invalid_key_type_in_context" time="0.003" timestamp="2020-01-16T13:46:59">
<system-out>
<![CDATA[]]> </system-out>
<system-err>
<![CDATA[]]> </system-err>
</testcase>
<testcase classname="Tests.Algorithms.AlgorithmInterfaceTestCase" file="Tests\Algorithms\__init__.py" line="55" name="test_invalid_key_value_in_context" time="0.000" timestamp="2020-01-16T13:46:59">
<system-out>
<![CDATA[]]> </system-out>
<system-err>
<![CDATA[]]> </system-err>
</testcase>
<testcase classname="Tests.Algorithms.AlgorithmInterfaceTestCase" file="Tests\Algorithms\__init__.py" line="42" name="test_missing_required_key_in_context" time="0.000" timestamp="2020-01-16T13:46:59">
<system-out>
<![CDATA[]]> </system-out>
<system-err>
<![CDATA[]]> </system-err>
</testcase>
<testcase classname="Tests.Algorithms.AlgorithmInterfaceTestCase" file="Tests\Algorithms\__init__.py" line="46" name="test_set_optional_key_in_context" time="0.001" timestamp="2020-01-16T13:46:59">
<system-out>
<![CDATA[]]> </system-out>
<system-err>
<![CDATA[]]> </system-err>
</testcase>
<testcase classname="Tests.Algorithms.AlgorithmInterfaceTestCase" file="Tests\Algorithms\__init__.py" line="60" name="test_wrong_context_type" time="0.001" timestamp="2020-01-16T13:46:59">
<system-out>
<![CDATA[]]> </system-out>
<system-err>
<![CDATA[]]> </system-err>
</testcase>
</testsuite>
Am I using the proper class JUnit to parse the file? I don’t have any test results. I understand that the file and line are extra properties not into the JUnit standards, but still the remaining should work, I should see those test as working. Any clue why is this an invalid file or why I don’t have any results?
|
une compagnie
RAPPROCHEZ LA DISTANCE |
|
Jérôme Godbout
Hi,
I will answer my own question, or at least the work around:
ditch xunit and use:
junit keepLongStdio: true, testResults: 'TestResults/**/*.xml';
This work, not sure why the xunit with JUnit is not able to handle this properly, I loose the threshold control, not the best solution under the sun, but it’s a tolerable workaround.
Sounds like a bad xsl validation for xunit. For a spec this old like JUnit, make me wonder how come this ain’t working well.
Strong opinion ahead, stop reading if you don’t like those. The more I dig into Jenkins, the more painful I find this thing to be, the API is unplaisant, the API doc is uterly bad to browse and imply you know the Java underhood, the groovy is not helping in any way, Jenkins and his plugin are buggy as hell, the only true focus is seem to be security. I start to wonder if I should just have Jenkins kick in python scripts and be done with it, at the very least I would have cross platform out of the box and file manipulation would not be such a nightmare. I would only need a good API to collect the results that actually work decently.
regards,
Jerome
--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to
jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/jenkinsci-users/YQXPR0101MB11738BDA5541824489CB8652CD360%40YQXPR0101MB1173.CANPRD01.PROD.OUTLOOK.COM.