[JIRA] (JENKINS-52152) xunit cannot read junit time attributes with more than three decimals or using exponential notation

7 views
Skip to first unread message

thomas.sondergaard@karoshealth.com (JIRA)

unread,
Jun 23, 2018, 3:41:01 PM6/23/18
to jenkinsc...@googlegroups.com
tsondergaard created an issue
 
Jenkins / Bug JENKINS-52152
xunit cannot read junit time attributes with more than three decimals or using exponential notation
Issue Type: Bug Bug
Assignee: Nikolas Falco
Components: xunit-plugin
Created: 2018-06-23 19:40
Priority: Minor Minor
Reporter: tsondergaard

xunit fails to parse junit xml files where the time attribute of the testcase element contains e.g 0.123456 or 1.6e-05, because it very narrowly defines the valid format to decimal notation with no more than three decimals.

 

INFO: [JUnit] - 34 test report file(s) were found with the pattern '**/*result.xml' relative to 'foo' for the testing framework 'JUnit'.
WARNING: The file 'foo/junit-result.xml' is an invalid file.
WARNING: At line 3 of file:foo/junit-result.xml:cvc-pattern-valid: Value '1.6e-05' is not facet-valid with respect to pattern '(([0-9]{0,3},)*[0-9]{3}|[0-9]{0,3})*(\.[0-9]{0,3})?' for type '#AnonType_timetestsuite'.
WARNING: At line 3 of file:foo/junit-result.xml:cvc-attribute.3: The value '1.6e-05' of attribute 'time' on element 'testsuite' is not valid with respect to its type, 'null'.
ERROR: Step ‘Publish xUnit test result report’ aborted due to exception: 
Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e)
Atlassian logo

thomas.sondergaard@karoshealth.com (JIRA)

unread,
Jun 23, 2018, 4:17:01 PM6/23/18
to jenkinsc...@googlegroups.com
tsondergaard updated an issue
Change By: tsondergaard
xunit fails to parse junit xml files where the time attribute of the testcase element contains e.g 0.123456 or 1.6e-05, because it very narrowly defines the valid format to decimal notation with no more than three decimals.

  This breaks parsing the junit output from e.g. [https://github.com/catchorg/Catch2] as well as an internal unit testing framework with use at my workplace.
{noformat}

INFO: [JUnit] - 34 test report file(s) were found with the pattern '**/*result.xml' relative to 'foo' for the testing framework 'JUnit'.
WARNING: The file 'foo/junit-result.xml' is an invalid file.
WARNING: At line 3 of file:foo/junit-result.xml:cvc-pattern-valid: Value '1.6e-05' is not facet-valid with respect to pattern '(([0-9]{0,3},)*[0-9]{3}|[0-9]{0,3})*(\.[0-9]{0,3})?' for type '#AnonType_timetestsuite'.
WARNING: At line 3 of file:foo/junit-result.xml:cvc-attribute.3: The value '1.6e-05' of attribute 'time' on element 'testsuite' is not valid with respect to its type, 'null'.
ERROR: Step ‘Publish xUnit test result report’ aborted due to exception: {noformat}
 

nfalco79@hotmail.com (JIRA)

unread,
Jul 5, 2018, 8:52:02 AM7/5/18
to jenkinsc...@googlegroups.com
Nikolas Falco closed an issue as Not A Defect
 

Explanation was in Wiki page and support in the linked issue

Change By: Nikolas Falco
Status: Open Closed
Resolution: Not A Defect
This message was sent by Atlassian JIRA (v7.10.1#710002-sha1:6efc396)

thomas.sondergaard@karoshealth.com (JIRA)

unread,
Jul 5, 2018, 9:21:01 AM7/5/18
to jenkinsc...@googlegroups.com
tsondergaard commented on Bug JENKINS-52152
 
Re: xunit cannot read junit time attributes with more than three decimals or using exponential notation

Nikolas Falco, could you clarify whether this is the explanation in the wiki page you refer to:

https://wiki.jenkins.io/display/JENKINS/xUnit+Plugin#xUnitPlugin-BreakingChanges

In case you are using a non java tools that declares to produce a JUnit compatible report, ensure the report respects one of the widely used supported schemas. If not than select as test type the build step "Custom Tool" and provide a XSL as described here.

 

I can understand the wish to pick and follow "standards", so I have reported an issue against http://maven.apache.org/surefire/maven-surefire-plugin/xsd/surefire-test-report.xsd to have it support more than three decimals.

https://issues.apache.org/jira/browse/SUREFIRE-1533

 

nfalco79@hotmail.com (JIRA)

unread,
Jul 5, 2018, 9:45:02 AM7/5/18
to jenkinsc...@googlegroups.com

Read the JENKINS-52392 for a possible solution.

About the issue opened to sufefire you wrote "Many test tools, also outside the Java world, use the junit format,"
You have to know that JUnit is a java library and does not define a standard for report. Maven surefire/Ant has defined an own schema to support JUnit library feature and the library that rely on java time (Date.getTime()) return millisecond. For this reason does not have sense precision more than milliseconds or as someone has open a defect to parse time in format 1-E10.

thomas.sondergaard@karoshealth.com (JIRA)

unread,
Jul 5, 2018, 10:01:03 AM7/5/18
to jenkinsc...@googlegroups.com

I believe there is nothing lost by loosening the format for the time attribute. If they feel different then other tools will have to adapt to that reality.

cmuller13@gmail.com (JIRA)

unread,
Nov 16, 2018, 8:42:03 AM11/16/18
to jenkinsc...@googlegroups.com

I understand that it is cleaner to validate the XML files but it brings a lot of issues when using several test frameworks.

We had to upgrade nose so that it now puts skipped instead of skip (which was fixed in nose2) and this was OK because it looked like an issue on nose part (when they say it "follows" junit standard). But here for catch2 it is not an issue on their part, they have better precision and it's good. What we did as a workaround is to patch the xsd file to allow for better precision of time attributes and it worked:

        • a/junit-10.xsd 2018-11-15 18:25:44.849045904 +0100*
          +++ b/junit-10.xsd 2018-11-15 18:27:26.705774044 +0100
          @@ -25,7 +25,7 @@
          <xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">
          <xs:simpleType name="SUREFIRE_TIME">
          <xs:restriction base="xs:string">
    • <xs:pattern value="(([0-9]{0,3},)[0-9]{3}|[0-9]{0,3})(\.[0-9]{0,3})?"/>*
      + <xs:pattern value="(([0-9]{0,3},)[0-9]{3}|[0-9]{0,3})(\.[0-9]{0,9})?"/>
      </xs:restriction>
      </xs:simpleType>

It would be nice that such feature be considered in the xunit plugin IMHO.

 

This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)

cmuller13@gmail.com (JIRA)

unread,
Nov 16, 2018, 8:44:03 AM11/16/18
to jenkinsc...@googlegroups.com
Christophe Muller edited a comment on Bug JENKINS-52152
I understand that it is cleaner to validate the XML files but it brings a lot of issues when using several test frameworks.

{{ We had to upgrade nose so that it now puts *{{skipped}}* instead of *{{skip}}* (which was fixed in nose2) and this was OK because it looked like an issue on nose part (when they say it "follows" junit standard). But here for catch2 it is not an issue on their part, they have better precision and it's good. What we did as a workaround is to patch the xsd file to allow for better precision of time attributes and it worked: }}
{{}}
* {code:java}
--- a/junit-10.xsd 2018-11-15 18:25:44.849045904 +0100 *
* +++ b/junit-10.xsd 2018-11-15 18:27:26.705774044 +0100 *
* @@ -25,7 +25,7 @@ *
* <xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"> *
* <xs:simpleType name="SUREFIRE_TIME"> *
* <xs:restriction base="xs:string"> *
* - <xs:pattern value="(([0-9] \ {0,3},)*[0-9] \ {3}|[0-9] \ {0,3})*(\.[0-9] \ {0,3})?"/> *
* + <xs:pattern value="(([0-9] \ {0,3},)*[0-9] \ {3}|[0-9] \ {0,3})*(\.[0-9] \ {0,9})?"/> *
* </xs:restriction> *
* </xs:simpleType> * {code}

It would be nice that such feature be considered in the xunit plugin IMHO.

 

cmuller13@gmail.com (JIRA)

unread,
Nov 16, 2018, 8:44:04 AM11/16/18
to jenkinsc...@googlegroups.com
Christophe Muller edited a comment on Bug JENKINS-52152
I understand that it is cleaner to validate the XML files but it brings a lot of issues when using several test frameworks.

{{We had to upgrade nose so that it now puts *skipped*}} instead of *{{skip}}* (which was fixed in nose2) and this was OK because it looked like an issue on nose part (when they say it "follows" junit standard). But here for catch2 it is not an issue on their part, they have better precision and it's good. What we did as a workaround is to patch the xsd file to allow for better precision of time attributes and it worked: }}
\{{}}

{code:java}
--- a/junit-10.xsd 2018-11-15 18:25:44.849045904 +0100
+++ b/junit-10.xsd 2018-11-15 18:27:26.705774044 +0100
@@ -25,7 +25,7 @@

<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">
<xs:simpleType name="SUREFIRE_TIME">

<xs:restriction base="xs:string">
- <xs:pattern value="(([0-9]{0,3},)*[0-9]{3}|[0-9]{0,3})*(\.[0-9]{0,3})?"/>
+ <xs:pattern value="(([0-9]{0,3},)*[0-9]{3}|[0-9]{0,3})*(\.[0-9]{0,9})?"/>
</xs:restriction>
</xs:simpleType>{code}

It would be nice that such feature be considered in the xunit plugin IMHO.

 

cmuller13@gmail.com (JIRA)

unread,
Nov 16, 2018, 8:44:04 AM11/16/18
to jenkinsc...@googlegroups.com
Christophe Muller edited a comment on Bug JENKINS-52152
I understand that it is cleaner to validate the XML files but it brings a lot of issues when using several test frameworks.

{{We had to upgrade nose so that it now puts * {{ skipped * }} * instead of *{{skip}}* (which was fixed in nose2) and this was OK because it looked like an issue on nose part (when they say it "follows" junit standard). But here for catch2 it is not an issue on their part, they have better precision and it's good. What we did as a workaround is to patch the xsd file to allow for better precision of time attributes and it worked:}}
\ {{}}
{code:java}
--- a/junit-10.xsd 2018-11-15 18:25:44.849045904 +0100
+++ b/junit-10.xsd 2018-11-15 18:27:26.705774044 +0100
@@ -25,7 +25,7 @@
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">
<xs:simpleType name="SUREFIRE_TIME">
<xs:restriction base="xs:string">
- <xs:pattern value="(([0-9]{0,3},)*[0-9]{3}|[0-9]{0,3})*(\.[0-9]{0,3})?"/>
+ <xs:pattern value="(([0-9]{0,3},)*[0-9]{3}|[0-9]{0,3})*(\.[0-9]{0,9})?"/>
</xs:restriction>
</xs:simpleType>{code}
It would be nice that such feature be considered in the xunit plugin IMHO.

 

nfalco79@hotmail.com (JIRA)

unread,
Nov 17, 2018, 2:16:02 AM11/17/18
to jenkinsc...@googlegroups.com

I had already explain the reason of why and which junit format this plugin supports.

For any custom junit format refer to this issue and the wiki documentation here

Marek.Ruszczak+jenkins@gmail.com (JIRA)

unread,
Nov 5, 2019, 9:13:04 AM11/5/19
to jenkinsc...@googlegroups.com

Ok supported formats are https://github.com/windyroad/JUnit-Schema/blob/master/JUnit.xsd and http://maven.apache.org/surefire/maven-surefire-plugin/xsd/surefire-test-report.xsd

Now take a look into first one:

<?xml version="1.0" encoding="UTF-8"?>

<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
	 elementFormDefault="qualified"
	 attributeFormDefault="unqualified">

    ......
    <xs:attribute name="time" type="xs:decimal" use="required">

    ......
</xs:schema>

Lets see definition of decimal in http://www.w3.org/2001/XMLSchema

XML Schema 1.1

An XML Schema schema document for XML Schema schema documents.

XMLSchema.xsd contains:

  <xs:simpleType name="decimal" id="decimal">
    <xs:annotation>
      <xs:appinfo>
        <hfp:hasFacet name="totalDigits"/>
        <hfp:hasFacet name="fractionDigits"/>
        <hfp:hasFacet name="pattern"/>
        <hfp:hasFacet name="whiteSpace"/>
        <hfp:hasFacet name="enumeration"/>
        <hfp:hasFacet name="maxInclusive"/>
        <hfp:hasFacet name="maxExclusive"/>
        <hfp:hasFacet name="minInclusive"/>
        <hfp:hasFacet name="minExclusive"/>
        <hfp:hasProperty name="ordered" value="total"/>
        <hfp:hasProperty name="bounded" value="false"/>
        <hfp:hasProperty name="cardinality" value="countably infinite"/>
        <hfp:hasProperty name="numeric" value="true"/>
      </xs:appinfo>
      <xs:documentation source="http://www.w3.org/TR/xmlschema-2/#decimal"/>
    </xs:annotation>
    <xs:restriction base="xs:anySimpleType">
      <xs:whiteSpace value="collapse" fixed="true" id="decimal.whiteSpace"/>
    </xs:restriction>
   </xs:simpleType>

And finally above leads to documentation which states

3.2.3.1 Lexical representation

*decimal* has a lexical representation consisting of a finite-length sequence of decimal digits (#x30-#x39) separated by a period as a decimal indicator. An optional leading sign is allowed. If the sign is omitted, "+" is assumed. Leading and trailing zeroes are optional. If the fractional part is zero, the period and following zero(es) can be omitted. For example: `-1.23, 12678967.543233, +100000.00, 210`.

3.2.3.2 Canonical representation

The canonical representation for *decimal* is defined by prohibiting certain options from the Lexical representation (§3.2.3.1). Specifically, the preceding optional "+" sign is prohibited. The decimal point is required. Leading and trailing zeroes are prohibited subject to the following: there must be at least one digit to the right and to the left of the decimal point which may be a zero.

For me it is clear value 12678967.543233 is legal for "time" and should not lead to an error as it does.

Nikolas Falco So IMO this should be reopened and fixed.

This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)
Atlassian logo
Reply all
Reply to author
Forward
0 new messages