[JIRA] (JENKINS-57615) xUnit fails to parse PHPUnit output when a skipped test also has system output

31 views
Skip to first unread message

captainhypertext@gmail.com (JIRA)

unread,
May 22, 2019, 11:00:02 AM5/22/19
to jenkinsc...@googlegroups.com
Captain Hypertext created an issue
 
Jenkins / Bug JENKINS-57615
xUnit fails to parse PHPUnit output when a skipped test also has system output
Issue Type: Bug Bug
Assignee: Nikolas Falco
Components: xunit-plugin
Created: 2019-05-22 14:59
Priority: Minor Minor
Reporter: Captain Hypertext

I'm using the latest version of the xUnit plugin (2.3.3) with the PHPUnit-3.x parser. It fails on a few tests because it can't parse the xml file, and it gives a somewhat misleading error:

 WARNING: The file 'myresult.xml' is an invalid file. WARNING: At line 7261 of file:myresult.xml:cvc-complex-type.2.4.d: Invalid content was found starting with element 'system-out'. No child element is expected at this point. FATAL: The result file 'myresult.xml' for the metric 'PHPUnit' is not valid. The result file has been skipped.

This seems to be happening because the test was skipped, but still had system output. Here's an example:

<testcase name="testMyTest" class="MyTestUnitTest" file="myUnitTest.php" line="320" assertions="157" time="45.286155">
    <skipped />
    <system-out>Some debug info...</system-out>
</testcase>

In this case there's a bunch of setup happening in the base class before the test even runs (which will ultimately be skipped), which is echoing out debug information. This system output is useful for debugging, and inconsequential to the result of the tests. I can alter my tests to not output anything, but parsing shouldn't fail here, or should at least produce a meaningful error message.
Add Comment Add Comment
 
This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)

captainhypertext@gmail.com (JIRA)

unread,
May 22, 2019, 11:01:03 AM5/22/19
to jenkinsc...@googlegroups.com
Captain Hypertext updated an issue
Change By: Captain Hypertext
I'm using the latest version of the xUnit plugin (2.3.3) with the PHPUnit-3.x parser. It fails on a few tests because it can't parse the xml file, and it gives a somewhat misleading error:
{noformat}

WARNING: The file 'myresult.xml' is an invalid file.
WARNING: At line 7261 of file:myresult.xml:cvc-complex-type.2.4.d: Invalid content was found starting with element 'system-out'. No child element is expected at this point.
FATAL: The result file 'myresult.xml' for the metric 'PHPUnit' is not valid. The result file has been skipped.{noformat}

This seems to be happening because the test was skipped, but still had system output. Here's an example:
{noformat}

<testcase name="testMyTest" class="MyTestUnitTest" file="myUnitTest.php" line="320" assertions="157" time="45.286155">
    <skipped />
    <system-out>Some debug info...</system-out>
</testcase>{noformat}

In this case there's a bunch of setup happening in the base class before the test even runs (which will ultimately be skipped), which is echoing out debug information. This system output is useful for debugging, and inconsequential to the result of the tests. I can alter my tests to not output anything, but parsing shouldn't fail here, or should at least produce a meaningful error message.

nfalco79@hotmail.com (JIRA)

unread,
May 22, 2019, 11:13:02 AM5/22/19
to jenkinsc...@googlegroups.com
Nikolas Falco commented on Bug JENKINS-57615
 
Re: xUnit fails to parse PHPUnit output when a skipped test also has system output

Could you post your job configuration? I would know your xunit step configuration.

captainhypertext@gmail.com (JIRA)

unread,
May 22, 2019, 11:33:02 AM5/22/19
to jenkinsc...@googlegroups.com

Nikolas Falco - Just uploaded my job's config.xml

captainhypertext@gmail.com (JIRA)

unread,
May 22, 2019, 11:33:02 AM5/22/19
to jenkinsc...@googlegroups.com
Captain Hypertext updated an issue
Change By: Captain Hypertext
Attachment: config.xml

nfalco79@hotmail.com (JIRA)

unread,
May 22, 2019, 4:37:04 PM5/22/19
to jenkinsc...@googlegroups.com
Nikolas Falco commented on Bug JENKINS-57615
 
Re: xUnit fails to parse PHPUnit output when a skipped test also has system output

Ok I understand, could you post to me the report file?

nfalco79@hotmail.com (JIRA)

unread,
May 22, 2019, 4:49:02 PM5/22/19
to jenkinsc...@googlegroups.com

I see that in the config.xml you are running "phpunit --log-junit .." that by documentation here it produce a "JUnit Ant like". You had configure to PHPUnit parser style that means it expects a different format.

Change PHPUnit 4.x -> JUnit and hope the JUnit produced is a really Ant style like the documentation describe.

nfalco79@hotmail.com (JIRA)

unread,
May 22, 2019, 4:50:01 PM5/22/19
to jenkinsc...@googlegroups.com
Nikolas Falco closed an issue as Not A Defect
 
Change By: Nikolas Falco
Status: Open Closed
Resolution: Not A Defect

nfalco79@hotmail.com (JIRA)

unread,
May 22, 2019, 4:54:02 PM5/22/19
to jenkinsc...@googlegroups.com
Nikolas Falco edited a comment on Bug JENKINS-57615
I see that in the config.xml you are running "phpunit --log-junit .." that by [documentation here|https://phpunit.readthedocs.io/en/8.1/logging.html] it produce a "JUnit Ant like". You had configure to configured PHPUnit parser style as report type that means it expects to parse a different format.

Change PHPUnit 4.x -> JUnit and hope that the JUnit produced is a really Ant style task format like the documentation describe (https://github . com/sebastianbergmann/phpunit/issues/3162).

captainhypertext@gmail.com (JIRA)

unread,
May 22, 2019, 4:57:01 PM5/22/19
to jenkinsc...@googlegroups.com

I had to switch from JUnit because it stopped parsing after we upgraded.  I think it's related to JENKINS-51919, it kept failing to parse because of the attributes PHPUnit includes such as "assertions", along with several more.  So I switched to PHPUnit-3.x (PHPUnit-4.x was not listed as an option, and my version of PHPUnit is 6, so I'm confused about that...).  I'll see if I can get one of those report files for you.

captainhypertext@gmail.com (JIRA)

unread,
May 22, 2019, 5:07:01 PM5/22/19
to jenkinsc...@googlegroups.com

Yes, I did think it was weird how things were fixed when I just switched the parser away from JUnit to PHPUnit, but hey, it works for everything except these cases.  The log types available for my version of PHPUnit is: junit, teamcity, testdox-html, testdox-text, testdox-xml, and reverse-list.

nfalco79@hotmail.com (JIRA)

unread,
May 22, 2019, 5:53:01 PM5/22/19
to jenkinsc...@googlegroups.com

The case is the same of other similar issue. The point is that PHPUnit does not produce a supported JUnit format. They declare to produce a format junit task like but is not true. The reason of issue https://github.com/sebastianbergmann/phpunit/issues/3162 where they would standardize the format in output.

In the Wiki is explained how to setup "In case you are using a non java tools that declares to produce a JUnit compatible report" (search the string between "")

captainhypertext@gmail.com (JIRA)

unread,
May 22, 2019, 6:03:02 PM5/22/19
to jenkinsc...@googlegroups.com

Yes, but this does not address my issue.  My issue is with the PHPUnit-3.x parser, which works fine except for this bug.  I am not using the JUnit parser.

nfalco79@hotmail.com (JIRA)

unread,
May 22, 2019, 6:08:02 PM5/22/19
to jenkinsc...@googlegroups.com

phpunit --log-junit produces a JUnit report.

captainhypertext@gmail.com (JIRA)

unread,
May 22, 2019, 6:18:01 PM5/22/19
to jenkinsc...@googlegroups.com

Yes, I know.  Please explain how the PHPUnit-3.x parser works better than the JUnit parser for parsing JUnit logs, or at least refer me to the author or proper documentation.  This is definitely not a phpUnit issue, this bug is an issue with the PHPUnit parser in xUnit.
Reply all
Reply to author
Forward
0 new messages