GoogleTest: XML output should contain an option for one file per class
Charles
please refer to http://code.google.com/p/googletest/issues/detail?id=114)
The basic issue is that the XML reports generated by GoogleTest
encapsulated all the results from all test classes (within a single
executable) into a single file. This is different from some of the
other xUnit frameworks (eg. Ant/JUnit) where the output from each test
class is store in a separate file.
My argument:
We are using Hudson as our CI server as well, like the original poster
has said. Since
JUnit outputs multiple XML files, in order to allow broader
compatibility with existing
tools, I believe GoogleTest should also offer an similar option to
output multiple XML files. I think this will also make it easier for
non-GoogleTest users to migrate their test cases while and keep
their current workflow intact.
I understand it's inefficient at times to generate multiple XML output
files for each
test (class), but as long as it is provided as an additional option (--
gtest_output
...), we can satisfy both camps of people.
Although a workaround is available, we prefer to keep our build system
clean and not
have to depend on Perl or other translation facilities.
Thoughts?
Charles
Charles
understand-Googletest-XML--p22182483.html), someone suggests Hudson
works if the outermost <testsuite> was replaced with <testsuites>.
ie.
## from
<testsuite>
<testsuite/>
</testsuite>
## to
<testsuites>
<testsuite/>
</testsuites>
This changes bypass the initial argument of creating multiple XML
files. Although it seem reasonable (for me) to have "testsuites" as
the outer most element, do you guys see any potentially issue with it?
Was there an XML schema used by GoogleTest to come up with the
existing file format?
Regards,
Charles
Zhanyong Wan (λx.x x)
Google Test was designed to match our internal tools which expect the
XML report to match the format of those produced by the "junit" Ant
task, which has "testsuite" as the root element.
The format Hudson uses corresponds to the "junitreport" Ant task,
which has "testsuites" as the root element.
Unfortunately, if we change Google Test's format, we will break the
build system integration at Google. I see two possible solutions:
1. Make Hudson support the "junit" format in addition to the
"junitreport" format.
2. Make Google Test support generating both formats, e.g. we can
continue to use --gtest_output=xml to select the current format, and
use --gtest_output=xmlreport to select the junitreport-style format.
We can work on #2, but given the limited resources, we may not be able
to get to it soon. Therefore, could you talk to the Hudson guys to
see if they can get #1 done quickly? Thanks,
>
> Regards,
> Charles
>
--
Zhanyong
Zhanyong Wan (λx.x x)
> understand-Googletest-XML--p22182483.html), someone suggests Hudson
> works if the outermost <testsuite> was replaced with <testsuites>.
>
> ie.
> ## from
> <testsuite>
> <testsuite/>
> </testsuite>
>
> ## to
> <testsuites>
> <testsuite/>
> </testsuites>
Have you done an experiment to confirm this? Thanks,
>
> This changes bypass the initial argument of creating multiple XML
> files. Although it seem reasonable (for me) to have "testsuites" as
> the outer most element, do you guys see any potentially issue with it?
> Was there an XML schema used by GoogleTest to come up with the
> existing file format?
>
> Regards,
> Charles
>
--
Zhanyong
Vlad Losev
Charles
> to get to it soon. Therefore, could you talk to the Hudson guys to
> see if they can get #1 done quickly? Thanks,
hardest thing for me is finding a definitive XML schema for "junit"
and "junitreport". If I had that information available, then it would
no doubt help me convince the Hudson guys to support both formats.
> Have you done an experiment to confirm this? Thanks,
Vlad:
> You can also try looking up a thread on this group titled 'Using gtest with
> hudson' - it discussed the issue in detail, and some people provided helpful
> workarounds.
>
> IMHO, we should not be providing output formats for every CI system out
> there. What we should do is to publish Google Test's event interface so it
> will be trivial to write listeners producing custom output formats.
>
> Regards,
> Vlad.
I would prefer if I don't have to use a workaround. I agree if you can
publish the interface to allow people to extend the file outputter.
However, if the format already matches "junit" ant task, then there is
not reason why it should not be supported... Again I just need to find
the official "junit" schema to help convince those working on Hudson.
Cheers,
Charles
Zhanyong Wan (λx.x x)
>
>
> Zhanyong:
>> We can work on #2, but given the limited resources, we may not be able
>> to get to it soon. Therefore, could you talk to the Hudson guys to
>> see if they can get #1 done quickly? Thanks,
> -- Certainly. The Hudson group is also open to suggestions. The
> hardest thing for me is finding a definitive XML schema for "junit"
> and "junitreport". If I had that information available, then it would
> no doubt help me convince the Hudson guys to support both formats.
>
>> Have you done an experiment to confirm this? Thanks,
> -- Not yet.
Could you try it and let us know the result? We need this information
to decide what we should do. Thanks.
> Vlad:
>> Charles -
>> You can also try looking up a thread on this group titled 'Using gtest with
>> hudson' - it discussed the issue in detail, and some people provided helpful
>> workarounds.
>>
>> IMHO, we should not be providing output formats for every CI system out
>> there. What we should do is to publish Google Test's event interface so it
>> will be trivial to write listeners producing custom output formats.
It is valuable to publish this interface nonetheless. However, I
think supporting native GoogleTest/Hudson integration is a better user
experience in this particular case. Look at the code we have for
producing the current XML report: it's not trivial although not super
complex. Asking the users to implement this will be yet another
workaround instead of a solution.
I agree that we shouldn't try to match every continuous build system
out there. That's impossible and a poor use of our time. However,
Hudson seems to be a particularly popular system and has lots of users
among *our* user community, and "junitreport" is a format made popular
by Ant, which is semi-standard in the Java world. I think supporting
"junitreport" will make it easier for many people to adopt Google
Test.
>> Regards,
>> Vlad.
> -- Thanks Vlad, I did read that post earlier, but like I said earlier,
> I would prefer if I don't have to use a workaround. I agree if you can
I agree that a workaround is not a nice user experience and we should
fix the root problem one way or another.
> publish the interface to allow people to extend the file outputter.
> However, if the format already matches "junit" ant task, then there is
> not reason why it should not be supported... Again I just need to find
> the official "junit" schema to help convince those working on Hudson.
Sadly I don't think the schema was ever published by the Ant project.
We had to reverse-engineer it. From reading the thread you forwarded,
my understanding is that the Hudson team understands the existence of
the two formats and the difference between them. Therefore I think
what we need to do to convince them is to show that "junit" is a
popular format used by many people (not just Google Test users).
There is at least one PHP developer on that thread complaining about
the same problem with Hudson.
> Cheers,
> Charles
>
--
Zhanyong
Charles
> Could you try it and let us know the result? We need this information
> to decide what we should do. Thanks.
result report"
## Default GoogleTest output aka: "junit" schema ##
<?xml version="1.0" encoding="UTF-8"?>
<testsuite tests="4" failures="0" disabled="0" errors="0" time="0.422"
name="AllTests">
<testsuite name="FileSystemInfoTest" tests="2" failures="0"
disabled="0" errors="0" time="0.062">
<testcase name="listPassTest" status="run" time="0.062"
classname="FileSystemInfoTest" />
<testcase name="listFailTest" status="run" time="0"
classname="FileSystemInfoTest" />
</testsuite>
<testsuite name="HttpClientTest" tests="2" failures="0" disabled="0"
errors="0" time="0.36">
<testcase name="getResponseSuccessTest" status="run" time="0.36"
classname="HttpClientTest" />
<testcase name="getResponseFailTest" status="run" time="0"
classname="HttpClientTest" />
</testsuite>
</testsuite>
Hudson output:
Recording test results
None of the test reports contained any result
finished: FAILURE
## Modified GoogleTest XML aka "junitreport" schema ##
<?xml version="1.0" encoding="UTF-8"?>
<testsuites tests="4" failures="0" disabled="0" errors="0"
time="0.422" name="AllTests">
<testsuite name="FileSystemInfoTest" tests="2" failures="0"
disabled="0" errors="0" time="0.062">
<testcase name="listPassTest" status="run" time="0.062"
classname="FileSystemInfoTest" />
<testcase name="listFailTest" status="run" time="0"
classname="FileSystemInfoTest" />
</testsuite>
<testsuite name="HttpClientTest" tests="2" failures="0" disabled="0"
errors="0" time="0.36">
<testcase name="getResponseSuccessTest" status="run" time="0.36"
classname="HttpClientTest" />
<testcase name="getResponseFailTest" status="run" time="0"
classname="HttpClientTest" />
</testsuite>
</testsuites>
Hudson output:
Recording test results
finished: SUCCESS
Charles
parse google test XML output".
https://hudson.dev.java.net/issues/show_bug.cgi?id=3007
I guess we have to wait and see...
peter.ki...@gmail.com
I am an ant developer and had a quick look at the formats used by
junit and junitreport.
junit ant task generates xml files with <testsuite> as a root element
and <testcase> as children.
junitreport ant task combines these testsuite root elements into one
xml file with <testsuites> as
the root element (which is used as input to an xslt processor).
so there is *no* case where a <testsuite> is nested within a
<testsuite> in xml used by <junit> or <junitreport>
Hudson can consume the start = testsuite; testsuite = element
testsuite { element testcase*} and
the start=testsuites; testsuites = element testsuites { testsuite* }
xnl files.
I will be placing a rnc schmena file for junit into ant 1.8 at some
stage soon.
Peter
Zhanyong Wan (λx.x x)
On Wed, Mar 11, 2009 at 1:46 AM, peter.ki...@gmail.com
<peter.ki...@gmail.com> wrote:
>
> Hi,
> I am an ant developer and had a quick look at the formats used by
> junit and junitreport.
> junit ant task generates xml files with <testsuite> as a root element
> and <testcase> as children.
> junitreport ant task combines these testsuite root elements into one
> xml file with <testsuites> as
> the root element (which is used as input to an xslt processor).
> so there is *no* case where a <testsuite> is nested within a
> <testsuite> in xml used by <junit> or <junitreport>
>
> Hudson can consume the start = testsuite; testsuite = element
> testsuite { element testcase*} and
> the start=testsuites; testsuites = element testsuites { testsuite* }
> xnl files.
Thanks for chiming in. Much appreciated!
Google Test's format was picked several years ago. It was based on
Ant's format at the time. Either it was never an exact match as I
thought, or the Ant format has evolved. In either case, it's now
clear that Google Test's format no longer matches "junit" now. Thanks
for confirming!
Given that, I think we should implement a "--gtest_output=xmlreport"
flag to produce the junitreport-style report.
> I will be placing a rnc schmena file for junit into ant 1.8 at some
> stage soon.
That will benefit many people. Thanks!
--
Zhanyong
Charles
How are your schedules, looking like? I can offer some help if you
guys are too busy.
Charles
Zhanyong Wan (λx.x x)
>
> Will this change make it into the next GoogleTest release?
Realistically, I don't think we'll be able to do it. :-( We'll make
sure it's part of the one after though. Until that, please use one of
the workarounds.
> How are your schedules, looking like? I can offer some help if you
> guys are too busy.
Thanks for the offer. This change will involve updating many Python
tests, so we'll do it ourselves this time.
Barring the unexpected, we should be able to release 1.3.0 early next week.
--
Zhanyong
Charles
>
> Thanks for the offer. This change will involve updating many Python
> tests, so we'll do it ourselves this time.
>
> Barring the unexpected, we should be able to release 1.3.0 early next week.
> --
> Zhanyong
upcoming release.
Charles
Zhanyong Wan (λx.x x)
I've thought more about this. Having both --gtest_output=xml and
--gtest_output=xmlreport isn't ideal as I suspect that the vast
majority of the users outside of Google will use the latter, making
the former pretty useless. Perhaps instead of adding
--gtest_output=xmlreport, we can just change Google Test's behavior
such that --gtest_output=xml will produce a junitreport-format report.
In theory, this change could break some users, but I suspect that
most people will welcome it in practice.
We cannot change the format used inside Google, at least not yet.
Therefore this means the XML format will be different in the internal
version of Google Test (at least for now). This is not ideal either,
but I suspect it's a lesser evil.
Since this doesn't require a new flag, it can be implemented
trivially, perhaps even in time for this release.
What do people think?
--
Zhanyong