XML results missing (Issue #426)

[Paul Smith]

Hi all. I'm running into a problem where my custom main() is invoking RUN_ALL_TESTS() multiple times, with different settings for GTEST_FLAG(filter). The problem is there is one test which MUST be run last, after all other tests are complete, and verify some things. It can't be done in each test, it can only be done after all tests are finished. I don't like this but I didn't write this test.

The problem is that we generate XML output using GTEST_OUTPUT, and every time RUN_ALL_TESTS() is invoked it overwrites all previous values in the output file: I can see in the code where the XML listener just opens the file in write mode.

So, I thought I could reset the output file name before each test, but unfortunately that doesn't work either because there are static (private) variables which make argument parsing idempotent. I guess there must be some special dispensation for resetting the filter argument since it seems valid to change that one, but changing GTEST_FLAG(output) has no effect.

I found this issue: https://github.com/google/googletest/issues/426 from two years ago but no response.

Anyone have any thoughts about this?

[Josh Kelley]

Can you move the "must run last" testing logic into a test environment TearDown function? See https://github.com/google/googletest/blob/master/googletest/docs/AdvancedGuide.md#global-set-up-and-tear-down. That meets the criteria of running after all other tests finish, and you can still use whatever assertions you want there (just like you can in a normal test), without having to do unexpected things to Google Test's execution order.

Otherwise, I'd take a look at the event listener API (https://github.com/google/googletest/blob/master/googletest/docs/AdvancedGuide.md#extending-google-test-by-handling-test-events) and see about doing my own custom XML event listener, using Google Test's XmlUnitTestResultPrinter as a starting point.

--

Josh Kelley

--

---
You received this message because you are subscribed to the Google Groups "Google C++ Testing Framework" group.
To unsubscribe from this group and stop receiving emails from it, send an email to googletestframework+unsub...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Paul Smith]

On Fri, 2017-09-29 at 12:04 -0400, Josh Kelley wrote:

Can you move the "must run last" testing logic into a test environment TearDown function? See https://github.com/google/googletest/blob/master/googletest/docs/AdvancedGuide.md#global-set-up-and-tear-down. That meets the criteria of running after all other tests finish, and you can still use whatever assertions you want there (just like you can in a normal test), without having to do unexpected things to Google Test's execution order.

I thought about it but I searched high and low in the docs and via Google to see whether it was valid to fail tests in TearDown and couldn't find anything. I've used many unit test frameworks (for example, Python unittest) where this is not allowed so I assumed it wasn't.

However, if it works that's a feasible alternative. If TearDown fails then what is shown? What test is marked as failed? I guess I'd need to try it out.

As a quick hack what I did was rename() the output file after each invocation of RUN_ALL_TESTS(), which worked (since my XML parser just looks for *.xml files and doesn't care about the exact name) but is a bit cheesy :).

I get that it's not easy to deal with multiple RUN_ALL_TESTS() writing to the same file, due to the structure of XML (you can't just more output at the end, necessarily). I just wish it was possible to reset the GTEST_FLAG(output) option after RUN_ALL_TESTS() is run, and have the next RUN_ALL_TESTS() recognize it.

Thanks for the hints Josh!

[Craig Scott]

You didn't mention how you're building gtest, but if you happen to be using CMake and can require at least CMake 3.7 as a minimum version, you may want to look into its test fixture support:

https://crascit.com/2016/10/18/test-fixtures-with-cmake-ctest/

The problem with trying to enforce order within gtest alone is that the tests can be driven individually with the --gtest_filter command line option. The test fixture support in CMake is designed specifically for scenarios like yours and can be used to robustly enforce whatever test order you want. 

--

---
You received this message because you are subscribed to the Google Groups "Google C++ Testing Framework" group.

To unsubscribe from this group and stop receiving emails from it, send an email to googletestframe...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Craig Scott
Melbourne, Australia
https://crascit.com