proposal: extend the output of --gtest_list_tests with test parameter values

[Zhanyong Wan (λx.x x)]

Hi gtest users,

When you run a gtest-based test program with command-line flag --gtest_list_tests, it prints out the names of the test cases and tests within the program instead of running them.  Currently the output looks like:

TestCase1.

  TestName1

  TestName2

TestCase2.

  TestName

This is not quite satisfactory for value-parameterized and type-parameterized tests, as their names consist of a base name and an index number, e.g.

TestCaseFoo/5

Here 5 means the test is for the 5th test parameter.  Since the actual (type or value) parameter is not printed, there's not enough information for a user to decide which tests to select (if he wants to run them selectively).

One option we discussed and rejected is to make the test parameter part of the test name, as in general the test parameter can contain all sorts of characters that aren't suitable to use in a --gtest_filter flag (and they can be arbitrarily long and awkward to refer to).

I propose to augment the output of -gtest_list_tests with the value of the test parameters (either type parameters or value parameters), e.g.

TestCaseFoo/0. TypeParam=int

  TestName1

  TestName2

TestCaseFoo/1. TypeParam=float

  TestName1

  TestName2

This will help the user make an informed decision on which tests to run.

Another option is to keep the existing behavior of --gtest_list_tests and introduce a new flag to trigger this more verbose output.  However, that makes the framework more complex to maintain and use, and IMO is not a good long-term plan.

The proposed change could break scripts that parse the output of --gtest_list_tests, but I think it's worth the temporary pain.  I will make sure all such scripts used within Google are fixed.

Thoughts?  Thanks,

--
Zhanyong

[Joey Oravec]

I think that's a positive change. I was confused in my original response -- as you stated, that info is all within the object and printed for test results, but Matthew wants to see it when listing the tests. This would be consistent with info that gtest already prints.

Your example was pretty simple for TypeParam. Assuming gtest handles both, we'd see every combination like:

TestCaseA/0. TypeParam = %s

TestCaseB/0. GetParam() = %s

TestCaseC/0. TypeParam = %s and GetParam() = %s

TestCaseD/0.

where %s can be a pretty crazy string for GetParam(), since it depends on operator<< for the parameter (often a structure or object). It's fine for a human to read but rather open-ended for a machine. My preferred approach for a GUI or test runner is usually to avoid gtest_main.cc and access the object model directly.

Can you comment on accessing type_param() and value_param() programmatically? Is all of the test list info public accessible through test_info? I know that some GUIs today call the test as a separate executable and attempt to parse the printout, but if it's available through the object model we shouldn't be so dependent.

-joey

--
 
---
You received this message because you are subscribed to the Google Groups "Google C++ Testing Framework" group.
To unsubscribe from this group and stop receiving emails from it, send an email to googletestframe...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Zhanyong Wan (λx.x x)]

Hi Joey,

Thanks for sharing your thoughts.

On Fri, Apr 5, 2013 at 8:53 PM, Joey Oravec <joeyo...@gmail.com> wrote:

I think that's a positive change. I was confused in my original response -- as you stated, that info is all within the object and printed for test results, but Matthew wants to see it when listing the tests. This would be consistent with info that gtest already prints.

Your example was pretty simple for TypeParam. Assuming gtest handles both, we'd see every combination like:

TestCaseA/0. TypeParam = %s

TestCaseB/0. GetParam() = %s

TestCaseC/0. TypeParam = %s and GetParam() = %s

This combination is not supported by gtest.  In theory type parameters and value parameters are orthogonal, but it's not trivial to combine both, and so far we haven't had a strong need for it.

 

TestCaseD/0.

where %s can be a pretty crazy string for GetParam(), since it depends on operator<< for the parameter (often a structure or object). It's fine for a human to read but rather open-ended for a machine.

That's true.  My plan is to replace the newline characters in the parameter value with something else (e.g. spaces) such that a parameter is always on a single line.  Then it's easy for a program to parse the test list.

 

My preferred approach for a GUI or test runner is usually to avoid gtest_main.cc and access the object model directly.

Can you comment on accessing type_param() and value_param() programmatically? Is all of the test list info public accessible through test_info? I know that some GUIs today call the test as a separate executable and attempt to parse the printout, but if it's available through the object model we shouldn't be so dependent.

The type/value parameter values (as strings) for the tests are already publicly accessible in the UnitTest object model, so it's not hard for a test runner to get them programmatically.  You can do it by defining a test event listener (https://code.google.com/p/googletest/wiki/AdvancedGuide) that traverses the UnitTest object in the OnTestProgramStart() event.  If you just want to read the test info without running the tests, throws an exception at the end of OnTestProgramStart() to abort the rest of the standard gtest flow.

--
Zhanyong

[Zhanyong Wan (λx.x x)]

This was implemented in trunk r652.  Cheers,

--
Zhanyong