Value-parameterized tests review request

[Vlad Losev]

Hi everyone,

After all the talk we finally have an implementation of the value-parameterized tests feature (discussed earlier). As a quick reminder, this feature allows you to define a test suite with a parameter of particular type and then use it with various values of that type. This comes very handy when you want to test your code with different flag values or environment variables, for example. The design document for this feature is available at http://docs.google.com/Doc?docid=dfhzgpkh_30gwjm92dj&hl=en.

We want to get your feedback and ideas about it before it is committed to the repository, and are posting it as a patch for review at http://codereview.appspot.com/8654. Feel free to comment on the feature in general or on the individual implementation details.

So what is it good for?

* If your code is affected by a value of an external parameter, and you want to test it with a set of values of that parameter.
* If you repeat the same assertions with multiple parameter values.
* If you want to run the same tests over different implementations of an interface.

And here is how you do it:

To write value-parameterized tests, first you should define a fixture class. It must be derived from testing::TestWithParam<T>, where T is the type of your parameter values. TestWithParam<T> is itself derived from testing::Test. T can be any copyable type. If it's a raw pointer, you are responsible for managing the lifespan of the pointed values.

class FooTest : public testing::TestWithParam<const char*> {

  // You can implement all the usual fixture class members here.
};

Then, use the TEST_P macro to define as many test patterns using this fixture as you want. The _P suffix is for "parameterized" or "pattern", whichever you prefer to think.

TEST_P(FooTest, DoesBlah) {
  // Inside a test, access the test parameter with the GetParam() method

  // of the TestWithParam<T> class:
  EXPECT_TRUE(foo.Blah(GetParam()));

  ...
}

TEST_P(FooTest, HasBlahBlah) {

  ...
}

Finally, you can use INSTANTIATE_TEST_CASE_P to instantiate the test case with any set of parameters you want. Google Test defines a number of functions for generating test parameters. They return what we call (surprise!) parameter generators. Here is a summary of them, which are all in the testing namespace:

Range(begin, end [, step])	Yields values {begin, begin+step, begin+step+step, ...}. The values do not include end. step defaults to 1.
Values(v1, v2, ..., vN)	Yields values {v1, v2, ..., vN}.
ValuesIn(container) and ValuesIn(begin,end)	Yields values from a C-style array, an STL-style container, or an iterator range [begin, end).
Bool()	Yields sequence {false, true}.
Combine(g1, g2, ..., gN)	Yields all combinations (the Cartesian product for the math savvy) of the values generated by the N generators. This is only available if your system provides <tr/tuple> header. If you are sure your system does, and Google Test disagrees, you can override it by defining GTEST_HAS_TR1_TUPLE=1. See comments in include/gtest/internal/gtest-port.h for more information.

The following statement will instantiate tests from the FooTest test case each with parameter values "meeny", "miny", and "moe".

INSTANTIATE_TEST_CASE_P(InstantiationName,
                        FooTest,
                        testing::Values("meeny", "miny", "moe"));

To distinguish different instances of the pattern (yes, you can instantiate it more than once), the first argument to INSTANTIATE_TEST_CASE_P is a prefix that will be added to the actual test case name. Remember to pick unique prefixes for different instantiations.

More documentation is available in comments in include/gtest/gtest-param-test.h. Samples are available in samples/sample7_unittest.cc and samples/sample8_unittest.cc.

Hope yo find it useful.

Best regards,
Vlad.

[Zhanyong Wan (λx.x x)]

Hi,

We are hoping to check this in soon. Does anyone have any comments
before it's done? Thanks!

--
Zhanyong

[Vlad Losev]

[adding the group so others may benefit, too]

Hi Peter,

On Thu, Dec 17, 2009 at 5:44 AM, pschwenn <psch...@gmail.com> wrote:

right in thinking that Value-Parameterization and Test Fixture
cannot be used together -- to put it another way: a TEST cannot be
both TEST_F... and TEST_P... ?

For value-parameterized tests, you always write the fixture class:

class MyFooTest : public TestWithParam<Foo> {
  public:
    Bar bar_;
};

And the tests themselves use that fixture:

TEST_P(MyFooTest, DoesBlah) {
  bar_.baz()
}

So in that respect, fixture tests can be considered a subset of value-parameterized tests.


HTH,
Vlad