proposal: parameterized tests
Zhanyong Wan (λx.x x)
Some of us have been discussing ways to extend Google Test to support
parameterized tests, which are test templates that can be instantiated
with different parameter values at run-time. Examples where they are
useful including:
- data-driven tests,
- testing various implementations of an interface,
- testing various instantiations of a class/function template.
Vlad Losev has put together a draft proposal at
https://docs.google.com/Edit?tab=view&docid=dfhzgpkh_30gwjm92dj .
Please take a look and send us your opinion! Thanks,
--
Zhanyong
Zhanyong Wan (λx.x x)
2008/8/6 Matteo Slanina <msla...@google.com>:
> I can't access it. Shouldn't this be in docs.corp instead of docs.google?
>
> -- Matteo
>> _______________________________________________
>> gunit-users mailing list
>> gunit...@google.com
>> https://mailman.corp.google.com/mailman/listinfo/gunit-users
>
>
--
Zhanyong
Zhanyong Wan (λx.x x)
In the Related Work section you listed some solutions in other testing
frameworks. Could you add a small example for each of them to
illustrate how it works? That helps a reader to understand these
approaches quickly without having to read through their documentation.
Thanks,
On Wed, Aug 6, 2008 at 4:01 PM, Zhanyong Wan (λx.x x) <w...@google.com> wrote:
--
Zhanyong
Zhanyong Wan (λx.x x)
I added a list of design goals to the doc. Copied here:
We are trying to produce a design that satisfies the needs of existing
users, is reasonably general, is easy to learn, and minimizes the user
effort by being declarative. Here are the concrete goals we are
trying to achieve:
* We should minimize the number of macros used, as macros pollute
every namespace and are easy to clash with user code or other
libraries.
* We should minimize the need for user-written boilerplate code.
In particular, the user shouldn't need to
* Each instantiation of a parameterized test should be uniquely
identified by a name, which can be printed by the --gtest_list_tests
flag and be used to select which tests to run in
--gtest_filter=PATTERN.
* Except in the parameterization-by-type case, the test parameters
don't need to be determined at compile time. In particular, the
number of test iterations and the values of the test parameters can be
calculated at run time, perhas based on information that's not
available at compile time. One example is to iterate the tests over
each line in a text file.
* Each iteration of a parameterized test should run on a new test
object, to prevent tests from interfering with each other.
* Ideally parameterized tests should work with authomatic test
registration (the way TEST and TEST_F work), such that a user doesn't
need to enumerate the tests in order to run them.
* The definition of a parameterized test "template" (used loosely
here) should be decoupled from the specification of the parameters,
such that a user can define the test "template" in one library and
reuse it with different parameters in different test programs.
* Test parameter specifications should be easily composable. If a
test has multiple, independent parameters, the user should be able to
specify the range of each parameter separately and easily compose the
specs into a single spec. ("Orthogonal constructs for expressing
orthogonal concerns.")
I know they may seem very abstract here. The idea is to use the list
to compare the relative merits of different proposals. While it's
hard or even impossible for a single design to satisfy all points,
we'd like to hit as many of them as possible.
Please let me know if you think there are other important goals we
should add. Thanks,
--
Zhanyong
Zhanyong Wan (λx.x x)
> Zhanyong - The list second item is unfinished. I guess this is what happens
> when two people are editing the document simultaneously :( Gotta kick those
> Writely folks...
Thanks. I've finished it:
"We should minimize the need for user-written boilerplate code. In
particular, the user shouldn't need to explicitly write template type
arguments in many places."
>
> - Vlad
>
> On Wed, Aug 6, 2008 at 6:21 PM, Zhanyong Wan (λx.x x) <w...@google.com>
--
Zhanyong
Zhanyong Wan (λx.x x)
Some high-level comments on your design:
- I agree with David that the distinction between TEST_P
(parameterized test without a fixture) and TEST_PF (parameterized
test with a fixture) may be unnecessary. Parameterized tests are
relatively rare, and are already more involved to write. Having to
write an empty test fixture doesn't seem a big price to pay.
Therefore I think TEST_P won't be very useful and can be removed.
- Obviously all tests sharing the same test fixture should be
parameterized with the same parameter type. Your TEST_PF macro takes
3 arguments (test-case name, test name, and parameter range), which
allows different tests in the test case to have different parameter
ranges. Without losing generality, I think we should be able to move
the specification of the parameter range from TEST_PF to the test
fixture class. In other words, all tests in the same test case will
have the same parameter range. This gets rid of the 3rd argument of
TEST_PF, which means we may be able to merge it with TEST_F - that
would be very nice.
My hunch is that in the majority of the cases, the user would want
to reuse the same parameter range for all (or most) tests in the
test case, so having to repeatedly specify it is tedious and
error-prone.
If the user wants a different parameter range for a test, it
probably means that the test logically doesn't belong to this test
case, so moving it to another test case may be the right thing
anyway.
- Instead of introducing some type list class templates (e.g. boost's
mpl::vector), there's tr1::tuple we can use. For example, we can
use tuple<int, bool, char> to represent a list of 3 types int, bool,
and char. Most people are already familiar with tuples, so this is
easier to learn. Also it avoids a dependency on Boost, which is not
as universally installed as tr1.
gcc's tr1::tuple implementation only supports 10 fields at most, I
think. However that should be enough for most cases.
- At least initially, I would not expose the design of the generator
class "interface" to the users. Treating it as internal details
allows us to improve it without worrying about backward
compatibility. The range, array, etc, generators you plan to
provide should be enough for most users (considering that they have
access to none now). When there are real needs from the users to
define their own generator classes, we can make the API public. At
that time the design can be driven by real user needs, instead of
what we think they might need.
- STL container classes are widely used and the concept of iterators
are well known. Chances are that the user's test parameters will
come from some STL container. When designing the generator class,
we should reuse STL's iterator concept as much as possible. Avoid
non-standard names and types like ValueType and etc.
- I'd try to avoid macros RANGE_PARAM_GENERATOR,
ARRAY_PARAM_GENERATOR, and CARTESIAN_PRODUCT_PARAM_GENERATOR. Are
they really necessary as macros? Here's the syntax I'd like to see:
// Creates a parameter iterator over { 1, 2, 3, 4, 5 }.
Range(1, 5)
const std::string names[] = { "john", "peter", "mary" };
// Creates a parameter iterator over the above static array.
ValuesIn(names);
std::list<char> foo = ...;
// Creates a parameter iterator over the elements in foo.
ValuesIn(foo)
// Creates a parameter iterator over the list { 1, 2, 3, 5, 8 },
// without needing to create an array or container first.
// Values() is overloaded to take 1 or many arguments.
Values(1, 2, 3, 5, 8)
// Creates a parameter iterator over the Cartesian product of
// 3 iterators.
make_tuple(Range(false, true), ValuesIn(name), Values(1, 2, 3))
Values() is overloaded to handle both C-style static arrays and
STL-style containers. We should be able to define these function
templates. They are much easier to use than macros.
To define a test fixture that's parameterized over a list of integers:
class FooTest : public testing::TestWithParam<int> {
public:
static testing::Param<int> GetParamGenerator() {
return Values(1, 2, 3, 5, 8);
}
};
To define a test using this fixture:
TEST_F(FooTest, Bar) {
// parameter will return 1, 2, 3, 5, and 8 in different iterations of
// the test.
EXPECT_GE(parameter(), 0);
}
--
Zhanyong
Zhanyong Wan (λx.x x)
Some explanation:
Param<int> is the type of a parameter generator that generates int
values. Perhaps Params<int> is a better name?
For multi-dimensional parameters:
class FooTest : public testing::TestWithParam<bool, int> {
public:
static testing::Params<bool, int> GetParamGenerator() {
return make_tuple(Values(false, true), Range(1, 5));
// Perhaps we could define Bool() as a shortcut for
// Values(false, true)?
}
};
To support Kurt's use case of mapping an integer to a parameter value:
class FooTest : public testing::TestWithParam<int> {
public:
static testing::Params<int> GetParamGenerator() {
return Range(0, 100);
}
FooTest() {
blah_ = SomeFunction(parameter());
}
protected:
Blah blah_;
};
To support David's use case of multiple bool flags:
class FooTest : public testing::TestWithParam<bool, bool, bool> {
public:
static testing::Params<bool, bool, bool> GetParamGenerator() {
return make_tuple(Bool(), Bool(), Bool());
}
FooTest() {
flag1 = get<0>(parameter());
flag2 = get<1>(parameter());
flag3 = get<2>(parameter());
}
};
--
Zhanyong
Kenton Varda
Kenton Varda
Zhanyong Wan (λx.x x)
2008/8/6 Kenton Varda <ken...@google.com>:
> I think describing this design as "parameterization" is somewhat misleading.
> Normally, something which is "parameterized" can be reused elsewhere with
> arbitrary values substituted for the parameter. There should be no need to
> maintain any single list of all the parameter values we care about.
I agree. This is what I tried to capture with the following design
goal, which I posted earlier on this thread:
"The definition of a parameterized test "template" (used loosely here)
should be decoupled from the specification of the parameters, such
that a user can define the test "template" in one library and reuse it
with different parameters in different test programs."
> Here,
> though, no such thing is possible.
This, I disagree. See more details below.
> All these "parameterized" tests are
> doing is iterating over a set of values that are already known to the
> "parameterized" code. This is not really parameterization; it's just syntax
> sugar that avoids an explicit loop.
Whatever you call it, my hunch is that this is the more common case.
> I think that a real "parameterized" test would be something I declare in one
> place and then reuse in other files. When I reuse the test, I should be
> able to write code which supplies an arbitrary parameter.
You can do it in my design like this:
// Defines the test "template".
class FooTest : public testing::TestWithParam<int> {
public:
Params<int> GetParamGenerator() { return param_; }
Params<int> params_;
};
TEST_F(FooTest, Bar) { ... }
// Binds the test "template" with actual parameters. You can do this
// in main() or in a TestEnvironment.
FooTest::params_ = Range(1, 100);
The actual syntax may not be accurate, and we may want to throw in
some syntactic sugar to simplify the syntax, but you get the idea.
Basically your APPLY_TEST() is done here by some kind of dependency
injection.
This is not the most pretty code, but it's straightforward and I don't
expect this to be a common scenario, so I'm fine with it. It doesn't
seem worth to introduce more magic macros to make this nicer.
> Here's one idea
> for what it might look like:
> // =====================
> // foo.h
>
> // This is the interface we are testing.
> class FooInterface {
> public:
> virtual bool IsLessThan(int a, int b) const = 0;
> };
>
> // =====================
> // foo_interface_test.h
>
> // FooInterfaceTest is a test suite parameterized over a FooInterface*.
> PARAMETERIZED_TEST(FooInterfaceTest, FooInterface*);
>
> // =====================
> // foo_interafce_test.cc
>
> // We can define tests normally.
> TEST_P(FooInterfaceTest, SomeMethod, impl) {
I was hoping to get rid of TEST_P and TEST_PF.
> EXPECT_FALSE(impl->IsLessThan(5, 2));
> }
>
> // We can also declare fixtures by subclassing.
> class BlahTest : public FooInterfaceTest {
> protected:
> virtual void SetUp(FooInterface* impl) {
> impl_ = impl;
> }
> FooInterface* impl_;
> };
>
> TEST_F(BlahTest, SomeMethod) {
> EXPECT_FALSE(impl_->IsLessThan(5, 2));
> }
>
> // =====================
> // foo_impl.h
>
> // An implementation of our interface which we'd like to test.
> class FooImpl: public FooInterface {
> ...
> };
>
> // =====================
> // foo_impl_test.cc
>
> FooImpl my_impl;
>
> // We reuse the parameterized FooInterfaceTest.
> APPLY_TEST(FooInterfaceTest, &my_impl);
The question is how to make automatic test registration work here.
> The above example declares a singleton FooImpl which would be used by all
> the tests. That's undesirable, of course, but the user can easily fix this
> by declaring a FooFactory interface and parameterizing the test on that
> rather than on FooInterface directly.
> This design could also be used to accomplish the data-driven pattern:
>
> PARAMETERIZED_TEST(IsPrimeTest, int);
> TEST_P(IsPrimeTest, TrueCase, value) {
> EXPECT_TRUE(IsPrime(value));
> }
> APPLY_TEST_MULTI(IsPrimeTest, {2, 3, 5, 7, 11, 13, 17});
> Thoughts?
> On Wed, Aug 6, 2008 at 4:01 PM, Zhanyong Wan (λx.x x) <w...@google.com>
> wrote:
>>
--
Zhanyong
Kenton Varda
// Binds the test "template" with actual parameters. You can do this
// in main() or in a TestEnvironment.
FooTest::params_ = Range(1, 100);
I was hoping to get rid of TEST_P and TEST_PF.
> // We reuse the parameterized FooInterfaceTest.> APPLY_TEST(FooInterfaceTest, &my_impl);The question is how to make automatic test registration work here.
Zhanyong Wan (λx.x x)
> On Wed, Aug 6, 2008 at 8:44 PM, Zhanyong Wan (λx.x x) <w...@google.com>
> wrote:
>>
>> // Binds the test "template" with actual parameters. You can do this
>> // in main() or in a TestEnvironment.
>> FooTest::params_ = Range(1, 100);
>
> So basically you're leaving it up to me to implement my own registration
> mechanism for parameter values.
This mechanism can be implemented by following the very simple pattern
I showed. And I realized there is a more straightforward way:
// foo_test_util.cc: defines the test "template".
class FooTest : public testing::TestWithParam<int> {
public:
// The user who instantiates FooTest is expected to define this.
Params<int> GetParamGenerator();
};
TEST_F(FooTest, Bar) { ... }
TEST_F(FooTest, Baz) { ... }
// my_test.cc: binds the test "template" with actual parameters.
Params<int> FooTest::GetParamGenerator() { return Range(1, 100); }
> Although this gives me the ability to do
> that, it seems like a rather convoluted way to accomplish what I want.
As I admitted, this is not the prettiest code. However I think it's
acceptable given that I don't expect people to do it often. I want to
avoid introducing some heavy constructs for relatively rare cases.
> What are the advantages over my approach?
The advantages include:
- It doesn't require to introduce any new, magic macros. Anyone
reading the code can immediately see what it's doing.
- It allows automatic test registration. Despite your claim, I doubt
how you can achieve automatic test registration with your approach.
More details below.
Also, please note that the design I presented only covers data-driven
tests and interface tests. Type-driven tests (i.e. testing various
instantiations of a class/function template) requires some different
constructs that are yet TBD. Automatic test registration probably
won't work there.
My hunch is that the mechanism for type-driven tests can also be used
for writing real parameterized tests. If that's case, we may not need
to duplicate the effort here.
>> I was hoping to get rid of TEST_P and TEST_PF.
>
> My example has a TEST_P but not a TEST_PF -- it just uses TEST_F and passes
> the parameter to SetUp(). I suppose TEST_P might not be necessary since
> creating a fixture is not that hard. Or maybe PARAMETERIZED_TEST itself
> could generate a class usable as a fixture.
>>
>> > // We reuse the parameterized FooInterfaceTest.
>> > APPLY_TEST(FooInterfaceTest, &my_impl);
>>
>> The question is how to make automatic test registration work here.
>
> I don't see what the problem is. APPLY_TEST registers an application of
> FooInterfaceTest to be run during RUN_TESTS().
The problem is that FooInterfaceTest is a test case that contains
multiple tests, which are implemented as sub-classes of
FooInterfaceTest. However, the FooInterfaceTest class itself has no
knowledge of these sub-classes. The syntax
APPLY_TEST(FooInterfaceTest, &my_impl) doesn't give the compiler
enough information to enumerate the tests in FooInterfaceTest.
--
Zhanyong
Kenton Varda
And I realized there is a more straightforward way:
> Although this gives me the ability to do> that, it seems like a rather convoluted way to accomplish what I want.As I admitted, this is not the prettiest code. However I think it's
acceptable given that I don't expect people to do it often. I want to
avoid introducing some heavy constructs for relatively rare cases.
> I don't see what the problem is. APPLY_TEST registers an application ofThe problem is that FooInterfaceTest is a test case that contains
> FooInterfaceTest to be run during RUN_TESTS().
multiple tests, which are implemented as sub-classes of
FooInterfaceTest. However, the FooInterfaceTest class itself has no
knowledge of these sub-classes. The syntax
APPLY_TEST(FooInterfaceTest, &my_impl) doesn't give the compiler
enough information to enumerate the tests in FooInterfaceTest.
Zhanyong Wan (λx.x x)
> On Wed, Aug 6, 2008 at 10:01 PM, Zhanyong Wan (λx.x x) <w...@google.com>
> wrote:
>>
>> And I realized there is a more straightforward way:
>
> Um, no. Sorry, but I refuse to define the same C++ symbol differently in
> different source files, even if they aren't intended to be linked together.
> Aside from being extremely bad style, there are far too many ways that
> could go horribly wrong, and far too many inflexibilities it introduces
> (like making it impossible to link the tests together into a single binary).
> You are aware that GCC will not tell you about multiply-defined symbols so
> long as they reside in different dynamic libraries, right?
Very good point! I take it back.
> Similarly, I find the "very simple pattern" you showed before to be
> unreasonable because it breaks if you have two usages of the same "template"
> test in the same binary. Hence, I have to implement my own registration
> mechanism.
>
>> > Although this gives me the ability to do
>> > that, it seems like a rather convoluted way to accomplish what I want.
>>
>> As I admitted, this is not the prettiest code. However I think it's
>> acceptable given that I don't expect people to do it often. I want to
>> avoid introducing some heavy constructs for relatively rare cases.
>
> OK, but the problem you are solving is one which already has a usable
> solution: people can just use an explicit loop.
One explicit loop is not enough though. You'll need to write the loop
for each test in your test case, immediately an O(N) syntactic cost.
This doesn't allow you to easily select which iteration to run using
--gtest_filter either. And you'd go through more trouble to get the
set-up/run/tear-down behavior where a set-up failure will cause the
run phase to be skipped. You can make it work but it's not exactly
usable.
> On the other hand, the
> problem that my solution solves does not have any usable existing solution.
> Your design happens to make a solution feasible but still requires a lot of
> additional work.
Agreed.
>> > I don't see what the problem is. APPLY_TEST registers an application of
>> > FooInterfaceTest to be run during RUN_TESTS().
>>
>> The problem is that FooInterfaceTest is a test case that contains
>> multiple tests, which are implemented as sub-classes of
>> FooInterfaceTest. However, the FooInterfaceTest class itself has no
>> knowledge of these sub-classes. The syntax
>> APPLY_TEST(FooInterfaceTest, &my_impl) doesn't give the compiler
>> enough information to enumerate the tests in FooInterfaceTest.
>
> This is why the PARAMETERIZED_TEST() macro was needed for declaring
> FooInterfaceTest.
I see. I thought it was just for declaring the test parameter type.
Sorry for my lack of imagination.
> The gtest implementation could hook in here to maintain a
> mapping from the parameterized test base class to all the registered
> APPLY_TESTs and all the registered TEST_Fs for that class.
> Yes, this involves "magic macros", but any usable solution to my problem is
> going to require some sort of static registration system, which almost
> always means magic macros will be involved.
I start to like this design now that you've shown that automatic test
registration can be made to work. I need more time to think through
it though.
Some fixable issues:
- Since each parameterized test case must maintain its own static test
registry, the following code will not work:
PARAMETERIZED_TEST(FooTest, int);
// Test cases ATest and BTest share the same test fixture definition.
typedef FooTest ATest;
TEST_P(ATest, Blah) { ... }
typedef FooTest BTest;
TEST_P(BTest, BlahBlah) { ... }
// Oops, ATest and BTest now use the same registry, while they
// each think they have their own.
Subclassing from FooTest has the same problem.
- The PARAMETERIZED_TEST macro makes it hard to customize the test
fixture class (e.g. specifying a list of base classes, defining some
non-trivial ctors, etc). You may be able to work around it but it's
convoluted.
- The name APPLY_TEST seems too functional for most C++ users. Also
I'd refer to test case instead of test. Perhaps REGISTER_TEST_CASE
is clearer.
- I'd like to merge APPLY_TEST and APPLY_TEST_MULTI into a single
construct.
- APPLY_TEST should let the user supply a name prefix for
distinguishing different instantiations of the same test "template"
in Google Test's output (and in --gtest_filter).
So I'm thinking about this revised plan:
- To define a parameterized test fixture, just write it in usual C++:
class FooTest : public TestWithParam<int> {
... whatever ...
};
The test registry for FooTest will be stored in static data members
of class testing::internal::TestRegistry<FooTest>, which Google Test
defines.
- To write parameterized tests using this fixtures, just use TEST_P
instead of TEST_F:
TEST_P(FooTest, DoesThis) { ... }
TEST_P(FooTest, DoesThat) { ... }
The difference is that TEST_P doesn't automatically add the test to
Google Test's global test list. Instead, it will add information
about the test into some static data members of
testing::internal::TestRegistry<FooTest>.
- To instantiate a parameterized test case:
// The second argument is a name prefix.
// The third argument is the test parameter generator.
REGISTER_TEST_CASE(FooTest, ModuleA, Values(1));
// This creates 2 tests:
// ModuleA/FooTest.DoesThis
// ModuleA/FooTest.DoesThat
// Each test will be run once.
REGISTER_TEST_CASE(FooTest, ModuleB, Values(2, 3, 5));
// This creates 2 tests:
// ModuleB/FooTest.DoesThis
// ModuleB/FooTest.DoesThat
// Each test will be run 3 times.
You can now use --gtest_filter=ModuleA/* or
--gtest_filter=ModuleB/FooTest.DoesThat/2 to select which tests to run.
- To reuse the same fixture in multiple test cases:
// Can't use typedef FooTest AbcTest. We need distinct types
// or the test registries will clash.
class AbcTest : public FooTest {};
TEST_P(AbcTest, ...) { ... }
...
class XyzTest : public FooTest {};
TEST_P(XyzTest, ...) { ... }
We can even make it a run-time error if the user uses typedef to
reuse the fixture. This mistake is indicated by two tests with
different test case names being registered into the same registry.
I haven't thought through all the implementation details, but this
looks promising. Thoughts?
--
Zhanyong
Zhanyong Wan (λx.x x)
One tricky issue is that when TEST_P(FooTest, ...) and
REGISTER_TEST_CASE(FooTest, ...) are in different translation units
(most likely), their order of execution is undefined. This means the
implementation must be careful not to depend on the relative order of
these macros. It should be doable though.
>
> - To reuse the same fixture in multiple test cases:
>
> // Can't use typedef FooTest AbcTest. We need distinct types
> // or the test registries will clash.
> class AbcTest : public FooTest {};
> TEST_P(AbcTest, ...) { ... }
>
> ...
>
> class XyzTest : public FooTest {};
> TEST_P(XyzTest, ...) { ... }
>
> We can even make it a run-time error if the user uses typedef to
> reuse the fixture. This mistake is indicated by two tests with
> different test case names being registered into the same registry.
>
> I haven't thought through all the implementation details, but this
> looks promising. Thoughts?
>
> --
> Zhanyong
>
--
Zhanyong
Jeffrey Yasskin
> Thanks for the feedback, Kenton.
>
> 2008/8/6 Kenton Varda <ken...@google.com>:
>> I think describing this design as "parameterization" is somewhat misleading.
>> Normally, something which is "parameterized" can be reused elsewhere with
>> arbitrary values substituted for the parameter. There should be no need to
>> maintain any single list of all the parameter values we care about.
>
> I agree. This is what I tried to capture with the following design
> goal, which I posted earlier on this thread:
>
> "The definition of a parameterized test "template" (used loosely here)
> should be decoupled from the specification of the parameters, such
> that a user can define the test "template" in one library and reuse it
> with different parameters in different test programs."
>
>> Here,
>> though, no such thing is possible.
>
> This, I disagree. See more details below.
>
>> All these "parameterized" tests are
>> doing is iterating over a set of values that are already known to the
>> "parameterized" code. This is not really parameterization; it's just syntax
>> sugar that avoids an explicit loop.
>
> Whatever you call it, my hunch is that this is the more common case.
I can't say whether it's more common or not, but the only thing I've
needed a parametrized test for that I couldn't do some other way was
testing implementations of interfaces and Concepts. For example, I'd
like to write one test that checks that a map implementation is
STL-compatible, and instantiate it in the test for each additional
type of map. There's no common base class for STL maps, so I can't use
a function like Values(). It's also important to instantiate the map
with a couple different value types, so the test has to be able to
take a template template parameter.
Because the STL Associative Container concept is built from several
other concepts, we'd want to be able to compose trees of these tests.
That is, the test for being a Container should be shared between the
Associative Container test and the Sequence test. It doesn't look like
any of the proposals now on the table will support that?
To test interfaces, we can use the value-parametrized tests, but it'd
still be easier for the user if they could pass the type of the
subclass instead of a factory for it.
>> I think that a real "parameterized" test would be something I declare in one
>> place and then reuse in other files. When I reuse the test, I should be
>> able to write code which supplies an arbitrary parameter.
>
> You can do it in my design like this:
>
> // Defines the test "template".
>
> class FooTest : public testing::TestWithParam<int> {
> public:
> Params<int> GetParamGenerator() { return param_; }
>
> Params<int> params_;
> };
>
> TEST_F(FooTest, Bar) { ... }
>
> // Binds the test "template" with actual parameters. You can do this
> // in main() or in a TestEnvironment.
> FooTest::params_ = Range(1, 100);
Ew. Don't we always tell people to pass parameters as arguments rather
than through global variables?
--
Namasté,
Jeffrey Yasskin
http://jeffrey.yasskin.info/
Zhanyong Wan (λx.x x)
Thanks for the comments.
I've changed the design to be based on Kenton's ideas (see my last
couple of messages on this thread), so some of your concerns should be
moot now. :-)
You're right that the proposals so far don't cover testing concepts,
or type-driven tests as I call them. That will require some different
mechanism and is yet to be designed. I plan to give it some thinking
soon.
Cheers,
--
Zhanyong
Jeffrey Yasskin
Thanks for looking into the type-driven tests. :)
One set of bikesheddy comments: Perhaps INSTANTIATE_TEST_CASE instead
of REGISTER_TEST_CASE. Although APPLY_TEST (or CALL_TEST) has the
benefit of being short. APPLY_TEST_P is also short and mentions
explicitly the kind of thing that's being applied. And the order of
the parameters to the macro should match the order they appear in the
--gtest_filter name. So if the macro call winds up as
WHATEVER_TEST(FooTest, ModuleB, Values(2, 3)), the name should be
"FooTest.DoesThat/ModuleB/2" or similar.
Jeffrey
Zhanyong Wan (λx.x x)
> Oops. Don't know why I didn't see those emails.
>
> Thanks for looking into the type-driven tests. :)
>
> One set of bikesheddy comments: Perhaps INSTANTIATE_TEST_CASE instead
> of REGISTER_TEST_CASE. Although APPLY_TEST (or CALL_TEST) has the
> benefit of being short. APPLY_TEST_P is also short and mentions
> explicitly the kind of thing that's being applied.
I don't quite like REGISTER_TEST_CASE either, so I'm considering plain
ADD_TEST_CASE now. It's shorter than INSTANTIATE_TEST_CASE but OTOH
the latter is more accurate. What do people think?
> And the order of
> the parameters to the macro should match the order they appear in the
> --gtest_filter name. So if the macro call winds up as
> WHATEVER_TEST(FooTest, ModuleB, Values(2, 3)), the name should be
> "FooTest.DoesThat/ModuleB/2" or similar.
Agreed. In fact I've already adjusted the order in the design doc on
Google Docs. :-)
--
Zhanyong
Zhanyong Wan (λx.x x)
I've updated http://docs.google.com/Doc?id=dfhzgpkh_30gwjm92dj with
the latest design (there are actually 3 designs and I'm in favor of #3
now). Please take a look and share your opinions. Thanks,
--
Zhanyong
Kenton Varda
I start to like this design now that you've shown that automatic testregistration can be made to work. I need more time to think through
it though.
Some fixable issues:
- Since each parameterized test case must maintain its own static test
registry, the following code will not work:
PARAMETERIZED_TEST(FooTest, int);
// Test cases ATest and BTest share the same test fixture definition.
typedef FooTest ATest;
TEST_P(ATest, Blah) { ... }
typedef FooTest BTest;
TEST_P(BTest, BlahBlah) { ... }
// Oops, ATest and BTest now use the same registry, while they
// each think they have their own.
Subclassing from FooTest has the same problem.
- The PARAMETERIZED_TEST macro makes it hard to customize the test
fixture class (e.g. specifying a list of base classes, defining some
non-trivial ctors, etc). You may be able to work around it but it's
convoluted.
- The name APPLY_TEST seems too functional for most C++ users. Also
I'd refer to test case instead of test. Perhaps REGISTER_TEST_CASE
is clearer.
- I'd like to merge APPLY_TEST and APPLY_TEST_MULTI into a single
construct.
- APPLY_TEST should let the user supply a name prefix for
distinguishing different instantiations of the same test "template"
in Google Test's output (and in --gtest_filter).
So I'm thinking about this revised plan:
- To define a parameterized test fixture, just write it in usual C++:
class FooTest : public TestWithParam<int> {
... whatever ...
};
The test registry for FooTest will be stored in static data members
of class testing::internal::TestRegistry<FooTest>, which Google Test
defines.
- To write parameterized tests using this fixtures, just use TEST_P
instead of TEST_F:
TEST_P(FooTest, DoesThis) { ... }
TEST_P(FooTest, DoesThat) { ... }
The difference is that TEST_P doesn't automatically add the test to
Google Test's global test list. Instead, it will add information
about the test into some static data members of
testing::internal::TestRegistry<FooTest>.
- To instantiate a parameterized test case:
// The second argument is a name prefix.
// The third argument is the test parameter generator.
REGISTER_TEST_CASE(FooTest, ModuleA, Values(1));
// This creates 2 tests:
// ModuleA/FooTest.DoesThis
// ModuleA/FooTest.DoesThat
// Each test will be run once.
REGISTER_TEST_CASE(FooTest, ModuleB, Values(2, 3, 5));
// This creates 2 tests:
// ModuleB/FooTest.DoesThis
// ModuleB/FooTest.DoesThat
// Each test will be run 3 times.
You can now use --gtest_filter=ModuleA/* or
--gtest_filter=ModuleB/FooTest.DoesThat/2 to select which tests to run.
- To reuse the same fixture in multiple test cases:
// Can't use typedef FooTest AbcTest. We need distinct types
// or the test registries will clash.
class AbcTest : public FooTest {};
TEST_P(AbcTest, ...) { ... }
...
class XyzTest : public FooTest {};
TEST_P(XyzTest, ...) { ... }
We can even make it a run-time error if the user uses typedef to
reuse the fixture. This mistake is indicated by two tests with
different test case names being registered into the same registry.
Zhanyong Wan (λx.x x)
> On Thu, Aug 7, 2008 at 1:24 AM, Zhanyong Wan (λx.x x) <w...@google.com>
> wrote:
>>
>> - Since each parameterized test case must maintain its own static test
>> registry, the following code will not work:
>>
>> PARAMETERIZED_TEST(FooTest, int);
>>
>> // Test cases ATest and BTest share the same test fixture definition.
>>
>> typedef FooTest ATest;
>> TEST_P(ATest, Blah) { ... }
>>
>> typedef FooTest BTest;
>> TEST_P(BTest, BlahBlah) { ... }
>>
>> // Oops, ATest and BTest now use the same registry, while they
>> // each think they have their own.
>>
>> Subclassing from FooTest has the same problem.
>
> IMO, this is desired behavior. When I do APPLY_TEST(FooTest, ...), I want
> both ATest and BTest to be executed on my parameters. If I didn't want
> this, I would declare ATest and BTest separately using independent
> PARAMETERIZED_TEST declarations.
I like the conciseness of using a single APPLY_TEST(FooTest, ...)
statement to instantiate all test cases derived from FooTest. OTOH,
relying on inheritance for specifying membership in a parameterized
test suite is a hack and can be very confusing to the users. When a
reader sees APPLY_TEST(FooTest, ...), it can be hard for him to know
which test cases are involved, without tediously searching through the
code base. Basically he needs to find all subclasses of FooTest,
which can be scattered in the source files and whose names may bear no
resemblance with FooTest at all, and then find all tests defined using
these subclasses. It's no longer a simple grep job.
>> - The PARAMETERIZED_TEST macro makes it hard to customize the test
>> fixture class (e.g. specifying a list of base classes, defining some
>> non-trivial ctors, etc). You may be able to work around it but it's
>> convoluted.
>
> My thought was that if you wanted a customized fixture, you'd subclass it.
This puts some artificial restrictions on how the user can structure
his fixture class hierarchy. In particular, it's now impossible to
reuse fixture classes across parameterized test suites. For example,
given
PARAMETERIZED_TEST(FooTest, int);
class ATest : public FooTest {
... some goodies ...
};
...
and I'm writing a related test suite:
PARAMETERIZED_TEST(BarTest, int);
I cannot reuse fixture ATest in tests in BarTest. Considering that
all tests in the same PARAMETERIZED_TEST suite share the same
parameter values, you have to use different PARAMETERIZED_TESTs if you
want to vary the parameter list, even when the two suites are about
the same subject and logically related. This means you'll often want
to share the same fixture definition between multiple
PARAMETERIZED_TESTs. Therefore this restriction is unreasonable.
> Again, this way multiple fixtures can be associated with a single
> PARAMETERIZED_TEST and invoked with a single APPLY_TEST, which is useful if
> the same interface needs to be tested in different circumstances that call
> for different test setups.
I think requiring the user to write APPLY_TEST() for each test case he
wants to instantiate is the lesser of the two evils. (Note that a
test case contains multiple tests.) Different test cases often
require different parameter lists anyway, so this may not be much more
work in practice. Also, the user may appreciate the flexibility of
being able to set different parameter lists for different test cases.
And we'll have one less magic macro to introduce.
>> - To define a parameterized test fixture, just write it in usual C++:
>>
>> class FooTest : public TestWithParam<int> {
>> ... whatever ...
>> };
>>
>> The test registry for FooTest will be stored in static data members
>> of class testing::internal::TestRegistry<FooTest>, which Google Test
>> defines.
>
> Unfortunately this means each parameterized test can only have a single
> fixture. Not sure how limiting this would be in practice.
As said above, I don't think this would be a big problem.
>> - To write parameterized tests using this fixtures, just use TEST_P
>> instead of TEST_F:
>>
>> TEST_P(FooTest, DoesThis) { ... }
>> TEST_P(FooTest, DoesThat) { ... }
>>
>> The difference is that TEST_P doesn't automatically add the test to
>> Google Test's global test list. Instead, it will add information
>> about the test into some static data members of
>> testing::internal::TestRegistry<FooTest>.
>
> Can't this be done automatically? testing::Test could have:
> static const bool gtest_internal_is_parameterized_ = false;
> Then TestWithParam could redefine this as true, then you could use template
> magic to detect the difference in TEST_F.
I want the distinction between the two to be explicit to the users.
TEST_P defines a test "pattern" that's not a real test until being
instantiated. TEST_F defines a real test immediately. Therefore,
conceptually TEST_F is NOT a special case of TEST_P. If we merge the
two constructs, people will wonder why their TEST_F didn't get run by
Google Test. It's more work to explain to them that not all TEST_Fs
are created equal.
Thanks,
--
Zhanyong
Kenton Varda
Zhanyong Wan (λx.x x)
> I think people are more likely to want to use multiple text fixtures with
> one parameterization than multiple parameterizations with one test fixture.
> However, I don't think this is a really big deal so I'll concede the point.
> I can pretty easily #define my own macro which expands to a series of
> APPLY_TESTs covering all my fixtures, so that clients don't have to update
> their code whenever I add a new fixture.
> Agreed on the rest.
Great.
Other people have been pretty quiet. What do other major-stake
holders think? Vlad, David, and Kurt?
--
Zhanyong
Vlad Losev
Oh, I am pretty late to the party, and still absorbing your exchange with Kenton. In that light, design #1 does indeed seem lacking. There are however some feartures in it I'd like to see in the final design.
Hi, Vlad,
Some high-level comments on your design:
- I agree with David that the distinction between TEST_P
(parameterized test without a fixture) and TEST_PF (parameterized
test with a fixture) may be unnecessary. Parameterized tests are
relatively rare, and are already more involved to write. Having to
write an empty test fixture doesn't seem a big price to pay.
Therefore I think TEST_P won't be very useful and can be removed.
- Obviously all tests sharing the same test fixture should be
parameterized with the same parameter type. Your TEST_PF macro takes
3 arguments (test-case name, test name, and parameter range), which
allows different tests in the test case to have different parameter
ranges. Without losing generality, I think we should be able to move
the specification of the parameter range from TEST_PF to the test
fixture class. In other words, all tests in the same test case will
have the same parameter range. This gets rid of the 3rd argument of
TEST_PF, which means we may be able to merge it with TEST_F - that
would be very nice.
My hunch is that in the majority of the cases, the user would want
to reuse the same parameter range for all (or most) tests in the
test case, so having to repeatedly specify it is tedious and
error-prone.
If the user wants a different parameter range for a test, it
probably means that the test logically doesn't belong to this test
case, so moving it to another test case may be the right thing
anyway.
- Instead of introducing some type list class templates (e.g. boost's
mpl::vector), there's tr1::tuple we can use. For example, we can
use tuple<int, bool, char> to represent a list of 3 types int, bool,
and char. Most people are already familiar with tuples, so this is
easier to learn. Also it avoids a dependency on Boost, which is not
as universally installed as tr1.
gcc's tr1::tuple implementation only supports 10 fields at most, I
think. However that should be enough for most cases.
- At least initially, I would not expose the design of the generator
class "interface" to the users. Treating it as internal details
allows us to improve it without worrying about backward
compatibility. The range, array, etc, generators you plan to
provide should be enough for most users (considering that they have
access to none now). When there are real needs from the users to
define their own generator classes, we can make the API public. At
that time the design can be driven by real user needs, instead of
what we think they might need.
- STL container classes are widely used and the concept of iterators
are well known. Chances are that the user's test parameters will
come from some STL container. When designing the generator class,
we should reuse STL's iterator concept as much as possible. Avoid
non-standard names and types like ValueType and etc.
- I'd try to avoid macros RANGE_PARAM_GENERATOR,
ARRAY_PARAM_GENERATOR, and CARTESIAN_PRODUCT_PARAM_GENERATOR. Are
they really necessary as macros? Here's the syntax I'd like to see:
// Creates a parameter iterator over { 1, 2, 3, 4, 5 }.
Range(1, 5)
const std::string names[] = { "john", "peter", "mary" };
// Creates a parameter iterator over the above static array.
ValuesIn(names);
std::list<char> foo = ...;
// Creates a parameter iterator over the elements in foo.
ValuesIn(foo)
// Creates a parameter iterator over the list { 1, 2, 3, 5, 8 },
// without needing to create an array or container first.
// Values() is overloaded to take 1 or many arguments.
Values(1, 2, 3, 5, 8)
// Creates a parameter iterator over the Cartesian product of
// 3 iterators.
make_tuple(Range(false, true), ValuesIn(name), Values(1, 2, 3))
Zhanyong Wan (λx.x x)
>> - I agree with David that the distinction between TEST_P
>> (parameterized test without a fixture) and TEST_PF (parameterized
>> test with a fixture) may be unnecessary. Parameterized tests are
>> relatively rare, and are already more involved to write. Having to
>> write an empty test fixture doesn't seem a big price to pay.
>> Therefore I think TEST_P won't be very useful and can be removed.
>
> I disagree. The best advantage Google Test has over other test frameworks,
> IMO, is the ease of use. This is very important. With Google Test one can
> start writing tests practically on a whim (try that with CppUnit!). I'd like
> us to keep the same ease of use with parametrized tests. Fixtures are
> necessary if you have to perform some set-up and/or tear-down work. If you
> just want to run some simple tests over a set of values you don't need
> fixtures.
I agree that ease-of-use is a top priority. Keep in mind that every
construct we add makes the framework a little bit more work to learn
(and a little harder to evolve). We have to weigh the
learning/maintaining cost against the value a construct brings.
My hunch is that parameterized tests are an advanced tool that won't
be needed a lot, which means having both TEST_P and TEST_PF may not be
worth it.
Also my philosophy is to add things slowly. Once some (mis)feature is
in, it's extremely hard to take out, and over time Google Test will
become a kitchen sink. We can always add more to the framework when
we have real user needs instead of hypothetical needs.
If you want a parameterized test but don't want a fixture, just define
an empty one:
class FooTest : public testing::TestWithParam<int> {};
You need to declare the parameter type somewhere anyway.
>> - Obviously all tests sharing the same test fixture should be
>> parameterized with the same parameter type. Your TEST_PF macro takes
>> 3 arguments (test-case name, test name, and parameter range), which
>> allows different tests in the test case to have different parameter
>> ranges. Without losing generality, I think we should be able to move
>> the specification of the parameter range from TEST_PF to the test
>> fixture class. In other words, all tests in the same test case will
>> have the same parameter range. This gets rid of the 3rd argument of
>> TEST_PF, which means we may be able to merge it with TEST_F - that
>> would be very nice.
>>
>> My hunch is that in the majority of the cases, the user would want
>> to reuse the same parameter range for all (or most) tests in the
>> test case, so having to repeatedly specify it is tedious and
>> error-prone.
>>
>> If the user wants a different parameter range for a test, it
>> probably means that the test logically doesn't belong to this test
>> case, so moving it to another test case may be the right thing
>> anyway.
>
> Hrm. On one hand, I am inclined to agree. Within one test case, running some
> tests over one set of values and another tests over a different set of
> values can lead to missed coverage. On the other hands imagine two
> data-driven tests each one testing a method in a class. The methods can have
> different boundary conditions, requiring different values. Requiring two
> different fixtures for that is excessive, IMO.
I agree it's not ideal, but the other choice seems poorer. We just
try to make the best trade-off we can.
To add a fixture is not very hard, just define an empty class:
class AnotherFixture : public BaseFixture {};
Also, having to specify the parameter list for each test gets too
verbose in the context of design #3...
> /After emergency reading up on tr1::tuples/ My original design implied using
> std::pair for Cartesian product generators, but tuples are better. Typelists
> are probably the best for parametrization by type, but may also be good for
> specifying type list for parametrization by interface.
What I meant is to use a tuple type to represent a type list in
type-driven tests. I was saying that there is no need to introduce a
new way to represent a list of types.
> We don't have to expose or document the generator interface to the users. In
> fact, the macros described in the document are designed to explicitly hide
> it. You invoke the macro and get this opaque entity you later pass as
> template parameter to ::testing::ParametrizedTest.
That's why I think the generator interface is an implementation detail
and doesn't need to be described in the design doc for the readers to
understand how the concept of generator works. In any case, it's a
minor issue.
BTW, you may have noticed that I use the "parameterized" spelling with
an "e" after "t". It seems to be more popular according to Google
search. :-)
Thanks,
--
Zhanyong
Vlad Losev
Could you please expand on the concept of the module? Is it just a syntactic construct to identify what instantiation of the test we are dealing with?
Thanks,
Vlad
Zhanyong Wan (λx.x x)
> Zhanyong -
>
> Could you please expand on the concept of the module? Is it just a syntactic
> construct to identify what instantiation of the test we are dealing with?
The name "ModuleA" was just an example. Any name goes. Its sole
purpose is for distinguishing between tests instantiated from the same
test pattern. You can think of it like a namespace.
--
Zhanyong
Vlad Losev
TEST_P(FooTest, DoesThis) { ... }
which is instantiated from
REGISTER_TEST_CASE(FooTest, Values(1, 2, 3));
and
int[] array = {3, 4, 5};
REGISTER_TEST_CASE(FooTest, ValuesIn(array));
it doesn't matter which "test case" the test FooTest.DoesThis.3 comes. In either case this is the same code being tested, with the same parameter.
By the way, I have an issue with FooTest.DoesThis/3 designation. The slash character is used in XPath expressions and if XPath is used to navigate results of the tests, it will make it harder to write such expressions.
- Vlad
Zhanyong Wan (λx.x x)
> Then I argue that modules are not really necessary. If you have
>
> TEST_P(FooTest, DoesThis) { ... }
> which is instantiated from
> REGISTER_TEST_CASE(FooTest, Values(1, 2, 3));
> and
> int[] array = {3, 4, 5};
> REGISTER_TEST_CASE(FooTest, ValuesIn(array));
>
> it doesn't matter which "test case" the test FooTest.DoesThis.3 comes. In
> either case this is the same code being tested, with the same parameter.
In my example, the number 3 is the index of the test iteration, not
the test parameter value. In general, some values are not printable,
or may print in a format that we don't like. We cannot use the
parameter value to identify a test iteration.
> By the way, I have an issue with FooTest.DoesThis/3 designation. The slash
> character is used in XPath expressions and if XPath is used to navigate
> results of the tests, it will make it harder to write such expressions.
I'm not familiar with XPath. Just curious: the test name is not an
element or attribute name in the XML report. It just appears in an
attribute value and is quoted. Will it still have problem with XPath?
If yes, I agree we need to change the naming scheme. Any suggestion?
--
Zhanyong
Vlad Losev
On Thu, Aug 7, 2008 at 5:38 PM, Vlad Losev <vl...@google.com> wrote:In my example, the number 3 is the index of the test iteration, not
> Then I argue that modules are not really necessary. If you have
>
> TEST_P(FooTest, DoesThis) { ... }
> which is instantiated from
> REGISTER_TEST_CASE(FooTest, Values(1, 2, 3));
> and
> int[] array = {3, 4, 5};
> REGISTER_TEST_CASE(FooTest, ValuesIn(array));
>
> it doesn't matter which "test case" the test FooTest.DoesThis.3 comes. In
> either case this is the same code being tested, with the same parameter.
the test parameter value. In general, some values are not printable,
or may print in a format that we don't like. We cannot use the
parameter value to identify a test iteration.
As for the modules -- I still think that situations when they are important will not be frequent enough to warrant a separate parameter. I suggest instead an optional post-clause similar to ones used in gMock:
REGISTER_TEST_CASE(FooTest, Values(1, 2, 3).WithName("ModuleA"));
I'm not familiar with XPath. Just curious: the test name is not an
> By the way, I have an issue with FooTest.DoesThis/3 designation. The slash
> character is used in XPath expressions and if XPath is used to navigate
> results of the tests, it will make it harder to write such expressions.
element or attribute name in the XML report. It just appears in an
attribute value and is quoted. Will it still have problem with XPath?
If yes, I agree we need to change the naming scheme. Any suggestion?
Zhanyong Wan (λx.x x)
>
>
> On Thu, Aug 7, 2008 at 5:45 PM, Zhanyong Wan (λx.x x) <w...@google.com>
> wrote:
>>
>> On Thu, Aug 7, 2008 at 5:38 PM, Vlad Losev <vl...@google.com> wrote:
>> > Then I argue that modules are not really necessary. If you have
>> >
>> > TEST_P(FooTest, DoesThis) { ... }
>> > which is instantiated from
>> > REGISTER_TEST_CASE(FooTest, Values(1, 2, 3));
>> > and
>> > int[] array = {3, 4, 5};
>> > REGISTER_TEST_CASE(FooTest, ValuesIn(array));
>> >
>> > it doesn't matter which "test case" the test FooTest.DoesThis.3 comes.
>> > In
>> > either case this is the same code being tested, with the same parameter.
>>
>> In my example, the number 3 is the index of the test iteration, not
>> the test parameter value. In general, some values are not printable,
>> or may print in a format that we don't like. We cannot use the
>> parameter value to identify a test iteration.
>
>
> Oh, it's much more readable to use values instead of indexes. Imagine
> looking looking up an item from a Cartesian product of three sequences by
> its flat index. That's why we should have a concept of a display value. We
We should print the parameter value somewhere in the test output -
that should solve your problem. However we shouldn't use it as part
of the test name.
Test names are used to identify the tests and must have certain properties:
1. They must be unique (at least for different parameter values).
2. They must work well with --gtest_filter.
3. They must work well in the XML report.
Using parameter values as part of the test names violates all 3:
1. Two different values may print as the same string (especially for
user-defined types).
2. They may contain shell meta characters that are hard to handle when
being passed on command lines.
3. As you said, they shouldn't contain "/", which we have no control
over.
> can define it for the basic types like integrals, floating point numbers,
> strings, and tuples and resort to indexes for everything else. And we can
> allow users to define display values for their own types. My design
> provided for display values -- check out the implementation.I
> As for the modules -- I still think that situations when they are important
> will not be frequent enough to warrant a separate parameter. I suggest
> instead an optional post-clause similar to ones used in gMock:
>
> REGISTER_TEST_CASE(FooTest, Values(1, 2, 3).WithName("ModuleA"));
I don't think the name prefix is a property of the parameter
generator, which the above syntax leads one to think.
I think it's very common that people would want to pick some
meaningful names to identify their test instantiation. If they don't
care about the name prefix, they can always leave it empty.
>> > By the way, I have an issue with FooTest.DoesThis/3 designation. The
>> > slash
>> > character is used in XPath expressions and if XPath is used to navigate
>> > results of the tests, it will make it harder to write such expressions.
>>
>> I'm not familiar with XPath. Just curious: the test name is not an
>> element or attribute name in the XML report. It just appears in an
>> attribute value and is quoted. Will it still have problem with XPath?
>> If yes, I agree we need to change the naming scheme. Any suggestion?
>
> I'd suggest a dot. It is already used as separator and people are already
> used to it.
We should avoid dot exactly because it's already in use. Doing that
will change the meaning of existing uses of filters like Foo.*. For
the same reason, we should avoid meta characters already used in
defining filter patterns (e.g. : and -).
--
Zhanyong
Vlad Losev
#define TEST_P_NO_FIXTURE
What I meant is to use a tuple type to represent a type list in
> /After emergency reading up on tr1::tuples/ My original design implied using
> std::pair for Cartesian product generators, but tuples are better. Typelists
> are probably the best for parametrization by type, but may also be good for
> specifying type list for parametrization by interface.
type-driven tests. I was saying that there is no need to introduce a
new way to represent a list of types.
That's why I think the generator interface is an implementation detail
> We don't have to expose or document the generator interface to the users. In
> fact, the macros described in the document are designed to explicitly hide
> it. You invoke the macro and get this opaque entity you later pass as
> template parameter to ::testing::ParametrizedTest.
and doesn't need to be described in the design doc for the readers to
understand how the concept of generator works. In any case, it's a
minor issue.
BTW, you may have noticed that I use the "parameterized" spelling with
an "e" after "t". It seems to be more popular according to Google
search. :-)
:P
Thanks,
--
Zhanyong
Matt Frantz
What about "parametric tests"?
Zhanyong Wan (λx.x x)
> #define TEST_P_NO_FIXTURE
We can, but I wouldn't until it's proven a problem. The amount of
work for writing a parameterized test dwarfs the one-liner, so this is
not the bottle neck.
>> What I meant is to use a tuple type to represent a type list in
>> type-driven tests. I was saying that there is no need to introduce a
>> new way to represent a list of types.
>
> What I meant were typelists as described in Alexandrescu's book. Under
Yes, I understood that.
> closer scrutiny, the tuple's implementation looks very much like those
> Alexandrescu's typelists. This means we'll be able to iterate over types in
> them and thus use them for type based parametrization. But I am still
> worried by the 10-parameter limitation.
If we make the implementation generic enough, which I believe we can
and should, it should work with multiple type list representations, be
it tuple or type list as in Alexandrescu's book. 10 types should be
enough for the vast majority of the cases. Only those who really want
more than 10 (probably bad style) need to deal with the much more
verbose syntax of Alexandrescu's type lists. This way we make common
things easy, while rare things still possible.
>> > We don't have to expose or document the generator interface to the
>> > users. In
>> > fact, the macros described in the document are designed to explicitly
>> > hide
>> > it. You invoke the macro and get this opaque entity you later pass as
>> > template parameter to ::testing::ParametrizedTest.
>>
>> That's why I think the generator interface is an implementation detail
>> and doesn't need to be described in the design doc for the readers to
>> understand how the concept of generator works. In any case, it's a
>> minor issue.
>
> On one hand, I'm all for encapsulation. On the other hand, the first thing
> people will start asking for will be custom generators,
We can make it public when people really ask for it.
> Dave Oliver being first among them ;)
His use cases are covered. To enumerate all possible bool flag combinations:
// This can be easily defined by the user, or we can provide it for
convenience.
Params<bool> Bool() { return Values(false, true); }
...
Combine(Bool(), Bool(), Bool())
To initialize a parameter from a varying int:
Range(0, 100)
...
MyParameter p = SomeFunction(GetParam());
--
Zhanyong
Zhanyong Wan (λx.x x)
I like the succinctness and would pick it if we were the first in the
field. However, xUnit-based frameworks like JUnit, TestNG,
Boost::test, and etc all appear to favor Parameterized Test. It's
best to be consistent with them in terminology.
--
Zhanyong
Vlad Losev
What David needs is not a Cartesian product of three bools, but a way to run tests over different flag combinations easily. With this design one has to write a fixture to set the flags:
class FooTest : public TestWithParam<tuple<bool, bool, bool> > {
public:
virtual void SetUp() {
GTEST_FLAG(A) = TestParam().get<0>();
GTEST_FLAG(B) = TestParam().get<1>();
GTEST_FLAG(C) = TestParam().get<2>();
}
};
TEST_P(ModuleA, FooTest, DoesSomething) {
...
}
...
Params<bool> Bool() { return Values(false, true); }
The flags have to be hard-coded in the fixture, and the tests are tied to the fixture. This means that it's impossible to reuse the tests for different flags. To really separate parameters from tests in this situation it will be necessary to write closures.
--
Zhanyong
Vlad Losev
Zhanyong Wan (λx.x x)
> What David needs is not a Cartesian product of three bools, but a way to run
> tests over different flag combinations easily. With this design one has to
> write a fixture to set the flags:
>
> class FooTest : public TestWithParam<tuple<bool, bool, bool> > {
> public:
> virtual void SetUp() {
> GTEST_FLAG(A) = TestParam().get<0>();
> GTEST_FLAG(B) = TestParam().get<1>();
> GTEST_FLAG(C) = TestParam().get<2>();
Just write
tie(FLAGS_A, FLAGS_B, FLAGS_C) = TestParam();
which is pretty declarative to me.
> }
> };
> TEST_P(ModuleA, FooTest, DoesSomething) {
> ...
> }
> ...
> Params<bool> Bool() { return Values(false, true); }
> APPLY_TEST(FooTest, Combine(Bool(), Bool(), Bool()));
>
> The flags have to be hard-coded in the fixture, and the tests are tied to
> the fixture.
The flags need to be listed somewhere. I think the above code looks
pretty clear. If the user wants to further reduce the boilerplate, he
can easily define a TEST_FLAGS macro in terms of Google Test's *public*
API.
> This means that it's impossible to reuse the tests for
> different flags.
First, it's possible. Just specify the test parameter as
Combine(Values(make_tuple(&FLAGS_A, &FLAGS_B, &FLAGS_C)),
Bool(), Bool(), Bool())
Second, I think it's a very rare case, and we shouldn't optimize for
it. You don't want to test how your code reacts to some *random*
collection of flags - that's aimless and wasteful. Instead, you
*know* that it interacts with certain flags, and want to enumerate
these flags. The set of flags you want to cover is determined by the
code you are testing.
Let me say again that I'm not opposed to opening the parameter
generator API. I'm just saying to do it later. We are already
undertaking a large piece of task, and have to do it in well-planned
steps. Committing to too much too early often results in regrets. :-)
Thanks,
--
Zhanyong
Zhanyong Wan (λx.x x)
> It occured to me that sequences can be combined in many ways, cartesian
> product being just one of them. There is concatenation and parallel merging,
> to name the most common.
In theory, yes. In practice, they don't occur equally frequently. I
expect Cartesian products to be the vast majority of the cases (they
correspond to the nested loops pattern).
It is good style to list all values explicitly when specifying the
test parameter, e.g. using
Values(1, 2, 5, 8, 100)
Complex parameter generator expressions make it hard to see what
values the test covers, and should be discouraged.
We could support more combinators later, but it's not urgent.
> In that light, isn't it better to rename Combine to
> avoid confusion?
I avoided Cartesian because that sounds scary to people who are not
math-savvy. I picked Combine() for the connotation of "combinations",
because what it does is to give you all possible combinations of the
fields.
I expect people to use Combine() a lot for multi-parameter cases, so
the name should be short and friendly.
Thanks,
--
Zhanyong
Zhanyong Wan (λx.x x)
> We should be careful to use or name Range consistently with C++0X's
> std::Range, which is a principle behind the "new" for loop: N2394, N2049.
Thanks for the pointers, David.
AFAIK, the concept proposal is still under review by the C++ standard
committee, so the Range concept is not finalized yet.
I agree we should make the parameter generators conform to the Range
concept, such that they can be used in for-loops and other code that
works with std::Range. We are still waiting for concept to be added
to the language, but at least we can design the API in a way that can
be easily migrated to std::Range.
--
Zhanyong
Zhanyong Wan (λx.x x)
# Since it may not be possible to evaluate a parameter generator
before main() (for example, it may rely on command-line flags that are
parsed in main(), or data structures that are initialized in main()),
it should not be evaluated until RUN_ALL_TESTS() is called.
# Parameter generators should conform to the std::Range concept
(N2394, N2049), such that when concept is added to C++ we can use
parameter generators in for-loops and use any value conforming to
std::Range as a parameter generator.
Cheers,
--
Zhanyong
Zhanyong Wan (λx.x x)
>>
>> > We should be careful to use or name Range consistently with C++0X's
>> > std::Range, which is a principle behind the "new" for loop: N2394,
>> > N2049.
>>
>> Thanks for the pointers, David.
>>
>> AFAIK, the concept proposal is still under review by the C++ standard
>> committee, so the Range concept is not finalized yet.
>
> Working group. Full wording is available, and each has been reviewed at
> least once by the CWG. However, some draughting issues remain to be resolved
> before moving into the Working Paper." (per the state of C++ Evolution
> N2597), so we are unlikely to see surprises.
We may not see any surprise at all. Still, my understanding is that
the proposed std::Range concept requires the syntax
std::Range<T>::begin(generator)
where T is the type of generator. Note that it doesn't use ADL here
and instead requires the type to be explicitly supplied. This syntax
is practical only when we have decltype() in the language, which we
cannot use yet.
Therefore, I don't think we can strictly conform to the std::Range
concept (not to mention that we are not allowed to define anything in
::std in the first place). We can make it as close as possible though
(and it's one more reason we may want to keep the generator API
internal for now). Am I missing something?
Thanks,
>> I agree we should make the parameter generators conform to the Range
>> concept, such that they can be used in for-loops and other code that
>> works with std::Range. We are still waiting for concept to be added
>> to the language, but at least we can design the API in a way that can
>> be easily migrated to std::Range.
>>
>> --
>> Zhanyong
>
>
>
> --
> Cheers!
>
> David
>
--
Zhanyong