proposal: disabling a test dynamically

2,297 views
Skip to first unread message

Zhanyong Wan (λx.x x)

unread,
Jul 17, 2008, 5:00:24 PM7/17/08
to Google C++ Testing Framework, Marc-Antoine Ruel
Hi,

In gtest you can mark a test as disabled by changing its name to
DISABLED_*. The test will still be compiled (so the code won't rot),
but will be skipped when you run the test binary.

Sometimes you may want to decide whether to disable a test at
run-time. For example, you may want to disable a test if it runs on a
machine with some incompatible configurations.

You could return early in a test if some necessary resources aren't
available. But that can be misleading as the test will be reported as
successful.

Therefore, Marc-Antoine proposed to allow a user to tell gtest that a
test should be disabled at runtime. Since gtest reports the number of
disabled tests, you won't mistakenly think they have passed. He's
also kindly provided a patch.

I think this is a useful feature, and have uploaded the patch to
http://codereview.appspot.com/2588 to be reviewed.

I have a different design in mind though. In the current design, the
user defines a *static* IsTestCaseDisabled() function to tell gtest
that whether an entire test case should be disabled. It would be more
useful to allow the decision to be made for each test individually.
(We don't want to force the user to turn on/off all tests in the same
test case at the same time.)

Also, such a decision may depend on information specific to each test,
so the user should be able to make it inside a test function (as
opposed to in SetUp(), for example).

I propose to add a public static function void
Test::DisableThisTest(), which can be called anytime during a test,
and will tell gtest that the *current* test is disabled and failures
in it should be ignored. Example:

TEST(FooTest, DoesBar) {
... figure out pre-conditions ...
if (conditions_not_met) {
DisableThisTest();
return;
}
... some assertions ...
}

Thoughts? Thanks,

--
Zhanyong

Zhanyong Wan (λx.x x)

unread,
Jul 17, 2008, 6:04:39 PM7/17/08
to Vlad Losev, Google C++ Testing Framework, Marc-Antoine Ruel
On Thu, Jul 17, 2008 at 2:59 PM, Vlad Losev <vl...@google.com> wrote:
> What would be semantics for a case when a user puts some assertions before
> the condition check? Will this test be considered disabled or failed?

I would say disabled. Then you don't have to worry about having to
call DisableThisTest() absolutely before anything could fail. After
all, to disable a test means that you are not interested in the result
of the test (although you are still interested in knowing that the
test is indeed disabled, such that you can revisit it later).

Thanks,

>> _______________________________________________
>> opensource-gtest mailing list
>> opensour...@google.com
>> https://mailman.corp.google.com/mailman/listinfo/opensource-gtest
>
>

--
Zhanyong

Zhanyong Wan (λx.x x)

unread,
Jul 17, 2008, 7:01:17 PM7/17/08
to Wink Saville, Google C++ Testing Framework, Marc-Antoine Ruel
2008/7/17 Wink Saville <wi...@google.com>:
> What about disabling in SetUp so time isn't wasted in SetUp/TearDown?

Sure, you can do that if you want. I would allow DisableThisTest() to
be called *anytime* in a test, including SetUp()/TearDown().

Note that you cannot always do that, as the decision may depend on
information that's only available in the test body.

> On Thu, Jul 17, 2008 at 2:00 PM, Zhanyong Wan (λx.x x) <w...@google.com>
> wrote:
>>

>> _______________________________________________
>> gunit-users mailing list
>> gunit...@google.com
>> https://mailman.corp.google.com/mailman/listinfo/gunit-users
>
>

--
Zhanyong

Zhanyong Wan (λx.x x)

unread,
Jul 17, 2008, 7:14:26 PM7/17/08
to Wink Saville, Google C++ Testing Framework, Marc-Antoine Ruel
2008/7/17 Wink Saville <wi...@google.com>:
> One thing that would be needed in SetUp would be which test is
> going to be executed, is that information available when SetUp is
> called? If so should it be passed as a parameter? Or maybe this

You can tell the name of the current test in SetUp() by calling a
function. See the Google Test advanced wiki.

> is related to the ParameterizedSetUp question I just saw?

Not directly, but the user should be able to use the idx information
in ParameterizedSetUp() to decide whether to disable the test, if we
go that route.

> -- Wink
>
> On Thu, Jul 17, 2008 at 4:01 PM, Zhanyong Wan (λx.x x) <w...@google.com>

--
Zhanyong

Marc-Antoine Ruel

unread,
Jul 17, 2008, 7:39:29 PM7/17/08
to Zhanyong Wan (λx.x x), Wink Saville, Google C++ Testing Framework
Let me explain the rationale why I'm using a static function in my patch.

My requirement where:

- For a disabled TestCase, none of its code is actually run. (Except
IsTestCaseDisabled)
- For a disabled TestCase, the TestCase is not instantiated,
especially its member variables.
- Disabling a TestCase is independent of the actual run, it should be
idempotent and invariant in time. Running the same executable on the
same computer two times in a row should result in the same TestCase
being disabled. This is obviously not enforced.
- Running the test executable will display the number of disabled
tests, invariant of the number of tests run (i.e. --gtest_filter)

This led me to using a static function. This has the advantage that
the situation that Vlad explains can't happen since nothing in the
test had run. If it happens, the IsTestCaseDisabled is too complex.

The reason I did this was explicitly for specific components that are
known to be unsupported on some OS version, so the
IsTestCaseDisabled() is really straight forward. Furthermore, the
current way it is designed permits to use a class template to abstract
this issue in multiple TestCase.

M-A

2008/7/17 Zhanyong Wan (λx.x x) <w...@google.com>:

Zhanyong Wan (λx.x x)

unread,
Jul 18, 2008, 7:27:24 PM7/18/08
to Marc-Antoine Ruel, Wink Saville, Google C++ Testing Framework
Thanks for the explanation, M-A. Please see my comments below.

2008/7/17 Marc-Antoine Ruel <mar...@gmail.com>:


> Let me explain the rationale why I'm using a static function in my patch.
>
> My requirement where:
>
> - For a disabled TestCase, none of its code is actually run. (Except
> IsTestCaseDisabled)

With my proposal, you can achieve this by calling DisableThisTest() at
the beginning of SetUp() or your test fixture's ctor.

> - For a disabled TestCase, the TestCase is not instantiated,
> especially its member variables.

If this is important, you could set up the data *after* deciding
whether to call DisableThisTest().

> - Disabling a TestCase is independent of the actual run, it should be
> idempotent and invariant in time. Running the same executable on the
> same computer two times in a row should result in the same TestCase
> being disabled. This is obviously not enforced.

You get the same degree of guarantee for repeatability if you call
DisableThisTest() at the beginning of SetUp() or the ctor.

> - Running the test executable will display the number of disabled
> tests, invariant of the number of tests run (i.e. --gtest_filter)

I'm not sure if the user should be told there's a disabled test Foo
when he has already explicitly said he's not interested in running
Foo. If he wants to know if Foo is disabled, he can always run the
binary without --gtest_filter, or he can run it with
--gtest_filter=Foo.

> This led me to using a static function. This has the advantage that
> the situation that Vlad explains can't happen since nothing in the
> test had run. If it happens, the IsTestCaseDisabled is too complex.

I don't see the situation Vlad described as a problem. The semantics
is clearly (and IMHO reasonably) defined in that case: a disabled test
doesn't affect the test binary's result, even if it has assertion
failures.

> The reason I did this was explicitly for specific components that are
> known to be unsupported on some OS version, so the
> IsTestCaseDisabled() is really straight forward. Furthermore, the
> current way it is designed permits to use a class template to abstract
> this issue in multiple TestCase.

You should be able to do the same using DisableThisTest(). Just do
the check at the beginning of the class template's ctor.

I can understand that the static function approach works well for your
requirements, and it is straightforward when all tests in the same
test case must be enabled or disabled together. I'm just saying that
as a general solution it is not as flexible as the DisableThisTest()
approach.

In general, I like designs that don't impose artificial restrictions.
Having to disable all tests in the test case together and not being
able to use test-specific information in the decision seem artificial
restrictions to me. Thanks,

--
Zhanyong

Zhanyong Wan (λx.x x)

unread,
Jul 18, 2008, 7:29:16 PM7/18/08
to Wink Saville, Google C++ Testing Framework, Marc-Antoine Ruel
2008/7/17 Wink Saville <wi...@google.com>:
>
>
> On Thu, Jul 17, 2008 at 4:14 PM, Zhanyong Wan (λx.x x) <w...@google.com>

> wrote:
>>
>> 2008/7/17 Wink Saville <wi...@google.com>:
>> > One thing that would be needed in SetUp would be which test is
>> > going to be executed, is that information available when SetUp is
>> > called? If so should it be passed as a parameter? Or maybe this
>>
>> You can tell the name of the current test in SetUp() by calling a
>> function. See the Google Test advanced wiki.
>
> I wonder if there was a SetUpEx which the user could override and
> might be implemented as:
>
> boolean SetUpEx(const char *test_case_name)
> {
> SetUp();
> return true;
> }
>
> If this routine returned true this test continues if false the
> test is disabled.

You can achieve the same effect by calling DisableThisTest() in
SetUp(). gtest will skip the test body if the test was already
disabled in the ctor or SetUp().
>
> Just a thought.


>
>
>>
>>
>> > is related to the ParameterizedSetUp question I just saw?
>>
>> Not directly, but the user should be able to use the idx information
>> in ParameterizedSetUp() to decide whether to disable the test, if we
>> go that route.
>

> Agreed.

--
Zhanyong

Marc-Antoine Ruel

unread,
Jul 18, 2008, 7:38:54 PM7/18/08
to Zhanyong Wan (λx.x x), Wink Saville, Google C++ Testing Framework
Ok fine, I'll look at cooking something up since I'd need this
functionality ASAP.

2008/7/18 Zhanyong Wan (λx.x x) <w...@google.com>:

Artur Czechowski

unread,
May 14, 2015, 9:38:19 AM5/14/15
to googletes...@googlegroups.com, wi...@google.com, w...@google.com
Hi guys,
Any chance this functionality would appear in googletest soon? I'd be glad to use it too.

Regards
    Artur

ma...@chromium.org

unread,
Jan 22, 2019, 4:30:43 PM1/22/19
to Google C++ Testing Framework
Closing the loop on this after ten years…

Reply all
Reply to author
Forward
0 new messages