How to disable some unittests at runtime?

[Xiaohan Wang (王消寒)]

Say I have a function:

bool IsFooSupported();

I only want to run FooTest.* when IsFooSupported() is true. There's no point to run those tests otherwise. Also, IsFooSupported() has to be a run time check. Is there a simple way to do that other than checking IsFooSupported() in each individual test?

This wasn't an issue on desktop chromium since features are enabled/disabled per platform. But on Android we enable/disable features based on Android releases, hence a runtime check.

Thanks,

Xiaohan

[Lei Zhang]

If it's not an arbitrary IsFooSupported(), but just the Android
version, you may be able to use the test runner to do the check and
--gtest_filter to filter out the tests you don't want to run.

> --
> --
> Chromium Developers mailing list: chromi...@chromium.org
> View archives, change email options, or unsubscribe:
> http://groups.google.com/a/chromium.org/group/chromium-dev

[Greg Thompson]

The Chrome Frame net tests dynamically create a gtest filter to disable tests at runtime. Perhaps you can use this as inspiration. See fake_external_tab.cc.

[Xiaohan Wang (王消寒)]

Thanks for the replies! I'll look into using FLAGS_gtest_filter to do the work.

[Paweł Hajdan, Jr.]

Why not just put something like this at the beginning of each test where you need it?

if (!IsFooSupported()) {

  LOG(INFO) << "Foo not supported, skipping test."

  return;

}

With the test launcher in base/test/launcher btw we might get a support for more explicitly skipping tests.

Paweł

[Xiaohan Wang (王消寒)]

This is what we are doing in some tests. But pretty much the whole test suite shouldn't run in this case. I feel we are wasting 4xN lines (N is the number of tests) of boilerplate code to skip the tests, not to mention the time wasted to setup/teardown the test and do nothing on the bots.

[Gabriel Charette]

Yes I very much agree that having to put it at the top of each test is very tedious and feels wrong (I've had to do this for Ash browser tests).

We should at least have a way for an entire test fixture to disable itself by calling some protected method from the fixture's SetUp() if it so desires.

[Xiaohan Wang (王消寒)]

I extended gtr@'s idea of modifying FLAGS_gtest_filter and added a TestSuite::DisableTests(const std::string& filter) function:

https://codereview.chromium.org/26883002/

It's pretty simple a change and works for me on Linux. It's not working on Android even if I see the correct filter (with disabled test). I am looking into it right now...

[Paweł Hajdan, Jr.]

For any solution other than an "if" in each test body, I think it'd be better to integrate it with code in base/test/launcher . Note that most of it is not used by default now since it's still in development.

Paweł

[Xiaohan Wang (王消寒)]

I looked at test_launcher code and I am not sure how it'll be used with TestSuite. Is there a design doc or example how it'll be used?

[Paweł Hajdan, Jr.]

Test launcher will be used to run the test suite. It's going to be the top-level component deciding which tests to run (not the test suite).

It can already be used by passing --brave-new-test-launcher to most unit test targets. Browser tests like content_browsertests and browser_tests already go through that code.

Please let me know if you have more specific questions. There is not really a design doc yet since some questions are still somewhat open (like test retry strategy).

Paweł