Singletons living across unit tests

[Satish Sampath]

I recently got to know that in unit tests singletons created by one test are not released at the end of that test. This causes state to be shared with subsequent tests and in some cases test failures.

Some tests such as src/base/debug/trace_event_win_unittest.cc use a ShadowingAtExitManager to release all singletons created by those set of tests. Have we considered adding this to the base testing::Test or other relevant base class so that singletons don't persist across tests?

--

Cheers

Satish

[Evan Martin]

I coulda sworn I fiddled with some test-running code that did this
already, but I can't find it right now.
I agree that we should scope all singletons to each single test.

> --
> Chromium Developers mailing list: chromi...@chromium.org
> View archives, change email options, or unsubscribe:
> http://groups.google.com/a/chromium.org/group/chromium-dev
>

[Satish Sampath]

I coulda sworn I fiddled with some test-running code that did this
already, but I can't find it right now.

Was it in src/base/test/test_suite.h which contains an AtExitManager ?

--

Cheers

Satish

[Paweł Hajdan, Jr.]

On Wed, Dec 8, 2010 at 22:14, Satish Sampath <sat...@chromium.org> wrote:

I recently got to know that in unit tests singletons created by one test are not released at the end of that test. This causes state to be shared with subsequent tests and in some cases test failures.

Some tests such as src/base/debug/trace_event_win_unittest.cc use a ShadowingAtExitManager to release all singletons created by those set of tests. Have we considered adding this to the base testing::Test or other relevant base class so that singletons don't persist across tests?

Yes. http://code.google.com/p/chromium/issues/detail?id=12710 describes the story.

It may be easier to run every test in a separate process. We already have infrastructure to do that (browser_tests), and it can easily be re-used for other tests, also outside of src/chrome.

Note that this is not just a question about tests, but also about how the code is designed. To run AtExit callbacks between tests, we have to change several assumptions in the non-test code, for example how LazyInstance works.

Furthermore, there are LeakySingletonTraits, that would mean Singletons surviving between tests.

[Satish Sampath]

Thanks for the link Pawel.

Note that this is not just a question about tests, but also about how the code is designed. To run AtExit callbacks between tests, we have to change several assumptions in the non-test code, for example how LazyInstance works.

Using a ShadowingAtExitManager seems to only release Singletons and LazyInstances created while running that test, and I was thinking that should be sufficient. Do you think otherwise?

 

Furthermore, there are LeakySingletonTraits, that would mean Singletons surviving between tests.

Yes using an atexit manager doesn't help here.

I agree that running tests in their own processes would take care of it all. Is that something you were looking to implement as part of that bug?

--

Cheers

Satish

[Paweł Hajdan, Jr.]

On Thu, Dec 9, 2010 at 11:25, Satish Sampath <sat...@chromium.org> wrote:

Using a ShadowingAtExitManager seems to only release Singletons and LazyInstances created while running that test, and I was thinking that should be sufficient. Do you think otherwise?

Well, many low-level things use Singletons and LazyInstances - for example MessageLoop. I think all the details are in the bug.

I agree that running tests in their own processes would take care of it all. Is that something you were looking to implement as part of that bug?

I was considering it, and I think there was a thread on chromium-dev (possibly the old one). There were performance concerns about the overhead of launching a separate process. However, there was no benchmark to either prove or disprove that those concerns are valid, so if you'd like to try and measure how separate processes impact test run time, that would be great!