How to warn that production code is (wrongly) used in unit testing?

[Christos]

Is there a way to guarantee that a location in the code can be reached in production but not when running tests? I want to remind people, who might use my code, to supply a mock, since not using a mock will make their tests slow and flaky.

[Václav Brožek]

It should be possible to test every piece of the code as if it were run in production, so the ability to distinguish between tests and production at run time sounds, at least philosophically, undesirable.

If people forget to use the mock and the tests will become slow and flaky, it will ultimately lead to bug reports about those tests and a good chance someone will fix that. If, instead, we made our code detect whether it is being tested, it might evolve non-testable bugs, which might take long to get discovered (if at all) and cause confusion.

So I would not worry about the reminder to use the mock.

Cheers,
Vaclav

On Fri, 20 May 2016 at 16:00 Christos <cfrou...@chromium.org> wrote:

Is there a way to guarantee that a location in the code can be reached in production but not when running tests? I want to remind people, who might use my code, to supply a mock, since not using a mock will make their tests slow and flaky.

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev

[Matthew Menke]

Actually, it should not be possible to run every piece of code as if it were run in production:  getaddrinfo, for example, depends on the configured system DNS server.  I agree this isn't worth doing, but if any tests are depending on the result of a real DNS lookup, it's a bug.

[Václav Brožek]

Fair point, Matthew. And I guess what I wrote might have been confusing. What I meant is that the production code should not care about whether is called from within a test. Of course the test code knows it is a test and should do appropriate mocking in the test set-up.

The case Christos is working on is more isolated than dependencies on the Internet: It is about mocking the backends for OSCrypt, so, e.g., on Mac, the tests would use a mocked Keychain instead of the real one. I agree that other than tests for the backend itself, the tests do benefit from using a mock. And Christos identified and fixed a couple of existing tests in his last CL which forgot to mock the backend. So I also see his point that preventing such omissions to set up the mock would make the tests faster and more reliable. I just don't think there is a good way to do this from the production code at run time.

Cheers,

Vaclav

[Erik Staab]

Would you mind pointing to the code in question or an example CL where you cleaned up a test to use a mock?

I'm not sure if a mechanism currently exists but since slow and flaky tests are an ongoing problem I think this is worth pursuing.

Thanks,

Erik

On Fri, May 20, 2016 at 7:00 AM, Christos <cfrou...@chromium.org> wrote:

Is there a way to guarantee that a location in the code can be reached in production but not when running tests? I want to remind people, who might use my code, to supply a mock, since not using a mock will make their tests slow and flaky.

--

[Václav Brožek]

I believe https://codereview.chromium.org/2010463002/diff/220001/chrome/browser/browsing_data/browsing_data_remover_unittest.cc is an example of a test where Christos replaced the real backend with the mocked one.

Cheers,

Vaclav