Proposal/question to make mocking easy

[Sugu Sougoumarane]

I don't remember having seen this idea proposed before, yet it feels simple enough that somebody must have thought of it. So, here it goes:

Could we allow the _test.go files to override a package that's imported by the non-test files?  If this were possible, then we can do away with some unnecessary interfaces, and also avoid injecting dependencies just for the sake of testability.

In Vitess, we have some non-elegant code that tries to mock out things like mysql, especially for creating specific failure scenarios. We do use the real thing as much as possible, but there are situations where it's not practical.

I've looked at gomock, but that seems to address a different requirement.

It feels like this can be done with just tooling changes without breaking backward compatibility.

If this has already been proposed, can you point me to the thread or issue?

[Burcu Dogan]

You still can achieve the same behavior by hiding real implementations with a build constraint.

See the package at https://gist.github.com/rakyll/48332d772025b1dc12e8

Use `go test -tags=test` to test against the fake Hello. The proposal might suggest that go test should set a build constraint (such as test) automatically -- which is not backwards compatible btw.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Sugu Sougoumarane]

Thanks Rakyll :). That's a feature I wasn't aware of. It does partially achieve what I was looking for, except as you said, it would not be backward compatible if the tag was automatically defined for tests.

What I had in mind was along the lines of defining new directives like the codegen ones.

In the worst case, we could look at just using these tags for tests, but it would require us to use custom command lines, which is against the spirit of go builds.

[roger peppe]

I'm not entirely sure that this would be a good idea (and I have mixed feelings about the example that Burcu showed too) because when tests fail (too often in code I'm not familiar with) I find myself tracing through to see what's going on, and this would make it quite unclear when something might be using a mock,  and therefore which code might be responsible for the failure.

Personally I quite like the approach of explicitly defining the external things that need to be mocked as variables.

e.g.
    // somepkgFoo is defined as a variable
    // so that it can be overridden for
    // testing purposes.
    var somepkgFoo = somepkg.Foo

and overriding them in tests. (It would be nice if the compiler could tell in the production code that the variable is never assigned and treat it exactly as if it had been defined as a regular function but usually the performance impact is negligible.)

This makes it straightforward to change some specific behaviour for the duration of a test.

This doesn't get you everywhere though - it is harder when the external functions return concrete types. Sometimes I interpose a little mocking facade that defines everything in terms of interfaces, which can work ok.

If you *could* override a package for  tests, there would be some interesting questions raised:
- would it override that package for every other package too?
- if not, would the types be compatible?
- what about external test harness code that happens to use the package that's being tested (this isn't that uncommon)? There are some grey areas here actually where you "kinda" have two versions of the package being tested in the system.
- how would you actually specify the overridden package?
- would the original package still be available to the tests?

FWIW I think this is an interesting discussion - we write a great deal of test code and how do to do it best is still an open question in my mind.

  cheers,
    rog.

[Jesper Louis Andersen]

On Sun, Oct 4, 2015 at 1:43 PM, roger peppe <rogp...@gmail.com> wrote:

I'm not entirely sure that this would be a good idea (and I have mixed feelings about the example that Burcu showed too) because when tests fail (too often in code I'm not familiar with) I find myself tracing through to see what's going on, and this would make it quite unclear when something might be using a mock,  and therefore which code might be responsible for the failure.

I think the idea is right and wrong at the same time:

* Right, because it is a link-time construction which "assembles" a variant of the code with mocks rather than concrete implementations. Dynamic overriding doesn't feel like being in the Go DNA to me, whereas Burcu's solution does.

* Wrong, because as you mention, it makes it hard to assign "blame" to the piece of code in question.

There are all kinds of solutions, but a good one is language specific. Other programming languages already have solutions, but it is not a priori given that these are transplantable onto Go.

--

J.

[Egon]

On Sunday, 4 October 2015 03:39:12 UTC+3, Sugu Sougoumarane wrote:

I don't remember having seen this idea proposed before, yet it feels simple enough that somebody must have thought of it. So, here it goes:

Could we allow the _test.go files to override a package that's imported by the non-test files?  If this were possible, then we can do away with some unnecessary interfaces, and also avoid injecting dependencies just for the sake of testability.

In Vitess, we have some non-elegant code that tries to mock out things like mysql, especially for creating specific failure scenarios. We do use the real thing as much as possible, but there are situations where it's not practical.

Can you point out the exact code locations where these problems occur? It would make the discussion clearer.

[Sugu Sougoumarane]

The main principles I'm trying to withhold are:

1. Write the code in the most readable form, like it should work on production.

2. Do not distort code for the sake of testing.

3. Do not intermix testing code with non-testing code.

Unfortunately, with the way the language and tooling is currently defined, it's pretty much not possible to achieve all the above three objectives.

Rog's idea of defining vars for what you want to mock comes very close, but it does break rule#2. However, it may be the best trade off given the constraints.

AFAIK, this is an unsolved problem for any language. But I think we have this unique opportunity to discuss what are the important principles we want to withhold for testing, and come up with solutions that no other language has thought of before.

[Sugu Sougoumarane]

Could we allow the _test.go files to override a package that's imported by the non-test files?  If this were possible, then we can do away with some unnecessary interfaces, and also avoid injecting dependencies just for the sake of testability.

In Vitess, we have some non-elegant code that tries to mock out things like mysql, especially for creating specific failure scenarios. We do use the real thing as much as possible, but there are situations where it's not practical.

Can you point out the exact code locations where these problems occur? It would make the discussion clearer.

For vitess, we created this layer which did not exist before: https://github.com/youtube/vitess/tree/master/go/sqldb. Prior to this, the code just imported the https://github.com/youtube/vitess/tree/master/go/mysql package and called its functions.

We avoided dependency injection by "injecting" an Engine name into the data structure, which is used to redirect the calls at the sqldb level.

Here are some tests that use the mock: https://github.com/youtube/vitess/blob/master/go/vt/tabletserver/query_executor_test.go.

Right now, anybody that needs to mock out a low level layer (or object) has to introduce an interface that was not needed till then, and figure out a way to replace the real thing with the mock. And this should preferably done with distorting the original code.

[Sugu Sougoumarane]

Right now, anybody that needs to mock out a low level layer (or object) has to introduce an interface that was not needed till then, and figure out a way to replace the real thing with the mock. And this should preferably done with distorting the original code.

Correction: "And this should preferably be done without distorting the original code." :)

[Marvin Renich]

* Burcu Dogan <j...@golang.org> [151003 21:18]:

> You still can achieve the same behavior by hiding real implementations with
> a build constraint.
>
> See the package at https://gist.github.com/rakyll/48332d772025b1dc12e8
>
> Use `go test -tags=test` to test against the fake Hello. The proposal might
> suggest that go test should set a build constraint (such as test)
> automatically -- which is not backwards compatible btw.

Note, though, that the build tools are explicitly excluded from the Go 1
compatibility promise.

...Marvin

[Egon]

On Sunday, 4 October 2015 19:51:48 UTC+3, Sugu Sougoumarane wrote:

The main principles I'm trying to withhold are:

1. Write the code in the most readable form, like it should work on production.

2. Do not distort code for the sake of testing.

3. Do not intermix testing code with non-testing code.

Unfortunately, with the way the language and tooling is currently defined, it's pretty much not possible to achieve all the above three objectives.

It should be possible to create a separate build tool that allows replacing whole packages; without making any changes to the language.

e.g.

// test-redirect vitess/go/mysql -> vitess/go/sqldb

During the test run do that replacement.

Obviously, I would be very wary of introducing this into go tools. It sounds like a hack solution.

Testing architecture is also your code architecture. To me it doesn't make sense if they aren't interrelated.

[Egon]

After skimming the whole thing. I don't completely understand what are the reasons for mocking it out. Is it performance or introducing certain behaviors?

One thing I am considering is mocking out MySQL. Let the driver connect to a local "mock MySQL" server that implements just enough.

server := mysqlmock.New("127.0.0.1:18565")

defer server.Shutdown()

// add any mocking behavior

conn := sql.Open("127.0.0.1:18565")

...

I know MySQL is big... but I'm guessing you are not using the whole thing. You could probably bootstrap a lot of things based on https://github.com/pingcap/tidb. Maybe there are other, more light-weight implementations that could be used.

The other thing I was thinking... maybe the interface is too low. But without going through the whole thing, it would be very difficult to judge that.

Let's say you have:

A -> B -> X

C -> D -> X

You currently introduced the abstraction boundary here:

A -> B -> | X

C -> D -> | X

Maybe B, D should depend on a local interface:

A -> B | -> X

C -> D | -> X

Or, expose B and D as interfaces:

A -> | B -> X

C -> | D -> X

Hopefully, it gives some additional ideas...

+ Egon

[Sugu Sougoumarane]

I think some form of formalization is better than creating a hack that addresses one use case.

We are able to debug a program without modifying its code, or profile it, or run a race detector. Somehow, not enough effort has gone into applying the same principles for testing.

[Julian Phillips]

On 04/10/2015 17:51, Sugu Sougoumarane wrote:
> The main principles I'm trying to withhold are:
> 1. Write the code in the most readable form, like it should work on
> production.
> 2. Do not distort code for the sake of testing.

This point in particular was why I originally wrote withmock/mocktest
(https://github.com/qur/withmock). The idea being to replace libraries
with auto-generated mocks when the tests are compiled. There are some
issues (particularly with types being used in public interfaces of other
packages), but it has largely worked for the project I was working on at
the time.

>> <javascript:>>

>>> email to golang-nuts...@googlegroups.com <javascript:>.

>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>

--
Julian

[Robert Johnstone]

I'm not following point #2.  In both approaches, the mock relies and an additional level of indirection.  The main difference is that in one case the indirection occurs at compile time (through the use of build flags) and in the other case the indirection occurs at runtime.  The indirection is perhaps more visible using rog's approach, but I don't see how it makes assigning blame easier.