Mocks Suck Reloaded

[Derek Greer]

Hey guys. First, I've found the book to be a good read. I also found
the original "Mocks Suck" thread to be extremely interesting, but it
seemed to have a low signal to noise ratio going on.

At various points, the topic touched on state-based testing verses
behavior-based testing, what kind of double is preferred when you need
behavior-based testing, the stylistic order in which behavior
assertion occurs within the test, strict-mocking vs. loose-mocking and
several other things. Of the topics discussed, the only one that
really seemed substantive was the issue of strict-mocking.

Style and nomenclature aside, Brian Swan's position seems to be
advocating against encoding the interaction between the SUT and it's
collaborators when that interaction isn't the subject of the test,
while Steve Freeman's position seems to be advocating the use of
strict-mocking as a feedback mechanism for being alerted that the
original context of the test may have changed and needs to be
revisited. From reading the book and watching Brian's presentation,
it seems both advocate using real objects when it doesn't concern the
subject of the test. Please correct me if I've misrepresented either
position or have otherwise grossly oversimplified the matter. To this
I say there are pros and cons to be gained by both approaches and I
think, like Nat's excellent points concerning the validity of viewing
things from a "Mockist TDD vs Statist TDD" perspective, I likewise see
room for both positions to be held depending on the nature of the
system you're developing.

Some of the other points seem to have more to do with style or
circumstance (i.e. the tools you're using) than a substantive
difference in approach as it pertains to the maintainability of the
system.

To split hairs a bit, Meszaros defines a Test Spy to be a test double
used for behavior verification of indirect outputs where the test, not
the double, performs assertions to verify that the expected
interaction occurred and he defines a Mock as a double which is
configured with expectations where the double, not the test, performs
assertions to verify that the expected interaction occurred.
Personally, I prefer the Context/Specification BDD style which,
similar to the AAA style, moves all observations to the end of the
test. For behavior verification, I usually use a mocking framework to
create the test double and explicitly verify that a particular
interaction occurred in an observation. Based on Meszaros definition
this is still a mock because, while my test is explicitly rather than
implicitly causing the mock to verify specific expectations were met,
it's still the mock that's doing the assert, not the test. Of course,
semantically it's still very much like a Test Spy.

I bring this up, not to be pedantic, but only to illustrate that
technically the preference one has over where the assertion comes
isn't really a distinction between mocks and test spies, but has more
to do with style and tooling.

Similarly, any hazards to using strict-mocks isn't a "Mocks Suck" or a
"Mocks Rule" issue, but a question of which kind of mocking you are
talking about and in what circumstances. When using loose mocking,
the objections often associated with older frameworks aren't as
applicable.

Anyway, just thought I'd share my perspective on the discussion for
what it's worth.


Derek Greer
http://aspiringcraftsman.com
http://twitter.com/derekgreer

[philip schwarz]

Derek,

you said:

"Steve Freeman's position seems to be advocating the use of
strict-mocking as a feedback mechanism for being alerted that the
original context of the test may have changed and needs to be
revisited. From reading the book and watching Brian's presentation,
it seems both advocate using real objects when it doesn't concern the
subject of the test."

Have you read the following sections of GOOS:

#1 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Chapter 20 - Listening to the Tests
...
Section: Too Many Expectations
...
Page: 243
...
We can make our intentions clearer by distinguishing between stubs,
simulations
of real behavior that help us get the test to pass, and expectations,
assertions we
want to make about how an object interacts with its neighbors.
...
Subsection: Write few Expectations:
A colleague, Romilly Cocking, when he first started working with us,
was surprised
by how few expectations we usually write in a unit test. Just like
“everyone” has
now learned to avoid too many assertions in a test, we try to avoid
too many
expectations. If we have more than a few, then either we’re trying to
test too large
a unit, or we’re locking down too many of the object’s interactions.

#2 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Chapter 24 Test Flexibility
...
Section: Precise Expectations
...
Subsection: Allowances and Expectations
...
Page: 277
...
jMock insists that all expectations are met during a test, but
allowances may be
matched or not. The point of the distinction is to highlight
what matters in a particular test. Expectations describe the
interactions that are
essential to the protocol we’re testing: if we send this message to
the object, we
expect to see it send this other message to this neighbor.
Allowances support the interaction we’re testing. We often use them as
stubs
to feed values into the object, to get the object into the right state
for the behavior
we want to test. We also use them to ignore other interactions that
aren’t relevant to the current test.
...
The distinction between allowances and expectations isn’t rigid, but
we’ve
found that this simple rule helps:
###############################################################
ALLOW QUERIES; EXPECT COMMANDS
COMMANDS are calls that are likely to have side effects, to change the
world outside
the target object. When we tell the auditTrail above to record a
failure, we expect
that to change the contents of some kind of log. The state of the
system will be
different if we call the method a different number of times.
QUERIES don’t change the world, so they can be called any number of
times,
including none. In our example above, it doesn’t make any difference
to the
system how many times we ask the catalog for a price.
###############################################################

The rule helps to decouple the test from the tested object. If the
implementation
changes, for example to introduce caching or use a different
algorithm, the test
is still valid. On the other hand, if we were writing a test for a
cache, we would
want to know exactly how often the query was made.

#3 On page 242-242 they give an example of a test with too many
expectations:

one(firstPart).isReady(); will(returnValue(true) ) ;
one(organizer).getAdj udicator(); will( returnValue(adjudicator) ) ;
one(adj udicator). findCase(firstParty, issue);
will(returnValue(case));
one(thirdParty).proceedWith(case);

and rewrite it as follows to highlight that the first three
expectations are stubs (i.e. they are there to get the test through to
the interesting part of the behaviour):

allowing(firstPart). isReady(); will(returnValue(true));
allowing(organizer). getAdjudicator() ;
will(returnValue(adjudicator));
allowing(adjudicator) . findCase(firstParty, issue) ;
will(returnValue(case));
one(thirdParty).proceedWith(case);

[Derek Greer]

I'm assuming by quoting sections from the book that you object to these comments in some way. Thinking back, aside from value objects I don't recall the book advocating the use of real objects at the unit test level, so I probably misrepresented his position here. Concerning the use of strict mocking, I'm not sure I recall the book discussing this explicitly, but based on the examples and his comments here I gather that Steve is an advocate of strict mocking. Was it the use of real objects part that you are objecting to?

Derek

[philip schwarz]

Hi Derek,

you said: "Thinking back, aside from value objects I don't recall the

book advocating the use of real objects at the unit test level, so I
probably misrepresented his position here."

I agree.

You also said: "Steve Freeman's position seems to be advocating the

use of strict-mocking as a feedback mechanism for being alerted that
the original context of the test may have changed and needs to be
revisited."

I don't think so.

In XUnit Test Patterns, Meszaros says (http://xunitpatterns.com/Mock
%20Object.html): Mock Objects can be either "strict" or
"lenient" (sometimes called "nice".) A "strict" Mock Object fails the
test if the calls are received in a different order than was specified
when the Mock Object was programmed. A "lenient" Mock Object tolerates
out-of-order calls.

The first time I looked at GOOS (as soon as it came out) I stopped at
the end of part III, "A Worked Example": I did not read part IV,
"Sustainable Test-Driven Development", nor the Appendix, "jMock2 Cheat
Sheet".

I then went back to Roy Osherove's "The Art of Unit Testing" (AOUT).
One of the things I like about AOUT, is that it contains the following
guideline, which I'll call "ONE MOCK PER TEST GUIDELINE":

"In a test where you test only one thing (which is how I recommend you
write tests), there should be no more than one mock object. All other
fake objects will act as stubs. Having more than one mock per test
usually means you are testing more than one thing, and this can lead
to complicated or brittle tests
...Combining stubs and mocks in the same test is a powerful technique,
but you must take care to have no more than one mock in each test. The
rest of the fake objects should be stubs that can't break your test.
Following this practice can lead to more maintainable tests that break
less often when internal code changes."

In the above, the definitions of stub, mock and fake differ from the
ones in Meszaros' "XUnit Test Patterns", and are as follows:

A STUB is a controllable replacement for an existing dependency (or
collaborator) in the system. By using a stub, you can test your code
without dealing with the dependency directly.

A MOCK OBJECT is a fake object in the system that decides whether the
unit test has passed or failed. It does so by verifying whether the
object under test interacted as expected with the fake object. There's
usually no more than one mock per test.

A FAKE is a generic term that can be used to describe either a stub or
a mock object (handwritten or otherwise), because they both look like
the real object. Whether a fake is a stub or a mock depends on how
it's used in the current test. If it is used to check an interaction
(asserted against), it is a mock object, otherwise it is a stub.

I reckon The "ONE MOCK PER TEST" guideline is important, and I value
AOUT because I think it is the first test/TDD book (that I have read)
in which I have seen it mentioned. So when I re-read AOUT and was
reminded of the guideline, I thought: GOOS is such a good book, surely
it will contain the equivalent of this guideline somewhere? So I went
back to GOOS and searched for the word 'stub'. It makes only 4
appearances, and three were in the parts I had not read: IV,
"Sustainable Test-Driven Development" and the Appendix, "jMock2 Cheat
Sheet" (the word fake does make many more appearances).

The passages containing 'stub' are the ones I reproduced in my
previous message. They seem to me to be closely related to the "ONE
MOCK PER TEST GUIDELINE" in that they exhort us to use mocks sparingly
("Write few Expectations"), and where possible, to use stubs
(allowances) to achieve "Test flexibility" by not 'locking down' too
many object interactions.

Given that GOOS advocates striving for test flexibility, I would be
surprised if it also advocated using strict mocking, except in those
cases where doing so is absolutely necessary.

In fact I had a look in GOOS and found the following:

"Invocation Order: jMock allows invocations on a mock object to be
called in any order; the expectations don’t have to be declared in the
same sequence. The less we say in the tests about the order of
interactions, the more flexibility we have with the implementation of
the code.... Only Enforce Invocation Order When It Matters...Sometimes
the order in which calls are made is significant, in which case we
addexplicit constraints to the test. Keeping such constraints to a
minimum avoids locking down the production code. It also helps us see
whether each case is necessary—ordered constraints are so uncommon
that each use stands out."

[Nat Pryce]

I think there's some confusion about terminology.

We use "stub" as a verb to mean, *allow* a message to be sent to a
collaborator, but do not *expect* it. (JMock 2 uses the term allowing
to avoid this confusion. JMock 1 used the term stub in its API).

So we suggest allowing queries, but expecting commands.

But we don't recommend limiting your tests to just one expectation and
everything else being allowed, and definitely not to just one mock and
the rest being stub objects.

Instead, expect the significant outgoing command messages that you are
testing for -- there may be more than one. Allow outgoing queries,
the results of which affect the behaviour under test. Don't constrain
order explicitly unless it is the behaviour being tested, and even
then do so as loosely as possible -- prefer state machines to
sequences, for example. And ignore any interactions that are not
germane to the behaviour being tested if you cannot avoid them
happening in some other way (in jMock you can use the ignoring clause
to ignore messages or entire objects).

Obviously there are situations where this does not hold. For example,
if testing a read-through cache, then you would care about the number
of times a query was made to pull data into the cache, so allowing the
query would not actually test the behaviour you want to test. But
most often, in my experience, it doesn't matter to the functionality
of an object how many times a query is performed.

--Nat

On 10 January 2011 23:37, philip schwarz

--
http://www.natpryce.com

[philip schwarz]

understood: thanks for the clear/comprehensive explanation

[Derek Greer]

By strict, I mean the behavior of throwing exceptions for any non-configured method invocation, not just out of order invocations. JMock provides the ignoring() and allowing() methods for explicitly denoting collaborations which aren't central to a particular specification and are used in the examples throughout the book.

Additionally, Steve stated the following in one of his responses:

"What does concern me is that all the interactions should be verified unless I turn them off explicitly."

Based on the design of JMock, how it's used in the book, and the above comment, it seems apparent Steve is an advocate of strict mocking.

That aside, the main points I was trying to make in my comments were that many of the differences expressed within the thread aren't really part and parcel to distinct paradigms of TDD, but are a result of opinions formed based upon the particular language and tooling being used and that the only real difference I saw was the view each side was taking on the subject of strict mocking. Concerning these positions, I submit that the positions aren't mutually exclusive but rather both techniques can be used depending on what's best at the time.

What's your opinion on these points?

Derek

[philip schwarz]

For what it's worth, I tried having a go at summarising how I think
the GOOS approach relates to the ONE MOCK PER TEST GUIDELINE (OMPTG).

They set up the following types of test doubles:

* Test doubles that allow the SUT to send collaborators the commands
that are not the subject of the test, and to send them queries, some
of which return indirect inputs required to set up the test
conditions:

OMPTG: one or more Meszaros stubs
GOOS: one or more jMock2-style 'mocks with allowances'

* Test double(s) that allow the SUT to send collaborators the commands
that are the subject of the test:

OMPTG: a single Meszaros mock
GOOS: one or more Meszaros mocks

On Jan 10, 11:50 pm, Nat Pryce <nat.pr...@gmail.com> wrote:

[J. B. Rainsberger]

On Tue, Jan 11, 2011 at 01:50, Nat Pryce <nat....@gmail.com> wrote:

  

Obviously there are situations where this does not hold.  For example,
if testing a read-through cache, then you would care about the number
of times a query was made to pull data into the cache, so allowing the
query would not actually test the behaviour you want to test.  But
most often, in my experience, it doesn't matter to the functionality
of an object how many times a query is performed.

I find that I often want to verify the parameters that the SUT passes to a query, so I'll set an expectation on the query for that purpose. I write this test separately from verifying how the SUT uses the result of that query.

Simplified example:

I run a coffee shop. I have registered customers. Every week, I run a report showing which customers have their birthday in the coming week. I offer them each a pound of free coffee beans if they come in to pick them up.

To design the Controller, I write these tests:

noMatchingCustomers:

    allowing customerRepository.findCustomersBornInDateRange(any) will return []

    model, view = controller.handleRequest

    assertEquals "NoFreeBeansReport", view.name

someMatchingCustomers:

    matchingCustomers = [customer1, customer2, customer3]

    allowing customerRepository.findCustomersBornInDateRange(any) will return matchingCustomers

    model, view = controller.handleRequest

    assertEquals "FreeBeansReport", view.name

    assertSame matchingCustomers, model["customers"]

Here I stub the query, per the usual advice. All good so far. Still, these don't force me to look for customers born in the next 7 days, so I set an expectation on the query checking the parameter.

askForCustomersBornInNextSevenDays:

    expect customerRepository.findCustomersBornInDateRange(January 4 upto January 10)

    controller.handleRequest(requestTimestamped(January 4))

Here, "upto" means up to and including.

So I don't /always/ stub queries, but I definitely /don't/ expect a query and an action in the same test. That's just crazy. :)

-- 

J. B. (Joe) Rainsberger :: http://www.jbrains.ca :: http://blog.thecodewhisperer.com
Diaspar Software Services :: http://www.diasparsoftware.com
Author, JUnit Recipes
2005 Gordon Pask Award for contribution to Agile practice :: Agile 2010: Learn. Practice. Explore.

[J. B. Rainsberger]

On Tue, Jan 11, 2011 at 02:55, Derek Greer <dbg...@gmail.com> wrote:

By strict, I mean the behavior of throwing exceptions for any non-configured method invocation, not just out of order invocations.  JMock provides the ignoring() and allowing() methods for explicitly denoting collaborations which aren't central to a particular specification and are used in the examples throughout the book.

 

<snip>

What's your opinion on these points?

http://bit.ly/faDpY7

[philip schwarz]

I am afraid I don't yet have enough TDD experience to have formed an
opinion. While I increase my experience, I keep looking for
information in the evidence-based opinions/findings of thought leaders
like Nat Pryce, Steve Freeman, J.B. Rainsberger, James Shore, Bob
Martin, Ron Jeffries, Jason Gorman, and many others.