Mocks Suck
Sageniuz
because I'm a big fanboy of the book and I've learned so much by
reading it, I was a little bit shocked as I saw Brian Swan's
presentation title "Mocks Suck".
Here's the link to the presentation:
http://www.testingtv.com/2010/12/20/mocks-suck-and-what-to-do-about-it/
I'd really recommend watching this presentation and would love to
discuss all Brian's criticism about "Mockist TDD".
Has somebody already seen the presentation? What are your thoughts?
Steve Freeman
The underlying issue, which I only really understood while writing the book, is that there are different approaches to OO. Ours comes under what Ralph Johnson calls the "mystical" school of OO, which is that it's all about messages. If you don't follow this, then much of what we talk about doesn't work well.
That said, I would also expect the balance of practices to be different in a dynamic language where we can't rely on the language to keep interfaces in sync--although the original ideas came from a(nother) bunch of Smalltalkers.
S.
Matt Wynne
My approach is, I hope, much more akin to what you'd pick up from reading GOOS. I'd say a significant part of Brian's scepticism about mocks is mostly because he hasn't ever worked with anyone who knows how to use them properly. He certainly seemed to be intrigued by the idea from watching my session that you didn't need to keep scurrying off down a rabbit warren each time you needed a collaborator class.
If you watch the videos, you'll also see we ended up with quite different designs, even for such a simple problem.
cheers,
Matt
ma...@mattwynne.net
07974 430184
Uberto Barbini
Looking at the presentation, it reminds me of a classical dialect
techniques: make your opponent ideas idiotic and then have fun to
demolish them.
It's used by politician everywhere, I hope this is not the start of a
new trend in tech conference.
I think it's completely ok dislike mocks, it's also perfectly ok to
ignore them, but if you present a session about mocks and you
misrepresent them in this way either you do on purpose or you didn't
any preparation for the session.
Said that, I agree that sometimes it's difficult (at least for me) to
choose how much mocking and where cut the layers... but you can always
refactor later.
cheers
Uberto
Brian Swan
I'd be happy to discuss why I dislike using mocks and prefer to use
stubs and spies in the cases where I want to verify that specific
messages were sent to collaborating objects.
I've been doing TDD for over 10 years and have seen numerous
situations where overusing mocks leads to brittle, hard to read test
cases. Having moved from Java to Ruby over that last couple of years
I'm seeing a lot of the problems I saw in Java code bases reappearing
in Ruby code bases. The goal of the presentation was to highlight
these problems, albeit by taking an extreme position.
I don't believe I particularly misrepresented mockist TDD in the
presentation but certainly emphasised the aspects that I dislike.
I completely agree with Steve's comment above that there are different
styles of OO (and TDD) and that if you don't follow the GOOS OO design
style much of the mock advice won't work well. What I find in practice
is that many teams don't recognise this and use mocks inappropriately.
By taking an extreme position in my presentation I hope that people
will discuss both views and as a result refine the advice on when to
use (or not use) mocks.
Regards
Brian
Uberto Barbini
I agree with pretty everything you wrote here, but I think you should
try to explain 2 things:
1. where exactly stubs and spies are better than mocks and why.
2. (more important) everything can be misused, so do you see something
wrong with the "correct" way to use mocks (let's take GOOS as
example).
> style much of the mock advice won't work well. What I find in practice
> is that many teams don't recognise this and use mocks inappropriately.
In my experience, most of teams don't use TDD at all. Most of the
misuses of mocks I saw they were on teams who wrote the test later or
in teams who were forced to use overcomplex architectures.
Said that, taking extreme positions is fine for me, but then what else
could you expect if not extreme reactions? ;)
cheers
Uberto
Steve Freeman
"Hammers suck. They strip the threads from my screws."
Yes, a lack of skill will lead to brittle code and tests. And there are plenty of broken TDD codebases that don't use mocks. And even more broken codebases that don't use TDD. No technique can survive inadequately trained developers.
I only had the slides to go on, so I don't know if the words altered the position. Personally, I'm very, very tired of "mocks are bad because X", "Yes, that's why we never do X" arguments. There's a lot of misunderstanding in the world, much of it perpetrated by Big Names, which leads to exactly the sort of mess you've been seeing. If the teams that you were working with had been Listening to the Tests, rather than just typing furiously, then they wouldn't have gone down that hole.
It might have been more productive (if less noticeable) to have done a symptoms and cures talk. To me, taking an extreme anti-mocks position is getting a bit old.
S.
Steve Freeman
> stubs and spies in the cases where I want to verify that specific
> messages were sent to collaborating objects.
personally, I don't mind that much whether interactions are specified before or after. I prefer before because that's what I'm used to and it looks a bit like a poor man's Design by Contract.
What does concern me is that all the interactions should be verified unless I turn them off explicitly. If the behaviour of the object under test changes, than I want my test to have changed too (preferably beforehand). If I need to default or ignore a significant proportion of the interactions then that's a clue that something's too big.
I would guess that there's an interesting project in the dynamic languages to write a tool that reads the expectations and cross-checks the signatures with candidate classes. In Smalltalk these could be defined as protocols. Ruby's less regular, so it would be harder to be consistent.
S.
Brian Swan
> 1. where exactly stubs and spies are better than mocks and why.
the same things you can do with mocks but without what I consider to
be the disadvantages of mocks.
I prefer that stubs are distinctly providing input and that spies are
checking indirect outputs. With mocks I find the distinction between
input and output less obvious (although I notice that JMock has added
'allowing' to add more distinction).
I also prefer that spies allow me to put asserts after the invocation
of the method being tested, I consider it to be much more readable and
again makes a clearer distinction between input and output.
> 2. (more important) everything can be misused, so do you see something
> wrong with the "correct" way to use mocks (let's take GOOS as
> example).
but I think what people generally don't appreciate is the relationship
between using mocks and the GOOS design style. What I find is that if
your design style differs, or is constrained by using a framework,
then mocking can lead to the brittle, unreadable test problems that I
have often seen.
Brian
Brian Swan
object under test changes. However the failure mode that most concerns
me with using mocks is when the behaviour doesn't change but a
refactoring of the object under test results in test failures because
the mock is effectively duplicating the implementation.
> If I need to default or ignore a significant proportion of the interactions then that's a clue that something's too big.
problem tests the options to consider also include changing the
testing style to a more stateful style.
Brian
J. B. Rainsberger
I'd be happy to discuss why I dislike using mocks and prefer to use
stubs and spies in the cases where I want to verify that specific
messages were sent to collaborating objects.
I've been doing TDD for over 10 years and have seen numerous
situations where overusing mocks leads to brittle, hard to read test
cases.
Having moved from Java to Ruby over that last couple of years
I'm seeing a lot of the problems I saw in Java code bases reappearing
in Ruby code bases. The goal of the presentation was to highlight
these problems, albeit by taking an extreme position.
I don't believe I particularly misrepresented mockist TDD in the
presentation but certainly emphasised the aspects that I dislike.
I completely agree with Steve's comment above that there are different
styles of OO (and TDD) and that if you don't follow the GOOS OO design
style much of the mock advice won't work well. What I find in practice
is that many teams don't recognise this and use mocks inappropriately.
By taking an extreme position in my presentation I hope that people
will discuss both views and as a result refine the advice on when to
use (or not use) mocks.
Diaspar Software Services :: http://www.diasparsoftware.com
Author, JUnit Recipes
2005 Gordon Pask Award for contribution to Agile practice :: Agile 2010: Learn. Practice. Explore.
Brian Swan
> Spies and mocks do the same thing, so I don't understand this statement.
> Spies only differ from mocks in when you set the expectations: spies after
> the action, mocks before.
the expectations are set to affect the readability of the test and
that asserting on a spy after the action to be more natural and
readable. In addition a mock will fail if the implementation invokes a
method on the collaborator that isn't specified in the test, whether
or not I care about checking it. With a spy I can make the choice of
what I want to assert in the test, making the test less brittle in the
face of refactoring the implementation.
>
> Overusing spies results in the same problems.
>
methods on the collaborator that I care about checking rather that
having to specify all (or explicitly ignore) the methods invoked by
the implementation on the mock.
> I have found that using verifying test doubles (spies and mocks both) as
> opposed to non-verifying test doubles (stubs) creates positive pressure on
> your design by highlighting unhealthy dependencies.
the influence on the design that comes from using mocks. I've always
had an interest in different techniques for software design and would
like to understand this aspect more. What I'm not sure about is how
much mocks influence the design versus mocks are useful when testing
code written in a certain style.
>
> As a result, I posit two central inappropriate uses of verifying test
> doubles:
>
> 1. When trying to change code with existing, deep design problems,
> 2. When modeling queries, rather than commands
>
Oriented Software Construction, his 'Advice to the advisors' about
giving 'always' or 'never' or qualified 'always' or 'never' advice.
What I'd like to find is the qualified advice on when to use mocks,
which lies somewhere between always and never.
> By taking an extreme position in my presentation I hope that people
>
> > will discuss both views and as a result refine the advice on when to
> > use (or not use) mocks.
>
> How clear have you made that intent in your presentation? I haven't read it,
> so don't interpret that as a passive aggressive question, because I
> genuinely don't know.
> --
Brian
Nat Pryce
> I like this advice. I remember something from Bertrand Meyers Object
> Oriented Software Construction, his 'Advice to the advisors' about
> giving 'always' or 'never' or qualified 'always' or 'never' advice.
>
> What I'd like to find is the qualified advice on when to use mocks,
> which lies somewhere between always and never.
That's what we set out to do in the book.
When writing the book we were very careful to avoid any absolute
statements and to describe *our experience* of using a variety of TDD
tools (including mocks), and the context in which we found the tools
useful. We deliberately did not write in absolute terms ("mocks are
great" or "mocks suck", for example), because any tool is applicable
in some context. Outside that context any tool becomes less helpful
and, far enough away from the sweet spot, a hindrance. We wanted to
let the readers understand enough about the contexts we have
experienced that they could hopefully map them to their own contexts
and make informed decisions as to which tools are appropriate for
them. Anything else is, I think, a disservice to the audience and just
continues the parlous state of debate and education the industry in
which technologies are described almost like fundamentalist religious
beliefs instead of in terms of engineering trade-offs. No technology
is bad, it's all context dependent -- even Spring might be useful
somewhere, maybe*...
--Nat
* only kidding, Spring fans!
Nat Pryce
> I prefer that stubs are distinctly providing input and that spies are
> checking indirect outputs. With mocks I find the distinction between
> input and output less obvious
That, I think, is the key difference in thinking. You're designing in
terms of inputs and outputs, mock objects (and jMock in particular)
aid designing in terms of state machines that communicate by message
passing protocols. In that model, a state transition is triggered by
an event (incoming message) and causes some observable effects
(outgoing messages). The transition rule -- trigger, effects, new
state -- is a single "chunk" of behaviour.
> (although I notice that JMock has added 'allowing' to add more distinction).
JMock has always had that feature, although it was called "stubs" in jMock 1.
--Nat
Nat Pryce
> I don't believe I particularly misrepresented mockist TDD in the
> presentation but certainly emphasised the aspects that I dislike.
Sorry to repeat myself again (and again (and again ...))
THERE IS NO SUCH THING AS MOCKIST TDD!
There are different ways of designing how a system is organised into
modules and the interfaces between those modules. In one design, you
might split the system into modules that communicate by leaving data
in shared data structures. In another, you might split the system
into modules that communicate by synchronous message-passing. In
another, modules might communicate by lazily calculated streams, or by
async messaging between concurrent actors, or by content-based pub/sub
events, or...
Depending on how you design inter-module interfaces, you'll need
different tools to unit-test modules (whether TDD or not).
Mock Objects are designed for test-driving code that is modularised
into objects that communicate by "tell, don't ask" style message
passing. In that style, there is little visible state to assert about
because state is an implementation detail -- housekeeping used to
coordinate the message-passing protocols.
In other styles of code, procedural or functional, assertions on
queried state works fine. (In asynchronous code, they do not because
of race conditions).
Even in a system that uses Mock Objects appropriately in its
unit-tests, a significant part is purely functional in my experience,
and so will not be tested with mock objects.
Talking about Mockist TDD vs Statist TDD leads people to think in
terms of adopting a "TDD style" and rigidly following a process,
rather than understanding how to select from a wide array of tools
those that are appropriate for the job at hand.
Rant over (this time!).
--Nat
--
http://www.natpryce.com
Dale Emery
I find where the expectations are set to affect the readability of the test and that asserting on a spy after the action to be more natural and readable.
In addition a mock will fail if the implementation invokes a method on the collaborator that isn't specified in the test, whether or not I care about checking it.
This is an aspect in the use of mocks that I genuinely don't "get", the influence on the design that comes from using mocks.
What I'd like to find is the qualified advice on when to use mocks, which lies somewhere between always and never.
Dale Emery
Consultant to software teams and leaders
Web: http://dhemery.com
Dale Emery
I have found that using verifying test doubles (spies and mocks both) as opposed to non verifying test doubles (stubs) creates positive pressure on your design by highlighting unhealthy dependencies.
Steve Freeman
>> If the behaviour of the object under test changes, than I want my test to have changed too (preferably beforehand).
>
> I too would expect to change the test first if the behaviour of the
> object under test changes. However the failure mode that most concerns
> me with using mocks is when the behaviour doesn't change but a
> refactoring of the object under test results in test failures because
> the mock is effectively duplicating the implementation.
Then that might a symptom of unbalanced relationships between the object and its collaborators, or that you should be using real objects in the test. One clue is if you're mocking out setters and getters, which suggests there isn't really an action in the collaboration.
>> If I need to default or ignore a significant proportion of the interactions then that's a clue that something's too big.
>
> I agree that people don't "listen to the tests" enough. In response to
> problem tests the options to consider also include changing the
> testing style to a more stateful style.
That might be an answer, or perhaps you're looking at the wrong interactions. I often find that tests that involve mocks are actually exercising a small cluster of objects: a small container and some helper and value objects. The focus should be on interactions that trigger changes in the environment.
S.
Steve Freeman
Winner of the Agile Alliance Gordon Pask award 2006
Book: http://www.growing-object-oriented-software.com
+44 (0) 797 179 4105
M3P Limited. http://www.m3p.co.uk
Registered office. 2 Church Street, Burnham, Bucks, SL1 7HZ.
Company registered in England & Wales. Number 03689627
Steve Freeman
> I prefer that stubs are distinctly providing input and that spies are
> checking indirect outputs. With mocks I find the distinction between
> input and output less obvious
That's a reasonable position. I prefer to think in terms of protocols between objects, so I like to keep the interactions together. Not a show stopper either way.
> (although I notice that JMock has added 'allowing' to add more distinction).
allowing has been there since the beginning. It was called stubbing in jMock 1, but Nat changed the terminology. I'm not sure where he took it from, but I know that it's used in Syntropy in event specifications. We've been banging on about "Stub queries, mock actions" for some years.
I think this also why Easymock has had such a pernicious effect, since it makes that distinction at the end of the line where it's hard to see.
S.
Uberto Barbini
>
> Better is obviously subjective but for me stubs and spies let me do
> the same things you can do with mocks but without what I consider to
> be the disadvantages of mocks.
still not sure what you mean with "disadvantages of mocks".
Also there are differences between (for example) mockito and JMock in
practical use.
> I also prefer that spies allow me to put asserts after the invocation
> of the method being tested, I consider it to be much more readable and
> again makes a clearer distinction between input and output.
I need to see the code here to understand what you mean.
Generally speaking I'm not very concerned by the "distinction between
input and output", I'm interested in testing the behaviour as a whole.
> I don't see anything wrong with the GOOS 'correct' way of using mocks
> but I think what people generally don't appreciate is the relationship
> between using mocks and the GOOS design style. What I find is that if
> your design style differs, or is constrained by using a framework,
> then mocking can lead to the brittle, unreadable test problems that I
> have often seen.
As Steve put it: "No technique can survive inadequately trained developers."
Anyway my target is improving my team skills to go near the GOOS
design style model.
So I want for them to learn to use mocks in the correct way, nothing less. ;)
Cheers
Uberto
Esko Luontola
> > Better is obviously subjective but for me stubs and spies let me do
> > the same things you can do with mocks but without what I consider to
> > be the disadvantages of mocks.
>
> still not sure what you mean with "disadvantages of mocks".
> Also there are differences between (for example) mockito and JMock in
> practical use.
JMock.
http://code.google.com/p/mockito/wiki/FAQ
> > I also prefer that spies allow me to put asserts after the invocation
> > of the method being tested, I consider it to be much more readable and
> > again makes a clearer distinction between input and output.
>
> I need to see the code here to understand what you mean.
> Generally speaking I'm not very concerned by the "distinction between
> input and output", I'm interested in testing the behaviour as a whole.
https://github.com/orfjackal/tdd-tetris-tutorial/blob/beyond/src/test/java/tetris/RemovingFullRowsTest.java
Uberto Barbini
> On Dec 31, 12:32 pm, Uberto Barbini <ube...@ubiland.net> wrote:
>> > Better is obviously subjective but for me stubs and spies let me do
>> > the same things you can do with mocks but without what I consider to
>> > be the disadvantages of mocks.
>>
>> still not sure what you mean with "disadvantages of mocks".
>> Also there are differences between (for example) mockito and JMock in
>> practical use.
>
> Mockito creates spies, not mocks. That is what makes it different from
> JMock.
>
> http://code.google.com/p/mockito/wiki/FAQ
technically speaking yes, but given its name, all its method names I
though it was included in the "mocks suck".
So at the end it's all about JMocks vs. Mockito? how boring...
>> I need to see the code here to understand what you mean.
>> Generally speaking I'm not very concerned by the "distinction between
>> input and output", I'm interested in testing the behaviour as a whole.
>
> Here is an example:
> https://github.com/orfjackal/tdd-tetris-tutorial/blob/beyond/src/test/java/tetris/RemovingFullRowsTest.java
I don't see the point. Why using JMocks instead of mockito in that
class would result in less brittle tests and better design?
cheers
Uberto
Brian Swan
On Dec 31, 12:58 am, Nat Pryce <nat.pr...@gmail.com> wrote:
> Outside that context any tool becomes less helpful
> and, far enough away from the sweet spot, a hindrance. We wanted to
> let the readers understand enough about the contexts we have
> experienced that they could hopefully map them to their own contexts
> and make informed decisions as to which tools are appropriate for
> them.
a much less nuanced approach to using mock objects than that presented
in the GOOS book, similar to the pattern explosion that is often seen
after reading the GoF book.
> Anything else is, I think, a disservice to the audience and just
> continues the parlous state of debate and education the industry in
> which technologies are described almost like fundamentalist religious
> beliefs instead of in terms of engineering trade-offs.
get people to notice and to have this discussion. I'd really like to
see more tools and techniques being upfront about their sweet spot/
trade offs, it's what first attracted me to both XP and Ruby on Rails.
Brian
Brian Swan
>
> That, I think, is the key difference in thinking. You're designing in
> terms of inputs and outputs, mock objects (and jMock in particular)
> aid designing in terms of state machines that communicate by message
> passing protocols. In that model, a state transition is triggered by
> an event (incoming message) and causes some observable effects
> (outgoing messages). The transition rule -- trigger, effects, new
> state -- is a single "chunk" of behaviour.
>
objects using design by contract so my tests follow a pattern of,
invoke a command and assert on queries. I'm rarely interested in
verifying the interactions between collaborators and when I need to I
find spies a better fit.
I'm not sure that the relationship between mock objects and design
style is well known. Would you say that using mock objects leads to
the design style you describe or that you favour that design style and
use mocks because they enable testing that design style?
Brian
Nat Pryce
> it's unfortunate that I had to use such a blunt instrument to
> get people to notice and to have this discussion.
You didn't. The discussion has been going on for over ten years. You
could have just joined in.
--Nat
Brian Swan
On Dec 31, 10:32 am, Uberto Barbini <ube...@ubiland.net> wrote:
>
> I need to see the code here to understand what you mean.
> Generally speaking I'm not very concerned by the "distinction between
> input and output", I'm interested in testing the behaviour as a whole.
demonstrating what I see as the disadvantages of poor readability and
brittleness.
https://gist.github.com/761102
Perhaps instead of input and output I should have said setup and
asserts. So when I read a test I like to be able to clearly see what
is setup code and what are asserts and I find that spies make that
clearer. Here is another example that I think demonstrates that.
https://gist.github.com/761110
>
> Anyway my target is improving my team skills to go near the GOOS
> design style model.
design style.
Brian
Brian Swan
On Dec 31, 2:11 pm, Uberto Barbini <ube...@ubiland.net> wrote:
>
> technically speaking yes, but given its name, all its method names I
> though it was included in the "mocks suck".
spy framework as an alternative to mocking. The terminology of the
Mockito api is unfortunate.
>
> So at the end it's all about JMocks vs. Mockito? how boring...
>
test spies as an alternative which may be more suitable depending on
your design style.
Brian
Steve Freeman
>> So at the end it's all about JMocks vs. Mockito? how boring...
>
> Not at all, it's about the misuse of mocks and making people aware of
> test spies as an alternative which may be more suitable depending on
> your design style.
I'm really struggling to see how a team that doesn't have the skills to write mocks properly will do better with spies. I can understand that they will get away with inadequate design for longer, but is that really a good thing?
S.
Uberto Barbini
>
> On Dec 31, 2:11 pm, Uberto Barbini <ube...@ubiland.net> wrote:
>>
>> technically speaking yes, but given its name, all its method names I
>> though it was included in the "mocks suck".
>
> I actually used Mocktio in the presentation as an example of a test
> spy framework as an alternative to mocking. The terminology of the
> Mockito api is unfortunate.
Is it really? I'm pretty sure that 90% of programmers cannot tell the
difference.
I saw your examples, in the first example first case I think you're
also testing RoR, that could be a good thing or a bad thing depending
on the contest.
In the second example I really don't understand why the second case is
less brittle than the first one.
>
>>
>> So at the end it's all about JMocks vs. Mockito? how boring...
>>
>
> Not at all, it's about the misuse of mocks and making people aware of
> test spies as an alternative which may be more suitable depending on
> your design style.
More suitable => yes. More solid => no.
I can provide you with lots of tests using mockito in the wrong way,
but what's the point?
cheers
Uberto
Steve Freeman
> Here is an example of a common use (abuse) of mocks from a Rails app
> demonstrating what I see as the disadvantages of poor readability and
> brittleness.
>
> https://gist.github.com/761102
This one has a singleton and it's expecting on a query, rather than stubbing, which we don't recommend. Also, I might factor out the book params to show that a value is being passed through.
> Perhaps instead of input and output I should have said setup and
> asserts. So when I read a test I like to be able to clearly see what
> is setup code and what are asserts and I find that spies make that
> clearer. Here is another example that I think demonstrates that.
>
> https://gist.github.com/761110
Here the expectation is buried in the setup, it should come afterwards. One of the advantages of the jmock initialiser hack is that it makes the protocol stand out. And the problem with the spy version is I haven't made explicit that I'm ignoring other calls rather than I've forgotten about them.
S.
Steve Freeman
> style is well known.
That's because a bunch of well-known names mischaracterised the technique from the very beginning, and we were too busy to write it up properly (and too unimportant to get heard). Dave Astels was the only author who took the trouble to understand. The damage continues to propagate, especially in the .Net world. That's why I'm so tired of these arguments.
> Would you say that using mock objects leads to
> the design style you describe or that you favour that design style and
> use mocks because they enable testing that design style?
There's a postscript in the book, copied on the mockobject.com website, that describes how we got there. In the spirit of Nat's recent rant, mocks arise naturally from doing responsibility-driven OO. All these mock/not-mock arguments are completely missing the point. If you're not writing that kind of code, people, please don't give me a hard time.
S.