When to use mocks in unit tests?
Tom Whittaker
component our software change interacts with because he insists that
we are suppose to completely isolate our unit test from other
components, which is what he calls a unit test.
We keep telling him that is not how to write unit tests, but he is the
manager, and so we've been doing it this way.
For example, I agree with the following web page's "Reasons for use"
of mocks.
http://en.wikipedia.org/wiki/Mock_object
Does anyone have the argument that supports the extreme position our
manager is taking? Or, a good book or paper that argues against this
extreme?
From what I've been seeing, it has created poorer quality software
because we are actually not finding bugs that use our real databases
and external web services until very late in the process. We just had
a feature freeze date, and we found a few major problems that day
largely because everyone was using mocks, and the mocked systems did
not behave the way our mocks did, and the higher level functional
tests created by QA missed a few things that a normal unit test would
of caught had we not been using mocks for everything.
Roy Smith
<9c9e542a-7573-4c37...@q40g2000prh.googlegroups.com>,
Tom Whittaker <tomwhit...@yahoo.com> wrote:
> I have a development manager who wants us to mock almost every single
> component our software change interacts with because he insists that
> we are suppose to completely isolate our unit test from other
> components, which is what he calls a unit test.
I agree with him so far. A unit test should test a single component in
isolation from all other components.
> Does anyone have the argument that supports the extreme position our
> manager is taking? Or, a good book or paper that argues against this
> extreme?
People who are into XP tend to be extreme :-) What he's saying is not
wrong, per-se, but it's also not the full picture.
It is useful to have true unit tests, which test components in isolation.
But, it's also essential (as you have discovered) to have tests which test
how the components interact. It sounds like your project is relying too
much on unit tests and doesn't have enough integration (system, end-to-end,
functional, whatever you want to call them) tests.
> From what I've been seeing, it has created poorer quality software
> because we are actually not finding bugs that use our real databases
> and external web services until very late in the process.
When dealing with external components like databases, it is important to
test against the real thing. If you test only against mocks, you risk your
code not working in the real world because:
* You have bugs in your mock (although a true mock should have such a
simple implementation that this is unlikely).
* Your understanding of how the external database is supposed to behave was
faulty.
* The database itself doesn't behave as it's supposed to behave (i.e. it
has bugs). Testing against the real thing is the only way you find these
problems.
So, your manager is right: real unit tests need mocks. And you're right
too: relying entirely on mocks will lead to problems. It's like the beer
commercial, "Less filling, tastes great!".
John Roth
Since you're posting in an XP group, I assume you're using Test
Driven Development, possibly full XP.
I will point out that several leading XP proponents, including Kent
Beck (who invented XP) and Ron Jeffries, do not use mocks except
for isolating external interfaces.
Whether or not to use mocks extensively is a --very-- contentious
issue, with the lines drawn rather firmly. I'm in the 'no, I don't use
mocks' camp. I feel that they tend to force too much up front design
and avoid facing the issue of continuous integration squarely.
As far as your manager is concerned, I suspect that he's confused
by the term 'test' in Test Driven Development. It means "development
using tests as a driver", not 'testing already written components.'
HTH
John Roth
Yahoo XP mailing list moderator
(NOT this list's moderator - it's unmoderated.)
John Roth
In looking at your message again, what occurs to me is that
you may not be doing sufficient testing. By that I don't mean
writing tests!
A fairly typical test setup in an XP shop might be to integrate and
run the test suite on the developers system(s) every few minutes,
trading off fidelity for speed (eg, an in-core data base), then run
the test suite on each integration (you are integrating all work at
least twice a day, right?) with an intermediate tradeoff of fidelity
for speed, and run the full test suite overnight, with full fidelity
except
where it's absolutely necessary to compromise for stability (eg,
an external e-mail server).
By full fidelity I mean a real data base configured the
same way as production. Also the test suite should go
all the way through packaging and installation on a
test system. This latter avoids problems that are only
discovered during deployment
HTH
John Roth
Ken
<snip>
> From what I've been seeing, it has created poorer quality software
> because we are actually not finding bugs that use our real databases
> and external web services until very late in the process.
<snip>
Hi. I think your problem is that you need to iterate from unit test
(with Mocks) to integration test (without Mocks) as a daily part of
your process. Sounds like you are only unit testing until late in the
game.
Note that mocks or stubs or fakes can have subtly different meanings.
Bottom line, though, is that you need them to unit test a system of
any complexity.
For unit testing, I hope you are using the xUnit framework, e.g.,
JUnit, CppUnitLite, DBUnit, etc. That enables you to automate test
execution and validation. Without that, unit testing does have nearly
as much return on investment.
Regarding books, my favorite books on unit testing are "Test-Driven
Development: A Practical Guide" and Michael Feather's "Working
Effectively with Legacy Code." I've also just started reading "xUnit
Test Patterns," and that is quite good. I suspect any of those would
have answers to your questions.
Ken
<snip>
> As far as your manager is concerned, I suspect that he's confused
> by the term 'test' in Test Driven Development. It means "development
> using tests as a driver", not 'testing already written components.'
>
I disagree with this. The notion of refactoring legacy code fits
quite nicely into TDD. The way you do that is when some new
requirement comes in, any code you have to touch that is not already
under test--gets put under test. Over time, you'll end up touching
all your code, so eventually it'll all get under test. Feather's
Working Effectively with Legacy Code is all about that.
John Roth
I may have not made myself clear. I was attempting a distinction
between "classical testing theory" and "test driven development."
What I was trying to say has nothing to do with refactoring.
BTW - Feather's book should be on every developer's shelf.
John Roth