How to handle refactoring with TDD and mocking/stubbing

[Jonathon McKitrick]

I use TDD and mocking/stubbing (conjure) to test each layer of my code.  The problem is when I change the function signature and the tests do not break, because the mocks/stubs do not know when their argument lists no longer agree with the underlying function they are mocking.  Is there a way to catch this?  Short of a test suite that eschews stubbing in favor of full setup/teardown of DB data for each test?

[Ashton Kemerling]

I've always done the full database setup and tear down thing, but that's made sufficiently performant with datomics in memory store. Consider using transactions to isolate tests, or use Midje, which is more designed for this kind of usage. 

--Ashton

Sent from my iPhone

On Dec 31, 2014, at 9:48 AM, Jonathon McKitrick <jmcki...@gmail.com> wrote:

I use TDD and mocking/stubbing (conjure) to test each layer of my code.  The problem is when I change the function signature and the tests do not break, because the mocks/stubs do not know when their argument lists no longer agree with the underlying function they are mocking.  Is there a way to catch this?  Short of a test suite that eschews stubbing in favor of full setup/teardown of DB data for each test?

--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Timothy Baldridge]

This is one of the main reasons why I try to stay clear of heavy use of invasive mocks. I can't tell you the amount of times I've looked at code and realized that because of mocks nothing was really being tested at all. Instead, think of mocks as the terminator in a test chain. That is to say, they only exist to allow the program to continue executing. Tests should only validate the behavior up to the call to the mock'd function, no further. Otherwise you're testing your mock, and that causes all sorts of problems (as you discovered). 

I don't recommend Midje at all. Many of the framework's mocking facilities (such as providing) are abominations. You shouldn't go around mucking with functions and re-deffing them on the fly. It may look cute, but I've lost countless hours to bugs and unexpected behavior related to Midje. IMO, stay clear of that. 

The pattern I do recommend is to break your system up into components. Each component should be testable by mocking its dependencies and then testing only that component. In a final test, stitch together all your components and run system tests against all components at once. If you do what Ashton suggests and use temporary databases, this will work quite well. This way you get the detailed errors of unit tests when things fail inside a component, as well as contractual tests between components. 

I highly recommend Stuart Sierra's Component Library for all of this: https://github.com/stuartsierra/component It forces you to define the dependencies of each component, and so mocking or stubbing out a component is almost trivial. 

Timothy Baldridge

--

“One of the main causes of the fall of the Roman Empire was that–lacking zero–they had no way to indicate successful termination of their C programs.”
(Robert Firth)

[Sean Corfield]

On Dec 31, 2014, at 8:24 AM, Timothy Baldridge <tbald...@gmail.com> wrote:
> This is one of the main reasons why I try to stay clear of heavy use of invasive mocks. I can't tell you the amount of times I've looked at code and realized that because of mocks nothing was really being tested at all. Instead, think of mocks as the terminator in a test chain. That is to say, they only exist to allow the program to continue executing. Tests should only validate the behavior up to the call to the mock'd function, no further. Otherwise you're testing your mock, and that causes all sorts of problems (as you discovered).

I agree with Timothy here: we have about 6kloc test code (for a production code base of about 24kloc) and only have 25 with-redefs to stub just 8 functions in our Expectations code. In addition we use Expectations’ side-effects to mock just one function in four tests. Whenever we find ourselves needing to stub or mock in tests, we usually try to refactor the code so it’s easier to test fully without that.

As for DB tear down & rebuild: we do a full tear down & rebuild at the start of running our suite - to create a baseline of test data - and have a few tests that tear down & rebuild their own specific test data but we try to keep this to a minimum (since it slows testing down). For some of the DB-heavy sections of our application, this is inevitable since we need to ensure all tests can be run multiple times without affecting their result, but we think carefully about any tests that have to do this.

Sean Corfield -- (904) 302-SEAN
An Architect's View -- http://corfield.org/

"Perfection is the enemy of the good."
-- Gustave Flaubert, French realist novelist (1821-1880)

[Brian Marick]

> I use TDD and mocking/stubbing (conjure) to test each layer of my code.
> The problem is when I change the function signature and the tests do not
> break, because the mocks/stubs do not know when their argument lists no
> longer agree with the underlying function they are mocking. Is there a
> way to catch this? Short of a test suite that eschews stubbing in favor
> of full setup/teardown of DB data for each test?

Could you give an example? I use mocks fairly heavily, and I don't seem
to have this problem. Perhaps it's because I change the tests before the
code?

That said, I don't mock out our persistent storage tests. That is, we
have facade namespaces over Monger (Mongo), Korma (Postgres), and Redis
(Carmine). The facades' tests go directly to the database. This is
plenty fast enough, and I rarely find writing the setup code much of a
burden. Part of the reason is that we explicitly allow "partially
constructed data" (as opposed to the OO advice that all constructors
should produce objects that satisfy all the invariants). The benefits
exceed the dangers, I believe.

Note: this is consistent with the advice of some of the "mockish TDD"
old-timers: "only mock objects you own".

A fair number of tests for the code that uses the facades goes to the
database too. Whether I use mocks or not is really a matter of my own
convenience.

[Brian Marick]

Timothy Baldridge wrote:
> I don't recommend Midje at all. Many of the framework's mocking
> facilities (such as providing) are abominations.

Hacker News notwithstanding, "idiosyncratic interface" is not a synonym
for abomination.

> It may look cute,
> but I've lost countless hours to bugs and unexpected behavior related to
> Midje.

An example would be handy right about now.

As the author of Midje, I protest. Lots of people use Midje for
production code without your problems. I've been using on production
code for coming on two years now.

C'mon: all of the main Clojure testing frameworks are opinionated, each
in different ways. They're all perfectly competent at what they do. They
don't hide bugs. They all have deterministic behavior. None of them are
all that complicated.

>
> The pattern I do recommend is to break your system up into components.
> Each component should be testable by mocking its dependencies and then
> testing only that component. In a final test, stitch together all your
> components and run system tests against all components at once.

I think this is good advice.

[Timothy Baldridge]

>> Hacker News notwithstanding, "idiosyncratic interface" is not a synonym for abomination.

True, however, reaching into code, and re-deffing a function's definition is. Not only is it not thread-safe (removing the possibility of ever running two tests in parallel), but it also hides main problem. If you need to use Midje's "providing" you wrote your code wrong. If your only way to test a given bit of code is to re-deff a var, then that var should be abstracted into a different component. Because at the end of the day, I don't care if 'foo' makes a call to 'bar', all I really care about is that 'foo' saves data to 'baz'.

Stuff like with-redefs and providing muck with a developer's mental model of the source code. So instead of being able to say "well foo calls baz here, so this should work". They have to think "well foo calls baz unless someone re-deffs it, in which case I haven't a clue what's going to happen, so let me go check every test that calls foo, or calls something that calls foo to make sure it isn't overriding baz somehow". 

>> Lots of people use Midje for production code without your problems.

Not in my experience. I've used Midje in two fairly large codebases (not including mine, on these I was brought in as a consultant). And not only were the tests brittle (doing simple refactoring to the codebase would break many tests unrelated to that code, and that weren't really bugs at all), but the developers themselves mentioned more than once that they wish they had gone with a simpler testing framework. 

So perhaps that is the problem. Midje is "easy", but not "simple". New Clojure developers pick up Midje because it has all the bells and whistles, but they lack the experience to use it properly, and so they end up with unmaintainable code. And so I assert that a simpler library, something that only provides deftest, assert and run-tests forces developers to think about the best way to test something, and to write their own macros (as patterns emerge). 

But hey, don't take my word for it, run a poll, see what users of Midje think and would like to see. Include the factors I mentioned above (# of years using Clojure, # of years using Midje, do they still use it, etc). 

Timothy

--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to

clojure+unsubscribe@googlegroups.com

For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- You received this message because you are subscribed to the Google Groups "Clojure" group.

To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Brian Marick]

Timothy Baldridge wrote:
> I assert that a simpler library, something that only provides deftest,
> assert and run-tests forces developers to think about the best way to
> test something, and to write their own macros (as patterns emerge)

A not-unreasonable idea. Midje supports that by also running
clojure.test tests, making it easy to migrate when you get tired of
writing your own functions to compare expected to actual results in
different ways.

It's also reasonable to simply not use the mocking parts of Midje (or
other frameworks). Then you'll have clojure.test with a syntax you may
hate or like, plus more checker functions. The documentation since 1.5
should make that easy.

[Brian Marick]

Timothy Baldridge wrote:
> Stuff like with-redefs and providing muck with a developer's mental
> model of the source code. So instead of being able to say "well foo
> calls baz here, so this should work". They have to think "well foo calls
> baz unless someone re-deffs it, in which case I haven't a clue what's
> going to happen, so let me go check every test that calls foo, or calls
> something that calls foo to make sure it isn't overriding baz somehow".

Here I have to disagree. Technically, I think the description is wrong,
but the bit about "mental model" is more important. Midje's mocking
behavior is directly inspired by /Structure and Interpretation of
Computer Programs/' idea of "programming by wishful thinking". You write
your code assuming that whatever convenience functions you need already
exist. Then you "discover" they don't and go and write them.

Midje (and similar tools) add two things to that idea:
1. they let you run the function you're working on and see it do
something right (or wrong) before you've finished all the functions it
uses.
2. the test cases for the current function contain examples of how its
missing functions are supposed to behave.

At the end of this, your program is a lattice of functions that use each
other. You also have a collection of tests that document the assumptions
functions make about the functions they use. See
https://github.com/marick/Midje/wiki/The-idea-behind-top-down-development

It turns out to be useful when the functions so described are - and this
is fuzzy - "meaningful ideas in the domain of the program". That is,
they are "about" some part of the problem the program is supposed to
solve. When they're not - when they're just "about" the implementation
of the function that calls them, you get the sort of fragile tests I
think you're complaining about.

It shouldn't be surprising that mocks require some skill, experience,
and study to use correctly. Just like, oh, actors or logic programming
or lazy sequences, they're a simple idea that has some subtleties and
lore and non-obvious consequences.

I recommend reading /Growing Object-Oriented Software: Driven by Tests/.
The ideas are readily translated to functional languages; I actually
think they work better there.

[Colin Yates]

(Happy new year all!)

I have thousands of lines of tests written using Midje and it was the second one I turned to when I started using Clojure full-time a couple of years ago. I think it would be fairer to say that Midje is powerful enough to hang yourself, but that doesn't make that power wrong. This is the good old power/not power dilema and caution should dfefinitely be used by newbies using Midje, particularly established OO developers to ensure they don't mis-use Midje's tools as a bridge to stay in the OO paradigm.

Having said that, I don't use midje anymore simply because of the IDE support. I did encounter some minor issues (like gettings facts working in macros for example) but that was almost certainly my issue and if I stayed on the "beaten track" as it were it was a delight to use and rock solid.

I gotta say Timothy, I think you are coming across a bit strong - it might not float your boat, but you need to back up "abomination". At the moment you risk people writing off your experience as "mis-used the tools, got in a mess and then blamed the tools" which would be a shame as your "paradigm shift/mental model" point is valid.

Brian - Midje rocks - keep up the good work ;).

clojure+u...@googlegroups.com

For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- You received this message because you are subscribed to the Google Groups "Clojure" group.

To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Sam Ritchie]

Agreed, Timothy - obviously the mental model gets more tangled when state mocking comes into play, but the fact is, sometimes you don't have the option (right away) of rewriting the code you're testing.

Midje has been great for the Cascalog community:
http://www.samritchie.io/testing-cascalog-with-midje/
http://www.samritchie.io/cascalog-testing-2-0/

The state mocking is just one piece of Midje. A bunch of its other features, such as its collection checkers and chatty checkers, are excellent.

Colin Yates wrote:

I have thousands of lines of tests written using Midje and it was the second one I turned to when I started using Clojure full-time a couple of years ago. I think it would be fairer to say that Midje is powerful enough to hang yourself, but that doesn't make that power wrong. This is the good old power/not power dilema and caution should dfefinitely be used by newbies using Midje, particularly established OO developers to ensure they don't mis-use Midje's tools as a bridge to stay in the OO paradigm.

--
Sam Ritchie (@sritchie)

Paddleguru Co-Founder
703.863.8561
www.paddleguru.com
Twitter // Facebook

[Timothy Baldridge]

>>  I think you are coming across a bit strong

That's probably true. At the end of the day I believe that tests should be written in the same language with the same semantics as the code they are testing. Midje does not recommend this. It is a multi-thousand line compiler that transforms a DSL into Clojure, introducing new semantics, and syntax. Simplicity always wins in my book. I'll take more verbose tests any day over a custom compiler wrapped in a macro. 

Timothy

--

You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Akos Gyimesi]

On Sat, Jan 3, 2015, at 02:46 AM, Brian Marick wrote:
>
> > I use TDD and mocking/stubbing (conjure) to test each layer of my code.
> > The problem is when I change the function signature and the tests do not
> > break, because the mocks/stubs do not know when their argument lists no
> > longer agree with the underlying function they are mocking. Is there a
> > way to catch this? Short of a test suite that eschews stubbing in favor
> > of full setup/teardown of DB data for each test?
>
> Could you give an example? I use mocks fairly heavily, and I don't seem
> to have this problem. Perhaps it's because I change the tests before the
> code?

Although the subject changed a little bit, I would be still interested
in your approach to refactoring if there is heavy use of mocking. Let me
give you an example:

Let's say I am writing a login form, trying to use the top-down approach
you described. My approach could be the following:

(unfinished check-pw)

(fact "login-form succeeds if user enters the correct password"
(login-form-success? {:username "admin" :password "secret"}) => true
(provided
(db/get-user "admin") => (contains (:password "my-secret-hash"))
(check-pw "my-secret-hash" "secret") => true))

(defn login-form-success? [user-input]
(let [user (db/get-user (:username user-input))]
(check-pw (:password user) (:password user-input))))

Then I finish the check-pw function and everything works.

Now, later that day I decide that I pass the whole user object to the
check-pw function. Maybe I want to use the user ID as a salt, or maybe I
just want to leave the possibility for checking password expiration,
etc. So I modify the test and the implementation of check-pw so that the
first parameter is the user object, not the password hash.

Suddenly my co-worker comes to me saying "hey, I need you on a meeting
right now!" I close my laptop, and an hour later I think "where were
we?..." I run all the tests, and they all pass, so I commit.

Except... I forgot to modify all the invocations of check-pw in both the
test and the implementation. Every test pass, so I have no way of
finding out the problem without careful code review or by examining the
stack traces from the live code.

While this bug is easy to catch, what if my function is mocked in
several places, and I fail to rewrite all of them properly?

Do you have any advice on what you would have done differently here to
avoid this bug?

Regards,
Akos

[Jonathon McKitrick]

Akos,

that is exactly the kind of problem I'm talking about!  Right down to the detail about stopping work and returning to the project later, and seeing all the tests pass!

--
Jonathon McKitrick

--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---

You received this message because you are subscribed to a topic in the Google Groups "Clojure" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/clojure/T8fIW27kDYE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to clojure+u...@googlegroups.com.

[Colin Yates]

I don't think there is an easy answer here, and note that this is a problem generic to mocking (i.e. not clojure or midje specific).

The usual advice applies though:
 - do you really need to mock? Unit testing is about the coarseness of granularity which is defined more by cohesion and abstractions than "one function and only this function" (e.g. remove the problem by not overly mocking)
 - make everything fail then fix rather than fix and then upgrade (i.e. update every instance of the call/mock to check-pw before the implementation of check-pw).
 - Clojure's lack of types means the compiler can't help. Schema or core.typed can. This isn't *the* answer, but I have found it very helpful.

As mentioned elsewhere, mocking in general is a very powerful tool, but it is does need wielding carefully. These problems are easier to swallow in strongly typed languages because of IDE support (changing parameters around in Java with IntelliJ is a matter of a few key presses for example).

Hope this helps.

[Timothy Baldridge]

I think the answer to questions like this is that you are testing the wrong thing, or more correctly you are writing incomplete tests. 

In your example, you stubbed out check-pw. But calling check-pw has a contract, a contract that (at the moment) only exists in your head, but a contract none-the-less. That contract needs to be tested, from both sides. Tests should invoke all instances of check-pw. In addition you should have a test that pairs a login form with a check-pw and runs tests against this system.

Some people call these tests "integration tests" or "system tests". But I think of them as contract tests. 

Here's a diagram of the problem:

login -----> check-pw

I've found that most code that uses mocking will test the login and the check-pw bits, but completely neglect testing the "arrow" between them, and when that happens, you get exactly the experience you described. 

The other thing I'd like to mention is that I have found it very valuable to sit down and think about what code is actually being hit by a test. In your example, if check-pw and the db are both mocked, what is actually being tested? In your example all you are testing with those functions mocked is that Clojure is capable of compiling a function that calls two other functions. I can't tell you how many times I've looked at mocked tests and realized that the only thing being tested is something like read-string, get, or destructuring. 

So my personal approach is this: write very coarse tests that exercise the entire system. These will catch the protocol mis-matches. Then if you want more detail for when tests do fail, write more specific tests. In short:

System (integration) tests: so I feel good about my codebase
Unit (smaller) tests: so I can figure out what went wrong when the larger tests fail. 

Timothy

--

You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Colin Yates]

+1 - I think we are saying the same thing (not sure if you meant to
reply to me?)

> You received this message because you are subscribed to a topic in the
> Google Groups "Clojure" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/clojure/T8fIW27kDYE/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to

[Akos Gyimesi]

Thank you for all the responses! To be honest, I hoped that someone
would explain why this mocking style is a good thing, and I just
misunderstand something about the "top-down development" that the Midje
wiki suggests:
https://github.com/marick/Midje/wiki/The-idea-behind-top-down-development

While this idea sounds perfectly reasonable when you write the code for
the first time, it seems to me that it will make your life very hard
when you try to modify that code later.

I agree with Timothy that the main problem with my example is that these
tests are checking almost nothing. In essence, the tests are just a
"shadow implementation" of the real code: every function call has a
corresponding mock call in the test, so whenever you modify the
implementation (without even changing the top-level result) you have to
modify the tests as well, and vica-versa. I once encountered a codebase
that had thousands of lines of such "tests", and it was a nightmare to
modify anything in it.

The thing that I like in the Midje article though is the process how the
code is written: I usually design things top-down, but implement it
bottom-up because that's what works easily with the tools. It would be
great (at least for me) if top-down thinking could be applied during
implementation as well.

Maybe the "big trick" is to use the mocking features for unfinished code
(they are indeed convenient in Midje), but remove them immediately after
the lower-level components are implemented. In some way it is similar to
the classic TDD cycle when you first write a fake method that satisfies
your test, and then "refactor" it to have a real implementation.

BTW my current approach is to test mostly without mocks, from as
high-level as reasonably possible:
- If possible, test a module from its public interface
- If this is too much pain (e.g. the execution has too many branches and
it is hard to exercise all of them), descend one level: take the
high-level components of the module and test these parts individually.
Descend another level if it's still too painful, (recurse). Always keep
a few integration tests that exercises the components together.
- Try to find balance between not going too deep vs being able to test
effectively. I usually end up with unit-testing a few complex low-level
functions, and have much higher-level tests for the rest of the code.
- Use mocks for undeterministic functions like (rand) or (current-time)
only

Although this approach is maybe far from classic TDD, I found that it
usually results in a test suite that is easy to maintain: the public
interfaces rarely change, so you can refactor most of the implementation
details without touching the tests. It also helps you think in
"contracts", as your focus is on the interface that is used between
components.

Regards,
Akos

[Brian Marick]

Akos Gyimesi wrote:

> Now, later that day I decide that I pass the whole user object to the
> check-pw function. Maybe I want to use the user ID as a salt, or maybe I
> just want to leave the possibility for checking password expiration,
> etc. So I modify the test and the implementation of check-pw so that the
> first parameter is the user object, not the password hash.
>
> Suddenly my co-worker comes to me saying "hey, I need you on a meeting
> right now!" I close my laptop, and an hour later I think "where were
> we?..." I run all the tests, and they all pass, so I commit.

What I'll often do in this case is start the change by marking every
test that contains `check-pw` "pending" (using Midje's `future-fact`).
Thereafter, running the test suite nags you about unfinished work.

The pending tests might include some set of mock tests, but it will also
contain direct tests of `check-pw`. (I might also find that some uses of
`check-pw` are in untested functions.)

Now to coding.

I would typically change the mock tests first. In that case, after you
return from the interruption, the direct test(s) of `check-pw` will
remind you that you're not finished.

You might instead work on the direct tests. In that case, the pending
mock tests will remind you that you're not done.



Things get more complicated with more code. For example, we right now
have nine (I think) Clojure services, plus a big library ("clojure
commons") of shared code. When I have to change commons code, I'll often
start by making a separate function (maybe called something obnoxious
like check-pw-change-in-progress, maybe something different than
`check-pw` but still sensible). I gradually migrate services over to the
new version in the way described above. Eventually the original function
can be deleted.

I'm sure this strategy is covered in Fowler's book on refactoring.

It's not always that simple. Monger 2.X was a big change from Monger 1.X
(every function in the API grew an extra argument). Rather than make a
minimal change to our getset.services.mongo, I did a
function-generates-many-functions thing. That was more "big bang", and
there was one problem that only got caught on the staging server by
manual testing.



In the past, I would sometimes deliberately break the function being
changed:

(defn check-pw [])

... and run all the tests. That could tell me two things:

1. "You have an awful lot of "unit" tests that have nothing to do with
passwords that nevertheless fail when `check-pw` [never mentioned in
these tests] is changed." That's bad, if only because debugging is
easier when only one test fails and it's a test that's "close to" the
code that changed.

2. "None of your end-to-end tests test check the password form." As
other people have noted, that means there's a class of bug that won't be
caught by the test suite. On the other hand, end-to-end tests are harder
to maintain than isolated unit tests. You have a tradeoff-type decision
to make. In my case, I would almost always decide to forego the
end-to-end test in this case. Instead, I'd make sure to exercise that
form a little more thoroughly before I counted the story Done.

(More generally: the closer the code is to the "surface" of the app, the
less need for end-to-end tests. It's the down deep code that makes me
nervous. Way back in 1998, I wrote a paper explaining that thinking (in
a different context): "When Should a Test Be Automated?"
http://www.exampler.com/testing-com/writings/automate.pdf)

I seem to have drifted out of the habit of doing this. Not sure why.
Maybe it's because doing in Clojure leads to multiple tests spewing huge
swaths of stack trace garbage at me, and seeing that depresses me.


Hope this is of use.

[Brian Marick]

Akos Gyimesi wrote:
> every function call has a
> corresponding mock call in the test, so whenever you modify the
> implementation (without even changing the top-level result) you have to
> modify the tests as well, and vica-versa. I once encountered a codebase
> that had thousands of lines of such "tests", and it was a nightmare to
> modify anything in it.

That used to bother me too, until I had an Experience. It was a browser
front end written in, heaven help me, Objective-J. (Not -C. -J.
http://en.wikipedia.org/wiki/Objective-J)

Because I am lousy at GUI design, I at some point had to make a big
revision. I remember distinctly looking at a file of maybe 20 tests and
realizing I was going to have to rewrite them all. I was discouraged
until I suddenly realized that every one of those tests represented
something that had once been important about the UI. They could be seen
as a record of design choices/decisions. Some of them were relevant to
the new UI, and I ended up being glad I could look at each test and ask
myself "Self, what would be an analogous test for the new UI?"

http://atulgawande.com/book/the-checklist-manifesto/

Since then, modifying tests hasn't bothered me as much. (But note: if a
simple change means modifying a whole lot of tests, you're probably
doing something wrong.)

In fact, I sometimes point out that I've lived through the transition
from a time when almost all programmers scorned and despised writing
tests and the few who did were oddballs, to a time when writing tests is
- if not as common as NOT writing tests used to be - not at all oddball.
BUT the same programmers who love writing tests despise REwriting tests.
Except for a few oddballs. Like me. The vanguard of a new revolution!

Or perhaps not.

> Maybe the "big trick" is to use the mocking features for unfinished code
> (they are indeed convenient in Midje), but remove them immediately after
> the lower-level components are implemented.

I believe some people do that. I don't. The main reason is that doing so
will very often require spending time setting up data. My coding is very
hashmap heavy. My test for a function will pass typically pass in a very
sparse map, containing only the keys referenced by that function. If I
now replace a call to a mock with a call to the real function (which
itself calls another real function, etc.), chances are I'll have to
construct a big old map like the ones that flow through it in production.

I have as strong an aversion to maintaining setup code as other people
have to maintaining mocking tests.

> BTW my current approach is to test mostly without mocks, from as
> high-level as reasonably possible:
> - If possible, test a module from its public interface
> - If this is too much pain (e.g. the execution has too many branches and
> it is hard to exercise all of them), descend one level: take the
> high-level components of the module and test these parts individually.
> Descend another level if it's still too painful, (recurse). Always keep
> a few integration tests that exercises the components together.

Interestingly (to me, at least), that's pretty much the approach I took
in my first book, /The Craft of Software Testing/, 1994. It took some
ferocious arguments on the C2 wiki around 2000 + buckling down and
really trying XP-style unit testing for a few thousand lines of Java to
change my mind. Then it took me many more years to finally figure out
what the London mocking people were talking about. (I was actually a
reviewer on the very first paper on mock objects, and I *completely*
missed the point. I apologized to Steve Freeman for that the first time
I met him, and he replied, "That's OK - everyone did.")

I don't claim my progression is inevitable for all Right-Thinking
People. I value some things more than other people, and I worry less
about some drawbacks than other people. Thus we might make different
choices.

[Allen Rohner]

On Wednesday, December 31, 2014 8:48:27 AM UTC-6, Jonathon McKitrick wrote:

I use TDD and mocking/stubbing (conjure) to test each layer of my code.  The problem is when I change the function signature and the tests do not break, because the mocks/stubs do not know when their argument lists no longer agree with the underlying function they are mocking.  Is there a way to catch this?  Short of a test suite that eschews stubbing in favor of full setup/teardown of DB data for each test?

I'll second the avoiding mocks, and avoiding midje votes.  

As for practical advice, I noticed that many times when I had mocking/stubbing problems in my tests, it was because my app was too tightly complected or stateful. I highly highly recommend using https://github.com/stuartsierra/component . Component strongly encourages you to program in a style where it's easy to e.g. replace your DB with an in-memory instance, and make it obvious which fns need DB access, encouraging you to split fns into pure and impure. Also, use core.async to communicate between separate components, where it makes sense. 

As a design rule, I prefer making I/O fns (things that hit the DB or a REST API), 'dumb', and perform all logic/processing in fns that just receive plain data, and return plain data. 

When you follow those two practices, your need for stubbing & mocking will drop significantly. As a bonus, your tests usually get faster!

Thanks,

Allen

[Sean Corfield]

On Jan 7, 2015, at 10:21 AM, Allen Rohner <aro...@gmail.com> wrote:
> As a design rule, I prefer making I/O fns (things that hit the DB or a REST API), 'dumb', and perform all logic/processing in fns that just receive plain data, and return plain data.

I’m curious: do you have wrapper functions for the DB reads/writes (i.e., a domain layer that just wraps the calls to your persistence library as well as managing the connections etc)?

Currently we primarily run Clojure as the embedded "Model" in a legacy MVC app so we can’t really use Stuart’s Component (at least, I don’t think we can, because we have a lot of entry points into the Clojure code) but we would like to figure out a cleaner way to separate DB access from our business logic on a per request basis...

[Allen Rohner]

On Wednesday, January 7, 2015 at 2:49:50 PM UTC-6, Sean Corfield wrote:

On Jan 7, 2015, at 10:21 AM, Allen Rohner <aro...@gmail.com> wrote:
> As a design rule, I prefer making I/O fns (things that hit the DB or a REST API), 'dumb', and perform all logic/processing in fns that just receive plain data, and return plain data.

In my current project, I'm using component + datomic, so I'll talk about that here.

 

I’m curious: do you have wrapper functions for the DB reads/writes (i.e., a domain layer that just wraps the calls to your persistence library as well as managing the connections etc)?

Yes. Finding a user looks like

(defn get-user-by-login [db login]

  (d/q '[:find (pull ?u [*]) :in $ ?login :where [?u :user/login ?login]] db login))

I'm passing the datomic DB value into the find function. 

Currently we primarily run Clojure as the embedded "Model" in a legacy MVC app so we can’t really use Stuart’s Component (at least, I don’t think we can, because we have a lot of entry points into the Clojure code) but we would like to figure out a cleaner way to separate DB access from our business logic on a per request basis...

The important part of component isn't using the actual library, it's the philosophy. The most important thing is the signatures on your fns. You "just" need to change all of your code to take the DB connection as an argument, rather than doing something like:

(defn get-user-by-login [login]

  (d/q '[:find...] db login))

The only difference from the first example is that the DB is no longer an argument, so now it's implicit global state. Fixing that is 90% of the benefit of component, the rest of component is designed to make starting the system simpler. Component may or may not be a good fit for you, but you can certainly get most of the benefit on your own. 

Thanks,

Allen