[GOOS] when you stub a query, how do you become confident that the correct query is made?

[philip schwarz]

In answering a comment left by Steve on his blog post "I Like Mocks, but I Distrust Spies" [1], J.B. Rainsberger says:

"Consider a controller that queries, then acts: it asks for all the customers with pending orders, then merges that with a view template. The controllers seems responsible for (at least) two things: (1) merging a valid set of customers with the view template and (2) choosing customers with pending orders (as opposed to some other set of customers). My question: since we stub the query, how do *you* become confident that the controller asks the model for the correct set of customers without setting an expectation on `Customers.where(order_status: :pending)`? This has always bothered me."

Maybe the answer is that although Steve and Nat find the simple rule to "Allow queries, expect commands" helpful, they also say that the distinction between allowances and expectations isn’t rigid, and so maybe there is some duality at play: maybe what they mean is that a call that the object under test makes on a collaborator can be treated as an allowance in one test and as an expectation in the next one.

Maybe there is support for this hypothesis in the following excerpts from chapter 24 of GOOS:

"jMock insists that all expectations are met during a test, but allowances may be matched or not. The point of the distinction is to highlight what matters in a particular test. Expectations describe the interactions that are essential to the protocol we’re testing: if we send this message to the object, we expect to see it send this other message to this neighbor.

Allowances support the interaction we’re testing. We often use them as stubs to feed values into the object, to get the object into the right state for the behavior we want to test. We also use them to ignore other interactions that aren’t relevant to the current test. For example, in “Repurposing sniperBidding()” we have a test that includes:

ignoring(auction);

allowing(sniperListener).sniperStateChanged(with(aSniperThatIs(BIDDING)));

                                          then(sniperState.is("bidding"));

The ignoring() clause says that, in this test, we don’t care about messages sent to the auction; they will be covered in other tests. The allowing() clause matches any call to sniperStateChanged() with a Sniper that is currently bidding, but doesn’t insist that such a call happens. In this test, we use the allowance to record what the Sniper has told us about its state. The method aSniperThatIs() returns a Matcher that checks only the SniperState when given a SniperSnapshot.

In other tests we attach “action” clauses to allowances, so that the call will return a value or throw an exception. For example, we might have an allowance that stubs the catalog to return a price that will be returned for use later in the test:

allowing(catalog).getPriceForItem(item); will(returnValue(74));

...

Like all “power tools,” ignoring() should be used with care. A chain of ignored objects might suggest that the functionality ought to be pulled out into a new collaborator. As programmers, we must also make sure that ignored features are tested somewhere, and that there are higher-level tests to make sure everything works together. In practice, we usually introduce ignoring() only when writing specialized tests after the basics are in place.

[1] http://blog.thecodewhisperer.com/2014/09/15/i-like-mocks/

Philip

[Nat Pryce]

On 27 September 2014 19:12, philip schwarz <philip.joh...@googlemail.com> wrote:

My question: since we stub the query, how do *you* become confident that the controller asks the model for the correct set of customers without setting an expectation on `Customers.where(order_status: :pending)`? This has always bothered me."

Triangulation.  I don't see it as being any different from input parameters.

If I pass a value in by stubbing a data source or by passing a parameter, how do I know that the code has really used that value?  I pass in different values in different tests, so that the tests cross-check each other and the implementation.  Ideally, do this automatically, with randomly chosen values, in a property test.

--Nat

--
http://www.natpryce.com

[J. B. Rainsberger]

This brings up a new question: why triangulate when you can simply check the parameters?

--jbrains

http://www.jbrains.ca :: http://blog.thecodewhisperer.com :: http://www.agiletutor.com

 

[Colin Vipurs]

I'll do it by using a fake instead of a stub* which is usually backed by a map so only the correct value will be returned for the correct input.  I'll use a stub where I absolutely 100% do not care about the interaction but need the collaborator to return me a value and I'll have other tests checking the correct input parameters were provided.

* not always

--

---
You received this message because you are subscribed to the Google Groups "Growing Object-Oriented Software" group.
To unsubscribe from this group and stop receiving emails from it, send an email to growing-object-oriente...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Maybe she awoke to see the roommate's boyfriend swinging from the chandelier wearing a boar's head.

Something which you, I, and everyone else would call "Tuesday", of course.

[Nat Pryce]

On 4 October 2014 17:06, J. B. Rainsberger <jbrain...@gmail.com> wrote:

On Sunday, 28 September 2014 22:02:09 UTC+1, Nat wrote:

On 27 September 2014 19:12, philip schwarz <philip.joh...@googlemail.com> wrote:

My question: since we stub the query, how do *you* become confident that the controller asks the model for the correct set of customers without setting an expectation on `Customers.where(order_status: :pending)`? This has always bothered me."

Triangulation.  I don't see it as being any different from input parameters.

This brings up a new question: why triangulate when you can simply check the parameters?

Good question.  My current thoughts...

1) Checking that the object under test has called a query on a collaborator does not test what it does with the result of the query.  I'd still want to triangulate the behaviour being tested by returning different values from the query.

2) If behaviour of an object depends on data elsewhere in the system, I don't really care how it gets that data*.  I think of a command or event emission that is intended to invoke activity in the rest of the system as an outcome of the object's behaviour, while a query for data that does not affect external state is an implementation detail.

* usually ... I care if I'm writing a caching policy or the data is remote or expensive to calculate, for example

--Nat

--
http://www.natpryce.com

[Tim Wood]

On 28 September 2014 22:02, Nat Pryce <nat....@gmail.com> wrote:

If I pass a value in by stubbing a data source or by passing a parameter, how do I know that the code has really used that value?  I pass in different values in different tests, so that the tests cross-check each other and the implementation.  Ideally, do this automatically, with randomly chosen values, in a property test.

Microsoft Research released an (experimental?) tool called "PEX" which uses symbolic execution and an SMT solver to try to find values that falsify such test properties in a directed way (for .NET programs).

And it can work (or was designed to work) with their .NET mock object framework called "Moles".

So, in essence, it uses a directed (better than random) search technique to try to find behaviors of the mocks that will cause the properties checked by the tests to be false and thus the test to fail.  And it gives up after some timeout if it can't (i.e. if the test always passes).

Pex does this by examining the conditional statements in the program, including the test properties, and the dependencies between the values in the program during an execution. It starts by executing from a random input. But then it calculates how to change the input so a different branch is taken at a heuristically chosen conditional. And uses this calculation to generate a new input for the next execution. Thus guaranteeing that each new input generated will take a new path through the program. Rather than random testing, which can fail to generate values that will take a new (uncovered) path, if examples of such a value are rare. i.e. randomly generating a specific integer required to take a path can be difficult, because there are very many integers --- PEX avoids this problem.

Last year I tried a few examples with PEX and it is pretty good at generating behaviors that falsify the properties. At least for unit-testing sized examples. Even if they are hard to find by ordinary random testing --- such as bugs needing input of a specific format, like a postcode, to trigger.

You can play with PEX online here: http://www.pexforfun.com/

Pex+Moles documentation: http://research.microsoft.com/en-us/projects/pex/documentation.aspx

[J. B. Rainsberger]

On Mon, Oct 6, 2014 at 5:20 AM, Nat Pryce <nat....@gmail.com> wrote:

On 4 October 2014 17:06, J. B. Rainsberger <jbrain...@gmail.com> wrote:

On Sunday, 28 September 2014 22:02:09 UTC+1, Nat wrote:

On 27 September 2014 19:12, philip schwarz <philip.joh...@googlemail.com> wrote:

My question: since we stub the query, how do *you* become confident that the controller asks the model for the correct set of customers without setting an expectation on `Customers.where(order_status: :pending)`? This has always bothered me."

Triangulation.  I don't see it as being any different from input parameters.

This brings up a new question: why triangulate when you can simply check the parameters?

Good question. 

Whew! I thought I was missing something really fundamental.

 

My current thoughts...

1) Checking that the object under test has called a query on a collaborator does not test what it does with the result of the query.  I'd still want to triangulate the behaviour being tested by returning different values from the query.

Yes. I think of two different kinds of tests:

1. Does my Controller correctly ask for Lapsed Accounts as of TODAY (instead of some other random date)?

2. When my Controller receives 0, 1, a few, or thousands of Account objects back, does it forward to the appropriate View?

For #1, I'd expect on the query, checking for TODAY.

For #2, I'd stub the query the various ways, as usual.

2) If behaviour of an object depends on data elsewhere in the system, I don't really care how it gets that data*.  I think of a command or event emission that is intended to invoke activity in the rest of the system as an outcome of the object's behaviour, while a query for data that does not affect external state is an implementation detail.

I get this, and agree, although I feel a difference that I can't articulate between "asks the right question (of the model)" and "handles the model's various responses gracefully".

Am I the only one?

-- 

J. B. (Joe) Rainsberger :: http://www.agiletutor.com :: http://www.jbrains.ca ::
http://blog.thecodewhisperer.com
Free Your Mind to Do Great Work :: http://www.freeyourmind-dogreatwork.com

[Nat Pryce]

On 6 October 2014 15:18, J. B. Rainsberger <m...@jbrains.ca> wrote:

Yes. I think of two different kinds of tests:

1. Does my Controller correctly ask for Lapsed Accounts as of TODAY (instead of some other random date)?

2. When my Controller receives 0, 1, a few, or thousands of Account objects back, does it forward to the appropriate View?

For #1, I'd expect on the query, checking for TODAY.

For #2, I'd stub the query the various ways, as usual.

For test kind #1, the stub itself usually guarantees that if it's asked for lapsed accounts as of TODAY, it'll return the data set up for TODAY in the tests, and if asked for lapsed accounts for some other date then usually it won't have any data set up for date requested. In that case the stub will either return an empty collection of results, or (more helpfully) will fail the test by throwing an assertion error or something similar.  In that case, the stub is checking the parameter(s) of the query, but not the specific message sent to the data holder object.

If the stub *has* been set up with some data for the wrong date, then that means that the test scenario involves two dates (TODAY and the other date) and the object has a bug that makes it use the wrong one in the query.  But since I'm testing a scenario that involves two dates I'll be mindful that I want the test to detect this kind of error and will set up the test data so that using the the data for the wrong date will make the object produce a different result that when it uses the data for the correct date.

--Nat

 

--
http://www.natpryce.com

[Steve Freeman]

One of the pernicious effects of some "helpful" mocking libraries is that they silently return default values. This can mean that a test reports unexpected behaviour that isn't obvious and leads to debugging. 

S

Steve Freeman

http://www.higherorderlogic.com

Written on a phone, so please allow for typos and short content.  

--

[Daniel Wellman]

Interesting.  I've often worked with folks and would try to explain "stub queries, but expect commands" and talk about how the stub might still pass if you change the production code and your stub arguments are flexible enough.  But then folks would ask, "But what about when the stub parameters change, won't your test fail or be invalid?", and I never had an answer that I felt great about.

So if I understand you correctly, you're saying you use that you use the stub to check the parameters are passed as expected, but the message sent to the object is less important.  Is that right?

Has your thinking changed about stubbing and queries changed since you wrote about finding queries being more often interrelated than commands?  I'm thinking of an example you gave with an OrderBook a few years ago on this list: https://groups.google.com/forum/#!searchin/growing-object-oriented-software/orderbook/growing-object-oriented-software/9hb-IuVVI6o/bgn6EqByqXwJ

(As I write this I am feeling quite foolish realizing that I'm quoting something written over *four years ago*.  I promise, I happened to be searching through this list just recently on a similar topic and found this message!)

Cheers,

Dan

--

[Steve Freeeman]

I think the point is to set up stubbing with a mapping between parameters and returned values that serves the purpose of the tests. If the parameters don't match, then fail in a useful way. That's why I'm one of the few left who like jmock and its pessimism: it fails when there's no match /and it tells me what happened/.

I think of stubs as pre-conditions. Here's the environment in which the post-conditions (i.e. assertions/expectations) are true. If the pre-conditions are missing, then we can't say what will happen. If we want to consider other pre-conditions, write another test.

S

[Daniel Wellman]

Thanks, Steve, that helps me understand a bit better.  The pre- and post-conditions example is particularly clear for me.

Dan

[Steve Freeman]

Good to hear. Not everyone has that background 

S

Steve Freeman

http://www.higherorderlogic.com

Written on a phone, so please allow for typos and short content.  

[Rick Mugridge]

I also think that jmock is great.