Partially ordering expectations
JJ
Hi all,
First off, I'd just like to say that Google Mock is great. It's made
my testing
life much easier. Thanks!
I'd like to talk about the test sequencing features. As explained in
the Cookbook,
it's possible to place expectations into an arbitrary directed acyclic
graph in
their sequencing, in theory. However, I'm finding this sometimes
extremely
cumbersome, and occasionally impossible, in practice. Let me give some
examples:
Suppose that I have a simple mock class
class MockA: public AInterface
{
MOCK_METHOD0(a, void());
MOCK_METHOD0(b, void());
MOCK_METHOD0(c, void());
MOCK_METHOD0(final, void());
};
and I want to expect that all of a(), b() and c() are called in
unspecified order,
and then final() is called after all three of them, my understanding
is that my
only recourse is to specify the following:
MockA myMock;
Sequence s, t, u;
EXPECT_CALL(myMock, a()).InSequence(s);
EXPECT_CALL(myMock, b()).InSequence(t);
EXPECT_CALL(myMock, c()).InSequence(u);
EXPECT_CALL(myMock, final()).InSequence(s, t, u);
This is starting to get a bit unwieldy, and would only get more so as
the number
of expectations increased. More telling, though, is that this setup
requires
knowledge of exactly how many expectations are going to be in the
initial group before
the expectation for final() can be created, so that the correct number
of sequences
can be created. Unfortunately, I don't think this is always the case:
consider the
situation in which the group of expectations is being created by a
polymorphic or
templated function, such as the following:
struct Tester
{
void setAllExpectations
{
MockA myMock;
//How many sequences would I create here? We don't know how
many
//expectations the virtual method is going to create, and we
can't even
//*call* it without that information, because we'd have to
pass all the
//sequences into it to replicate the previous code.
setABCExpectations(myMock);
EXPECT_CALL(myMock, final());
}
protected:
virtual void setABCExpectations(MockA& myMock) = 0;
};
Now, unless I've just missed a much more natural way to do what I want
to do there,
it's now impossible to create the desired ordering.
Has anyone else run into this situation? Is it worth adding
functionality to
enhance the ordering possibilities for expectations?
Thanks for your time,
JJ
One possible natural syntax for such an enhancement follows, provided
for the
purposes of discussion -- I don't claim to be sufficiently conversant
with the
internal code of Google Mock to know how difficult it would be to make
this
happen, though.
MockA myMock;
Sequence s;
{
Group g(s);
//alternatively an object of type InGroup could obviate the need
for the
//InGroup clauses on the expectations, in the same way as
InSequence objects.
EXPECT_CALL(myMock, a()).InGroup(g);
EXPECT_CALL(myMock, b()).InGroup(g);
EXPECT_CALL(myMock, c()).InGroup(g);
}
EXPECT_CALL(myMock, final()).InSequence(s);
Semantically, the above Group object would have to create some kind of
temporary
collection of Expectations that could be inserted in the sequence s
instead of
an individual expectation. The group would naturally be complete when
all expectations
within it were complete, and therefore only then could the call to
final() succeed.
Chris Pickel
One alternative would be to forward the first three calls to a
different expectation:
EXPECT_CALL(myMock, a())
.WillOnce(Invoke(&check, &CheckPoint::Check));
EXPECT_CALL(myMock, b())
.WillOnce(Invoke(&check, &CheckPoint::Check));
EXPECT_CALL(myMock, c())
e.WillOnce(Invoke(&check, &CheckPoint::Check));
{
InSequence s;
EXPECT_CALL(check, Check()).Times(3);
EXPECT_CALL(myMock, final());
}
As you've defined "setABCExpectations()" it's probably impossible, as
you haven't provided a channel for more information to go down or up.
But, if you change it to:
virtual int setABCExpectations(MockA& myMock, CheckPoint& check);
then you should have no trouble using the pattern above. You could
replace CheckPoint with a class that would do the bookkeeping itself,
and save you from having to count the number of calls and return it.
Zhanyong Wan (λx.x x)
On Thu, Jun 4, 2009 at 3:18 AM, JJ <jo...@autonomy.com> wrote:
>
> Hi all,
>
> First off, I'd just like to say that Google Mock is great. It's made
> my testing
> life much easier. Thanks!
Thanks for the feedback!
I agree that your use case is not well supported right now (you can
use the trick Chris suggested, but it doesn't scale well when your
expectations are more complex. A more natural encoding of your intent
will make the test much easier to read.).
Our philosophy is to keep Google Mock conceptually as simple as
possible. Therefore we tend to wait for actual needs to come up
before we implement something. That way we can be more sure that our
design does solve the user's problem, and that we don't bloat Google
Mock with "features" that don't match the reality.
Something like this might be made to work. However, it's unclear to
me what the relationship between groups and sequences should be. The
"Group g(s);" syntax suggests that there's some connection between g
and s, but what is it? I can understand it in your specific example,
but what about the general case? How do you associate one group with
multiple sequences? And how do you associate multiple groups with one
sequence? What's the semantics of these combinations? Is the
semantics sound? In the end, it can get muddy quickly.
As you stated, the problem you are having is that the InSequence()
syntax requires you to know the number of sequences when you *write*
the code, which is not always possible. We can fix this by allowing
InSequence() (or use a different name like InSequenceSet()) to accept a
container of sequences. Then the above program can be written as:
MockA myMock;
SequenceSet ss;
EXPECT_CALL(myMock, a()).InSequence(ss.AddNew());
EXPECT_CALL(myMock, b()).InSequence(ss.AddNew());
EXPECT_CALL(myMock, c()).InSequence(ss.AddNew());
EXPECT_CALL(myMock, final()).InSequenceSet(ss);
Notes:
- "Set" emphasizes that the order of the elements is unimportant.
- ss.AddNew() creates a new Sequence, adds it to ss, and returns the
Sequence.
- An expectation can accepts multiple InSequence()s and/or multiple
InSequenceSet()s.
It's easy to see that the semantics of InSequenceSet(ss) is sound, as
it can be viewed as a syntactic sugar of InSequence(s1, ..., sn) where
s1, ..., and sn are the members of ss. Being a syntactic sugar also
means a user can learn this construct easily.
Thoughts?
--
Zhanyong
Zhanyong Wan (λx.x x)
One more note:
- We provide SequenceSet as a convenience. However, the user is not
restricted to it. You can pass any STL-style containers to
InSequenceSet().
>
> It's easy to see that the semantics of InSequenceSet(ss) is sound, as
> it can be viewed as a syntactic sugar of InSequence(s1, ..., sn) where
> s1, ..., and sn are the members of ss. Being a syntactic sugar also
> means a user can learn this construct easily.
>
> Thoughts?
>
> --
> Zhanyong
>
--
Zhanyong
Zhanyong Wan (λx.x x)
JJ
better:
the real version of this code is part of a testing library itself, and
so
I want use of it to be as simple as possible.
simply by
construction from multiple sequences, in the same way as the
InSequence clauses
on EXPECT_CALL currently work. Multiple groups associated with a
sequence doesn't
seem like a problem to me - a group isn't really taking a sequence,
but the next
position within that sequence. Therefore, constructions like the
following should
just work, defining (a, b)->(middle)->(c, d)->(final):
Sequence s;
Group g(s);
EXPECT_CALL(myMock, a()).InGroup(g);
EXPECT_CALL(myMock, b()).InGroup(g);
EXPECT_CALL(myMock, middle()).InSequence(s);
Group h(s);
EXPECT_CALL(myMock, c()).InGroup(h);
EXPECT_CALL(myMock, d()).InGroup(h);
EXPECT_CALL(myMock, final()).InSequence(s);
Semantically, I see Group as an application of the Composite pattern
to Expectation.
Implementation of most of the features of Expectations as the obvious
pass-throughs to each Expectation in the Group -- so it would respond
to
satisfaction queries by checking whether each of its expectations was
satisfied,
have prerequisites added to it by adding them to each of its
expectations,
and so on.
Construction of a Group g in a Sequence s with members e1, e2 (so of
the form
(predecessor)->g(e1, e2)->(successor)) could then still be viewed as
constructing
little sequences s1 (predecessor->e1->successor) and s2 (predecessor-
>e2->successor),
although I imagine that would probably not be the implementation.
>
> As you stated, the problem you are having is that the InSequence()
> syntax requires you to know the number of sequences when you *write*
> the code, which is not always possible. We can fix this by allowing
> InSequence() (or use a different name like InSequenceSet()) to accept a
> container of sequences. Then the above program can be written as:
>
> MockA myMock;
> SequenceSet ss;
> EXPECT_CALL(myMock, a()).InSequence(ss.AddNew());
> EXPECT_CALL(myMock, b()).InSequence(ss.AddNew());
> EXPECT_CALL(myMock, c()).InSequence(ss.AddNew());
>
> EXPECT_CALL(myMock, final()).InSequenceSet(ss);
>
> Notes:
>
> - "Set" emphasizes that the order of the elements is unimportant.
> - ss.AddNew() creates a new Sequence, adds it to ss, and returns the
> Sequence.
> - An expectation can accepts multiple InSequence()s and/or multiple
> InSequenceSet()s.
>
> It's easy to see that the semantics of InSequenceSet(ss) is sound, as
> it can be viewed as a syntactic sugar of InSequence(s1, ..., sn) where
> s1, ..., and sn are the members of ss. Being a syntactic sugar also
> means a user can learn this construct easily.
>
> Thoughts?
clearly
theoretically sufficient to impose arbitrary orderings on
expectations, any
additional features added must be equivalent to some Sequence-based
form. It's
therefore good if it's clear exactly what that equivalence is, and
this certainly
has that.
I'd be hoping to remove the syntactic requirement for the a, b and c
expectations to
specify any sequencing at all, but perhaps an implicit InSequenceSet
in the same
style as InSequence could fix that (I really like InSequence. It's
proven very useful to me).
However, I think I can still break this particular suggestion by
making a fairly small
modification to my example:
If we add another function to MyMock called initial() that is expected
to be called before
a, b, and c in the same way as final() should be called after them, I
don't see how that fits
into this SequenceSet construct. The SequenceSet does not yet contain
the Sequences that the
expectations for a, b, and c will create, and as before in principle
we don't know how many of
them there will actually be, so we can't construct them at expectation
time for initial().
While semantically I think it is still clear what the following should
do, I no longer see how it can be
broken down into combinations of existing Sequence types.
MockA myMock;
SequenceSet ss;
EXPECT_CALL(myMock, initial()).InSequenceSet(ss);
EXPECT_CALL(myMock, a()).InSequence(ss.AddNew());
EXPECT_CALL(myMock, b()).InSequence(ss.AddNew());
EXPECT_CALL(myMock, c()).InSequence(ss.AddNew());
EXPECT_CALL(myMock, final()).InSequenceSet(ss);
(I'm now departing rather more from my present use case, though, so
your
point above about not implementing things before they're needed may be
important).
What if instead of final(), there were again multiple expectations,
with the only
sequencing requirements being that every expectation in the first
group was sequenced
before every expectation in the second group ((a, b, c)->(d, e, f)).
Would this not then require creation of
a SequenceSet for each of the expectations in one of the groups, or a
SequenceSetSet?
It's better than the current syntactic requirements, but I think not
as good as it
could be.
SequenceSet ss, tt, uu;
EXPECT_CALL(myMock, a()).InSequence(ss.AddNew(), tt.AddNew(), uu.AddNew
());
EXPECT_CALL(myMock, b()).InSequence(ss.AddNew(), tt.AddNew(), uu.AddNew
());
EXPECT_CALL(myMock, c()).InSequence(ss.AddNew(), tt.AddNew(), uu.AddNew
());
EXPECT_CALL(myMock, d()).InSequenceSet(ss, tt, uu);
EXPECT_CALL(myMock, e()).InSequenceSet(ss, tt, uu);
EXPECT_CALL(myMock, f()).InSequenceSet(ss, tt, uu);
Can you suggest how to handle these cases?http://groups.google.com/
group/googlemock/browse_thread/thread/888b6cec8d82675e
--JJ
>
> --
> Zhanyong
Zhanyong Wan (λx.x x)
One nice thing about the sequence abstraction is that it guarantees
(by construct) that the graph is acyclic. In other words, it prevents
the user from making the mistake of defining a cyclic ordering. I
would like whatever constructs we add to have this property too.
IIUIC, the following code using Group:
Sequence s;
Group g(s);
EXPECT_CALL(mock, a()).InGroup(g);
EXPECT_CALL(mock, b()).InSequence(s);
EXPECT_CALL(mock, c()).InGroup(g);
would result in a() and c() being in the same group g, and g -> b() ->
g. This is a cyclic graph. So I guess we need to impose some
restrictions on the usage of groups.
These are excellent points. I don't have a good answer yet, but will
keep thinking about it and update the list if I make any progress.
I'd encourage other people to do the same too. Thanks,
>
> --JJ
>
>>
>> --
>> Zhanyong
>
--
Zhanyong
Zhanyong Wan (λx.x x)
To say that B(), C(), and D() must be after A():
Expectation a = EXPECT_CALL(foo, A());
EXPECT_CALL(foo, B()).After(a);
EXPECT_CALL(foo, C()).After(a);
EXPECT_CALL(foo, D()).After(a);
To say that E() must be after B(), C(), and D():
ExpectationSet es;
es += EXPECT_CALL(foo, B());
es += EXPECT_CALL(foo, C());
es += EXPECT_CALL(foo, D());
EXPECT_CALL(foo, E()).After(es);
To say that X(), Y(), and Z() must be after B(), C(), and D():
ExpectationSet es;
es += EXPECT_CALL(foo, B());
es += EXPECT_CALL(foo, C());
es += EXPECT_CALL(foo, D());
EXPECT_CALL(foo, X()).After(es);
EXPECT_CALL(foo, Y()).After(es);
EXPECT_CALL(foo, Z()).After(es);
Notes:
- A variable of type Expectation identifies an EXPECT_CALL(). To bind an
Expectation variable x with an EXPECT_CALL(), just write "x =
EXPECT_CALL(...)...;"
- An ExpectationSet is a set of Expectation values. The += operator
can be used to add an Expectation to an ExpectationSet.
- EXPECT_CALL() accepts a new clause .After(e1, ..., en), where each e
is either an Expectation or an ExpectationSet. This adds the
constraint that the current expectation can be matched only after
all of e1, ..., and en have been matched.
- We'll continue to allow the .InSequence() syntax, which can be
viewed as a syntactic sugar for .After(). Suppose we have two
EXPECT_CALLs in the same sequence s:
EXPECT_CALL(foo, X()).InSequence(s);
...
EXPECT_CALL(foo, Y()).InSequence(s);
we can always rewrite them as:
Expectation e = EXPECT_CALL(foo, X());
...
EXPECT_CALL(foo, Y()).After(e);
- The ordering is still guaranteed to be acyclic by construct. In
EXPECT_CALL(...).After(e1, ..., en);
e1, ..., and en can only reference EXPECT_CALLs defined *before* the
current one. Hence a cycle is impossible.
Thoughts?
2009/6/8 Zhanyong Wan (λx.x x) <w...@google.com>:
--
Zhanyong
JJ
between the expectations are in a way that the Sequence constructions
are and my Group constructions were not, and as you say it's still
clear that the graph will definitely remain acyclic.
It's even clear how to construct overlapping sets of expectations, if
one happened to need those, via the use of an Expectation variable
that can then be added to each of the ExpectationSets individually.
The syntactic burden is slim, on a par with the EXPECT_CALL InSequence
construct, which is great. I have a couple of suggestions that might
improve some cases even more, I think.
Firstly, I think that the implicit InSequence is one of the best
features of the sequencing constructs. I'd therefore suggest that an
implicit InExpectationSet construction, if possible, would also be
great. I could use that to syntactically simplify my use case (the set
of expectations being defined in a subfunction, remember), because the
subfunction need not know about the ExpectationSet if it was implicit.
Secondly, what about the possibility of setting some After(ESet) at
definition time of an ExpectationSet, meaning that all expectations
added to the set would have After(ESet) called on them? I think that
this neatens some constructions:
ExpectationSet as;
as += EXPECT_CALL(foo, A(1));
as += EXPECT_CALL(foo, A(2));
ExpectationSet bs;
bs += EXPECT_CALL(foo, B(1)).After(as);
bs += EXPECT_CALL(foo, B(2)).After(as);
EXPECT_CALL(foo, C(1)).After(bs);
EXPECT_CALL(foo, C(2)).After(bs);
...
becomes
ExpectationSet as;
as += EXPECT_CALL(foo, A(1));
as += EXPECT_CALL(foo, A(2));
ExpectationSet bs(After(as));
bs += EXPECT_CALL(foo, B(1));
bs += EXPECT_CALL(foo, B(2));
ExpectationSet cs(After(bs));
cs += EXPECT_CALL(foo, C(1));
cs += EXPECT_CALL(foo, C(2));
...
I prefer the latter on two grounds. First, and most importantly, it's
better at saying things once and only once, as a large group of
expectations that are all sequenced after the same expectation(s) need
only have that specified once. Second, it means that fewer objects
need be passed around between functions. If you have a subfunction
that's going to create some expectations in a set, you need to pass
the set into the subfunction, which is all fine and dandy. But if you
also want them to be sequenced after something else, then without this
construction you also have to pass *that* into the subfunction,
whereas with this syntax only the one ExpectationSet need be passed
in.
It's easy to see that this is still acyclic by construction, since the
After clause on the ExpectationSet can still only refer to
expectations defined before it is.
The reason for the slightly strange construction syntax is purely for
clarity of what's going on. In the line:
ExpectationSet bs(After(as));
the After class has no purpose except to get the word 'after' into the
syntactic construction. We could instead pass the ExpectationSet 'as'
directly to the constructor of bs, but that would make it much less
clear what bs was going to do with the expectations in as.
Alternatively we could make After(const ExpectationSet& es) a member
function of ExpectationSet, but that would open up a whole other can
of worms if some expectations were added to the set before After was
called. It would still work, but would be unconscionably messy.
Overall, I think my proposed syntax is clearer than both of these.
Lastly, again, my main thought here is that I like it. I propose only
additions, not changes.
JJ
>
> read more »
Zhanyong Wan (λx.x x)
Thanks for the feedback and great suggestions. I like them.
Since the improvements you suggested are nice to have but not
indispensable, I'd like to implement the basic constructs first, see
how people are using them, and then decide what to do with the
improvements based on more user feedback. Since some people may have
got lost in this discussion, I'll start a new thread to solicit
comments on the design. Thanks,
--
Zhanyong