proposal: don't warn on uninteresting calls

[Zhanyong Wan (λx.x x)]

Hi,

In Google Mock's terms, we say that a call to method F() on mock object m is "uninteresting" if you don't have any EXPECT_CALL on it.  When an uninteresting call happens, Google Mock prints a warning message.  The rationale is that this call might not be supposed to happen, so we'll let you take a look and determine whether it indicates a bug.

If you decide that the call is legitimate, you can suppress the warning by adding an

  EXPECT_CALL(m, F()).Times(AnyNumer());

Or, if you would rather not see such warnings ever, you can run the test with --gmock_verbose=error, which means "don't print any message that's less severe than an error."

So, while it's possible to suppress this warning properly, it requires some work.  And no one likes extra work.  An unfortunate consequence of this is that people start to take the path of least resistance and write expectations like

  EXPECT_CALL(m, F());

even when they don't really care about this call.  As is, the above statement *enforces* that m.F() is called exactly once, even though it's not a bug if F() is not called at all or called more than once.  This makes the tests brittle as they depend on the specific implementation at the moment.  Later, when someone needs to change the implementation without changing the code's end behavior, suddenly many tests start to break when they shouldn't.  And the poor person has to fix a whole slew of unrelated tests.  You can imagine how happy he/she will be.

In my experience, this is among the top of the gripes people have about Google Mock, and it's understandably so.  We need to make it easier for Google Mock users to do the right thing.

For that, I'd like to stop treating uninteresting calls as warnings.  By default, Google Mock should not print anything when an uninteresting call happens.  However, you can still see them by turning up Google Mock's verbosity level (e.g. run the test with --gmock_verbose=info).  Then, doing the right thing (not write any EXPECT_CALL on F()) becomes easier than the hack of adding an EXPECT_CALL().

Sounds good?  Thanks,
--
Zhanyong

[Zhanyong Wan (λx.x x)]

Liam,

On Wed, Jan 23, 2013 at 11:59 AM, liam mail <liam...@googlemail.com> wrote:

On 23 January 2013 19:47, Zhanyong Wan (λx.x x) <w...@google.com> wrote:
> Hi,
>
> In Google Mock's terms, we say that a call to method F() on mock object m is
> "uninteresting" if you don't have any EXPECT_CALL on it.  When an
> uninteresting call happens, Google Mock prints a warning message.  The
> rationale is that this call might not be supposed to happen, so we'll let
> you take a look and determine whether it indicates a bug.
>
> If you decide that the call is legitimate, you can suppress the warning by
> adding an
>
>   EXPECT_CALL(m, F()).Times(AnyNumer());
>
> Or, if you would rather not see such warnings ever, you can run the test
> with --gmock_verbose=error, which means "don't print any message that's less
> severe than an error."
>

I thought the correct way to suppress uninteresting calls was to use
::testing::NiceMock ?

You are right - NiceMock is one way to suppress uninteresting call warnings on a mock.  However, since this is not the default behavior, the test author needs to make some effort (i.e. write NiceMock<MockFoo> instead of MockFoo) to use it.  In practice, many people don't do that (perhaps they don't know about NiceMock, or perhaps they aren't sure they should suppress all uninteresting calls on the mock object).

With my proposal, NiceMock becomes the default (and unnecessary), so it's harder to do the wrong thing.

--Liam

--
Zhanyong

[Zhanyong Wan (λx.x x)]

Hi Jorge,

I don't see a conflict between TDD and my proposal.  If you do TDD, you want to write the test (the "spec") first.  That means you'll want to write explicit EXPECT_CALLs() to specify the contract of your code under test.

On Wed, Jan 23, 2013 at 12:32 PM, <jmec...@gmail.com> wrote:

Hi,
What about people doing TDD, I guess this feature is quite handy for them!

Best Regards
Jorge Costa

Sent from my iPad

--
Zhanyong

[Daniel Walker]

I strongly disagree with the proposal - as annoying as 'uninteresting call' can be, it's sometimes essential to know that nothing ELSE is happening.

Thus, NiceMock is a nice option (which I didn't previously know about and will use when I'm sure I don't care about uninteresting calls) but I would prefer to be in control of my expectations.

[Billy Donahue]

I'm with Daniel.

I've been saved by the reporting of unexpected calls.

Sounds like NiceMock needs more publicity.

Maybe post it over the toilet or something.

--
 
 
 

[Zhanyong Wan (λx.x x)]

Daniel and Billy,

I agree that sometimes seeing uninteresting calls is useful.  The question is which default behavior *on average* leads to better tests for *most* people.  I've observed *many* tests that are too strict because the test author didn't think carefully about what the expectations should be.  These tests became a huge liability for some big projects, to the extent that they are considering banning Google Mock completely.

The proposal is not about taking away your options.  If you prefer, you can still use --gmock_verbose=info on the command line to see uninteresting calls.  Or you can set the flag in your code.  In the future, we may even define NaggyMock, which is similar to NickMock/StrictMock, except that it generates warnings on uninteresting calls.  Essentially, I propose to change the default behavior from naggy to nice.

Admittedly, this makes it more work to create a naggy mock, but it's a net win if a nice mock is a better choice than a naggy one more often than not.  And I believe that is the case.

Thanks,

--
Zhanyong

[Billy Donahue]

On Wednesday, January 23, 2013, Zhanyong Wan (λx.x x) wrote:

Daniel and Billy,

I agree that sometimes seeing uninteresting calls is useful.  The question is which default behavior *on average* leads to better tests for *most* people.

And deciding whether the differential gain balances the cost of changing the default.

I've observed *many* tests that are too strict because the test author didn't think carefully about what the expectations should be.  These tests became a huge liability for some big projects, to the extent that they are considering banning Google Mock completely.

"No way! Why should I change? He's the one who sucks!"

Seriously, can't these guys just as readily set flags or wrap their mocks in NiceMock?

I don't have the exposure you have had to these cases, but banning gmock because of the uninteresting call logs (which has an easy cure) sounds extreme.

The uninteresting calls are only advisory, aren't they? I REALLY like seeing these messages because they often show that calls are coming into the object that I didn't realize were even happening. This is essential to know this stuff when developing a test, and it should be made really clear how to turn it off when it isn't wanted. an uninteresting call takes a default action that the programmer didn't specify, like returning false or zero or ignoring a notification, etc. If the call held a pointer argument, maybe the mock should have taken ownership or modified its pointee, but instead the argument will be ignored and the test author didn't even know the call happened unless the call is logged. I think this change might make bugs harder to find, and make tests harder to get right.

The proposal is not about taking away your options.  If you prefer, you can still use --gmock_verbose=info on the command line to see uninteresting calls.

Or you can set the flag in your code.

Obviously it can be extraordinarily difficult to do that for thousands of tests in their various call environments and launcher scripts, etc. I'm sure you're well aware of this, though. My time is as valuable as my options are. In other words, I would like the option to do nothing.

 

In the future, we may even define NaggyMock, which is similar to NickMock/StrictMock, except that it generates warnings on uninteresting calls. 

Essentially, I propose to change the default behavior from naggy to nice.

Suggest "Naggy" => "Nagging".  Naggy isn't a real word.

"NagMock" is nice and short, but also derogatory and would discourage use.

If you're going to do this, It would be better to have access to NagMock well beforehand. I will then be able to start on changing all of our tests to use this wrapper, which will probably take several days. For the time being it will give the same effect as no wrapper at all, but when the default changes to Nice, at least we will be prepared. I don't like it, though.

--
Zhanyong

[Zhanyong Wan (λx.x x)]

On Wed, Jan 23, 2013 at 7:10 PM, Billy Donahue <billyd...@google.com> wrote:

On Wednesday, January 23, 2013, Zhanyong Wan (λx.x x) wrote:

Daniel and Billy,

I agree that sometimes seeing uninteresting calls is useful.  The question is which default behavior *on average* leads to better tests for *most* people.

And deciding whether the differential gain balances the cost of changing the default.

As an interesting data point, I've observed the long debate between the EasyMock camp and Mockito camp among Java programmers at Google.  In EasyMock it's easier to create mocks that aren't nice.  As a result it leads to people over-specifying their expectations, which makes the tests brittle and hard-to-maintain over time.  In contrast, Mockito makes it easier to create nice mocks, and you have to do extra to actual verify an expectation.  The Mockito camp won the debate, and most people agreed that making mocks nice by default leads to more maintainable tests and tests whose intention is more obvious.  There's valuable lesson to be learned from this.

I've observed *many* tests that are too strict because the test author didn't think carefully about what the expectations should be.  These tests became a huge liability for some big projects, to the extent that they are considering banning Google Mock completely.

"No way! Why should I change? He's the one who sucks!"

Seriously, can't these guys just as readily set flags or wrap their mocks in NiceMock?

Apparently, people like to follow the path of least resistance.

I don't have the exposure you have had to these cases, but banning gmock because of the uninteresting call logs (which has an easy cure) sounds extreme.

This may not be the only reason, but significantly contributes to the perception that gmock leads to bad tests.

The uninteresting calls are only advisory, aren't they? I REALLY like seeing these messages because they often show that calls are coming into the object that I didn't realize were even happening. This is essential to know this stuff when developing a test, and it should be made really clear how to turn it off when it isn't wanted.

I sympathize with you.  Unfortunately it's not always possible to please everyone.  This is not an easy decision, but I think it will lead to overall better tests.  I suggest to use gmock_verbose=info when developing tests if needed.

 

an uninteresting call takes a default action that the programmer didn't specify,

Not necessarily true.  The programmer may have specified the default action using ON_CALL().  The style I advocate is to define the mock's behavior (stubs) using ON_CALL(), and write EXPECT_CALL() to verify interaction (instead of specifying behavior).  Obviously this is just a guideline and not strict rule.

 

like returning false or zero or ignoring a notification, etc. If the call held a pointer argument, maybe the mock should have taken ownership or modified its pointee, but instead the argument will be ignored and the test author didn't even know the call happened unless the call is logged.

I had exactly the same doubts when learning the Mockito philosophy.  Nice mocks could definitely hide bugs.  However, it turns out that in practice, if you have an uninteresting call that you don't know about and haven't set an action for, it _often_ leads to a test failure somewhere, due to the default action of returning 0 or NULL not making sense to the caller.  This was confirmed by my own experience.  Overall, the problem of over-specified tests due to non-nice mocks out-weighs the problem of under-specified tests due to nice mocks.

 

I think this change might make bugs harder to find, and make tests harder to get right.

The proposal is not about taking away your options.  If you prefer, you can still use --gmock_verbose=info on the command line to see uninteresting calls.

Or you can set the flag in your code.

Obviously it can be extraordinarily difficult to do that for thousands of tests in their various call environments and launcher scripts, etc. I'm sure you're well aware of this, though. My time is as valuable as my options are. In other words, I would like the option to do nothing.

 

In the future, we may even define NaggyMock, which is similar to NickMock/StrictMock, except that it generates warnings on uninteresting calls. 

Essentially, I propose to change the default behavior from naggy to nice.

Suggest "Naggy" => "Nagging".  Naggy isn't a real word.

 I think it's a real word:

https://www.google.com/search?q=define+naggy

It means "prone to nag", which is subtly different from "nagging".  Since generating warnings on uninteresting calls means the mock is inclined to nag, but not necessarily currently nagging, I believe "naggy" is the concept I'm after.

"NagMock" is nice and short, but also derogatory and would discourage use.

If you're going to do this, It would be better to have access to NagMock well beforehand. I will then be able to start on changing all of our tests to use this wrapper, which will probably take several days. For the time being it will give the same effect as no wrapper at all, but when the default changes to Nice, at least we will be prepared. I don't like it, though.

Sounds fair.  Thanks,

--
Zhanyong

[Billy Donahue]

I'm okay as long as I can specify the kind of mock I want explicitly.

Is there a way to make this easier maybe? Currently I need to make a mock class and then make a wrapper template to specify the strictness level. Can I "bake in" the strictness when defining the Mock class somehow or is that a silly thing to want?

On Thursday, February 7, 2013, Zhanyong Wan (λx.x x) wrote:

On Wed, Jan 23, 2013 at 7:10 PM, Billy Donahue <billyd...@google.com> wrote:

On Wednesday, January 23, 2013, Zhanyong Wan (λx.x x) wrote:

Daniel and Billy,

I agree that sometimes seeing uninteresting calls is useful.  The question is which default behavior *on average* leads to better tests for *most* people.

And deciding whether the differential gain balances the cost of changing the default.

As an interesting data point, I've observed the long debate between the EasyMock camp and Mockito camp among Java programmers at Google.  In EasyMock it's easier to create mocks that aren't nice.  As a result it leads to people over-specifying their expectations, which makes the tests brittle and hard-to-maintain over time.  In contrast, Mockito makes it easier to create nice mocks, and you have to do extra to actual verify an expectation.  The Mockito camp won the debate, and most people agreed that making mocks nice by default leads to more maintainable tests and tests whose intention is more obvious.  There's valuable lesson to be learned from this. 

I've observed *many* tests that are too strict because the test author didn't think carefully about what the expectations should be.  These tests became a huge liability for some big projects, to the extent that they are considering banning Google Mock completely.

"No way! Why should I change? He's the one who sucks!"

Seriously, can't these guys just as readily set flags or wrap their mocks in NiceMock?

Apparently, people like to follow the path of least resistance.

 

Always. And so do I. If they are the type to ban gmock

 

I don't have the exposure you have had to these cases, but banning gmock because of the uninteresting call logs (which has an easy cure) sounds extreme.

This may not be the only reason, but significantly contributes to the perception that gmock leads to bad tests.

The uninteresting calls are only advisory, aren't they? I REALLY like seeing these messages because they often show that calls are coming into the object that I didn't realize were even happening. This is essential to know this stuff when developing a test, and it should be made really clear how to turn it off when it isn't wanted.

I sympathize with you.  Unfortunately it's not always possible to please everyone.  This is not an easy decision, but I think it will lead to overall better tests.  I suggest to use gmock_verbose=info when developing tests if needed.

 

an uninteresting call takes a default action that the programmer didn't specify,

Not necessarily true.  The programmer may have specified the default action using ON_CALL().  The style I advocate is to define the mock's behavior (stubs) using ON_CALL(), and write EXPECT_CALL() to verify interaction (instead of specifying behavior).  Obviously this is just a guideline and not strict rule.

 

like returning false or zero or ignoring a notification, etc. If the call held a pointer argument, maybe the mock should have taken ownership or modified its pointee, but instead the argument will be ignored and the test author didn't even know the call happened unless the call is logged.

I had exactly the same doubts when learning the Mockito philosophy.  Nice mocks could definitely hide bugs.  However, it turns out that in practice, if you have an uninteresting call that you don't know about and haven't set an action for, it _often_ leads to a test failure somewhere, due to the default action of returning 0 or NULL not making sense to the caller.  This was confirmed by my own experience.  Overall, the problem of over-specified tests due to non-nice mocks out-weighs the problem of under-specified tests due to nice mocks.

 

I think this change might make bugs harder to find, and make tests harder to get right.

The proposal is not about taking away your options.  If you prefer, you can still use --gmock_verbose=info on the command line to see uninteresting calls.

Or you can set the flag in your code.

Obviously it can be extraordinarily difficult to do that for thousands of tests in their various call environments and launcher scripts, etc. I'm sure you're well aware of this, though. My time is as valuable as my options are. In other words, I would like the option to do nothing.

 

In the future, we may even define NaggyMock, which is similar to NickMock/StrictMock, except that it generates warnings on uninteresting calls. 

Essentially, I propose to change the default behavior from naggy to nice.

Suggest "Naggy" => "Nagging".  Naggy isn't a real word.

 I think it's a real word:

https://www.google.com/search?q=define+naggy

It means "prone to nag", which is subtly different from "nagging".  Since generating warnings on uninteresting calls means the mock is inclined to nag, but not necessarily currently nagging, I believe "naggy" is the concept I'm after.

Appearance in a dictionary doesn't really indicate propriety of a word anymore.

Dictionaries are soft and descriptive these days. And they need to get off my lawn.

I still think it gives the connotation that the mock is cranky and emitting unjustified complaints.

[Zhanyong Wan (λx.x x)]

Update: NaggyMock was checked into Google's internal code base yesterday.  It should be integrated to the public version soon.  Thanks.

On Fri, Feb 8, 2013 at 6:02 AM, Billy Donahue <billyd...@google.com> wrote:

I'm okay as long as I can specify the kind of mock I want explicitly.

Is there a way to make this easier maybe? Currently I need to make a mock class and then make a wrapper template to specify the strictness level. Can I "bake in" the strictness when defining the Mock class somehow or is that a silly thing to want?

You may do this:

// Define your mock class as usual.

class NiceMockFoo : ... {

  ...

};

typedef NaggyMock<NiceMockFoo> MockFoo;

Then the nagginess is baked in MockFoo.

However, I suggest that you don't rush to that.  Try to use nice mocks for a while and see how it goes.  Thanks,

--
Zhanyong

[Billy Donahue]

On Wed, Feb 13, 2013 at 12:48 PM, Zhanyong Wan (λx.x x) <w...@google.com> wrote:

Update: NaggyMock was checked into Google's internal code base yesterday.  It should be integrated to the public version soon.  Thanks.

On Fri, Feb 8, 2013 at 6:02 AM, Billy Donahue <billyd...@google.com> wrote:

I'm okay as long as I can specify the kind of mock I want explicitly.

Is there a way to make this easier maybe? Currently I need to make a mock class and then make a wrapper template to specify the strictness level. Can I "bake in" the strictness when defining the Mock class somehow or is that a silly thing to want?

You may do this:

// Define your mock class as usual.

class NiceMockFoo : ... {

  ...

};

typedef NaggyMock<NiceMockFoo> MockFoo;

Then the nagginess is baked in MockFoo.

That's not baked in, it's in a wrapper.

Sounds like the answer is no.

[Zhanyong Wan (λx.x x)]

On Wed, Feb 13, 2013 at 9:51 AM, Billy Donahue <billyd...@google.com> wrote:

On Wed, Feb 13, 2013 at 12:48 PM, Zhanyong Wan (λx.x x) <w...@google.com> wrote:

Update: NaggyMock was checked into Google's internal code base yesterday.  It should be integrated to the public version soon.  Thanks.

On Fri, Feb 8, 2013 at 6:02 AM, Billy Donahue <billyd...@google.com> wrote:

I'm okay as long as I can specify the kind of mock I want explicitly.

Is there a way to make this easier maybe? Currently I need to make a mock class and then make a wrapper template to specify the strictness level. Can I "bake in" the strictness when defining the Mock class somehow or is that a silly thing to want?

You may do this:

// Define your mock class as usual.

class NiceMockFoo : ... {

  ...

};

typedef NaggyMock<NiceMockFoo> MockFoo;

Then the nagginess is baked in MockFoo.

That's not baked in, it's in a wrapper.

I don't understand what you are looking for.  Why do you care about whether MockFoo is defined as a wrapper or not?

Suppose that we expose an API to let you declare a mock class as naggy as you define it.  Then you may use it like this (not exact syntax):

class MockFoo : ... {

  MAKE_THIS_MOCK_NAGGY();

  ...

};

Compared with the typedef version, this is the same number of lines.  Why is this much better?

Thanks,

--
Zhanyong

[Billy Donahue]

On Wed, Feb 13, 2013 at 12:57 PM, Zhanyong Wan (λx.x x) <w...@google.com> wrote:

On Wed, Feb 13, 2013 at 9:51 AM, Billy Donahue <billyd...@google.com> wrote:

On Wed, Feb 13, 2013 at 12:48 PM, Zhanyong Wan (λx.x x) <w...@google.com> wrote:

Update: NaggyMock was checked into Google's internal code base yesterday.  It should be integrated to the public version soon.  Thanks.

On Fri, Feb 8, 2013 at 6:02 AM, Billy Donahue <billyd...@google.com> wrote:

I'm okay as long as I can specify the kind of mock I want explicitly.

Is there a way to make this easier maybe? Currently I need to make a mock class and then make a wrapper template to specify the strictness level. Can I "bake in" the strictness when defining the Mock class somehow or is that a silly thing to want?

You may do this:

// Define your mock class as usual.

class NiceMockFoo : ... {

  ...

};

typedef NaggyMock<NiceMockFoo> MockFoo;

Then the nagginess is baked in MockFoo.

That's not baked in, it's in a wrapper.

I don't understand what you are looking for.  Why do you care about whether MockFoo is defined as a wrapper or not?

Suppose that we expose an API to let you declare a mock class as naggy as you define it.  Then you may use it like this (not exact syntax):

class MockFoo : ... {

  MAKE_THIS_MOCK_NAGGY();

  ...

};

Compared with the typedef version, this is the same number of lines.  Why is this much better?

I didn't suggest that it was much better. I'd settle for "slightly easier".

I'm looking at the prospect of changing a lot of mock classes.

[liam mail]

"No way! Why should I change? He's the one who sucks!"

I know this was said in jest but it is the way I feel :(

Correct me if I am wrong but the purpose and reason for this change
seems to be that people did not use the library correctly and the
library did not provide stubs,doubles or fakes only mocks?

Mocks define the actions the will be preformed and the order of these
actions, these actions will be asserted on. I just do not see how you
can have a mock (ie not a NiceMock) and it's behaviour is not
interesting.

"Mock objects always use behavior verification..."
http://martinfowler.com/articles/mocksArentStubs.html

--Liam

[Billy Donahue]

I'm not going to deliberately weaken my tests. I don't see how any good will come of that.

--
Zhanyong