proposal: allow gtest/gmock to be used with other C++ testing frameworks
Zhanyong Wan (λx.x x)
I heard from some people that they want to use Google Mock, but they
are using some C++ testing framework other than Google Test, and for
various reasons they cannot switch to Google Test yet. Since Google
Mock only works with Google Test now, they have to skip it.
We have planned to open up Google Mock more such that you can use it
with any C++ testing framework of your choice. Here's the background
and how I propose to do it:
- There are tons of C++ testing frameworks out there. It's impossible
to adapt Google Mock for each of them. Even if we only target the
most popular ones, it would still be a lot of work, and people will
keep asking us when their favorite frameworks will be supported.
- We could define an interface that each testing framework needs to
implement in order to hook up with Google Mock, and ask the users to
implement this interface themselves. However, this just moves work
around, and makes the barrier of entry pretty high. And we'll be
busy incorporating patches the users submitted for their favorite
frameworks.
- Luckily, all the C++ testing frameworks I know treat uncaught
exceptions as test failures. (A testing framework would be pretty
useless if a test could throw an uncaught exception and still is
declared successful.) Therefore, all we need to do is to make
Google Mock throw an exception when a failure needs to be reported.
Then a user can use Google Mock with his current testing framework,
be it CppUnit, CxxTest, UnitTest++, whatever.
- Actually, why do we stop with Google Mock? If we make Google Test
do this, a user can start using ASSERT_THAT and other assertions
from Google Test without giving up his current testing framework.
Perhaps he'll be more likely to switch to Google Test once he has
some experience with it.
- I propose to add a command-line flag --gtest_throw_on_failure to
Google Test. When this (or the GTEST_THROW_ON_FAILURE environment
variable) is specified (the user can also set it in the code),
Google Test will throw an exception when there's a failure. Since
Google Mock uses Google Test for error reporting, no additional work
needs to be done there - it will just work.
- What if the user compiles his code with exceptions disabled? This
is pretty unlikely as most C++ testing frameworks use exceptions for
error reporting. If, however, the user is using a testing framework
that doesn't use exceptions, and he absolutely must disable
exceptions in his build, we'll let Google Test crash when there's a
failure and --gtest_throw_on_failure is specified. The crash should
be treated by the testing framework or test runner as a failure, so
we are fine. One annoying thing is that the crash will kill the
entire program, so not-yet-started test methods won't have a chance
to run. But that's not a huge problem.
- When exceptions are disabled, gtest_throw_on_failure is not an
accurate name, as the behavior is "crash" instead, but I cannot
think of a better name.
Thought? Thanks,
--
Zhanyong
Keir Mierle
--
Zhanyong
Zhanyong Wan (λx.x x)
Google Test parses its own command line flags without using gflags.
> Keir
>
>>
>> --
>> Zhanyong
>
>
--
Zhanyong
Vlad Losev
Greetings,
I heard from some people that they want to use Google Mock, but they
are using some C++ testing framework other than Google Test, and for
various reasons they cannot switch to Google Test yet. Since Google
Mock only works with Google Test now, they have to skip it.
We have planned to open up Google Mock more such that you can use it
with any C++ testing framework of your choice. Here's the background
and how I propose to do it:
- There are tons of C++ testing frameworks out there. It's impossible
to adapt Google Mock for each of them. Even if we only target the
most popular ones, it would still be a lot of work, and people will
keep asking us when their favorite frameworks will be supported.
What's wrong with just providing a boolean predicate so that the user can use other framework's ASSERT_TRUE equivalent? The predicate can output explanatory text in case it returns false.
Zhanyong Wan (λx.x x)
>
>
> On Tue, Jan 6, 2009 at 4:40 PM, Zhanyong Wan (λx.x x) <w...@google.com>
> wrote:
>>
>> Greetings,
>>
>> I heard from some people that they want to use Google Mock, but they
>> are using some C++ testing framework other than Google Test, and for
>> various reasons they cannot switch to Google Test yet. Since Google
>> Mock only works with Google Test now, they have to skip it.
>>
>> We have planned to open up Google Mock more such that you can use it
>> with any C++ testing framework of your choice. Here's the background
>> and how I propose to do it:
>>
>> - There are tons of C++ testing frameworks out there. It's impossible
>> to adapt Google Mock for each of them. Even if we only target the
>> most popular ones, it would still be a lot of work, and people will
>> keep asking us when their favorite frameworks will be supported.
>
> What's wrong with just providing a boolean predicate so that the user can
> use other framework's ASSERT_TRUE equivalent? The predicate can output
> explanatory text in case it returns false.
to solve here is to allow Google *Mock* to be used with an arbitrary
C++ testing framework. Allowing Google Test assertions in other
testing frameworks is a nice by-product. I didn't say anything about
providing a predicate being wrong, and I don't see how it contributes
to allowing using Google Mock with other C++ testing frameworks.
Zhanyong
Chandler Carruth
>
> I heard from some people that they want to use Google Mock, but they
> are using some C++ testing framework other than Google Test, and for
> various reasons they cannot switch to Google Test yet. Since Google
> Mock only works with Google Test now, they have to skip it.
>
> We have planned to open up Google Mock more such that you can use it
> with any C++ testing framework of your choice. Here's the background
> and how I propose to do it:
>
> - There are tons of C++ testing frameworks out there. It's impossible
> to adapt Google Mock for each of them. Even if we only target the
> most popular ones, it would still be a lot of work, and people will
> keep asking us when their favorite frameworks will be supported.
>
> - We could define an interface that each testing framework needs to
> implement in order to hook up with Google Mock, and ask the users to
> implement this interface themselves. However, this just moves work
> around, and makes the barrier of entry pretty high. And we'll be
> busy incorporating patches the users submitted for their favorite
> frameworks.
this approach, as it seems a much more obvious approach. If throwing
an exception is all we need, couldn't we instead ask for a function to
call at the end of a test which causes the test to fail? We could even
default this to the appropriate mechanism for GoogleTest when present,
and a function which throws the exception otherwise; but still allow
each test framework to customize what GoogleMock does in the event of
failing expectations. Any patches to support GoogleMock from a
particular framework would go to that framework, not us. What's
blocking this?
-Chandler
Zhanyong Wan (λx.x x)
For this to work, we have to guarantee that the function is always
called by a test. Also, since any part of the test (including the
destructor of the test fixture) may generate a failure, the function
must be called at the very end of the test, e.g. in the destructor of
a base class of the test fixture. For testing frameworks that don't
use test fixtures, the mechanism will need to be different. Therefore
this puts more burden on the user to set it up properly. It's easy
for the user to do it wrong and get false positives.
> but still allow
> each test framework to customize what GoogleMock does in the event of
> failing expectations.
The approach I'm proposing is a low-hanging fruit. It is easy to
implement, requires little work from the users, and instantly enables
Google Mock to work with arbitrary C++ testing frameworks. The
pay-off is huge compared with the effort.
Of course, a solution tailored for a specific testing framework can
provide a better experience for users of that framework, compared with
the generic solution. If, for example, Google Mock talks to CppUnit's
error reporting mechanism directly, it's quite likely to generate
better failure messages than the throwing approach. I'm totally open
to that. Since that doesn't conflict with what I'm proposing now, and
requires more work, it can/should be done later.
> Any patches to support GoogleMock from a
> particular framework would go to that framework, not us. What's
> blocking this?
We don't control those testing frameworks, so there's no guarantee
that their owners would want to do that. Also I'm not sure
coordinating with these teams will save us much time compared with
hosting the patches ourselves.
Thanks,
--
Zhanyong
Chandler Carruth
Hmm. I think you're imagining a much more complex (and better for it,
as you point out below) solution, but maybe I'm misunderstanding what
you intend to do with exceptions. I'll try to explain, and that way
you can cut off my ramblings as soon as I'm off, and ignore the rest.
My impression was that upon any invalid call or upon the destruction
of a mock object with unmet expectations, an exception would be
thrown. My suggestion was that instead of necessarily throwing an
exception, GoogleMock calls an arbitrary function, potentially
supplied by the testing framework, to say "There's an error!". The
default such function in the presence of GoogleTest would be the
current behavior; in the absense it could throw the exception as
you've described; yet a testing framework could simply pass in the
function to call which flags "There's an error!".
If the above doesn't work, or I'm misunderstanding how the exceptions
will work (quite possible, I don't use them much, so I'm not nearly as
familiar as I should be), then by all means proceed with the proposal.
I like the approach, just trying to provide a slightly improvement for
the non-exception case.
-Chandler
Zhanyong Wan (λx.x x)
I see. I misunderstood you. Thanks for clarifying.
Google Mock was actually designed with the flexibility you described
and the implementation is partially there already. Therefore we are
on the same page. See FailureReporterInterface in
http://code.google.com/p/googlemock/source/browse/trunk/include/gmock/internal/gmock-internal-utils.h.
We just need to move it out of the internal namespace, and add an API
for replacing the default failure reporter. These should be
straightforward.
I imagine that more advanced users will find it useful to implement
FailureReporterInterface by themselves. My plan is to finish the
throwing approach first, as it's much easier to use for novice users,
who need our help more. Once that's done, we can flesh out the
FailureReporterInterface approach to allow more customization.
Thanks,
--
Zhanyong
Vlad Losev
I agree with Chandler: the failure reporting interface must be the first alternative. All the reporting method implementation has to do is to raise a failure and every testing framework has a macro or a method to force a failure which can be used for the purpose. We should provide an implementation for Google Test and a throwing implementation as a fallback for other frameworks. The reference implementation for a couple of other popular testing frameworks should be put into documentation. That will cover the majority of our user and will give the rest an excellent idea of how to implement such a function for a test framework of their choice.
Regards,
Vlad.
Keir Mierle
Zhanyong Wan (λx.x x)
"must" is a strong word. :-)
I don't think the failure reporting interface conflicts with the flag
I'm proposing. From a practical point of view, I think it gives us a
bigger bang for the buck if we implement the throw_on_failure flag
first, and I tried to explain why I think this is the case. I'm
totally fine with fleshing out the reporting interface afterwards.
Could you clarify why you think it has to be done first?
> All the reporting method implementation has to do is to raise a
> failure and every testing framework has a macro or a method to force a
> failure which can be used for the purpose. We should provide an
> implementation for Google Test and a throwing implementation as a fallback
> for other frameworks. The reference implementation for a couple of other
> popular testing frameworks should be put into documentation. That will cover
> the majority of our user and will give the rest an excellent idea of how to
> implement such a function for a test framework of their choice.
I agree with the plan. We only differ in the prioritization of the
two mechanisms.
What I like about this:
- Better failure messages tailored for each testing framework.
What I don't like about it:
- More time/work for us to implement.
- Works for fewer potential users.
- Requires more work from (at least some of) the users.
- Still doesn't allow Google Test assertions to be used in tests.
The last point is concerning, as once a user starts to use Google
Mock, it's easy for him to also use some Google Test assertions in the
code, thinking that they would work while their failures are silently
ignored.
--
Zhanyong
Chandler Carruth
> I agree with the plan. We only differ in the prioritization of the
> two mechanisms.
For reference, I'm good with either ordering, and didn't have a
problem with your priorities.
>
> What I like about this:
>
> - Better failure messages tailored for each testing framework.
>
> What I don't like about it:
>
> - More time/work for us to implement.
> - Works for fewer potential users.
> - Requires more work from (at least some of) the users.
> - Still doesn't allow Google Test assertions to be used in tests.
I'm starting to not like this last reason though...
>
> The last point is concerning, as once a user starts to use Google
> Mock, it's easy for him to also use some Google Test assertions in the
> code, thinking that they would work while their failures are silently
> ignored.
I actually think we should explicitly forbid using GoogleTest
assertions unless using GoogleTest proper. I don't see any benefit to
this, and lots of potential confusion. I think GoogleMock should make
any internal usage of GoogleTest completely opaque to the user, it
should be hidden and off limits.
If it is an explicit goal to allow GoogleTest assertions outside of
GoogleTest tests, I think that's a separate undertaking, and I'm a
long way from sold on it. I think this should stay focused on
GoogleMock.
-Chandler
Zhanyong Wan (λx.x x)
No worries.
> If the user wants to use gmock
> but not gtest, then they are not using either gflags or gtest's minimal port
> of gflags. I would assume they have their own test main() and command line
> parsing, with different syntax; in which case we cannot assume that the
> command line flags will be parsed.
> We could make it a requirement that the command line flags are parsed by
> gtest, but I suspect that would be awkward for some of the test suites.
Google Mock has to be initialized. We require the user to call
InitGoogleMock() in main(), which will parse the flags for both Google
Mock and Google Test.
> If
> they have to switch to using parts of gtest main, why not just use gtest
> entirely?
If they want to use Google Mock, they need to initialize it in main(),
which will take care of initialization Google Test as a dependent.
There can be many reasons why the user may not be able to switch to
gtest entirely: the organization may be reluctant and they need to try
it out in some pilots first, or their continuous build system is tied
with the testing framework they are currently using, for example.
> Another option is to make a compile time define to make gtest
> throw exceptions.
I think this is less flexible and harder for the user to set up.
> Keir
>
>>
>> > Keir
>> >
>> >>
>> >> --
>> >> Zhanyong
>> >
>> >
>>
>>
>>
>> --
>> Zhanyong
>
>
--
Zhanyong
Zhanyong Wan (λx.x x)
> On Wed, Jan 7, 2009 at 11:06 AM, Zhanyong Wan (λx.x x) <w...@google.com> wrote:
>> I agree with the plan. We only differ in the prioritization of the
>> two mechanisms.
>
> For reference, I'm good with either ordering, and didn't have a
> problem with your priorities.
Cool.
>>
>> What I like about this:
>>
>> - Better failure messages tailored for each testing framework.
>>
>> What I don't like about it:
>>
>> - More time/work for us to implement.
>> - Works for fewer potential users.
>> - Requires more work from (at least some of) the users.
>> - Still doesn't allow Google Test assertions to be used in tests.
>
> I'm starting to not like this last reason though...
>
>>
>> The last point is concerning, as once a user starts to use Google
>> Mock, it's easy for him to also use some Google Test assertions in the
>> code, thinking that they would work while their failures are silently
>> ignored.
>
> I actually think we should explicitly forbid using GoogleTest
> assertions unless using GoogleTest proper. I don't see any benefit to
> this, and lots of potential confusion. I think GoogleMock should make
> any internal usage of GoogleTest completely opaque to the user, it
> should be hidden and off limits.
I understand your philosophy. OTOH, we'll have to go out of our way
to make Google Mock's use of Google Test completely internal,
especially since macros don't obey namespaces. It's a big (if not
huge) undertaking and I'm not sure it will pay off. I'm not looking
forward to it myself.
Instead of preventing Google Mock users from using Google Test, I
would just advertise Google Test as a public component of Google Mock.
It's available there for you to use, but you don't have to use it. If
you decide to use it, you can choose which bits from it you want,
instead of being forced to accept it in its entirety. I think this is
a good thing.
> If it is an explicit goal to allow GoogleTest assertions outside of
> GoogleTest tests, I think that's a separate undertaking, and I'm a
> long way from sold on it.
This is actually much, much easier than prohibiting Google Test
assertions to be used outside of Google Test tests.
> I think this should stay focused on
> GoogleMock.
Unless we can enforce that Google Test assertions cannot be used
outside of Google Test tests (which I don't think we can do without a
lot of effort), it's a real danger that a Google Mock user would
inadvertently use them and end up with tests passing that should fail.
False positives are a big no-no for testing frameworks, so this is a
big problem.
Allowing Google Test assertions also means that the user doesn't have
to wonder why EXPECT_CALL works while other EXPECT_* macros don't.
--
Zhanyong
Vlad Losev
I'm OK with using C++ exceptions to signal failures. But when the exceptions are disabled, the only option to report a failure is to crash the program. I think this is too drastic. I am suspicious of the idea of intercepting a crash and continuing the program execution. For all you know, the program's state may be now corrupted and cannot be trusted. The only prudent way to go in such situation is to clean up and terminate the program ASAP. The alternatives may include infinite loops, stack overflows, etc. It's kinda tolerable /cringes/ when a user explicitly requests it (--gtest_catch_exceptions) but making it default behavior is plain evil. :-/
I agree with the plan. We only differ in the prioritization of the
> All the reporting method implementation has to do is to raise a
> failure and every testing framework has a macro or a method to force a
> failure which can be used for the purpose. We should provide an
> implementation for Google Test and a throwing implementation as a fallback
> for other frameworks. The reference implementation for a couple of other
> popular testing frameworks should be put into documentation. That will cover
> the majority of our user and will give the rest an excellent idea of how to
> implement such a function for a test framework of their choice.
two mechanisms.
What I like about this:
- Better failure messages tailored for each testing framework.
What I don't like about it:
- More time/work for us to implement.
- Works for fewer potential users.
- Requires more work from (at least some of) the users.
- Still doesn't allow Google Test assertions to be used in tests.
The last point is concerning, as once a user starts to use Google
Mock, it's easy for him to also use some Google Test assertions in the
code, thinking that they would work while their failures are silently
ignored.
Like Chandler, I don't like the idea of using Google Test assertions with other frameworks. Since there is no exception object format agreed between all test frameworks, all the contextual information is lost: captured failed expression, any user-provided message, even file name and line number information. Google Mock users may get an impression that Google Test is somehow inferior to their current framework.
Vlad Losev
We can abort (GTEST_CHECK_?) if Google Test assertions are run outside of RUN_ALL_TESTS(). That should be pretty easy.
.
Zhanyong Wan (λx.x x)
We can't really. They are called ad hoc tests and are supported by
Google Test, although not recommended or well advertised. See
AdHocTest in http://code.google.com/p/googletest/source/browse/trunk/test/gtest_output_test_.cc.
--
Zhanyong
Zhanyong Wan (λx.x x)
definitely not ideal, but I don't think it's a show stopper,
especially since that *most* C++ testing frameworks require exceptions
to be enabled.
exception, so the test output will still contain all the information.
The way to invite users is to make it easy for them to take small
steps to try out the product, without demanding them to completely
switch from the product they are currently using. There's a
significant market (people who are looking for better testing
frameworks but cannot yet abandon their existing framework) that
Google Test can tap. I'd try to enter that market rather than staying
out of it.
You can find many things not ideal about the gtest_throw_on_failure
approach. I welcome suggestions on how to improve it. Like most
engineering effort, the goal is not to find a perfect solution (which
probably doesn't exist), but to maximize the return on our investment.
Cheers,
Zhanyong
Vlad Losev
Well, if it comes to that I would rather Google Mock to be another one: we can require the use of Google Test or the use of exceptions. Otherwise we may be making a really naughty present to our users: making them debug failures induced by the crashes Google Test hides.
Chandler Carruth
> Unless we can enforce that Google Test assertions cannot be used
> outside of Google Test tests (which I don't think we can do without a
> lot of effort), it's a real danger that a Google Mock user would
> inadvertently use them and end up with tests passing that should fail.
> False positives are a big no-no for testing frameworks, so this is a
> big problem.
>
> Allowing Google Test assertions also means that the user doesn't have
> to wonder why EXPECT_CALL works while other EXPECT_* macros don't.
Ok, I see where you're coming from, but I'd like to find a middle
ground. Specifically, I agree that the worst possible thing are false
positives, and I'm fine with the internal GoogleTest expectations and
assertions throw exceptions in the event of errors without a validly
registered failure reporting interface. I disagree with Vlad in that I
think this is a perfectly acceptable degradation policy, and will even
provide a stacktrace to the precise failure and output describing it.
Not a big deal.
Now, as a separate issue, I'd like to find a way to not advertise the
GoogleTest stuff that GoogleMock uses because we may want to make it
truly internal at a later date. I understand this may be hard, and I'm
not trying to get it done for this project, which should be about
low-hanging fruit. I just don't want people to start using and thus
depending on aspects of GoogleTest when they're actually using
GoogleMock within some other testing framework. Is that an acceptable
goal?
Consider that it will increasingly tie the projects, and may actively
degrade the testing quality for users as well. It then becomes murky
what happens when they turn GoogleMock off, because they will suddenly
lose the ability to make GoogleTest assertions. They will be impacted
by any change to the behavior of the GoogleTest assertions, despite
not using GoogleTest. The list goes on. I think that the separation
can eventually be made, and would be good to make, so I'm just worried
about people building up dependencies prior to that separation.
-Chandler
Zhanyong Wan (λx.x x)
>> > I'm OK with using C++ exceptions to signal failures. But when the
>> > exceptions are disabled, the only option to report a failure is to crash
>> > the
>> > program. I think this is too drastic. I am suspicious of the idea of
>> > intercepting a crash and continuing the program execution. For all you
>> > know,
>> > the program's state may be now corrupted and cannot be trusted. The only
>> > prudent way to go in such situation is to clean up and terminate the
>> > program
>> > ASAP. The alternatives may include infinite loops, stack overflows, etc.
>>
>> I acknowledged this limitation in my original post. This is
>> definitely not ideal, but I don't think it's a show stopper,
>> especially since that *most* C++ testing frameworks require exceptions
>> to be enabled.
>
> Well, if it comes to that I would rather Google Mock to be another one: we
> can require the use of Google Test or the use of exceptions. Otherwise we
> may be making a really naughty present to our users: making them debug
> failures induced by the crashes Google Test hides.
I don't see why we want to lose the users who need to disable
exceptions.
Let's quantify the analysis: only a tiny fraction of the users will
see this degraded behavior (crashing on failure). If they use Google
Test as their main testing framework, they won't see it. If they use
another framework that requires exceptions (the majority of the case),
they won't see it. If they use a framework that doesn't require
exceptions, but their project doesn't disable exceptions, they still
won't see it. I don't think this is something to lose sleep over.
Cheers,
--
Zhanyong
Vlad Losev
Then why bother with the flag at all? If Google Test is used as the primary test framework then use its native reporting capabilities. If it is not used as such, require the exceptions. This covers all three cases you mentioned. Do not support the rest of the configurations since they are going to be such a tiny minority. My concern is that by supporting such configuration with the suggested hack will set us up for troubleshooting related user problems in the future.
Cheers,
--
Zhanyong
Zhanyong Wan (λx.x x)
How does Google Test reliably know whether it is used as the primary
testing framework? I think this knowledge is best provided by the
user, and the flag is the channel.
> This covers all three cases you mentioned.
> Do not support the rest of the configurations since they are going to be
> such a tiny minority. My concern is that by supporting such configuration
> with the suggested hack will set us up for troubleshooting related user
> problems in the future.
This is a general, supported, solution, so by definition it's not a hack. :-)
We will always be helping the users troubleshoot. If not this, then
something else. Why do you think that this one is more concerning?
Thanks,
>
>>
>> Cheers,
>>
>> --
>> Zhanyong
>
>
--
Zhanyong
Chandler Carruth
I'm not following this reasoning. You would actually have to write
more code to disable non-exception support explicitly. Throwing an
exception when they are turned off, by default, without any extra
steps, does what Zhanyong is proposing. Exceptions themselves are what
provide this behavior. Why is this a bad thing?
-Chandler
>
>>
>> Cheers,
>>
>> --
>> Zhanyong
>
>
Vlad Losev
When the exceptions are disabled, Zhanyong is proposing to crash the program by initiating a segfault on Linux or GPF on Windows. My position is that for the test runner up the stack this is indistinguishable from really bad situations and the program should not continue after this. If the crash was in the user code, the program state may be corrupted and an attempt to continue can produce undefined behavior.
-Chandler
>
>>
>> Cheers,
>>
>> --
>> Zhanyong
>
>
Vlad Losev
You once told me that a hack is something that works under some specific circumstances but not others. By that definition I deem it a hack. :-P
We will always be helping the users troubleshoot. If not this, then
something else. Why do you think that this one is more concerning?
Why troubleshoot something that we don't have to?
Zhanyong Wan (λx.x x)
> On Wed, Jan 7, 2009 at 11:53 AM, Zhanyong Wan (λx.x x) <w...@google.com> wrote:
>> Unless we can enforce that Google Test assertions cannot be used
>> outside of Google Test tests (which I don't think we can do without a
>> lot of effort), it's a real danger that a Google Mock user would
>> inadvertently use them and end up with tests passing that should fail.
>> False positives are a big no-no for testing frameworks, so this is a
>> big problem.
>>
>> Allowing Google Test assertions also means that the user doesn't have
>> to wonder why EXPECT_CALL works while other EXPECT_* macros don't.
>
> Ok, I see where you're coming from, but I'd like to find a middle
> ground. Specifically, I agree that the worst possible thing are false
> positives, and I'm fine with the internal GoogleTest expectations and
> assertions throw exceptions in the event of errors without a validly
> registered failure reporting interface. I disagree with Vlad in that I
> think this is a perfectly acceptable degradation policy, and will even
> provide a stacktrace to the precise failure and output describing it.
> Not a big deal.
Glad that we've made progress!
> Now, as a separate issue, I'd like to find a way to not advertise the
> GoogleTest stuff that GoogleMock uses because we may want to make it
> truly internal at a later date. I understand this may be hard, and I'm
> not trying to get it done for this project, which should be about
> low-hanging fruit. I just don't want people to start using and thus
> depending on aspects of GoogleTest when they're actually using
> GoogleMock within some other testing framework. Is that an acceptable
> goal?
This was my original vision too. However, after thinking more about
the logistics and the benefit-cost-ratio, I'm less thrilled about it
now.
Obviously it's bad to ask the users to depend on Google Mock's
implementation details - no argument there. The problem can be solved
by making it clear to the users that:
- Google Test can be used as the primary testing framework, or you can
set --gtest_throw_on_failure and use its assertions with other
testing frameworks.
- Google Mock comes with a bundled copy of Google Test. That copy
should be viewed as a sidekick for Google Mock, and NOT an internal
implementation detail.
- When using Google Mock, it's fine to also use the bundled Google
Test, either bits of it or the entirety.
As long as we make people aware that they are depending on Google Test
when they use Google Test assertions + Google Mock + another testing
framework, we should be fine.
> Consider that it will increasingly tie the projects,
I think the key is to make sure people understand that they are
depending on Google Test when they use bits of it. I don't think this
is particularly hard to achieve.
> and may actively
> degrade the testing quality for users as well.
Can you elaborate on this?
> It then becomes murky
> what happens when they turn GoogleMock off, because they will suddenly
> lose the ability to make GoogleTest assertions.
Not really, they can turn off Google Mock and continue to use Google
Test assertions.
> They will be impacted
> by any change to the behavior of the GoogleTest assertions, despite
> not using GoogleTest. The list goes on.
We need to make them aware that they are using Google Test.
> I think that the separation
> can eventually be made, and would be good to make, so I'm just worried
> about people building up dependencies prior to that separation.
When we make that separation eventually, Google Mock + Google Test
assertions + another testing framework will still be a valid
combination, so people won't lose their investment.
I hope I don't sound defensive. Just trying to explain why I think
your concerns aren't insurmountable. After considering all the
issues, I still dislike the gtest_throw_on_failure flag the least, for
the reasons I have listed. :-) That doesn't mean I'm set on the idea
though - I'm still open to suggestions on how to improve it and better
alternatives.
Thanks,
>
> -Chandler
--
Zhanyong
Zhanyong Wan (λx.x x)
I plan to work on this soon, unless someone comes up with new
arguments to convince me that I shouldn't do it. If you feel strongly
about it, please speak up. Thanks!
--
Zhanyong
Vlad Losev
Hi,my quip with gmock is that in the case where a default action is not available, the gmock framework aborts the entire application with the dreaded message:"The mock function has no default action set, and its return type has no default value set".my desired outcome is for the specific test to fail, and that the test application will continue to run the other tests.I think I can implement my desired behavior if I could replace the default FailureReporterInterface with my own implementation, which throws exception,and if I understand correctly, this is exactly what this thread is discussing ...is there a patch available that allows replacing the default FailureReporterInterface instance?
aviad rozenhek
Sent from my iPhone
aviad rozenhek
On Wednesday, September 5, 2012 11:23:31 PM UTC+3, Vlad Losev wrote:
On Sat, Aug 25, 2012 at 8:39 AM, aviad rozenhek <avi...@gmail.com> wrote:Hi,my quip with gmock is that in the case where a default action is not available, the gmock framework aborts the entire application with the dreaded message:"The mock function has no default action set, and its return type has no default value set".my desired outcome is for the specific test to fail, and that the test application will continue to run the other tests.I think I can implement my desired behavior if I could replace the default FailureReporterInterface with my own implementation, which throws exception,and if I understand correctly, this is exactly what this thread is discussing ...is there a patch available that allows replacing the default FailureReporterInterface instance?
What you face has little to do with reporting errors. If you mock a method, Google Mock will provide a default implementation. If that method returns a type, Google Mock's implementation needs to return something. Google Mock calls that type's default value. It defines default values for C++ built-in types, strings, and pointers. But if it's a user-defined type, Google Mock has no idea of what its default value should be. So you have to provide the default value for the types you define. If you don't, and Google Mock encounters the situation where it has to return a value of such a type, it doesn't know how to proceed and all it can do is to stop the program.
Zhanyong Wan (λx.x x)
support from the compiler, so it cannot throw by default. However, if
we make it generate an ASSERT_* failure before aborting the process, a
user can set ::testing::GTEST_FLAG(throw_on_failure) to true to cause
an exception to be thrown in this case.
http://code.google.com/p/googletest/wiki/AdvancedGuide#Letting_Another_Testing_Framework_Drive
Are you interested in contributing a patch, Aviad?
--
Zhanyong
Zhanyong Wan (λx.x x)
that, so no patch is required.
> user can set ::testing::GTEST_FLAG(throw_on_failure) to true to cause
> an exception to be thrown in this case.
> http://code.google.com/p/googletest/wiki/AdvancedGuide#Letting_Another_Testing_Framework_Drive
>
> Are you interested in contributing a patch, Aviad?
>
> --
> Zhanyong
Zhanyong
Chris F
-Chris
Zhanyong Wan (λx.x x)
On Mon, Jan 14, 2013 at 9:37 AM, Chris F <zil...@gmail.com> wrote:
> Hi Zhanyong,
>
> I've posted on the thread, but thought I'd ask you directly since we're
> pressed for time. No problem if you ignore it since you're probably busy,
> and don't have time to answer emails like this, but I thought I'd try.
>
> I was able to fit Googlemock in CppUnit, but my failing Googlemock tests are
> NOT counted in the CppUnit test framework, that is, even though I see my
> failures in the console output, my test runner (QtTestRunner or
> UnitTestRunner) still show 0 errors. How should I manifest my G-mock errors
> in CppUnit?
See http://code.google.com/p/googletest/wiki/AdvancedGuide#Letting_Another_Testing_Framework_Drive
If you use googlemock or googletest with CppUnit, you need to turn on
throw_on_failure and define/invoke the test methods in CppUnit's way.
You cannot use googletest's TEST() and RUN_ALL_TESTS() macros, as they
will handle test failures themselves instead of passing them to
CppUnit as you want. Inside the test methods you define, you can use
googlemock and googletest constructs (e.g. EXPECT_CALL, EXPECT_EQ)
without problems.
I tried the following and it works correctly (the failure was reported
and the program exits with a non-0 code):
#include "third_party/gtest/include/gtest/gtest.h"
void Test1() {
FAIL();
}
void Test2() {
EXPECT_EQ(1, 2);
}
int main(int argc, char **argv) {
::testing::GTEST_FLAG(throw_on_failure) = true;
testing::InitGoogleTest(&argc, argv);
Test1();
Test2();
return 0;
}
>
> -Chris
>
> On Tuesday, January 6, 2009 7:40:17 PM UTC-5, one wrote:
>>
>> Greetings,
>>
>> I heard from some people that they want to use Google Mock, but they
>> are using some C++ testing framework other than Google Test, and for
>> various reasons they cannot switch to Google Test yet. Since Google
>> Mock only works with Google Test now, they have to skip it.
>>
>> We have planned to open up Google Mock more such that you can use it
>> with any C++ testing framework of your choice. Here's the background
>> and how I propose to do it:
>>
>> - There are tons of C++ testing frameworks out there. It's impossible
>> to adapt Google Mock for each of them. Even if we only target the
>> most popular ones, it would still be a lot of work, and people will
>> keep asking us when their favorite frameworks will be supported.
>>
>> - We could define an interface that each testing framework needs to
>> implement in order to hook up with Google Mock, and ask the users to
>> implement this interface themselves. However, this just moves work
>> around, and makes the barrier of entry pretty high. And we'll be
>> busy incorporating patches the users submitted for their favorite
>> frameworks.
>>
Zhanyong Wan (λx.x x)
Sorry for the long delay. You're right: the throw_on_failure flag
doesn't solve the problem as the exception will still kill the
program.
I think we should make gmock throw a runtime_error if no default
action is available *and* exceptions are enabled. That runtime_error
will be caught by gtest and properly reported, and the next test
method can continue to run. I verified that this works.
I'll see if I can get this done soon. Thanks,
On Sun, Sep 23, 2012 at 8:40 AM, joey <jo...@rayv.com> wrote:
> the error is not reported as a test error because it is caught in the
> following code in gtest.cc:2127
>
> } catch (const GoogleTestFailureException&) { // NOLINT
> // This exception doesn't originate in code under test. It makes no
> // sense to report it as a test failure.
> throw;
Zhanyong
Zhanyong Wan (λx.x x)
Zhanyong