exception handling

[deuberger]

Is there a way to make Google Test handle exceptions by default? I
don't really want to clutter up my testing code with a bunch of
try...catch blocks just to see the basic exception information.

I did find this bug that looks like it may be intended to solve the
problem:

http://code.google.com/p/googletest/issues/detail?id=265

Is that correct? Do you know when that will make it to a release
version? Will it do anything to handle segfaults rather than just
aborting everything?

Thanks for the help; I'm rather new to Google Test and unit testing in
C++, so forgive me if I've overlooked something obvious.

- Daniel

[Steve Jaffe]

A workaround I'm using is to wrap the entire test block in try/catch, and in the
catch report the failure (using gtest FAIL() macro). And I've wrapped my
try/catch in my own macros, so it looks like:

TEST( "MyTestGroup", "MyTest" )
BEGINTEST
{
//test body
}
ENDTEST

Perhaps not ideal, but it works.

As for segfaults, you could probably install a signal handler which could then
report failure to gtest, but I haven't tried this.

[Zhanyong Wan (λx.x x)]

On Wed, Sep 8, 2010 at 9:01 AM, deuberger <daniel.n...@gmail.com> wrote:
> Is there a way to make Google Test handle exceptions by default?  I
> don't really want to clutter up my testing code with a bunch of
> try...catch blocks just to see the basic exception information.
>
> I did find this bug that looks like it may be intended to solve the
> problem:
>
> http://code.google.com/p/googletest/issues/detail?id=265

It's tracked by
http://code.google.com/p/googletest/issues/detail?id=313, which I just
created.

The issue depends on
http://code.google.com/p/googletest/issues/detail?id=44 and
http://code.google.com/p/googletest/issues/detail?id=312. The former
has just been fixed in the trunk, and Vlad will be working on the
latter soon.

> Is that correct?  Do you know when that will make it to a release
> version?

You can check out the latest revision from the SVN trunk. It's rather
stable. The next release will probably be one month away -- that's
just my best guess.

> Will it do anything to handle segfaults rather than just
> aborting everything?

No, it doesn't try to handle segfaults. Only C++ exceptions and SEH exceptions.

>
> Thanks for the help; I'm rather new to Google Test and unit testing in
> C++, so forgive me if I've overlooked something obvious.
>
> - Daniel
>

--
Zhanyong

[deuberger]

Thanks Zhanyong and Steve. I actually used both of these solutions
and both work great. So now, Google Test will catch my exceptions by
default (assuming I flipped the flag) and I've been using macros as
Steve suggested to handle my own exceptions.

Is it possible to extend Google Test to print additional information
when handling our own exceptions? Right now it just prints: 'Unknown C
++ exception thrown in the test body.'. I basically just want it to
print the name of the function and whatever what() returns.

> As for segfaults, you could probably install a signal handler which could then
> report failure to gtest, but I haven't tried this.

Would this be the best approach? Are there any plans to make Google
Test handle segfaults? What about reasons for doing or not doing so?

Thanks!

[Zhanyong Wan (λx.x x)]

On Wed, Sep 22, 2010 at 12:28 PM, deuberger <daniel.n...@gmail.com> wrote:
> Thanks Zhanyong and Steve.  I actually used both of these solutions
> and both work great.  So now, Google Test will catch my exceptions by
> default (assuming I flipped the flag) and I've been using macros as
> Steve suggested to handle my own exceptions.
>
> Is it possible to extend Google Test to print additional information
> when handling our own exceptions?

Yes. Try derive your exception class from std::exception.

>  Right now it just prints: 'Unknown C
> ++ exception thrown in the test body.'.  I basically just want it to
> print the name of the function and whatever what() returns.
>
>> As for segfaults, you could probably install a signal handler which could then
>> report failure to gtest, but I haven't tried this.
>
> Would this be the best approach?

I don't worry much about segfaults. They will cause the test to fail.
It's not ideal that they abort the test, but that just means you have
to fix the test crash before you can see the result of the rest of the
test methods, or you can use --gtest_filter to exclude the crashing
test method. If you run into crashes a lot, you probably want to
rewrite your tests to crash less often.

>  Are there any plans to make Google
> Test handle segfaults?  What about reasons for doing or not doing so?

No, we don't plan to.

If you don't want to run the test body in a child process, it's
impossible to recover cleanly from a segfault (think about object
destructors that get skipped), so it makes no sense to continue with
other test methods when one test method segfaults. That defeats the
whole purpose of gtest handling segfaults.

If you run the test body in a child process, when it's possible to
recover cleanly. There is a proposed patch to do that exactly.
However, it adds a lot of complexity to gtest, isn't essential, and
has its gotcha too (the same ones as death tests, see the advanced
guide wiki for more info), so I prefer not to accept the patch.

--
Zhanyong

[deuberger]

> Yes.  Try derive your exception class from std::exception.

We do that, but the problem is that the default handlers from Google
Test don't print any useful information (for derived exceptions). It
just says:

unknown file: Failure
Unknown C++ exception thrown in the test body.

So I still end up adding my own handler that looks something like
this:

catch(MyException &me)
{
cout << "MyException: " << me.what() << endl;
ADD_FAILURE();
}

It seems like at least printing the return from what() would be
helpful by default. If not though, is there a way to add handlers for
classes derived from std::exception so that they same handlers can be
used in all of my tests rather than having to insert them manually
everywhere (using macros).

> No, we don't plan to.

As far as segfaults, thanks for the the info. It's helpful to
understand the reasoning.

Thanks again.

[Joey Oravec]

2010/9/23 Zhanyong Wan (λx.x x) <w...@google.com>

>> As for segfaults, you could probably install a signal handler which could then
>> report failure to gtest, but I haven't tried this.
>
> Would this be the best approach?

I don't worry much about segfaults.  They will cause the test to fail.
 It's not ideal that they abort the test, but that just means you have
to fix the test crash before you can see the result of the rest of the
test methods, or you can use --gtest_filter to exclude the crashing
test method.  If you run into crashes a lot, you probably want to
rewrite your tests to crash less often.

 

While I respect that you're not concerned with segfaults, I think crashes are common for people who do black-box testing. Imagine my boss hands me a specification and asks me to test my coworker's code. I will call into that unknown code with:

 

- Expected values, expecially min/max

- Out of range values

- Previously valid values (like a closed handle)

- Nasty strings (test buffer overflows)

- Malformed input files

- Actions repeated lots of times

- Valid byte sequences with faults randomly inserted

- ...etc

 

Imagine my job is to respond with the XML output and say "fix these problems". All of the examples above are things that he may not have considered, and typically result in a crash. In that situation it's ideal if gtest can run every test and report "crashed" for the tests that crashed. That way I can let it run all 1000+ tests overnight and get a full set of results in the morning, even if some tests crash.

 

 

If you run the test body in a child process, when it's possible to
recover cleanly.  There is a proposed patch to do that exactly.
However, it adds a lot of complexity to gtest, isn't essential, and
has its gotcha too (the same ones as death tests, see the advanced
guide wiki for more info), so I prefer not to accept the patch.

 

Sorry I've been away from the office since posting that patch. I agree your comments, especially about the complexity. However I searched through old postings and found these threads:

 

Aug 31: [googletest] Exceptions while running the tests

May 7: [googletest] run tests in a separate environment (as in process or thread)

Jun 14: [googletest] RFC: Adding a timeout for integration tests

Jul 1: [googletest] Test to avoid segfaulting

Sep 8: [googletest] exception handling

Sep 16: [chromium-dev] avoid CHECK and DCHECK in test code

 

That's a handfull of examples where people have asked how to run the entire set of tests, despite the reality that some might crash. It seems like crash-tolerance is essential for some kinds of testing, and is most effective when the feature lives inside gtest (for reasons discussed in previous postings). I don't know that the proposed separate process approach is the best solution but the topic deserves careful consideration.

 

-joey

[Keith Ray]

You might want to look at "check":

"Check is a unit testing framework for C. It features a simple
interface for defining unit tests, putting little in the way of the
developer. Tests are run in a separate address space, so Check can
catch both assertion failures and code errors that cause segmentation
faults or other signals."

http://check.sourceforge.net/

or "CUnitWin32"

"It has to work in Microsoft Windows... Tests should be independent.
That means global/static variables HAVE to be in a known state. That
means each test has to execute as a separate process."

http://code.google.com/p/cunitwin32/

2010/9/23 Joey Oravec <joeyo...@gmail.com>

--
--
C. Keith Ray
 Web: http://industriallogic.com
 Twitter: @CKeithRay, @IndustrialLogic

Amplify Your Agility
Coaching | Training | Assessment | eLearning

[deuberger]

> I don't know that the proposed separate process approach
> is the best solution but the topic deserves careful consideration.

I can see both sides of the issue here. Certainly, it would be great
if Google Test handled segfaults; however, I understand that it might
add a lot of unnecessary complexity. We're using Google Test on a
large project split into a lot of small subprojects (libraries and
processes). We opted to write separate tests for each of those and
then wrote a little script that will find and run all of those tests.
The script also outputs the JUnit style XML reports and aggregates
them into one report. Lastly, it uses the gcovr python script to
produce a Cobertura XML coverage report.

It might be helpful to have a separate test runner either part of
Google Test or as a related project (kind of like Google Mock) that
would make it easy to run lots of related tests with streamlined
reporting. That would also solve a good portion of the segfault
problem without adding any complexity to the core Google Test
framework.