GTest handling of Segmentation Faults

[Jonathan Norman]

Hi there,

I am currently in the process of shifting my Unit testing duties over
to Google Test. Everything is looking good but after scanning the
features of GTest, I couldn't find anything concerning unexpected
segmentation faults or crashes. I understand that there are Death
Tests for tests that should explicitly crash, but what about tests
that unexpectedly crash? I don't want to run a Unit test and have it
trash the build machine!

Any links or resource concerning this would be great.
Cheers,
Jon

[Greg Miller]

Hi, Jonathan.

I'm not sure exactly what you want to know about seg faults or crashes, but if the unit test binary crashes for some reason it will have a non-zero exit status indicating that it crashed. Your test runner should interpret this as a failure. I'm not sure what you mean about trashing the build machine.

Thanks,
Greg

[Joey Oravec]

Jon -

I face similar problems -- when testing an unknown DLL my tests pass NULL or out of range values. They're very effective at causing a crash or corrupting the internal state of the process so badly it would affect all subsequent tests. My only option to make each test truly independent is to run each test in a separate process.

I posted a patch (against an older version of gtest) that was discussed but not accepted:  http://groups.google.com/group/googletestframework/browse_thread/thread/1d8c40cb5f876005/ef1fecb1c3f3bd46?lnk=gst&q=joey+oravec+process#ef1fecb1c3f3bd46 

This question comes up every 3-4 months on the mailing list. The common answer is "use a runner" but that means the runner must duplicate things that gtest should do best: collecting and reporting test results, running subsets of tests, etc. Maybe it's time to revisit my proposal and get this feature into core gtest code.

-joey

On Tue, Mar 27, 2012 at 6:46 AM, Jonathan Norman <jonatha...@gmail.com> wrote:

[Dirk Meister]

Any extension that would solve the problem that a crashing test
crashes the complete test run would be great.

I posted another approach last August, in which all tests are executed
independently. The test result state is transmitted so that the
framework, e.g. with recorded properties, XML output, ... works as
expected. I would have done the work if I would have known the patch
of you, Joey.

The main problem of extending an external test runner is that a lot of
work gtest does really well has to be copied. This can be seen on the
stuck(?) approach to extend the proposed external Python script by an
xml output, which is important to a lot of people.

--Dirk

[Jonathan Norman]

Hi Greg,

So my question is more focused at the use case where a unit test runs a piece of code from my runtime that then causes an unexpected program crash; a seg fault or some other such fatal error.

Since this unit testing will be fired off automatically on a build machine upon each new commit, I need to ensure that such events will be contained my GTest and that it won't cause GTest to fail, and fill up the build machine with core dumps in its death throes!

Cheers,

Jon

[Jonathan Norman]

Hi Joey,

thanks for your suggestion, I read through the patch with interest, but I think that, at the current time, it is best for me to stick with the core GTest code in the case of any other problems arising.

Do you currently use your custom patch to run your tests? If not, how do you run them? At current I am using ANT to build/compile/link my tests with GTest and then a simple <exec> task to execute them.

At the moment I am leaning towards creating a separate executable for each test suite and running them independently. At least in this case, faulty code will only wipe out a single set of test suite results

as opposed to that of all the test suites. It seems to be the easiest way to implement the running of my tests, without writing a tonne of perl and bash wrappers to ensure safe process execution and exit.

Cheers,

Jon

[Cliff Berg]

Hi Jonathan -

The problem with that approach is that it is antithetical to continuous integration, in which developers run tests locally and expect some to pass and some to fail - if the whole thing crashes, one can't get very much information about what is working and what is not, because the test run aborted at the first null pointer exception. In addition, in an CI environment, integration tests are often run as part of the CI process, and if the first unexpected failure (e.g., null pointer) causes the test run to crash, one will not have useful CI integration test runs. In general, a robust test framework will contain failures so that the test process can complete and one can see which tests pass and which fail.

Cheers,

Cliff

[Cliff Berg]

Actually, why not use the macro,

EXPECT_NO_THROW(statement)

which is defined in gtest.h? It expands to  GTEST_TEST_NO_THROW_(...), which catches any exception produced by the statement.

[Omar Shaikh]

Cliff, 

I believe that that statement cannot catch a segmentation fault. Segmentation faults are a system level thing and can cause undefined behavior. Exceptions are a lot easier to handle. I am still looking for ways to address this, but having separate executables is the way for now.