Failing Tests on Uncaught Exceptions, automatically?

[Andreas Zoufal]

Hi folks,

I'm using CppUTest for a while now, and recently I recognized that tests doesn't fail if my code under test throws an exception. To clarify this, the exceptions is NOT expected, therefore no CHECK_THROWS() is used. It is an real error in program logic, typically invalid arguments to function calls or so.

The test application terminates in this case, and to avid this and to show the test from the exception I'm wrapping everything in main() into a try/catch block, printing out the test on a caught exception.

This way I get some information for the exception, but I don't see in which test case the exception was thrown. Of course I can wrap every test case in a try/catch block using FAIL() on catching, but I wonder, isn't there a more general feature available? Or is it easy to implement a test plugin or similar for this?

Kind regards,

Andi

[Bas Vodde]

Hi Andi,

*grin*

Thats exactly why the exceptions aren’t caught. When you let the exception through and let it crash and do a backtrace then that is AFAIK the easiest way to figure out where and how it crashed. After catching the exception, it is very hard to figure out where it came from (not even know if that is possible :( )

Bas

> --
> You received this message because you are subscribed to the Google Groups "cpputest" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to cpputest+u...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.

[Andreas Zoufal]

Hi Bas,

 

Ok, I get your point. This makes sense for debugging. In my current project (Visual Studio 2010) I run the test application automatically as a post-build step, with the minimum output in the output console. on failing tests I get the error message and can simply click the message to go to the failing test. A crashing application because of uncaught exceptions doesn't help much in that case. I have either to fire up the debugger or to run the test from console again with -v option.

 

I'm thinking about some option like the one in the mocking extensions (mock().crashOnFailure()). I'm not sure if it is technically possible to catch all exceptions an re-throw them after writing some error output? With an simple example like blow it seems to work... But anyway, it will not be worth if it breaks some rules or would be too complicated ;)

 

TEST(SomeTest, UncaughtException)
{
  try {
    throw std::runtime_error("testing...");
  } catch (const std::exception& e) {

    std::cout << "*** Exception caught in "<< __FUNCTION__ << ": '" << e.what() << "'" << std::endl;

    throw(e);
  }
}

[Steve Hill]

Hi,

Just to reopen an old post, I've just encountered this issue. When running a test that throws an exception (not an expected exception), the test suite crashes and Windows pops up a dialog. Were someone to commit some code that broke a test in this way, this will be a problem for our regression builds (since these are unattended and, in some cases, on headless machines where the dialog could go unnoticed for a long time). Could CppUTest be extended to catch any unhandled exceptions and cause the test to fail gracefully in that case? I understand the point about debugging so it would seem sensible to only enable this if required (preferably at compile time).

Thanks,

S.

[Bas Vodde]

Hi Steve,

Ok, that sounds like a good reason. Could you raise an issue on github for this?

Thanks!

Bas

--
You received this message because you are subscribed to the Google Groups "cpputest" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cpputest+u...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Steve Hill]

Hi Bas,

Thanks for the reply. I will do!

S.

[A. Robert S.]

Hi Steve,

This issue is more tricky than it might seem. What you are asking for only makes sense for exceptions that allow the code to continue executing safely.

This is the case with (non-signalling) floating point exeptiosn. I wrote a plugin that does pretty much what you want with floating point "exceptions". If they signal, even in this case, the code cannot continue and has to crash.

Assuming it was safe to continue (or you could somehow ensure it was), you would need a try...catch block inside your test. You can't have a "try", say, in your setup, and then catch it in your teardown (which pretty much rules out implementing this as a plugin).

Still you could have something like this:

TEST(catch, catchmeifyoucan)
{
    try
    {
        function_to_test();
    }
    catch(...)
    {
        FAIL("Unexpected exception");
    }
}

This would allow the tests to continue if and only if that particular exception leaves the system in a state that's safe to continue from.

I don't see how this could be implemented easily in CppUTest though. I also don't think an "unexpected" exception can be assumed to leave the system in a safe state.

[A. Robert S.]

Hi again,

Actually, come to think of it, CppUtest already contains such a feature:

mytestswithpossibleexceptions.exe -p

This will run each test in its own process and report failures for tests that threw an exception. It is safe, because the process that ran into the exception is terminated as it should be.

Regards,
Robert

On Tuesday, April 5, 2016 at 11:30:26 AM UTC+2, Steve Hill wrote:

[A. Robert S.]

Here is a minimal example:

#include "CppUTest/CommandLineTestRunner.h"
#include "CppUTest/TestHarness.h"
#include "CppUTest/TestRegistry.h"

TEST_GROUP(catch)
{
};

TEST(catch, catchHello)
{
    throw "hello";
}

TEST(catch, goodTest)
{
}

TEST(catch, anotherThrower)
{
    throw "world";
}

int main(int argc, char** argv)
{
    TestRegistry::getCurrentRegistry()->setRunTestsInSeperateProcess();
    return RUN_ALL_TESTS(argc, argv);
}

This will produce the following output:

./OneTimeThrowAwayTests.exe
terminate called after throwing an instance of 'char const*'

OneTimeThrowAwayTests.cpp:18: error: Failure in TEST(catch, anotherThrower)
        Failed in separate process - killed by signal 6

..terminate called after throwing an instance of 'char const*'

OneTimeThrowAwayTests.cpp:9: error: Failure in TEST(catch, catchHello)
        Failed in separate process - killed by signal 6

.
Errors (2 failures, 3 tests, 0 ran, 0 checks, 0 ignored, 0 filtered out, 202 ms)

Note that only the failure and number of tests statistics are available. How many tests ran, how many checks, etc. is lost to the child process.

Regards,
Robert

P.S. I'd always recommend using -p instead of hard coding this into main(), but for the minimal example, you can run it just as is with no extra parameters.

P.P.S Using a TestTestingFixture and letting it run the test in a separate process keeps the statistics intact. This is more useful for situations where a crash is expected and should occur.

Regards,
Robert

On Tuesday, April 5, 2016 at 11:30:26 AM UTC+2, Steve Hill wrote:

[Steve Hill]

I've tried that but I get:

     -p doesn't work on Visual C++ as it is lacking fork.

However, I have found half a fix. If the main, I can do this which (as it calls exit rather than abort) doesn't allow the JIT debugger, etc. to get in the way:

static void TestTerminate()
{
   exit(EXIT_FAILURE);
}

int main(int ac, char** av)
{
   set_terminate(TestTerminate);

   return CommandLineTestRunner::RunAllTests(ac, av);
}

What I would like to do, however, is to output at least that the test that has caused the process to terminate. Is it possible to get this information?

S.

[Bas Vodde]

Hi Robert,

Can’t we have a catch all sttement in the try-catch around the test?

Then it does:

  if (canCrashDueExceptions) throw;

So, it rethrows the exception if this options is off, so you can run-time turn on/off this.

Or am I missing something important?

Thanks,

Bas

--
You received this message because you are subscribed to the Google Groups "cpputest" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cpputest+u...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[A. Robert S.]

Hi,

@Steve that's right, Visual C++ doesn't support fork(). Cygwin does.

@Bas, yes, it would then have to be around the entire test. Not exactly sure where in the code that would have to be. It mustn't catch exceptions due to test termination (failure).

I am really not sure that this would be a good idea, because we have no idea what exception might have occured, and it may not be safe to continue.

Also, what should happen when we are using setjmp / longjmp ? #ifdef ?

Another route for Steve to go would be running each test on its own with CTest, or any other script that first outputs the names of all test and then runs each test individually, based on the names. It's a bit of overhead at runtime, but probably not much more than using -p.

Regards,
Robert

[Steve Hill]

Thanks for the discussion around my issue. I think that I've now got a workable solution (though I can still raise the issue on github, if you would like).
In my main, I sniff the command line switches and, if I find "--debug", I remove it and run the test without doing anything else; this fulfils the interactive debugging use-case.
If I don't find "--debug", I set the terminate function, as above, so that it outputs the following on stderr:

ERROR: Terminating unit-test suite - perhaps due to an unhandled exception?
        Try running with --debug to investigate

That solves the automated use-case. Hence, both my use-cases are covered. I don't know whether this is something that is worth building into CppUTest or not; I'll leave that for you guys to decide.

Thanks again for all the discussion around this, which has resulted in my being able to resolve the issue.

S.

[Bas Vodde]

Hi,

@Bas, yes, it would then have to be around the entire test. Not exactly sure where in the code that would have to be. It mustn't catch exceptions due to test termination (failure).

I am really not sure that this would be a good idea, because we have no idea what exception might have occured, and it may not be safe to continue.

It would be right under the current catch. Just add a new catch block to that.

Also, what should happen when we are using setjmp / longjmp ? #ifdef ?

Too bad :)

Bas

[Bas Vodde]

Hi,

I still think it would be useful to add. I can see how in automated test run cases, a crash wouldn’t be preferred.

Bas