EXPECT versus ASSERT in tests

[Evan Martin]

gtest provides test macros in two forms: EXPECT_foo and ASSERT_foo.
The former still reports a failure, but allows the test to continue
past that point.
The latter aborts the test if the expectation isn't held.

In theory the EXPECT form allows you to get more test output on test failure.

In practice it makes it easy to write bugs, like this one:
FilePath dir;
EXPECT_TRUE(SetupTempDirectory(&dir));
DoWorkWith(dir);
When SetupTempDirectory failed, the test continued running with an
empty directory!
This seems to have caused a bot to commit suicide today, when
DoWorkWith decided to recursively delete the directory, thinking it
was a temporary one.

I think as a general rule, we should always use the ASSERT_ forms
unless the author of the test knows exactly the consequences of
EXPECT.

Please be cognizant of this when writing and reviewing test code. I'm
also considering cleaning up some files with a sed script.

[Roger Tawa]

Hi Evan,

Your general rule is good, but keep in mind that you cannot use the
ASSERT_xxx form inside functions that do not return void. Doing so
generates compiler error C2440 (in visual studio).

Thanks,
Roger

-

> --
> Chromium Developers mailing list: chromi...@chromium.org
> View archives, change email options, or unsubscribe:
>    http://groups.google.com/a/chromium.org/group/chromium-dev
>

[Peter Kasting]

On Wed, Jul 28, 2010 at 6:30 PM, Evan Martin <ev...@chromium.org> wrote:

I think as a general rule, we should always use the ASSERT_ forms
unless the author of the test knows exactly the consequences of
EXPECT.

That seems too harsh.  It's only necessary to use ASSERT for things that have side effects (in addition to the obvious use on tests which would result in a subsequent crash if failed).

To put it differently, enforcing this rule of thumb seems no easier than enforcing "real" correct ASSERT usage.  I doubt most reviewers look closely enough at test code to catch either.  And I don't know how one would write an automated check.

PK

[Carlos Pizano]

On Jul 29, 10:46 am, Peter Kasting <pkas...@google.com> wrote:
> On Wed, Jul 28, 2010 at 6:30 PM, Evan Martin <ev...@chromium.org> wrote:
> > I think as a general rule, we should always use the ASSERT_ forms
> > unless the author of the test knows exactly the consequences of
> > EXPECT.
>
> That seems too harsh.  It's only necessary to use ASSERT for things that
> have side effects (in addition to the obvious use on tests which would
> result in a subsequent crash if failed).

Use the ASSERT form to prevent the test from crashing

MyFoo* foo = MakeFoo(testvalue);
ASSERT(NULL != foo);
EXPECT(5, foo->GimmeFive());

[Peter Kasting]

On Thu, Jul 29, 2010 at 4:35 PM, Carlos Pizano <c...@chromium.org> wrote:

On Jul 29, 10:46 am, Peter Kasting <pkas...@google.com> wrote:
> (in addition to the obvious use on tests which would
> result in a subsequent crash if failed).

Use the ASSERT form to prevent the test from crashing

MyFoo* foo = MakeFoo(testvalue);
ASSERT(NULL != foo);
EXPECT(5, foo->GimmeFive());

Yes, that is exactly what my parenthetical note was referring to.  That is the typical use of ASSERT.

PK