Using asserts in test code

[Luke Plant]

Hi all,

This is a question of test code style.

In tests in my own projects, I use both the Python 'assert' statement
and the unittest TestCase.assert* methods, with the following
distinction:

* The 'assert' statement is used to make assertions about the
assumptions built in to *this* bit of code (i.e. the test itself).

* The TestCase.assert* methods are used to make assertions about the
outcome of the bit of code that is being tested.

This is consistent with the use of the assert statement in other code
and makes lots of sense to me. When reading tests it helps me see
straight away what are the assumptions and sanity checks in the test,
and what is actually being tested. It also has the advantage that if you
have some setup code in a test, and it has assert statements in it, it
can be moved out of a TestCase method without being altered.

Looking at Django's test suite, I can find a few instances where this
pattern is followed (e.g. [1], [2]). Sometimes the assert statement is
also used where it is impractical to call TestCase methods (e.g. [3]).
There are a few instances where the 'assert' statement is called when it
should be a TestCase method (e.g. [4]). And there are many instances
where TestCase asserts are called, when IMO it should be an assert
statement. (e.g. [5]).

So, the question is, going forward do we:

1) Use the pattern I outlined above?
2) Use a different pattern (e.g. always use TestCase assert methods
where it is possible)?
3) Simply not care about any of this?

My preference is to encourage 1). With existing tests, I would be
tempted to change the handful of instances which use the assert
statement when it should be TestCase assert method, but I wouldn't
bother with the many instances which are the other way around, unless I
was in the area.

Regards,

Luke

[1]
http://code.djangoproject.com/browser/django/trunk/tests/regressiontests/templates/tests.py#L347
[2]
http://code.djangoproject.com/browser/django/trunk/tests/regressiontests/utils/simplelazyobject.py#L70
[3]
http://code.djangoproject.com/browser/django/trunk/tests/regressiontests/model_forms_regress/models.py#L37
[4]
http://code.djangoproject.com/browser/django/trunk/tests/modeltests/model_forms/tests.py#L35
[5]
http://code.djangoproject.com/browser/django/trunk/tests/modeltests/serializers/tests.py#L69

--
"Smoking cures weight problems...eventually..." (Steven Wright)

Luke Plant || http://lukeplant.me.uk/

[Ian Clelland]

On Thu, Dec 23, 2010 at 5:44 AM, Luke Plant <L.Pla...@cantab.net> wrote:
> Hi all,
>
> This is a question of test code style.
>
> In tests in my own projects, I use both the Python 'assert' statement
> and the unittest TestCase.assert* methods, with the following
> distinction:
>
> * The 'assert' statement is used to make assertions about the
> assumptions built in to *this* bit of code (i.e. the test itself).
>
> * The TestCase.assert* methods are used to make assertions about the
> outcome of the bit of code that is being tested.

I like this pattern, at least as a notational style -- it is helpful
to know when your tests are failing because they are making incorrect
assumptions, rather than your application code being incorrect.

The danger, of course, is that a failing 'assert' in your test setup
could still be a bug in your code, rather than in the tests, and if
you blindly fix the "bug" in the tests, then you end up solidifying a
real bug in the code. (If your unit tests are factored properly,
though, then the assertions you make about your test should themselves
be tested in another location, so you should be able to rely on them.)

> Looking at Django's test suite, I can find a few instances where this
> pattern is followed (e.g. [1], [2]). Sometimes the assert statement is
> also used where it is impractical to call TestCase methods (e.g. [3]).
> There are a few instances where the 'assert' statement is called when it
> should be a TestCase method (e.g. [4]). And there are many instances
> where TestCase asserts are called, when IMO it should be an assert
> statement. (e.g. [5]).

From a technical standpoint, I don't know if I see a real difference
-- using "self.assert_(expr)" in a TestCase should never be
impractical -- at least not more than calling "assert" directly.
unittest.TestCase.assert_ just does the test manually, and raises
AssertionError if it is passed anything false.

The only difference between the two should be the fact that the raw
"assert" can be optimized away. You'd have to call 'python -O
manage.py test' deliberately for that, and I don't know why you'd ever
want to optimize away parts of your test code.

> So, the question is, going forward do we:
>
> 1) Use the pattern I outlined above?
> 2) Use a different pattern (e.g. always use TestCase assert methods
> where it is possible)?
> 3) Simply not care about any of this?

I'm +1 for #1, for consistency and clarity, but not for any technical reasons.

--
Regards,
Ian Clelland
<clel...@gmail.com>

[Karen Tracey]

On Thu, Dec 23, 2010 at 8:44 AM, Luke Plant <L.Pla...@cantab.net> wrote:

Hi all,

This is a question of test code style.

In tests in my own projects, I use both the Python 'assert' statement
and the unittest TestCase.assert* methods, with the following
distinction:

* The 'assert' statement is used to make assertions about the
assumptions built in to *this* bit of code (i.e. the test itself).

Perhaps it is not a problem when using bare asserts for this purpose, but I have grown to dislike asserts in testcases because they give so little information. For example, I am working with test code that uses bare asserts all over the place instead of the TestCase methods and when these tests fail I get errors like this:

Traceback (most recent call last):
  File [snipped]
    assert res.status_code == 200
AssertionError

Well, great, status wasn't 200 but what was it? That's information I'd really rather get by default, so I much prefer assertEquals, which tells me:

Traceback (most recent call last):
  File [snipped]
    self.assertEqual(res.status_code, 202)
AssertionError: 302 != 200

That actually gives me enough information that I might be able to fix the problem straight off.

Do you not find the paucity of information provided by bare assert to be annoying?

Karen

[Ian Clelland]

Test cases should probably be using the two-argument form of assert:

assert (res.status_code == 200), "The status code returned was incorrect"

or even

assert (res.status_code == 200), ("%s != 200" % res.status_code)

At least some of the examples that Luke pointed to are using that
style, and it definitely makes more sense than just asserting a bare
fact, with no explanation.

Regards,
Ian Clelland
<clel...@gmail.com>

[Dave Smith]

My team has adopted the convention of prepending "Sanity:" to the message of
any assertion whose purpose is to verify that things are set up correctly for the
'main act' assertions. This helps us cut through the 'something is broken, but it's
not here' noise  when a change causes a bunch of tests to fail. This works equally
well with 2-arg assert or assertEquals() et al.

Dave

Regards,
Ian Clelland
<clel...@gmail.com>

--

You received this message because you are subscribed to the Google Groups "Django developers" group.
To post to this group, send email to django-d...@googlegroups.com.
To unsubscribe from this group, send email to django-develop...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.

[Łukasz Rekucki]

Now imagine you have to do that 20 times using different operators (
==, <, >, is, in). The second thing is exactly what assertEqual() does
(at least for integers), so why duplicate it ? OTOH, it's a shame
unittest doesn't let you add a message prefix (at least I couldn't
find it), only replaces the default.

--
Łukasz Rekucki

[Luke Plant]

On Thu, 2010-12-23 at 11:46 -0500, Karen Tracey wrote:
> Well, great, status wasn't 200 but what was it? That's information I'd
> really rather get by default, so I much prefer assertEquals, which
> tells me:
>
> Traceback (most recent call last):
> File [snipped]
> self.assertEqual(res.status_code, 202)
> AssertionError: 302 != 200
>
> That actually gives me enough information that I might be able to fix
> the problem straight off.
>
> Do you not find the paucity of information provided by bare assert to
> be annoying?

Yes, I agree that in those cases bare asserts are a pain. So perhaps the
assert statement is only a good idea when 1) you are documenting an
assumption in the test code, 2) there is no TestCase.assert* method that
could be easily used to provide a more helpful error.

Luke

[Will Hardy]

Maybe we could add a keyword argument to Django's TestCase.assert*() methods that raises an AssertionError instead of failing the test when the method's condition is not met (I assume that the test should error instead of failing).

eg:

self.assertEqual(1, 1, test_assumption=True)

Cheers,

Will

[Alex Gaynor]

--

You received this message because you are subscribed to the Google Groups "Django developers" group.
To post to this group, send email to django-d...@googlegroups.com.
To unsubscribe from this group, send email to django-develop...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.

Maybe I'm doing testing fundamentally wrong but i don't really care if a test failed or had an exception, I treat them the same, some bug needs to be fixed.  To that end I prefer writing everything using the assert* methods on TestCase, they give more informative error messages.

Alex

--
"I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero
"Code can always be simpler than you think, but never as simple as you want" -- Me

[Russell Keith-Magee]

On 25/12/2010, at 3:33 AM, Alex Gaynor <alex....@gmail.com> wrote:

On Fri, Dec 24, 2010 at 12:56 PM, Will Hardy <e.wil...@gmail.com> wrote:

Maybe we could add a keyword argument to Django's TestCase.assert*() methods that raises an AssertionError instead of failing the test when the method's condition is not met (I assume that the test should error instead of failing).

eg:

self.assertEqual(1, 1, test_assumption=True)

Cheers,

Will

--
You received this message because you are subscribed to the Google Groups "Django developers" group.
To post to this group, send email to django-d...@googlegroups.com.
To unsubscribe from this group, send email to django-develop...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.

Maybe I'm doing testing fundamentally wrong but i don't really care if a test failed or had an exception, I treat them the same, some bug needs to be fixed.  To that end I prefer writing everything using the assert* methods on TestCase, they give more informative error messages.

I can see what Luke is driving at; there is a significant difference between asserting test preconditions and asserting test conditions. I have usually resorted to simply documenting that a test is a precondition, rather than attempting to use a less capable API. 

There is also a significant difference between Error and Fail. In theory, an error *should* be able to tell you that something unexpected has changed in your test environment (such as a preconditon change).

The problem is that there isn't a rich API for raising errors, and as a result, the Error/Fail distinction isn't as helpful as it should be. If such an API existed, I'd be all in favor of using it. However, it doesn't exist (at least, to my knowledge).

In short, I suppose I've got a case of Meh. Yes, there is a distinction to be made, but the tools don't exist to easily make that distinction, and I don't see it as Django's role to start reshaping test methodologies and building those capabilities as part of Django's test tools. I'd rather stick to documenting the distinction in situ when appropriate or helpful.

Yours,

Russ Magee %-)

[Will Hardy]

It's true that there isn't any rich API for the Error/Fail distinction. The best I can think of would be to create a custom exception eg FailedAssumption, which is raised when the TestCase.failedException (same as AssertionError) is caught while testing an assumption.

Here are four ways I can think of implementing the above, each with varying convenience/aesthetics. A few of these can be added to django.utils.unittest.TestCase for convenience, or provided as separate functions/classes to prevent any potential backwards incompatibility or to just keep TestCase clean.

1. plain old try/except in the test code:

    try:

        self.assertEqual(2,3)

    except self.failedException, e:

        raise FailedAssumption(e.message)

2. wrap tests self.assert* in a special function/method

    self.assumption(self.assertEqual, 2, 3)

    where the "method looks something like this:

    def assumption(self, func, *args, **kwargs):

        try:

            func(*args, **kwargs)

        except self.failureException, e:

            raise FailedAssumption(e.message)

3. wrap a series of test assumptions with state changing methods:

    self.start_assumption_tests()

    self.assertEqual(2, 3)

    self.finish_assumption_tests()

    where the methods look something like this:

    def start_assumption_tests(self):

        self._failureException = self.failureException

        self.failureException = FailedAssumption

    def finish_assumption_tests(self):

        self.failureException = self._failureException

4. use a fancy py2.5 "with" syntax:

    with self.assumptions():

        self.assertEqual(2,3)

    where the start_ and finish_ methods in the previous incarnation are __enter__ and __exit__

(NB none of these actually have to be in the TestCase class)

Cheers,

Will

[Alex Gaynor]

--

You received this message because you are subscribed to the Google Groups "Django developers" group.
To post to this group, send email to django-d...@googlegroups.com.
To unsubscribe from this group, send email to django-develop...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.

It seems to me that all of these suffer from a problem, over engineering what should really be "# If this fails we have problems". ;)