Inconsistent Django test results depending upon how the test is called in Django 1.5.1 running on Python 2.7.4

[Paul Whipp]

I have test cases that pass when tested individually, pass when the full app is tested but fail when the tests for the entire project are run:

(lsapi)~ $ django test
Creating test database for alias 'default'...
.................................................................................................................................................s.....................................................................................................................................x..........................................................................................................................................................................................................................................................Exception RuntimeError: 'maximum recursion depth exceeded' in <function remove at 0x13447d0> ignored
Exception RuntimeError: 'maximum recursion depth exceeded' in <function remove at 0x13447d0> ignored
Exception RuntimeError: 'maximum recursion depth exceeded' in <function remove at 0x13447d0> ignored
...ss........E.....................................................................................................................E.E.Setting content_object to liquid app one
.EEE.EEEE...............................................................................................................
======================================================================
ERROR: test_get_direct_notes (lsapi.tests.DirectGetTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "ls-api/lsapi/tests.py", line 869, in _method
....
  File "ls-core/lscore/model/note.py", line 37, in company
    return self.content_object.company
AttributeError: 'NoneType' object has no attribute 'company'
...


(lsapi)~ $ django test lsapi.NotesTestCase.test_wrong_user_cannot_put
Creating test database for alias 'default'...
.
----------------------------------------------------------------------
Ran 1 test in 0.241s

OK
Destroying test database for alias 'default'...


(lsapi)~ $ django test lsapi
Creating test database for alias 'default'...
...................................................ss..................................................................................................................................Setting content_object to liquid app one
...................Exception RuntimeError: 'maximum recursion depth exceeded while calling a Python object' in <_ctypes.DictRemover object at 0x46dac70> ignored
.....................................................................................................
----------------------------------------------------------------------
Ran 303 tests in 71.469s

OK (skipped=2)
Destroying test database for alias 'default'...

The 'django' command is an alias for django-admin.py. So the tests pass if run as a single case or as a test suite when testing the entire app but fail with errors if run when running the tests for all of the apps in the project.

Investigating the error in the full suite test: It is the result of an attribute being None when it 'should' have a value. The attribute is a content_object set up in a fixture.

I added a debugging setattr on the class and nothing is setting it to None so it seems that the fixture code is not being properly executed to set up the object leading to the errors. The fixture is using content types because it is for 'notes' that can be associated with various model classes:

notes.py:

...
class Note(TimeStampedModel):
    content_type = models.ForeignKey(ContentType)
    object_id = models.PositiveIntegerField()
    content_object = generic.GenericForeignKey('content_type', 'object_id')
...
    def __setattr__(self, key, value):
	if key == 'content_object':
	    if value is None:
		import pdb; pdb.set_trace()
	    print('Setting content_object to {0}'.format(value))
	return super(Note, self).__setattr__(key, value)

local_test.json:

...
 {"pk": 1, "model": "lscore.note",
  "fields": {"content_type": 16,
	     "object_id": 2,
	     "created_by": 4,
	     "text": "This is the note text"}},
...

Thinking on this further, GenericForeignKey is likely to work by accessing the content_type/object_id fields and the fixtures code will be responsible for putting these directly into the database. On this basis, I trapped the error and investigated the database at the point when it occurred in the full suite test. Sure enough, the content object has not been set in the test database when the error occurs. The above fixture works fine in the other test invocations and all of the fixtures are in a single file which is the only fixtures file specified and loaded for the test case.

Looking at the recursion errors which only occur on the full suite testing and which might conceivably have something to do with this, I added sys.setrecursionlimit(3000) to the relevant tests.py file. This eliminated the recursion errors but had no effect on the test errors described above.

So at this stage I still have a test suite that works fine when tested individually or as an app (e.g. 'django-admin.py lsapi') but which consistently fails if tested when all of the project's apps are tested together (e.g. 'django-admin.py test').

Can anyone shed any light on this or let me know how I might address the problem?

[Russell Keith-Magee]

This is an example of a really annoying edge cases that exists sometimes in Django's own test suite. Despite all our best efforts, sometimes unit tests don't clean up after themselves, so the order of execution matters. 

The test passes when it runs by itself because it gets a completely new environment when it runs. However, the test fails when run in the full suite because one of the tests that proceeds it leaves behind some detritus in the test database that effects the execution of the failing test.

Can anyone shed any light on this or let me know how I might address the problem?

There's two tasks here:

1) Find the individual test that, when paired with the failing test, causes test failure.

2) Fix the test that causes the problem.

The first task is laborious, but not especially difficult. Test execution order in Django is predictable, so you just need to do a binary search until you find a pair of tests that causes the problem. Lets say your suite contains apps A, B, C and D, and C.TestModel.test_problem that is causing the problem. Run:

 ./manage.py test A C.TestModel.test_problem

 ./manage.py test B C.TestModel.test_problem

 ./manage.py test D C.TestModel.test_problem

Normally, one of these three test executions will cause the failure, and the other two will pass. Let's B is the causes the failure. Now run:

 ./manage.py test B.TestModel1 C.TestModel.test_problem

 ./manage.py test B.TestModel2 C.TestModel.test_problem

 ./manage.py test B.TestModel3 C.TestModel.test_problem

Where TestModel[n] are the various test cases in the B app. Again, one of these will fail. Repeat again for individual tests; you should then have the problem narrowed down to a single test pair that causes the failure. Sometimes it will actually be every test in a given test case -- this will happen if the problem is actually being caused by the setUp method in a TestCase.

You've now identified the cause of the problem; now you just need to fix it.

This is a harder to answer without knowing the exact problem. However, you've suggested that the problem might be related to content types and fixtures. This is a common cause of the problem in Django's own test suite as well. It's caused because Django has an internal cache of content types to speed up the lookup process, and this cache *isn't* flushed at the start of each test by default. In most situations, it doesn't need to be, because the content types won't change between tests. However, if you're creating new temporary content types, or you've got fixtures that modify content types, the cache values will end up being incorrect, and will cause problems. 

*If* this is the problem, you just need to manually flush the content type cache at the end of the test that is causing the problem.

def tearDown(self):

    ContentType.objects.clear_cache()

The content type cache will be automatically repopulated on next use, so you don't have to worry about setting anything up again.

If content types aren't the problem, then you'll need to go spelunking to find the cause; essentially, you're looking for anything that *isn't* cleaned up at the end of the problem test that will cause the failing test to have a problem. Caches, stale files, and manually executed SQL are all examples of possible causes.

Hope that helps!

Yours,

Russ Magee %-)

[Paul Whipp]

Hi Russ,

Thanks for your detailed response, deserving of this detailed investigation:

I eventually found that it is the simple declaration of a model within the test suites that is causing the problem:

django.contrib.contents.tests causes the problem - even if I removed all of the tests and just imported the file:

I copied and pared the file down to this which still causes the error:

from django.db import models


class ConcreteModel(models.Model):
    name = models.CharField(max_length=10)


class ProxyModel(ConcreteModel):
    class Meta:
	proxy = True

If I remove the ProxyModel class the error changes and becomes a recursion depth error.

Renaming the classes does not get rid of the error (and the class names are not used in my apps).

The infinite recursion problem feels somehow related. My models do do something unusual: Much of the application revolves around models belonging to a company so, for convenience, all models have a company property that returns the company that owns them - it often does this by referencing objects on it. In the case of a company, the company property returns self. In the case of a note (the model where we're getting this problem), the content_object is a company and the note's company method accesses its content_object company attribute:

class Note(TimeStampedModel):
    content_type = models.ForeignKey(ContentType)
    object_id = models.PositiveIntegerField()
    content_object = generic.GenericForeignKey('content_type', 'object_id')
    ...

    @property
    def company(self):
	return self.content_object.company

class Company(BaseModel, ModelWithRandomisedToken):
    ...

    @property
    def company(self):
	return self

I think this is progress. Declaring ANY model class within the test suite causes the infinite recursion issue on the access. So my theory is that something in the django caching somewhere is getting partly updated when the model is declared but not fully sorted so the caches are broken with respect to content types. This has two symptoms - object references being None and infinite recursion, depending upon how the error gets encountered.

I'm guessing that there is some 'tidying up' of content types after all the models are imported that does not get done for a model arbitrarily defined in the test suite.

Does it sound like I am going in the right direction?

Thanks again for your help.

Cheers, Paul

--
You received this message because you are subscribed to a topic in the Google Groups "Django users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/django-users/OedhPc5bXYE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to django-users...@googlegroups.com.
To post to this group, send email to django...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CAJxq848%3DnNnpDF7tRbdUXFBkv2dM_1EgSifFvKpjfOw8JKrbng%40mail.gmail.com.

For more options, visit https://groups.google.com/groups/opt_out.

[Russell Keith-Magee]

Hi Paul,

The short answer is yes -- it sounds like you're on the right track. I'd need to sit down with actual code to confirm anything, butthe behaviour you're describing would be consistent with a content type caching issue. It's certainly an interesting presentation, but that might be partially due to the specifics of your own code.

Regarding tidying up --  keep in mind that there are two ways to skin this particular cat. While it's certainly good (and preferable) practice for every test to clean up after itself, if you've got a test that isn't under your control (e.g., something in Django's test suite), but is still causing problems, one option is to add the cleanup at the *start* of your own test -- i.e., rather than cleaning up after yourself, make sure everything is clean before you start. It's not ideal, but it works.

 

Yours,

Russ Magee %-)

[Ruben Alves]

I had a similar problem and I solved it by making all tests inherit from `django.test.TestCase`.  I had some tests inheriting from `unittest.TestCase`, which doesn't reset the database after the test is finished.

I found the answer on https://stackoverflow.com/a/436795/6490637