Random auth_permission + content_type errors during test runs

[Greg Taylor]

I've got a project running Django 1.4, Postgres 3, Python 2.7. Our tests get about halfway through, then we get a sudden flood of errors showing that the django_content_type table may be empty:

......................................................EEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE

Once the errors stop, they continue to the end of the test run. Here's the traceback for all of the errors:

Traceback (most recent call last):

File "/home/cbweb/.virtualenvs/coursebook/local/lib/python2.7/site-packages/django/test/testcases.py", line 502, in __call__

self._pre_setup()

File "/home/cbweb/.virtualenvs/coursebook/local/lib/python2.7/site-packages/django/test/testcases.py", line 463, in _pre_setup

self._fixture_setup()

File "/home/cbweb/.virtualenvs/coursebook/local/lib/python2.7/site-packages/django/test/testcases.py", line 475, in _fixture_setup

call_command('flush', verbosity=0, interactive=False, database=db)

File "/home/cbweb/.virtualenvs/coursebook/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 150, in call_command

return klass.execute(*args, **defaults)

File "/home/cbweb/.virtualenvs/coursebook/local/lib/python2.7/site-packages/raven/contrib/django/management/__init__.py", line 37, in new_execute

return original_func(self, *args, **kwargs)

File "/home/cbweb/.virtualenvs/coursebook/local/lib/python2.7/site-packages/django/core/management/base.py", line 232, in execute

output = self.handle(*args, **options)

File "/home/cbweb/.virtualenvs/coursebook/local/lib/python2.7/site-packages/django/core/management/base.py", line 371, in handle

return self.handle_noargs(**options)

File "/home/cbweb/.virtualenvs/coursebook/local/lib/python2.7/site-packages/django/core/management/commands/flush.py", line 77, in handle_noargs

emit_post_sync_signal(set(all_models), verbosity, interactive, db)

File "/home/cbweb/.virtualenvs/coursebook/local/lib/python2.7/site-packages/django/core/management/sql.py", line 189, in emit_post_sync_signal

interactive=interactive, db=db)

File "/home/cbweb/.virtualenvs/coursebook/local/lib/python2.7/site-packages/django/dispatch/dispatcher.py", line 172, in send

response = receiver(signal=self, sender=sender, **named)

File "/home/cbweb/.virtualenvs/coursebook/local/lib/python2.7/site-packages/django/contrib/auth/management/__init__.py", line 54, in create_permissions

auth_app.Permission.objects.bulk_create(objs)

File "/home/cbweb/.virtualenvs/coursebook/local/lib/python2.7/site-packages/django/db/models/manager.py", line 140, in bulk_create

return self.get_query_set().bulk_create(*args, **kwargs)

File "/home/cbweb/.virtualenvs/coursebook/local/lib/python2.7/site-packages/django/db/models/query.py", line 421, in bulk_create

transaction.commit(using=self.db)

File "/home/cbweb/.virtualenvs/coursebook/local/lib/python2.7/site-packages/django/db/transaction.py", line 153, in commit

connection.commit()

File "/home/cbweb/.virtualenvs/coursebook/local/lib/python2.7/site-packages/django/db/backends/__init__.py", line 234, in commit

self._commit()

File "/home/cbweb/.virtualenvs/coursebook/local/lib/python2.7/site-packages/django/db/backends/postgresql_psycopg2/base.py", line 235, in _commit

return self.connection.commit()

IntegrityError: insert or update on table "auth_permission" violates foreign key constraint "content_type_id_refs_id_728de91f"

DETAIL: Key (content_type_id)=(1) is not present in table "django_content_type".

This is pretty bizarre, but it's probably something we're doing. Has anyone ran into something like this before? I can't find anywhere that we're deleting content types (which the error seems to suggest is happening).

[Russell Keith-Magee]

Hi Greg,

It's not a common error, but I have seen it. 

There are two possible causes.

The first is a bad fixture. If your test has a fixture file, and that fixture file includes or references a permission by primary key value, you'll have problems. This is because content types are programmatically allocated at runtime, and the order (and thus the IDs) aren't deterministic. If you reference ID X, there's no guarantee that X will exist, or that it is the content type you want. 

The fix here is to use a natural key to reference content types, permissions, and any other dynamically created content. See the following link for details.

https://docs.djangoproject.com/en/dev/topics/serialization/#natural-keys

The second possibility is that you've got a corrupted content type cache. Django's content type system has an internal cache for performance reasons. This is unrelated to memcache or anything like that - it's a simple in-process, in-memory cache to avoid repeated lookups for content types. 

In most cases, it's completely transparent. You look up a content type for your model class, and the content type is stored. The next time you look for the same content type, it's only a dictionary lookup, not a database hit.

In testing, the same is true - the first hit is expensive, future hits are free.

However - during testing - there are certain use cases that can cause problems. If your tests are creating new content types (e.g., defining and registering a new model), this changes the values in your content type table, invalidating your cache. Django doesn't have any builtin capability to invalidate the content type cache, and so you end up getting errors about invalid constraints, or non existent models, and so on.

The solution - find the test case where you're doing this, and in the setUp() and tearDown() methods, invoke ContentType.objects.clear_cache(). This flushes the contenttype cache, and makes sure that each test runs with a clean starting point.

I hope that helps!

Yours,

Russ Magee %-)

[Aaron Lelevier]

 

 Russel,

Thank you for posting this great answer on why this error may be occurring.  I was getting this error, and for the life of me (1-2 days), couldn't figure out the issue.  Thank you!!