[Django] #23727: IntegrityError with TransactionTestCase and sqlite
Django
-----------------------------------+--------------------
Reporter: tctimmeh | Owner: nobody
Type: Bug | Status: new
Component: Testing framework | Version: 1.7
Severity: Normal | Keywords:
Triage Stage: Unreviewed | Has patch: 0
Easy pickings: 0 | UI/UX: 0
-----------------------------------+--------------------
Using a TransactionTestCase with a sqlite database raises an
IntegrityError in Django 1.7. This is the error I'm seeing:
{{{
Traceback (most recent call last):
File "/tmp/djangomigrate/venv/local/lib/python2.7/site-
packages/django/test/testcases.py", line 182, in __call__
self._pre_setup()
File "/tmp/djangomigrate/venv/local/lib/python2.7/site-
packages/django/test/testcases.py", line 754, in _pre_setup
self._fixture_setup()
File "/tmp/djangomigrate/venv/local/lib/python2.7/site-
packages/django/test/testcases.py", line 797, in _fixture_setup
connections[db_name]._test_serialized_contents
File "/tmp/djangomigrate/venv/local/lib/python2.7/site-
packages/django/db/backends/creation.py", line 428, in
deserialize_db_from_string
obj.save()
File "/tmp/djangomigrate/venv/local/lib/python2.7/site-
packages/django/core/serializers/base.py", line 173, in save
models.Model.save_base(self.object, using=using, raw=True)
File "/tmp/djangomigrate/venv/local/lib/python2.7/site-
packages/django/db/models/base.py", line 618, in save_base
updated = self._save_table(raw, cls, force_insert, force_update,
using, update_fields)
File "/tmp/djangomigrate/venv/local/lib/python2.7/site-
packages/django/db/models/base.py", line 699, in _save_table
result = self._do_insert(cls._base_manager, using, fields, update_pk,
raw)
File "/tmp/djangomigrate/venv/local/lib/python2.7/site-
packages/django/db/models/base.py", line 732, in _do_insert
using=using, raw=raw)
File "/tmp/djangomigrate/venv/local/lib/python2.7/site-
packages/django/db/models/manager.py", line 92, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/tmp/djangomigrate/venv/local/lib/python2.7/site-
packages/django/db/models/query.py", line 921, in _insert
return query.get_compiler(using=using).execute_sql(return_id)
File "/tmp/djangomigrate/venv/local/lib/python2.7/site-
packages/django/db/models/sql/compiler.py", line 920, in execute_sql
cursor.execute(sql, params)
File "/tmp/djangomigrate/venv/local/lib/python2.7/site-
packages/django/db/backends/utils.py", line 65, in execute
return self.cursor.execute(sql, params)
File "/tmp/djangomigrate/venv/local/lib/python2.7/site-
packages/django/db/utils.py", line 94, in __exit__
six.reraise(dj_exc_type, dj_exc_value, traceback)
File "/tmp/djangomigrate/venv/local/lib/python2.7/site-
packages/django/db/backends/utils.py", line 65, in execute
return self.cursor.execute(sql, params)
File "/tmp/djangomigrate/venv/local/lib/python2.7/site-
packages/django/db/backends/sqlite3/base.py", line 485, in execute
return Database.Cursor.execute(self, query, params)
IntegrityError: columns app_label, model are not unique
}}}
I can reproduce this behaviour using the default project template, created
using startproject and startapp, and nothing more than a single
TransactionTestCase with two tests and the `serialized_rollback` option.
Here is a minimal sample project that demonstrates the problem:
https://github.com/tctimmeh/djangomigrate. Running `manage.py test
mtestapp` results in the error above.
It looks like the django.contrib.contenttypes app has a post-migrate
script that populates the content type table. However, using the
serialized_rollback option to refresh any migrated data also inserts saved
duplicates into the content type table, which results in this exception.
--
Ticket URL: <https://code.djangoproject.com/ticket/23727>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
Reporter: tctimmeh | Owner: nobody
Type: Bug | Status: new
Component: Testing framework | Version: 1.7
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by timgraham):
* needs_better_patch: => 0
* needs_docs: => 0
* needs_tests: => 0
* stage: Unreviewed => Accepted
--
Ticket URL: <https://code.djangoproject.com/ticket/23727#comment:1>
Django
Reporter: tctimmeh | Owner: nobody
Type: Bug | Status: new
Component: Testing framework | Version: 1.7
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by khoobks):
I realise that this is '''not a good workaround''' but as a temporary
measure I found that the following works in my situation.
I subclassed the TransactionTestCase and added the following code.
{{{#!python
def _fixture_setup(self):
super(MyTransactionTestCase, self)._fixture_setup()
for app_config in apps.get_app_configs():
update_contenttypes(app_config)
create_permissions(app_config)
create_default_site(app_config)
def _fixture_teardown(self):
signals.post_migrate.disconnect(create_default_site,
sender=apps.get_app_config('sites'))
signals.post_migrate.disconnect(update_contenttypes)
signals.post_migrate.disconnect(create_permissions,
dispatch_uid="django.contrib.auth.management.create_permissions")
signals.post_migrate.disconnect(update_group_permissions)
super(MyTransactionTestCase, self)._fixture_teardown()
signals.post_migrate.connect(update_contenttypes)
signals.post_migrate.connect(create_permissions,
dispatch_uid="django.contrib.auth.management.create_permissions")
signals.post_migrate.connect(create_default_site,
sender=apps.get_app_config('sites'))
}}}
The the overridden {{{_fixture_teardown}}} disables the
{{{post_migrate}}} signal handlers from executing and creating the
problematic rows in the database. This allows the
{{{TransactionTestCase}}} to successfully reload the original database
state. The handlers are then manually invoked in the overriden
{{{_fixture_setup}}} in order to ensure that for subsequent tests, the
ContentTypes, Sites and Permissions are available.
--
Ticket URL: <https://code.djangoproject.com/ticket/23727#comment:2>
Django
Reporter: tctimmeh | Owner: nobody
Type: Bug | Status: new
Component: Testing framework | Version: 1.7
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by khoobks):
* cc: khoobks (added)
--
Ticket URL: <https://code.djangoproject.com/ticket/23727#comment:3>
Django
Reporter: tctimmeh | Owner: nobody
Type: Bug | Status: new
Component: Testing framework | Version: 1.7
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by dpoirier):
* cc: dpoirier@… (added)
--
Ticket URL: <https://code.djangoproject.com/ticket/23727#comment:4>
Django
Reporter: tctimmeh | Owner: nobody
Type: Bug | Status: new
Component: Testing framework | Version: 1.7
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by dpoirier):
FYI, I see this with Postgres as well, so it doesn't appear to be SQLite
specific.
--
Ticket URL: <https://code.djangoproject.com/ticket/23727#comment:5>
Django
Reporter: tctimmeh | Owner: tbeadle
Type: Bug | Status: assigned
Component: Testing framework | Version: 1.7
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by tbeadle):
* owner: nobody => tbeadle
* status: new => assigned
--
Ticket URL: <https://code.djangoproject.com/ticket/23727#comment:6>
Django
Reporter: tctimmeh | Owner: tbeadle
Type: Bug | Status: assigned
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by timgraham):
* has_patch: 0 => 1
Comment:
I had a look at the [https://github.com/django/django/pull/4510 PR], but
I'm not confident enough to commit it. Also, some documentation might be
needed (the loss of the `post_migrate` signal in this scenario might be
backwards incompatible for some users).
--
Ticket URL: <https://code.djangoproject.com/ticket/23727#comment:7>
Django
Reporter: tctimmeh | Owner: tbeadle
Type: Bug | Status: assigned
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by ldd):
* cc: ldd (added)
--
Ticket URL: <https://code.djangoproject.com/ticket/23727#comment:8>
Django
Reporter: tctimmeh | Owner: tbeadle
Type: Bug | Status: assigned
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by timgraham):
* needs_tests: 0 => 1
Comment:
Added a comment with an idea about a test on the pull request.
--
Ticket URL: <https://code.djangoproject.com/ticket/23727#comment:9>
Django
-----------------------------------+------------------------------------
Reporter: tctimmeh | Owner: tbeadle
Type: Bug | Status: assigned
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
--
Ticket URL: <https://code.djangoproject.com/ticket/23727#comment:10>
Django
-----------------------------------+------------------------------------
Reporter: tctimmeh | Owner: tbeadle
Type: Bug | Status: assigned
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by timgraham):
* needs_tests: 1 => 0
Comment:
[https://github.com/django/django/pull/5168 PR] with test added.
--
Ticket URL: <https://code.djangoproject.com/ticket/23727#comment:11>
Django
-----------------------------------+------------------------------------
Reporter: tctimmeh | Owner: tbeadle
Component: Testing framework | Version: 1.7
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Tim Graham <timograham@…>):
* status: assigned => closed
* resolution: => fixed
Comment:
In [changeset:"d3fdaf907db6a5be4d0391532d7e65688c19e851" d3fdaf90]:
{{{
#!CommitTicketReference repository=""
revision="d3fdaf907db6a5be4d0391532d7e65688c19e851"
Fixed #23727 -- Inhibited the post_migrate signal when using
serialized_rollback.
When using a TransactionTestCase with serialized_rollback=True,
after creating the database and running its migrations (along with
emitting the post_migrate signal), the contents of the database
are serialized to _test_serialized_contents.
After the first test case, _fixture_teardown() would flush the
tables but then the post_migrate signal would be emitted and new
rows (with new PKs) would be created in the django_content_type
table. Then in any subsequent test cases in a suite,
_fixture_setup() attempts to deserialize the content of
_test_serialized_contents, but these rows are identical to the
rows already in the database except for their PKs. This causes an
IntegrityError due to the unique constraint in the
django_content_type table.
This change made it so that in the above scenario the post_migrate
signal is not emitted after flushing the tables, since it will be
repopulated during fixture_setup().
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/23727#comment:12>
Django
-----------------------------------+------------------------------------
Reporter: tctimmeh | Owner: tbeadle
Type: Bug | Status: closed
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by romgar):
I have seen that this patch has been applied to Django 1.9.x.
For previous Django versions, I guess we can resolve this problem by
setting:
`TEST_NON_SERIALIZED_APPS = ['django.contrib.contenttypes']`
as proposed in StackOverflow issue
http://stackoverflow.com/questions/29226869/django-transactiontestcase-
with-rollback-emulation/35359897 by @se7entyse7en. Is it right ?
--
Ticket URL: <https://code.djangoproject.com/ticket/23727#comment:13>