[Django] #23640: StaticLiveServerTestCase does not properly respect data migrations
Django
----------------------------+----------------------------------------------
Reporter: eldamir | Owner: nobody
Type: Bug | Status: new
Component: Testing | Version: 1.7
framework | Keywords: data migration, functional tests
Severity: Normal | Has patch: 0
Triage Stage: Unreviewed | UI/UX: 0
Easy pickings: 0 |
----------------------------+----------------------------------------------
When data migrations exists for a project, it is expected that each test
case has these migrations run before running the test.
That is indeed the case for django.tests.TestCase (unit tests), but it is
NOT working for
django.contrib.staticfiles.testing.StaticLiveServerTestCase (functional
tests).
Migrations seem to be run for each class, e.g.
MyTestCase(StaticLiveServerTestCase), but not for the methods of this
class. That is, if MyTestCase has 3 methods with tests, the migrations are
only loaded for the first one, and then the database is wiped, and the
objects from the data migration is no longer present for the next two
methods.
A minimal reproduction of this issue - along with a guide to the issue and
which files to look at - can be found here:
https://github.com/eldamir/django_migration_bug
--
Ticket URL: <https://code.djangoproject.com/ticket/23640>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
Reporter: eldamir | Owner: nobody
Type: Bug | Status: new
Severity: Normal | Resolution:
Keywords: data migration, | Triage Stage:
functional tests | Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Changes (by eldamir):
* needs_better_patch: => 0
* needs_tests: => 0
* needs_docs: => 0
Old description:
> When data migrations exists for a project, it is expected that each test
> case has these migrations run before running the test.
> That is indeed the case for django.tests.TestCase (unit tests), but it is
> NOT working for
> django.contrib.staticfiles.testing.StaticLiveServerTestCase (functional
> tests).
>
> Migrations seem to be run for each class, e.g.
> MyTestCase(StaticLiveServerTestCase), but not for the methods of this
> class. That is, if MyTestCase has 3 methods with tests, the migrations
> are only loaded for the first one, and then the database is wiped, and
> the objects from the data migration is no longer present for the next two
> methods.
>
> A minimal reproduction of this issue - along with a guide to the issue
> and which files to look at - can be found here:
> https://github.com/eldamir/django_migration_bug
New description:
When data migrations exist for a project, it is expected that each test
case has these migrations run before running the test.
That is indeed the case for django.tests.TestCase (unit tests), but it is
NOT working for
django.contrib.staticfiles.testing.StaticLiveServerTestCase (functional
tests).
Migrations seem to be run for each class, e.g.
MyTestCase(StaticLiveServerTestCase), but not for the methods of this
class. That is, if MyTestCase has 3 methods with tests, the migrations are
only loaded for the first one, and then the database is wiped, and the
objects from the data migration are no longer present for the next two
methods.
A minimal reproduction of this issue - along with a guide to the issue and
which files to look at - can be found here:
https://github.com/eldamir/django_migration_bug
--
--
Ticket URL: <https://code.djangoproject.com/ticket/23640#comment:1>
Django
Reporter: eldamir | Owner: nobody
Type: Bug | Status: new
Severity: Normal | Resolution:
functional tests | Needs documentation: 0
Has patch: 0 | Patch needs improvement: 0
Needs tests: 0 | UI/UX: 0
Easy pickings: 0 |
-------------------------------------+-------------------------------------
Changes (by bmispelon):
* stage: Unreviewed => Accepted
Comment:
Hi,
I can reproduce the reported issue and it does appear to be a bug indeed.
Thanks.
--
Ticket URL: <https://code.djangoproject.com/ticket/23640#comment:2>
Django
Reporter: eldamir | Owner: nobody
Type: Bug | Status: new
Severity: Normal | Resolution:
Keywords: data migration, | Triage Stage: Accepted
functional tests | Needs documentation: 0
Has patch: 0 | Patch needs improvement: 0
Easy pickings: 0 |
-------------------------------------+-------------------------------------
Comment (by gchp):
The problem appears to be with
{{{django.test.testcases.LiveServerTestCase}}} which inherits from
{{{TransactionTestCase}}}. Changing this class to inherit from
{{{TestCase}} seems to solve the issue, though I'm not sure if this is
ideal. The docstring on LiveServerTestCase mentions a reason as to why it
doesn't inherit from TestCase:
> Note that it inherits from TransactionTestCase instead of TestCase
because
> the threads do not share the same transactions (unless if using in-
memory
> sqlite) and each thread needs to commit all their transactions so that
the
> other thread can see the changes.
I'm not sure how true this is, as when changing LiveServerTestCase to
inherit from TestCase, the issue is fixed and there are no failing tests.
Not sure whether this means the above statement is no longer true, or
whether there simply are not enough tests to prove it.
--
Ticket URL: <https://code.djangoproject.com/ticket/23640#comment:3>
Django
Reporter: eldamir | Owner: gchp
Type: Bug | Status: assigned
Component: Testing framework | Version: 1.7
Severity: Normal | Resolution:
Keywords: data migration, | Triage Stage: Accepted
functional tests | Needs documentation: 0
Has patch: 0 | Patch needs improvement: 0
Easy pickings: 0 |
-------------------------------------+-------------------------------------
Changes (by gchp):
* owner: nobody => gchp
* status: new => assigned
--
Ticket URL: <https://code.djangoproject.com/ticket/23640#comment:4>
Django
Reporter: eldamir | Owner: gchp
Type: Bug | Status: assigned
Component: Testing framework | Version: 1.7
Severity: Normal | Resolution:
Keywords: data migration, | Triage Stage: Accepted
functional tests | Needs documentation: 0
Has patch: 0 | Patch needs improvement: 0
Easy pickings: 0 |
-------------------------------------+-------------------------------------
Comment (by tchaumeny):
It looks like a more generic problem with `TransactionTestCase` which does
not restore properly the data from migrations between tests.
I added a test demonstrating the problem :
https://github.com/tchaumeny/django/compare/ticket_23640#diff-
3d02be4b6d6ab6d1ac6fe6bbddc3a844R1
--
Ticket URL: <https://code.djangoproject.com/ticket/23640#comment:5>
Django
Reporter: eldamir | Owner: gchp
Type: Bug | Status: assigned
Component: Testing framework | Version: 1.7
Severity: Normal | Resolution:
Keywords: data migration, | Triage Stage: Accepted
functional tests | Needs documentation: 0
Has patch: 0 | Patch needs improvement: 0
Easy pickings: 0 |
-------------------------------------+-------------------------------------
Comment (by gchp):
It appears that this problem only exists with migrations. When I use
fixtures generated by {{{manage.py dumpdata}}} the problem goes away. The
example below works without issue.
{{{
class MyTests(TransactionTestCase):
fixtures = ['data.json']
def test_fixtures_exist(self):
self.assertEqual(MyModel.objects.count(), 4)
def test_fixtures_still_exist(self):
self.assertEqual(MyModel.objects.count(), 4)
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/23640#comment:6>
Django
Reporter: eldamir | Owner: gchp
Type: Bug | Status: assigned
Component: Testing framework | Version: 1.7
Severity: Normal | Resolution:
Keywords: data migration, | Triage Stage: Accepted
functional tests | Needs documentation: 0
Has patch: 0 | Patch needs improvement: 0
Easy pickings: 0 |
-------------------------------------+-------------------------------------
Comment (by tchaumeny):
According to the documentation this might be the expected behavior, not a
bug: ''Any initial data loaded in migrations will only be available in
TestCase tests and not in TransactionTestCase tests, and additionally only
on backends where transactions are supported (the most important exception
being MyISAM).'' — see
https://docs.djangoproject.com/en/1.7/topics/testing/overview/#rollback-
emulation
--
Ticket URL: <https://code.djangoproject.com/ticket/23640#comment:7>
Django
Reporter: eldamir | Owner: gchp
Type: Bug | Status: assigned
Component: Testing framework | Version: 1.7
Severity: Normal | Resolution:
Keywords: data migration, | Triage Stage: Accepted
functional tests | Needs documentation: 0
Has patch: 0 | Patch needs improvement: 0
Easy pickings: 0 |
-------------------------------------+-------------------------------------
Comment (by gchp):
You can use fixtures generated by dumpdata, though, if you need.
--
Ticket URL: <https://code.djangoproject.com/ticket/23640#comment:8>
Django
Reporter: eldamir | Owner: gchp
Component: Testing framework | Version: 1.7
Keywords: data migration, | Triage Stage: Accepted
functional tests | Needs documentation: 0
Has patch: 0 | Patch needs improvement: 0
Easy pickings: 0 |
-------------------------------------+-------------------------------------
Changes (by gchp):
* status: assigned => closed
* resolution: => wontfix
Comment:
Resolving this ticket per tchaumeny's comment above.
--
Ticket URL: <https://code.djangoproject.com/ticket/23640#comment:9>
Django
Reporter: eldamir | Owner: gchp
Type: Bug | Status: closed
Component: Testing framework | Version: 1.7
Severity: Normal | Resolution: wontfix
Keywords: data migration, | Triage Stage: Accepted
functional tests | Needs documentation: 0
Has patch: 0 | Patch needs improvement: 0
Easy pickings: 0 |
-------------------------------------+-------------------------------------
Comment (by eldamir):
Well, with Django 1.7 we were finally presented with an alternative to
fixtures, since fixtures are very costly to maintain, whereas data
migrations are very easy.
If this issue is intended behaviour as stated, isn't there a way to do the
functional testing like with StaticLiveTestCase? Is there another base
case I can inherit from, so that I won't have to deal with this issue?
--
Ticket URL: <https://code.djangoproject.com/ticket/23640#comment:10>
Django
Reporter: eldamir | Owner: gchp
Type: Bug | Status: closed
Component: Testing framework | Version: 1.7
Severity: Normal | Resolution: wontfix
Keywords: data migration, | Triage Stage: Accepted
functional tests | Needs documentation: 0
Has patch: 0 | Patch needs improvement: 0
Easy pickings: 0 |
-------------------------------------+-------------------------------------
Comment (by gchp):
Inheriting from TestCase and StaticLiveServerTestCase appears to work,
though I don't know what other issues that will cause.
{{{
class MyTests(TestCase, StaticLiveServerTestCase):
...
}}}
Does that work for your use case?
--
Ticket URL: <https://code.djangoproject.com/ticket/23640#comment:11>
Django
Reporter: eldamir | Owner: gchp
Type: Bug | Status: closed
Component: Testing framework | Version: 1.7
Severity: Normal | Resolution: wontfix
Keywords: data migration, | Triage Stage: Accepted
functional tests | Needs documentation: 0
Has patch: 0 | Patch needs improvement: 0
Easy pickings: 0 |
-------------------------------------+-------------------------------------
Comment (by tchaumeny):
You should not inherit from both those classes as they are doing very
different things. What you need is to set `serialized_rollback = True` in
the body of your test case, which should make the migrated data restored
for each test function.
--
Ticket URL: <https://code.djangoproject.com/ticket/23640#comment:12>
Django
Reporter: eldamir | Owner: gchp
Type: Bug | Status: closed
Component: Testing framework | Version: 1.7
Severity: Normal | Resolution: wontfix
Keywords: data migration, | Triage Stage: Accepted
functional tests | Needs documentation: 0
Has patch: 0 | Patch needs improvement: 0
Easy pickings: 0 |
-------------------------------------+-------------------------------------
Comment (by gchp):
Yeah, just after writing that I realised that was probably a bad idea.
For me, I also need to set `available_apps = ['<app-name>']` in order for
serialized_rollback to work.
--
Ticket URL: <https://code.djangoproject.com/ticket/23640#comment:13>
Django
Reporter: eldamir | Owner: gchp
Type: Bug | Status: closed
Component: Testing framework | Version: 1.7
Severity: Normal | Resolution: wontfix
Keywords: data migration, | Triage Stage: Accepted
functional tests | Needs documentation: 0
Has patch: 0 | Patch needs improvement: 0
Easy pickings: 0 |
-------------------------------------+-------------------------------------
Comment (by eldamir):
Thank you kindly for the last two comments. My tests now run as expected
after setting
{{{
serialized_rollback = True
}}}
and
{{{
available_apps = ['<app-name>']
}}}
Is there a note of these settings in the documentation? Maybe there should
be. It is not something I would have found myself. Thanks for your assist!
--
Ticket URL: <https://code.djangoproject.com/ticket/23640#comment:14>
Django
Reporter: eldamir | Owner: gchp
Type: Bug | Status: closed
Component: Testing framework | Version: 1.7
Severity: Normal | Resolution: wontfix
Keywords: data migration, | Triage Stage: Accepted
functional tests | Needs documentation: 0
Has patch: 0 | Patch needs improvement: 0
Easy pickings: 0 |
-------------------------------------+-------------------------------------
Comment (by eldamir):
I do get another error now when using the solution mentioned above, which
seems a bit weird too.
After the first test case runs, before getting to the setUp method of the
second test case, I get
{{{
Error
Traceback (most recent call last):
File "/home/ruben/v3env/lib/python3.4/site-
packages/django/db/backends/utils.py", line 65, in execute
return self.cursor.execute(sql, params)
File "/home/ruben/v3env/lib/python3.4/site-
packages/django/db/backends/sqlite3/base.py", line 486, in execute
return Database.Cursor.execute(self, query, params)
sqlite3.IntegrityError: UNIQUE constraint failed:
django_content_type.app_label, django_content_type.model
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/ruben/v3env/lib/python3.4/site-
packages/django/test/testcases.py", line 182, in __call__
self._pre_setup()
File "/home/ruben/v3env/lib/python3.4/site-
packages/django/test/testcases.py", line 754, in _pre_setup
self._fixture_setup()
File "/home/ruben/v3env/lib/python3.4/site-
packages/django/test/testcases.py", line 797, in _fixture_setup
connections[db_name]._test_serialized_contents
File "/home/ruben/v3env/lib/python3.4/site-
packages/django/db/backends/creation.py", line 428, in
deserialize_db_from_string
obj.save()
File "/home/ruben/v3env/lib/python3.4/site-
packages/django/core/serializers/base.py", line 173, in save
models.Model.save_base(self.object, using=using, raw=True)
File "/home/ruben/v3env/lib/python3.4/site-
packages/django/db/models/base.py", line 618, in save_base
updated = self._save_table(raw, cls, force_insert, force_update,
using, update_fields)
File "/home/ruben/v3env/lib/python3.4/site-
packages/django/db/models/base.py", line 699, in _save_table
result = self._do_insert(cls._base_manager, using, fields, update_pk,
raw)
File "/home/ruben/v3env/lib/python3.4/site-
packages/django/db/models/base.py", line 732, in _do_insert
using=using, raw=raw)
File "/home/ruben/v3env/lib/python3.4/site-
packages/django/db/models/manager.py", line 92, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/home/ruben/v3env/lib/python3.4/site-
packages/django/db/models/query.py", line 921, in _insert
return query.get_compiler(using=using).execute_sql(return_id)
File "/home/ruben/v3env/lib/python3.4/site-
packages/django/db/models/sql/compiler.py", line 920, in execute_sql
cursor.execute(sql, params)
File "/home/ruben/v3env/lib/python3.4/site-
packages/django/db/backends/utils.py", line 65, in execute
return self.cursor.execute(sql, params)
File "/home/ruben/v3env/lib/python3.4/site-packages/django/db/utils.py",
line 94, in __exit__
six.reraise(dj_exc_type, dj_exc_value, traceback)
File "/home/ruben/v3env/lib/python3.4/site-
packages/django/utils/six.py", line 549, in reraise
raise value.with_traceback(tb)
File "/home/ruben/v3env/lib/python3.4/site-
packages/django/db/backends/utils.py", line 65, in execute
return self.cursor.execute(sql, params)
File "/home/ruben/v3env/lib/python3.4/site-
packages/django/db/backends/sqlite3/base.py", line 486, in execute
return Database.Cursor.execute(self, query, params)
django.db.utils.IntegrityError: UNIQUE constraint failed:
django_content_type.app_label, django_content_type.model
}}}
What seems to cause this is some automatically generated SQL calls from
Django:
{{{
UPDATE "django_content_type" SET "name" = ?, "app_label" = ?, "model" = ?
WHERE "django_content_type"."id" = ? ('log entry', 'admin', 'logentry', 1)
INSERT INTO "django_content_type" ("id", "name", "app_label", "model")
SELECT ? AS "id", ? AS "name", ? AS "app_label", ? AS "model" (1, 'log
entry', 'admin', 'logentry')
}}}
As far as I can tell, a row is updated, setting some values, and then a
row is inserted which violates the UNIQUE constraint.
I don't have the insight to work around this. I will just use fixtures for
now :(
--
Ticket URL: <https://code.djangoproject.com/ticket/23640#comment:15>