[Django] #23372: Test suite initial database construction is too slow
Django
--------------------------------------+----------------------
Reporter: manfre | Owner: nobody
Type: Cleanup/optimization | Status: new
Component: Testing framework | Version: 1.7-rc-3
Severity: Release blocker | Keywords:
Triage Stage: Unreviewed | Has patch: 0
Easy pickings: 0 | UI/UX: 0
--------------------------------------+----------------------
On my primary development system (Intel i7 quad core with SSD drives), it
takes nearly 20 minutes to get from executing {{{runtests.py}}} to the
start of the first test. Most of the time (~18 minutes) is spent on the
"loaddata" call_command in the migrate command's sync_apps method.
https://github.com/django/django/blob/master/django/core/management/commands/migrate.py#L292
For comparison, using the same test environment and only changing the
version of Django (to stable/1.6.x), it takes less than 2 minutes to get
to the first test.
--
Ticket URL: <https://code.djangoproject.com/ticket/23372>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
Reporter: manfre | Owner: nobody
Type: | Status: new
Cleanup/optimization | Version: 1.7-rc-3
Component: Testing framework | Resolution:
Severity: Release blocker | Triage Stage:
Keywords: | Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by manfre):
* needs_better_patch: => 0
* needs_tests: => 0
* needs_docs: => 0
Comment:
Set as release blocker because this is an order of magnitude performance
regression added by new functionality.
--
Ticket URL: <https://code.djangoproject.com/ticket/23372#comment:1>
Django
Reporter: manfre | Owner: nobody
Type: | Status: new
Cleanup/optimization | Version: 1.7-rc-3
Component: Testing framework | Resolution:
Severity: Release blocker | Triage Stage:
Keywords: | Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by claudep):
You didn't mention the database on which you encounter this problem,
default SQLite?
It would be nice to track a bit further what is so slow inside `loaddata`.
--
Ticket URL: <https://code.djangoproject.com/ticket/23372#comment:2>
Django
Reporter: manfre | Owner: nobody
Type: | Status: new
Cleanup/optimization | Version: 1.7-rc-3
Component: Testing framework | Resolution:
Severity: Release blocker | Triage Stage:
Keywords: | Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by timgraham):
MS SQL Server I presume. I'd guess the slowness might be the
`transaction.atomic()` calls that happen in `loaddata` (one for app app in
Django's test suite) even if no fixtures exist.
--
Ticket URL: <https://code.djangoproject.com/ticket/23372#comment:3>
Django
Reporter: manfre | Owner: nobody
Type: | Status: new
Cleanup/optimization | Version: 1.7-rc-3
Component: Testing framework | Resolution:
Keywords: | Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by andrewgodwin):
* severity: Release blocker => Normal
Comment:
Not a release blocker, I'm afraid; at this point release blocker is
reserved for bugs that either crash Django or remove features that
previously existed. I'm not willing to delay the 1.7 release, which is
literally on the verge of happening, for test slowdown, or we'll literally
never get round to releasing.
Notably, the code in question has not been touched by migrations; we do a
dumpdata as part of #22487, but that doesn't seem to be part of this. I'm
not sure what the cause of the regression could be.
--
Ticket URL: <https://code.djangoproject.com/ticket/23372#comment:4>
Django
Reporter: manfre | Owner: nobody
Type: | Status: new
Cleanup/optimization | Version: 1.7-rc-3
Component: Testing framework | Resolution:
Severity: Normal | Triage Stage:
Keywords: | Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by aaugustin):
Not a blocker for 1.7 indeed, but fairly high priority for 1.7.1.
--
Ticket URL: <https://code.djangoproject.com/ticket/23372#comment:5>
Django
Reporter: manfre | Owner: nobody
Type: Cleanup/optimization | Status: new
Component: Testing framework | Version: 1.7-rc-3
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by timgraham):
* stage: Unreviewed => Accepted
--
Ticket URL: <https://code.djangoproject.com/ticket/23372#comment:6>
Django
Reporter: manfre | Owner: nobody
Type: Cleanup/optimization | Status: new
Component: Testing framework | Version: 1.7-rc-3
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by shaib):
I've seen `loaddata` take eons on SQL Server, and even in the context of
migrations (with South). As far as we could figure out, the problem was
not transaction management per se, but the handling of constraints: For
`loaddata` to avoid complex and sometimes non-satisfiable ordering
requirements, it needs FK constraints to be deferred while it works. On
core databases, each for its own reasons, this is not an issue.
SQL Server does not support defering in the normal sense -- what you need
to do is turn the constraints off in the beginning of worlk and back on in
the end. I don't remember if this is an operation you can do within a
transaction, but for sure it is not tied to a transaction -- you can turn
the constraints off and then run several transactions before turning them
back on. When you do turn them back on, however, you can do it `WITH CHECK
OPTION` -- which loaddata does, or else it might put constraint-violating
data into the database.
SQL Server does not -- perhaps, given the semantics, cannot -- keep track
of data changed while constraints were off. When you turn them on `WITH
CHECK OPTION`, it checks every single record in the database -- or, at
least, takes long enough to do so.
We had a project with several dozen apps, and relatively simple sets of
migrations on a full database (for upgrades) would take minutes on Oracle
and hours on SQL Server. Then we told SQL Server users to use `migrate
--all --no-initial-data` (we didn't have any initial data anywhere), and
sanity returned.
I'm not sure what this implies with regards to a solution. Some of the
problem should go away if `loaddata` only defered constraints when initial
data actually existed. Perhaps there's a way to avoid the check in most
cases (say, most data loaded in the Django test suite can be given a free
ticket because it is verified by the other backends).
--
Ticket URL: <https://code.djangoproject.com/ticket/23372#comment:7>
Django
Reporter: manfre | Owner: nobody
Type: Cleanup/optimization | Status: new
Component: Testing framework | Version: 1.7
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by timgraham):
* version: 1.7-rc-3 => 1.7
--
Ticket URL: <https://code.djangoproject.com/ticket/23372#comment:8>
Django
Reporter: manfre | Owner: nobody
Type: Cleanup/optimization | Status: new
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by RobertPeters):
I have an application in 1.7 that I have been building for a while. I have
about 75 migrations. When I run
{{{
python manage.py test
}}}
with SQLite, it takes about 30 seconds to create the test database. In 1.6
this was almost instant (in fact, I just confirmed this. I ran my tests in
an old environment at 1.6 and the test database was created instantly. I
ran "pip install django --upgrade" and re-ran the tests. It took 30
seconds to create the test database).
I don't know if my issue is the same as this issue (being as I'm not using
MSSQL Server), but I'd just like to add that there appears to be a
significant slow down when going from 1.6 to 1.7, regardless of the
database type.
I'm using "django.db.backends.sqlite3" on Windows 7 64-bit with a Core i7
processor and 8 GB ram.
--
Ticket URL: <https://code.djangoproject.com/ticket/23372#comment:9>
Django
Reporter: manfre | Owner: nobody
Type: Cleanup/optimization | Status: new
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by timgraham):
Robert, when you run your tests in 1.7+ all migrations are applied, which
takes several seconds per migration on 1.7 as reported in #23745. The
performance of that will be improved in 1.8 as described in that ticket.
You can check if you can use the
[https://docs.djangoproject.com/en/dev/ref/django-admin/#django-admin-
option---keepdb --keepdb] option in 1.8. Also, we might add an option to
disable migrations when testing in the future. But ultimately, no, I don't
think your issue is the one tracked by this ticket.
--
Ticket URL: <https://code.djangoproject.com/ticket/23372#comment:10>
Django
Reporter: manfre | Owner: nobody
Type: Cleanup/optimization | Status: new
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by coagulant):
* cc: baryshev@… (added)
--
Ticket URL: <https://code.djangoproject.com/ticket/23372#comment:11>
Django
Reporter: manfre | Owner: nobody
Type: | Status: closed
Cleanup/optimization |
Component: Testing framework | Version: 1.7
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by MarkusH):
* status: new => closed
* resolution: => needsinfo
Comment:
If this ''is'' related to the performance of the migration operations -
which doesn't seem to be completely clear at this point - the issue should
be fixed or at leased reduced by an order of magnitude by the
optimizations the migrate process received for 1.8.
I'm closing this with ''needsinfo'' for the reasons posted in [comment:10
comment 10]
--
Ticket URL: <https://code.djangoproject.com/ticket/23372#comment:12>
Django
Reporter: manfre | Owner: nobody
Type: | Status: closed
Cleanup/optimization |
Component: Testing framework | Version: 1.7
Severity: Normal | Resolution: needsinfo
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by timgraham):
I wonder if this might be better on master (1.9) after commits like
67235fd4ef1b006fc9cdb2fa20e7bb93b0edff4b and
4aa089a9a9504c4a833eee8161be013206da5d15 -- the `migrate` command should
be more "lightweight" after them.
--
Ticket URL: <https://code.djangoproject.com/ticket/23372#comment:13>
Django
Reporter: manfre | Owner: nobody
Type: Cleanup/optimization | Status: new
Component: Testing framework | Version: 1.7
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by timgraham):
* status: closed => new
* has_patch: 0 => 1
* resolution: needsinfo =>
Comment:
The [https://github.com/django/django/pull/5438 proposed PR] for Django
1.8 reduces the time of the entire suite on MSSQL from about 7.75 hours to
5.25 hours (benchmarks from @manfre). I'm not particularly proud of that
hack, but it seems worthwhile unless someone can spot a problem with it.
--
Ticket URL: <https://code.djangoproject.com/ticket/23372#comment:14>
Django
Reporter: manfre | Owner: nobody
Type: Cleanup/optimization | Status: new
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by timgraham):
* needs_better_patch: 0 => 1
Comment:
Anssi has proposed [https://github.com/django/django/pull/5801 an
alternate approach].
--
Ticket URL: <https://code.djangoproject.com/ticket/23372#comment:15>
Django
Reporter: manfre | Owner: nobody
Type: Cleanup/optimization | Status: new
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Comment (by akaariai):
A yet new alternate approach is here:
https://github.com/django/django/pull/5905. The idea is to skip disabling
constraints when no fixture files are found in loaddata. The 1.8 test
suite runs in 2500 seconds on my laptop after the patch (plus one line fix
in django-mssql, see the pull request).
The previously proposed alternate approach takes off another 10 minutes
from the runtime of the test suite on mssql, yielding a runtime of around
30 minutes. The changes needed there are a lot more invasive, as we will
need to pre-walk the objects in the fixture files. This could lead to
performance problems. The change might be worth it for master branch, but
backpatching seems too risky.
--
Ticket URL: <https://code.djangoproject.com/ticket/23372#comment:16>
Django
are found
-------------------------------------+-------------------------------------
Reporter: manfre | Owner: nobody
Type: | Status: new
Cleanup/optimization |
Component: Testing framework | Version: 1.7
Severity: Normal | Resolution:
| checkin
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by timgraham):
* needs_better_patch: 1 => 0
* stage: Accepted => Ready for checkin
--
Ticket URL: <https://code.djangoproject.com/ticket/23372#comment:17>
Django
are found
-------------------------------------+-------------------------------------
Type: | Status: closed
Cleanup/optimization |
Component: Testing framework | Version: 1.7
Keywords: | Triage Stage: Ready for
| checkin
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Tim Graham <timograham@…>):
* status: new => closed
* resolution: => fixed
Comment:
In [changeset:"ee9f4686b19e2b4a68f5cb4f9d61dc045c1d4c63" ee9f4686]:
{{{
#!CommitTicketReference repository=""
revision="ee9f4686b19e2b4a68f5cb4f9d61dc045c1d4c63"
Fixed #23372 -- Made loaddata faster if it doesn't find any fixtures.
Django's test suite often tries to load fixture files from apps that have
no fixtures at all. This creates a lot of unnecessary disabling and
enabling of constraints which can be expensive on some database.
To speed this up, loaddata now first checks if any fixture file matches.
If no fixture file is matched, then the command exits before disabling
and enabling of constraints is done.
The main benefit of this change is seen on MSSQL, where tests on
Django 1.8 run hours faster.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/23372#comment:18>
Django
are found
-------------------------------------+-------------------------------------
Type: | Status: closed
Cleanup/optimization |
Component: Testing framework | Version: 1.7
Severity: Normal | Resolution: fixed
Keywords: | Triage Stage: Ready for
| checkin
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Tim Graham <timograham@…>):
In [changeset:"99569b22d93737f6304ffe808ebf1d0d711fedd9" 99569b22]:
{{{
#!CommitTicketReference repository=""
revision="99569b22d93737f6304ffe808ebf1d0d711fedd9"
[1.9.x] Fixed #23372 -- Made loaddata faster if it doesn't find any
fixtures.
Django's test suite often tries to load fixture files from apps that have
no fixtures at all. This creates a lot of unnecessary disabling and
enabling of constraints which can be expensive on some database.
To speed this up, loaddata now first checks if any fixture file matches.
If no fixture file is matched, then the command exits before disabling
and enabling of constraints is done.
The main benefit of this change is seen on MSSQL, where tests on
Django 1.8 run hours faster.
Backport of ee9f4686b19e2b4a68f5cb4f9d61dc045c1d4c63 from master
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/23372#comment:19>