[Django] #25930: Django-migrations & sqlite3 fail when migration-history is non-linear
Django
-------------------------------+--------------------
Reporter: vanschelven | Owner: nobody
Type: Uncategorized | Status: new
Component: Uncategorized | Version: 1.8
Severity: Normal | Keywords:
Triage Stage: Unreviewed | Has patch: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------+--------------------
1. We started seeing problems in our tests (which were the only place
where we used sqlite3), when running the tests with all migrations (as is
the default). The problem as it showed up: lots of tests failing,
complaining that a certain column wasn't existing in the DB. Despite the
fact that a migration existed to create the column, and despite the fact
that the migrations seemed fine.
2. As [stated in the
documentation](https://docs.djangoproject.com/en/1.9/topics/migrations/#sqlite),
Django doesnt' actually run alter table statements when altering sqlite
tables via migrations. Rather, it fully recreates the tables based on it
3. Consider the following graph (oldest migration at the bottom):
{{{
M
|
/ \
B C
\ /
|
A
}}}
For example's sake, let's assume `B` and `C` add columns to some table `b`
and `c` respectively. `M` does nothing, it's only there to provide a
single sink/target for django-migrations. I.e. it's a migration which only
consists of dependencies. Let's assume that Django-migrations chooses the
ordering `A` -> `B` -> `C` -> `M` (this is non-essential, the analogous
problem will show up with the other possible ordering (C before B).
4. As far as I understand, django-migrations keeps track of an internal
representation of the table after each migration; after migration `B` this
contains `b` but not `c`; after migration `C` this contains `c` but not
`b`; after migration `M` this contains both `b` and `c`.
5. Migrations `B` and `C` contain alterations, triggering a full
recreation of the table. (This is the behavior as stated in the
documentation, and I've also seen this in practice by printing all
generated queries)
6. At migration `C` this means: a full recreation of the table, but
without the column `b`. This is the source of problem (the problem being:
the complaints about column `b` not existing)
7. Migration `M` contains nothing, triggering nothing, even though django-
migrations has the right view of what the table should look like happen at
this point. (it has the merged view of both paths). This is the point in
time where the problem isn't solved, even though it could be.
8. Another look at point '7' reveals that it can be used for a workaround:
add a statement that amounts to "alter to the status quo", i.e. a
statement that looks like an alter statement but doesn't actually alter
anything. This statement will force a full recreation of the table,
containing both columns `b` and `c`. Even though I haven't deeply inpected
how django-migrations works, the workaround works. This causes me to think
that the hypothesis is indeed correct.
I've observed the above on Django 1.7; I have not checked other
installations for similar problems.
Observed in the wild, including the workaround; I do not currently have
the time available to provide an executable clean test which does not
refer to our production-code.
Still, I hope the write-up of the problem as I understand it is useful
--
Ticket URL: <https://code.djangoproject.com/ticket/25930>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
Reporter: vanschelven | Owner: nobody
Type: Uncategorized | Status: new
Component: Uncategorized | Version:
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by charettes):
* needs_better_patch: => 0
* version: 1.8 =>
* needs_tests: => 0
* needs_docs: => 0
Comment:
Hi @vanschelven,
Thanks to your very detailed report I managed
[https://github.com/charettes/django-
ticketing/commit/621c8ca210b3253af12da4ae77c8ae19e8d5086f to reproduce]
against the latest Django 1.7.
The good news is that while I didn't bisect the exact commit but the issue
seems to be fixed on Django 1.8 which is the oldest supported version at
this point.
Please confirm 1.8 solves your reported issue. If it does please resolve
this ticket as invalid as Django 1.7 is not supported anymore.
--
Ticket URL: <https://code.djangoproject.com/ticket/25930#comment:1>
Django
Reporter: vanschelven | Owner: nobody
Type: Bug | Status: new
Component: Migrations | Version:
Severity: Normal | Resolution:
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by charettes):
* type: Uncategorized => Bug
* component: Uncategorized => Migrations
--
Ticket URL: <https://code.djangoproject.com/ticket/25930#comment:2>
Django
Reporter: vanschelven | Owner: nobody
Type: Bug | Status: new
Component: Migrations | Version:
Severity: Normal | Resolution:
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by bmispelon):
Seems to have been fixed by 1aa3e09c2043c88a760e8b73fb95dc8f1ffef50e
(found using Simon's reproduction steps).
--
Ticket URL: <https://code.djangoproject.com/ticket/25930#comment:3>
Django
Reporter: vanschelven | Owner: nobody
Type: Bug | Status: closed
Component: Migrations | Version:
Severity: Normal | Resolution: fixed
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Changes (by timgraham):
* status: new => closed
* resolution: => fixed
--
Ticket URL: <https://code.djangoproject.com/ticket/25930#comment:4>