[Django] #24178: Moving production data to a new database
Django
-------------------------------+--------------------
Reporter: liwee | Owner: nobody
Type: Uncategorized | Status: new
Component: Migrations | Version: 1.7
Severity: Normal | Keywords:
Triage Stage: Unreviewed | Has patch: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------+--------------------
Hi,
When I move my production data to a new server, ideally I try to follow
the steps
Old server
1. ftp source files from repo
2. run migration
3. backup database
New server
4. ftp source files from repo (migration directory is non-existent)
5. restore database
6. run migration (faked)
The problem occurs when there are some model changes in repo and steps 1 &
2 are left out.
E.g.
In old server, both database and model.py match, i.e. both have foo1:
{{{
class foo(models.Model):
foo1 = models.CharField()
}}}
however, the model.py in repo has been rename from foo1->foo2
and when step 6 is executed, migration detects the table foo and fakes the
initial migration. it assumes that the table foo contains foo2 according
to model.py while in reality the table foo contains foo1
{{{
class foo(models.Model):
foo2 = models.CharField()
}}}
at this point, no errors are detected. the problem occurs during the next
code change, i.e. foo2->foo3. migration cannot locate foo2 in the table
foo.
{{{
class foo(models.Model):
foo3 = models.CharField()
}}}
Is it possible to add some sort of field verification when the initial
migration is faked at step 6?
--
Ticket URL: <https://code.djangoproject.com/ticket/24178>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
Reporter: liwee | Owner: nobody
Type: Uncategorized | Status: closed
Component: Migrations | Version: 1.7
Severity: Normal | Resolution: needsinfo
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by timgraham):
* status: new => closed
* needs_docs: => 0
* resolution: => needsinfo
* needs_tests: => 0
* needs_better_patch: => 0
Comment:
Sorry, but I can't really make sense of what you are doing. Maybe try
avoiding making database schema and code changes at the same time if
that's what's going on?
--
Ticket URL: <https://code.djangoproject.com/ticket/24178#comment:1>
Django
Reporter: liwee | Owner: nobody
Type: Uncategorized | Status: closed
Component: Migrations | Version: 1.7
Severity: Normal | Resolution: needsinfo
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by aaugustin):
I would also suggest using a (D)VCS for moving code around rather than
FTP'ing uncommitted changes. That would certainly spare you lots of
trouble.
--
Ticket URL: <https://code.djangoproject.com/ticket/24178#comment:2>
Django
Reporter: liwee | Owner: nobody
Type: Uncategorized | Status: closed
Component: Migrations | Version: 1.7
Severity: Normal | Resolution: needsinfo
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by liwee):
I am no expert in Django but I was just wondering if it is possible to
change how Django handle migration with existing tables.
{{{
Run python manage.py migrate. Django will see that the tables for the
initial migrations already exist and mark them as applied without running
them.
}}}
The above is taken from Django documentation. Instead of marking them as
applied unconditionally, it would be really helpful if Django could throw
out some sort of error if the existing database structure doesn't match
model.py (via inspectdb?).
--
Ticket URL: <https://code.djangoproject.com/ticket/24178#comment:3>
Django
Reporter: liwee | Owner: nobody
Type: Uncategorized | Status: closed
Component: Migrations | Version: 1.7
Severity: Normal | Resolution: needsinfo
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by timgraham):
The idea is that all changes to your database schema should have Django
migrations. If you are changing the schema in other ways, then you are on
your own.
--
Ticket URL: <https://code.djangoproject.com/ticket/24178#comment:4>
Django
Reporter: liwee | Owner: nobody
Type: Uncategorized | Status: closed
Component: Migrations | Version: 1.7
Severity: Normal | Resolution: needsinfo
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by carljm):
I do think it's odd that `migrate` will auto-fake your initial migrations
if it sees that the tables already exist, as documented in the "upgrading
from South" section that liwee quoted. I would think that situation should
raise an error instead. I see how the auto-fake behavior is convenient in
the upgrade-to-1.7 scenario, but as liwee points out, it's making some
hard-to-justify assumptions. I think it should require a special flag to
trigger that behavior.
(Of course it would be nice to keep the auto-fake behavior AND verify that
the db state actually matches the models, as liwee suggested, but I don't
think that's practically feasible at all.)
I'll look into this more to verify that I'm not missing anything and
perhaps file a new bug to require a flag for auto-fake.
--
Ticket URL: <https://code.djangoproject.com/ticket/24178#comment:5>
Django
Reporter: liwee | Owner: nobody
Type: Uncategorized | Status: closed
Component: Migrations | Version: 1.7
Severity: Normal | Resolution: needsinfo
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by liwee):
{{{
(Of course it would be nice to keep the auto-fake behavior AND verify that
the db state actually matches the models, as liwee suggested, but I don't
think that's practically feasible at all.)
}}}
I didn't mean for both to work at the same time.
1. For initial migration with existing tables, Django will first check if
db state match the model
2. if match, continue to auto-fake
3. if doesn't match, raise an error and auto-fake is aborted
User will adjust the db table manually and rerun initial migration again
--
Ticket URL: <https://code.djangoproject.com/ticket/24178#comment:6>
Django
Reporter: liwee | Owner: nobody
Type: Uncategorized | Status: closed
Component: Migrations | Version: 1.7
Severity: Normal | Resolution: needsinfo
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by carljm):
I understand. That's the same thing I meant. But I think you are under-
estimating the practical difficulty of implementing "Django will first
check if db state match the model" correctly. The code for "inspectdb"
doesn't really get us very far in that direction.
--
Ticket URL: <https://code.djangoproject.com/ticket/24178#comment:7>
Django
Reporter: liwee | Owner: nobody
Type: Uncategorized | Status: closed
Component: Migrations | Version: 1.7
Severity: Normal | Resolution: needsinfo
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by carljm):
I filed #24184 to follow up on the issue of the auto-fake behavior being
unsafe.
--
Ticket URL: <https://code.djangoproject.com/ticket/24178#comment:8>
Django
Reporter: liwee | Owner: nobody
Type: Uncategorized | Status: closed
Component: Migrations | Version: 1.7
Severity: Normal | Resolution: needsinfo
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by liwee):
This might be a very stupid idea ...
Assume the database already exist, say database name is mydb
Migration will
1. create another db called mydb_compare using model.py
2. export structure sql (no data) for mydb
3. export structure sql for mydb_compare
4. do a diff for these 2 files?
--
Ticket URL: <https://code.djangoproject.com/ticket/24178#comment:9>