[Django] #23237: KeyError when upgrading to the new migrations system (related to git and .pyc)
Django
.pyc)
---------------------------------+----------------------------------
Reporter: hheimbuerger | Owner: nobody
Type: Uncategorized | Status: new
Component: Migrations | Version: 1.7-rc-2
Severity: Release blocker | Keywords: git, pyc, migrations
Triage Stage: Unreviewed | Has patch: 0
Easy pickings: 0 | UI/UX: 0
---------------------------------+----------------------------------
There is a bit of an annoying gotcha connected to the upgrade path for the
new migrations, git and the .pyc files. I would recommend to at least add
this to the upgrade path docs at
https://docs.djangoproject.com/en/dev/topics/migrations/#upgrading-from-
south.
Here's what happened:
1. I followed the [https://docs.djangoproject.com/en/dev/topics/migrations
/#upgrading-from-south upgrade path instructions] pretty much down to the
letter: deleted my old South migration files, deleted the .pyc files, ran
`makemigrations` and `migrate`. Everything worked fine and life was good.
2. I committed these changes to my git repository. (Of course, I have
`*.pyc` in my `.gitignore`.)
3. My coworkers switched to my branch and upgraded to Django 1.7rc2.
4. They ran `manage.py migrate` and got this:
{{{
System check identified no issues (0 silenced).
Unhandled exception in thread started by <function wrapper at 0x03B75630>
Traceback (most recent call last):
File "c:\Python27\lib\site-packages\django\utils\autoreload.py", line
222, in wrapper
fn(*args, **kwargs)
File "c:\Python27\lib\site-
packages\django\core\management\commands\runserver.py", line 106, in
inner_run
self.check_migrations()
File "c:\Python27\lib\site-
packages\django\core\management\commands\runserver.py", line 158, in
check_migrations
executor = MigrationExecutor(connections[DEFAULT_DB_ALIAS])
File "c:\Python27\lib\site-packages\django\db\migrations\executor.py",
line 17, in __init__
self.loader = MigrationLoader(self.connection)
File "c:\Python27\lib\site-packages\django\db\migrations\loader.py",
line 48, in __init__
self.build_graph()
File "c:\Python27\lib\site-packages\django\db\migrations\loader.py",
line 241, in build_graph
self.graph.add_dependency(key, parent)
File "c:\Python27\lib\site-packages\django\db\migrations\graph.py", line
42, in add_dependency
raise KeyError("Dependency references nonexistent parent node %r" %
(parent,))
KeyError: u"Dependency references nonexistent parent node (u'myapp',
u'0001_initial')"
}}}
Obviously, the cause is that the `myapp/migrations/0001_initial.pyc` is
still there, and while git replaced the `myapp/migrations/0001_initial.py`
with the new migration one, the `myapp/migrations/0001_initial.pyc` is
still the one from South. The error message is a bit weird, so it took us
a few hours to figure out what's going on here.
We had the same issue on three different coworkers' machines, all running
some modern Windows (7–8), all running Python 2.7.4+ (32-bit) and
SourceTree as the git client.
----
I see three ways this could be tackled:
First of all, a warning about this should probably be added to the upgrade
path instructions, to at least save others from spending time to debug
this.
Second, the error message could probably be improved. Apparently, when
trying to load the `myapp/migrations/0001_initial.pyc`, it detected that
this wasn't a proper Django 1.7 migration file (correct) and just silently
ignored it (not nice), later leading to that graph dependecy error. But
why silently ignore this and not fail fast:
"myapp/migrations/0001_initial.pyc is an invalid migration file!"
Third, there's probably a way for the management command to actively
detect stale bytecode caches, but that sounds fairly difficult to
implement.
--
Ticket URL: <https://code.djangoproject.com/ticket/23237>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
.pyc)
Reporter: hheimbuerger | Owner: nobody
Type: Uncategorized | Status: new
Component: Migrations | Version: 1.7-rc-2
Keywords: git, pyc, | Triage Stage:
migrations | Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by hheimbuerger):
* needs_better_patch: => 0
* needs_tests: => 0
* needs_docs: => 0
Comment:
I should add, in case anyone finds this ticket because you have the same
issue and just want to fix it, the solution is simple:
Run `del /s *.pyc` on Windows or something like `rm -R "*.pyc"` (untested)
on Linux in your Django project root. Now the migrations will work fine.
--
Ticket URL: <https://code.djangoproject.com/ticket/23237#comment:1>
Django
.pyc)
Reporter: hheimbuerger | Owner: nobody
Type: Uncategorized | Status: new
Component: Migrations | Version: 1.7-rc-2
Keywords: git, pyc, | Triage Stage:
migrations | Unreviewed
Needs tests: 1 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
Changes (by hheimbuerger):
* needs_better_patch: 0 => 1
* needs_tests: 0 => 1
* needs_docs: 0 => 1
--
Ticket URL: <https://code.djangoproject.com/ticket/23237#comment:2>
Django
.pyc)
Reporter: hheimbuerger | Owner: nobody
Type: Uncategorized | Status: new
Component: Migrations | Version: 1.7-rc-2
Keywords: git, pyc, | Triage Stage:
migrations | Unreviewed
Has patch: 0 | Needs documentation: 1
Needs tests: 1 | Patch needs improvement: 1
Comment (by anonymous):
The Linux command is `find . -name '*.pyc' -ls -exec rm {} \;`
--
Ticket URL: <https://code.djangoproject.com/ticket/23237#comment:3>
Django
.pyc)
Reporter: hheimbuerger | Owner: nobody
Type: Uncategorized | Status: new
Component: Migrations | Version: 1.7-rc-2
Keywords: git, pyc, | Triage Stage:
migrations | Unreviewed
Has patch: 0 | Needs documentation: 1
Needs tests: 1 | Patch needs improvement: 1
Changes (by CollinAnderson):
* cc: cmawebsite@… (added)
Comment:
a simplification: `find -name '*.pyc' -delete`, or on OS X: `find . -name
'*.pyc' -delete`
--
Ticket URL: <https://code.djangoproject.com/ticket/23237#comment:4>
Django
.pyc)
Reporter: hheimbuerger | Owner: nobody
Type: Uncategorized | Status: closed
Component: Migrations | Version: 1.7-rc-2
Severity: Release blocker | Resolution: invalid
Keywords: git, pyc, | Triage Stage:
migrations | Unreviewed
Has patch: 0 | Needs documentation: 1
Needs tests: 1 | Patch needs improvement: 1
Changes (by andrewgodwin):
* status: new => closed
* resolution: => invalid
Comment:
Python should be detecting that the bytecode cache is older than the
Python file and renegerating it - if that's not working you have more
serious issues here than what Django can provide, like machines with wrong
timezones set (it's possible this is a Windows git issue).
There's no way Django can combat system-level issues like this - if your
system tells Python that this bytecode cache is newer than this source
file, then Python's going to use it, unless you've told python to turn off
bytecode (which I suggest you do, given this error - you need to set the
PYTHONDONTWRITEBYTECODE environment variable somehow), and this isn't a
migration-specific bug, as you'll see other errors with this behaviour.
I'm not going to add something to the upgrade path docs, as this isn't an
upgrade path bug; the ignoring is deliberate, to fix the real bug that was
here before (stale .pyc files from South without a corresponding new-named
migration); and the detection of stale bytecode caches is already done by
Python.
Closing as INVALID, sorry, as it's outside the scope of Django.
--
Ticket URL: <https://code.djangoproject.com/ticket/23237#comment:5>
Django
.pyc)
Reporter: hheimbuerger | Owner: nobody
Type: Uncategorized | Status: closed
Component: Migrations | Version: 1.7-rc-2
Severity: Release blocker | Resolution: invalid
Keywords: git, pyc, | Triage Stage:
migrations | Unreviewed
Has patch: 0 | Needs documentation: 1
Needs tests: 1 | Patch needs improvement: 1
Comment (by Andrew Godwin <andrew@…>):
In [changeset:"51673c146e48ff6230eace39bac18fd555607bee"]:
{{{
#!CommitTicketReference repository=""
revision="51673c146e48ff6230eace39bac18fd555607bee"
Don't treat .pyc/pyo files as migrations. Refs #23237 among others.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/23237#comment:6>
Django
.pyc)
Reporter: hheimbuerger | Owner: nobody
Type: Uncategorized | Status: closed
Component: Migrations | Version: 1.7-rc-2
Severity: Release blocker | Resolution: invalid
Keywords: git, pyc, | Triage Stage:
migrations | Unreviewed
Has patch: 0 | Needs documentation: 1
Needs tests: 1 | Patch needs improvement: 1
Comment (by Andrew Godwin <andrew@…>):
In [changeset:"267630ad50c69ebfe594de37a0636264aa5be7d6"]:
{{{
#!CommitTicketReference repository=""
revision="267630ad50c69ebfe594de37a0636264aa5be7d6"
[1.7.x] Don't treat .pyc/pyo files as migrations. Refs #23237 among
others.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/23237#comment:7>
Django
.pyc)
Reporter: hheimbuerger | Owner: nobody
Type: Uncategorized | Status: closed
Component: Migrations | Version: 1.7-rc-2
Severity: Release blocker | Resolution: invalid
Keywords: git, pyc, | Triage Stage:
migrations | Unreviewed
Has patch: 0 | Needs documentation: 1
Needs tests: 1 | Patch needs improvement: 1
Comment (by hheimbuerger):
Okay. But at any rate, your latest changes seem to fix the issue for me
(at least in the most immediate case), so thanks!
--
Ticket URL: <https://code.djangoproject.com/ticket/23237#comment:8>
Django
.pyc)
Reporter: hheimbuerger | Owner: nobody
Type: Uncategorized | Status: closed
Component: Migrations | Version: 1.7-rc-2
Severity: Release blocker | Resolution: invalid
Keywords: git, pyc, | Triage Stage:
migrations | Unreviewed
Has patch: 0 | Needs documentation: 1
Needs tests: 1 | Patch needs improvement: 1
Changes (by hheimbuerger):
* cc: henrik@… (added)
--
Ticket URL: <https://code.djangoproject.com/ticket/23237#comment:9>