Bytecode in Migrations

[abhillman]

I have gotten bitten by lingering bytecode in migrations on several occasions. Steps to reproduce are a little bit complex, but here is an rough example:

Local Box:

— have *.pyc rule in .gitignore

— create some migrations

— commit them

Server Box:

— pull repository

— execute migrations

Local Box:

— remove migrations

— commit

— create new initial migration

Server Box:

— pull repository

— remove migrations history from db

— run fake initial migration — get an error here

At this point, the server box has a migrations directory that looks something like this (bytecode is still around because it was in the gitignore):

— migrations 

    — __init__.py

    — 0001_initial.py

    — 0001_initial.pyc

    — 0002_second.pyc

    — 0003_third.pyc

When running "python manage migrate" the bytecode appears to be referenced, which often causes errors when running the migrations. When using git, for example "git clean -df" clears out the problem. What I am wondering is if it might make sense to make a deliberate attempt to ignore bytecode without accompanying *.py files. This appears to be an issue because of the way that migrations dynamically import python code, but I am not sure. Perhaps the problem is more subtle as I am not deeply familiar with the way that migrations work.

aryeh

[Collin Anderson]

Hah. Check out this ticket :) https://code.djangoproject.com/ticket/23406

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/1dbf1434-b24a-47c9-b45b-ae04530d7867%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[abhillman]

I am a little confused, Colin. I am running django 1.7.4, but seem to have come across an issue whereby lingering *.pyc files broke the "python manage.py migrate" command. But the ticket suggests that bytecode is in fact currently ignored. Do you know what's up here? Don't want to clutter up the ticket with yet another comment necessarily.

[abhillman]

Just checked the source that I was using. Definitely 1.7.4 and the local version of django I have reads:

 95             for name in os.listdir(directory):
 96                 if name.endswith(".py"):
 97                     import_name = name.rsplit(".", 1)[0]
 98                     if import_name[0] not in "_.~":
 99                         migration_names.add(import_name)

which is expected. Yet I am absolutely certain that migrations were failing until I cleared out old bytecode. Perhaps there was some other bytecode that was causing the problem which was not immediately related to the migrations, but I am not certain.

On Wednesday, February 11, 2015 at 1:06:05 PM UTC-8, Collin Anderson wrote:

[Andrew Godwin]

Your problem is likely that the pyc file for the new initial migration somehow seems newer than the py file and so Python is using it over the new source file.

I'm not sure how this happens if it's a new migration, but I've seen it happening when switching git branches before. We have pyc files turned off in dev to stop this.

Andrew

--

[Stan]

I noticed the same pb yesterday (runserver exception) when doing some git bisect on django repo because of some pyc content_type migration file.
Removing the pyc files after each git bisect good|bad fixed the issue.