Upgrading Django (to 1.7)

[Andrew Pinkham]

Hi,
I am writing a series of articles based on the presentation I gave at DjangoCon US 2014. I put the first article online earlier today, and figured members of this list might be interested in the material.

For all of the material:
afrg.co/updj17/

For the article:
afrg.co/updj17/a1/

I plan to post the next three in the series on each of the coming Wednesdays.

Any feedback appreciated. I hope you find the material helpful.

Andrew

[Sabine Maennel]

Dear Andrew, 

I read your article and it is great. I am really looking forward to your followups. I was hoping for something like this went I attended Django Con Europe. I struggle with the understanding the migrations the most. Usually in the early phases of development I do not want migrations to bother me since my database design just still changes too often for keeping track. I find the django admin a very useful tool to test your database design with real data and detect flaws early on. So will your followups contain advise on that? How can I restart migrations all over once my project is ready for deployment? 

Thanks so much for writing these very useful articles! Do you mind if I post about them in the Django user facebook user group, or do you want to do that, since I think these articles will be very helpful to most Django developers.

      with kind regards and keep going!

           Sabine

[Fred Stluka]

Andrew,

Good stuff!  Thanks!  We're still on 1.4, but planning a move
soon to 1.7.  Very helpful.

--Fred

---

Fred Stluka -- mailto:fr...@bristle.com -- http://bristle.com/~fred/
Bristle Software, Inc -- http://bristle.com -- Glad to be of service!
Open Source: Without walls and fences, we need no Windows or Gates.

---

[Andrew Pinkham]

Hi Sabine,
Thank you! It is really nice getting positive feedback.

The migrations section (Part II next week!) is a very basic introduction, aimed at people who have never used South or native migrations. One of the key takeaways (hopefully) of the article is the fact that it is easy to manually edit migrations, and that you are encouraged to do so.

Concerning early development cycles and migration reset, if you are pre-deploy, you currently have three options:
- disable migrations entirely during early development (will cease to be possible in Django 1.9; **inadvisable**)
- `squashmigrations` once models are ready
- delete all migrations and start from scratch once models are ready

I would recommend one of last two. The choice really comes down to how your engineering team is set up. If you are in a position where you can get the change to everyone at once, then the third option is viable (but keep a backup!). However, if you have independent developers, this may be dangerous/frustrating, and `squashmigrations` is likely a better option. The `squashmigrations` command is also what you should use in the long term. I do not cover it's use, but you may find the following documentation worthwhile:

https://docs.djangoproject.com/en/1.7/topics/migrations/#squashing-migrations

Unfortunately, I will not cover the admin in my article. Personally, I prefer the command line, as I am able to mover more quickly than the admin panel. This is completely subjective, of course, and not a criticism of the admin. For testing data, however, you may find data migrations to be useful in some situations, and I advise for their use. Part II will walk through multiple data migrations implementations in both Django 1.6 and Django 1.7.

Note that data migrations cannot be automatically squashed. If you use data migrations, this will result in migration file editing down the line. This should not be a problem, but I also don't want that to surprise you.

I hope this and future articles are helpful! Thanks again for the positive feedback.

Andrew

PS
I do not mind if you share my article. In fact, I would be delighted if you did. Please do!

[Andrew Pinkham]

Fred,
Thank you!

Good luck with the upgrade!

Andrew

[Fred Stluka]

Sabine,

I'm not sure about the new 1.7 Migrations, but with South,
restarting your migrations is easy. 

I'm currently doing it on my project, because we have a
long convoluted history of changes before and after we
started using South, and have made some mistakes along
the way (like editing migrations and fixtures that they use
after running them -- bad idea!).  Since we're at a nice
stable point now, and are certain we'll never need to
migrate back to an earlier version, I'm currently deleting
all the old migrations and creating new clean ones that get
us to our current state.

Steps:

- Delete all rows from south_migrationhistory table.

- For each app
   - Delete old migration files from migrations folders
   - Run: python manage.py convert_to_south appname
      to create and fake the 0001_initial.py migration.
      This creates a file in the migrations folder and adds
      row to the south_migrationhistory claiming that the
      migration has already been run.

- For each app with models mapped to DB tables that
   contain data you would want to pre-load into a newly
   created DB (lookup values like countries, states,
   statuses, etc.):
   - Run: python manage.py datamigration appname migration_name
      to create a new migration named 0002_migration_name.py
      with empty forwards() and backwards() methods.

- For each such DB table:
   - Run: python manage.py dumpdata --indent=4 appname.ModelName > ./appname/fixtures/table_name.default.json
   - Add to forwards() method:
        call_command("loaddata", "table_name")

--Fred

---

Fred Stluka -- mailto:fr...@bristle.com -- http://bristle.com/~fred/
Bristle Software, Inc -- http://bristle.com -- Glad to be of service!
Open Source: Without walls and fences, we need no Windows or Gates.

---

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.
To post to this group, send email to django...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/934a13d6-d5fd-4684-9bb1-e6d323778cda%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Benjamin Scherrey]

Great summary - looking forward to the rest. Especially like some of the backgrounder insights also explaining "why" certain things were done. I think this often helps developers understand and use features better than even the otherwise excellent and detailed online docs. Wonder if the Django doc pages couldn't have corresponding backgrounder pages containing such in the future.

-- Ben

Andrew

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.
To post to this group, send email to django...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/D05EC989-C8DB-41CE-AA23-E1D73C25A623%40andrewsforge.com.

For more options, visit https://groups.google.com/d/optout.

--
Chief Systems Architect Proteus Technologies

Chief Fan Biggest Fan Productions

Personal blog where I am not your demographic.

This email intended solely for those who have received it. If you have received this email by accident - well lucky you!!

[bobhaugen]

Andrew, thanks for the article, and the more-to-come.

When you get to data migrations, you might want to cover a problem I ran into with them: when I had a data migration in the middle of a sequence of other migrations, and then later wanted to set up a new environment, which meant running all of the migrations in sequence from the beginning, the data migration was now running in the latest models.py code, not the code in which it initially worked. So it did not work any more.

In particular, the data migration was a precursor to another schema migration were a field would be removed. The data migration was to move the existing data in the field-to-be-removed to a different field. So when the data migration ran in the longer sequence, the field it was trying to move data from was now gone from the code.

I suspect there are a lot of other ways to shoot yourself in the foot with data migrations, so we have stopped using them and just use our own data migration scripts outside of the sequence of schema migrations.

[Fred Stluka]

Bob,

You can control the order in which migrations run.

For South, see "depends_on" and "needed_by":
- http://south.readthedocs.org/en/latest/dependencies.html

For Django 1.7 migrations, see "dependencies":
- https://docs.djangoproject.com/en/dev/topics/migrations/#dependencies

--Fred

---

Fred Stluka -- mailto:fr...@bristle.com -- http://bristle.com/~fred/
Bristle Software, Inc -- http://bristle.com -- Glad to be of service!
Open Source: Without walls and fences, we need no Windows or Gates.

---

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.
To post to this group, send email to django...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/9f4e1b6a-4051-478d-9a09-43318266ed7f%40googlegroups.com.

[bobhaugen]

The problem we ran into was not with the order of migrations, it was that all of the migrations were running with the latest models.py code. The data migration was trying to move data from one model field to another model field, and the "from" field had been deleted from models.py. 

This was awhile ago, and my memory might be faulty. Do the latest migrations give you any way to deal with that situation? I mean, do they migrate the models in models.py in sync with the the database schema migrations?

I can see where a data migration might work in a schema migration sequence if you expressed it all in SQL, just dealing with the database alone, but we of course wrote in Python using the Django ORM.

[Andrew Pinkham]

Benjamin,
Thank you!

The next two parts of the article will focus on why certain changes were made. Please let me know if you think there were changes that I should have written about. Depending on my schedule and feedback, I would be willing to consider adding to the material if I've missed anything.

I'm happy to expand on my opinion if desired, but I don't think the documentation should have history/background sections. The documentation is already an enormous amount of work, and I like the fact that is sticks to being reference documentation.

However, I think the emails from Aymeric Augustin, Russell Keith-Magee, and Christopher Medrela documenting their work are incredibly helpful. Andrew Godwin even kept a blog! I would love it if people continued this trend. I will be linking to their work in Part III because of how useful and interesting the emails/blog-posts are. I think we owe them all a huge thank you for the effort.

Finally, the wiki contains some historical/background documents. I would not be averse to more pages like that, nor would I be averse to the release notes pointing to the emails mentioned above.

Andrew

[Andrew Pinkham]

Hi Bob,
Thank you!

I spend a lot of Part II talking about schema and data migrations in both South and native Django 1.7 migrations. When building data migrations, you should be using historical models. In Django 1.7, you can request these models directly from the app registry built on the fly by the migration system and passed to the methods specified in the `RunPython` operation. This avoids using the models as they exist in the `models.py` file. This is similar to South's requirement you use the `orm` variable to interact with models, as Fred mentions.

If you would like more depth on the topic, then Part II is for you. If you would like to explore the topic before Wednesday, this is material available in the video. The code in the slides and github repo may be quite helpful as well.

afrg.co/updj17

Data migrations and schema migrations should play very well with each other. As I mentioned to Sabine, the only instance I had any difficulty was when I squashed migrations. As I had been alternating systematically between schema migrations and data migrations, the automatic squash tool was unhelpful. Thankfully, as the migration files are now easy to edit, I had absolutely no issue modifying the squash migration file myself.

Unfortunately, I do not go into detail about this in the article. Hopefully, the number of times we directly modify the migration files will allow people to figure this out for themselves. If that is not the case, and enough people ask, I will happily write a short article about the process.

If you have any questions about data migrations after reading Part II, please ask them here!

Andrew

[Andrew Pinkham]

Hi,
Upgrading Django (to 1.7) Part II: Migrations in Django 1.6 and 1.7 is now available!

For the article:
afrg.co/updj17/a2/

For all of the material:
afrg.co/updj17/

[Carsten Fuchs]

Hi Andrew,

Am 2014-10-02 um 05:36 schrieb Andrew Pinkham:
> Any feedback appreciated. I hope you find the material helpful.

Also from me a hearty thanks for your article series!
It's very helpful, and very much appreciated!

Best regards,
Carsten


--
Cafu Engine • www.cafu.de
an open-source 3D engine for multiplayer games and simulations

[Stodge]

I will read this thanks. Does it cover this issue?

django.core.exceptions.AppRegistryNotReady: The translation infrastructure cannot be initialized before the apps registry is ready. Check that you don't make non-lazy gettext calls at import time.

I don't know if this has been a pain to many people, but I had to rollback to 1.6. That's a post for another day but I'm curious if you hit it.

[Andrew Pinkham]

Carsten,
Thank you for reading and for the positive feedback!

Andrew

[Andrew Pinkham]

Stodge,
Part II focuses on migrations, and deals in passing with AppConfig objects via data-migrations. Part III will take a closer look at the app registry, as we will create a custom AppConfig objects for `roundtable`. There is a (very) short section in Part III about import problems caused by custom user settings. I do not supply a solution to the problem, as every project will have a different solution, and the section is simply meant to let developers know of the potential problem.

I think your issue may be related. The issue is that Django must build the app registry before it can take (some) other actions. You likely have a problem in your import sequence. This should not prevent you from upgrading to 1.7. If you provide the traceback of the error and the associated code, I (or the mailing list) may be able to help figure out the problem.

I hope my article series provides some insight about the app registry, as well.

Hope that helps,
Andrew

[Fred Stluka]

On 10/1/14 11:36 PM, Andrew Pinkham wrote:

Any feedback appreciated. I hope you find the material helpful.

Andrew,

Excellent 2nd article.  Thanks! 

I especially liked the clear explanation of how South migrations
interact badly with initial_data fixtures.  OK to use fixtures, but
load them explicitly via "loaddata" from data migrations. Don't
put them in "initial_data" fixtures.

Also, good explanations of:
- South in general
- The reason why there's a models dictionary (the "frozen model")
   in each migration file, and why to use orm.ModelName
- Differences from Django 1.7 migrations
- How to get started with Django 1.7 migrations
- Unicode vs ASCII issues
- Use of objects.bulk_create()

Thanks!

[Andrew Pinkham]

Hi Fred,
Thanks again for the input! Glad you liked Part II.

Andrew

[Andrew Pinkham]

Hi,
Upgrading Django (to 1.7) Part III: Django 1.7's New Features is now available!

For the article:
afrg.co/updj17/a3/

For all of the material:
afrg.co/updj17/

[Fred Stluka]

Andrew,

Excellent 3rd article.  Thanks again!

I've recommended it to my team, summarizing it as:
- South vs Django 1.7 Native Migrations
- Upgrading from South to Django 1.7 Native Migrations
- New "App Registry"
- New "System Check Framework"
  -- Invoke via: % python manage.py check
  -- Also runs implicitly as part of most manage.py commands.
  -- Has hooks to add your own checks to enforce project standards, like:
     --- All Models must have a __str__ method.
- New "Custom Querysets"
  -- Often better than writing a custom manager
- Improved prefetch_related()
- New "Custom Lookups"
- Improved form error handling


The App Registry section is not entirely clear to me, probably
because I'm coming from 1.4, not from 1.6.

It might help if you explained how the App Registry relates to
1.4 concepts.  Like, I assume it replaces INSTALLED_APPS, plus
a whole lot more, right?

Otherwise, can you suggest where I can find an article similar to
yours that gets me from 1.4 to 1.5 and on to 1.6 before I use
yours to get me from 1.6 to 1.7?

[Andrew Pinkham]

Hi Fred,
Thanks again for the positive feedback!

I'm sorry I lost you at the App Registry section. I'm afraid I can't recommend an article to help with the topic, either. Given your questions, I can see I missed the mark. Let me give it another go (warning: this is neither technically reviewed nor copy edited).

Django uses the `INSTALLED_APPS` setting to find all of the apps in a project, and to build a list of apps to run and use, called an app registry. The way Django builds the app registry has changed radically from Django 1.6 to Django 1.7.

Django 1.6 (and before, but I admit I don't know how far back) used the `AppCache` class to maintain a list of these apps. The system was problematic because each `AppCache` instance shared state - any change to one `AppCache` object lead to a change in all `AppCache` objects. Because of migrations (and the need for historical/frozen models), the `AppCache` had to change in Django 1.7.

Django 1.7 now provides an `Apps` class to be the app registry. Each instance is separate from others, allowing the migration system to build historical apps and models. You can fetch the `Apps` object (not class!) for your own project with:

from django.apps import apps as django_apps

Referred to as the master registry (the app registry for your project as it runs), `django_apps` is an instance of `Apps`, which contains a list of `AppConfig` objects.

Starting in Django 1.7 all apps now have an `AppConfig` object attached to them. These are built automatically by Django, meaning many developers will be able to get away without ever creating an `/app_name/apps.py` file for their projects. The ability to override `AppConfig`, however, is quite useful. It allows for the implementation of a `ready()` method (seen in the admin `AppConfig` subclass) as well as the ability to explicitly set the app label, allowing developers to easily rectify namespace conflicts.

Note that we avoid a namespace error thanks to `as django_apps` with `AppConfig`. I recommend you do this even if there is no namespace error.

from django.apps import apps as django_apps
from . import apps

To deal with this list of `AppConfig` objects (and the models within), all `Apps` instances sport a brand new API. We saw it throughout Part II and Part III of the article, using it to help us with data migrations and to build our checks.

The only trick with `Apps` is that it must be fully built and configured before Django can take certain actions, including loading the custom user model, or using parts of the translation system. I touched a little bit on it in Part III, but plan to expand on it in Part IV.

The bottom line is that the app registry works mostly in the background by building a master registry and performing many of its key functions without developers ever knowing (just as many developers never knew about Django 1.6's `AppCache`). However, understanding the system even a little allows for the developer to better edit migrations and create checks, and I suspect that interacting with apps via this API will become commonplace.

Fred: Is the above helpful? Feel free to ask more questions.

On Oct 9, 2014, at 4:53 PM, Fred Stluka <fr...@bristle.com> wrote:
> Otherwise, can you suggest where I can find an article similar to
> yours that gets me from 1.4 to 1.5 and on to 1.6 before I use
> yours to get me from 1.6 to 1.7?

No, but I hope the process I lay out in Part IV will help you indirectly.

Andrew

[Fred Stluka]

Andrew,

Yes, that explains a lot.  I'd recommend including this explanation
in one of the articles of your current series.

BTW, one other thing to keep in mind is that we 1.4 users don't
know what an apps.py file is, so we don't appreciate not having
to create one, since we never did before.

Thanks again!

--Fred

---

Fred Stluka -- mailto:fr...@bristle.com -- http://bristle.com/~fred/
Bristle Software, Inc -- http://bristle.com -- Glad to be of service!
Open Source: Without walls and fences, we need no Windows or Gates.

---

[Andrew Pinkham]

Hi,
Thanks to feedback from Fred and a few others, I've started editing Part III to make it more accessible.

Part IV was originally set to be published today. Unfortunately, it is not ready to be published.

As soon as the new Part III and Part IV are available, I will let you know.

Thanks,
Andrew

[Fred Stluka]

Sounds good!  Thanks!

--Fred

---

Fred Stluka -- mailto:fr...@bristle.com -- http://bristle.com/~fred/
Bristle Software, Inc -- http://bristle.com -- Glad to be of service!
Open Source: Without walls and fences, we need no Windows or Gates.

---

[Andrew Pinkham]

Hi,
I have just updated Upgrading Django (to 1.7) Part III: Django 1.7's New Features. The App Registry and System Check Framework sections have been updated to be more accessible.

For the article:
afrg.co/updj17/a3/

For all of the material:
afrg.co/updj17/

Part IV will be available before the end of the day!

Andrew

[Andrew Pinkham]

Hi,
Upgrading Django (to 1.7) Part IV: Upgrade Strategies is now available!

For the article:
afrg.co/updj17/a4/

For all of the material:
afrg.co/updj17/

I've also provided a checklist of upgrade steps, as well as a list of all the links used throughout the series:
afrg.co/updj17/cl/
afrg.co/updj17/l/

[Timothy W. Cook]

Excellent resource.

Thanks Andrew.

Andrew

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.
To post to this group, send email to django...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/73E604D5-964A-48CE-A0ED-F0856DE5AD2B%40andrewsforge.com.

For more options, visit https://groups.google.com/d/optout.

--

============================================
Timothy Cook
LinkedIn Profile:http://www.linkedin.com/in/timothywaynecook

MLHIM http://www.mlhim.org

[Fred Stluka]

Andrew,

Thanks for the shoutout as a reviewer.  Would be better to
link it to my one-many company's corporate site than to
just my links page.  Use:  http://bristle.com

I'm re-posting my privately emailed comments on the latest
version here so people can find them as part of this thread.

Looks good!

You did a great job of explaining the bits that I previously
didn't get:
- Relationship of INSTALLED_APPS to App Registry
- History of the App Registry
- AppCache, Apps, AppConfig
- apps.py
- Custom AppConfig

Also:
- app registry vs. master registry
- label vs. name in AppConfig
- Need for renamed imports
- Exact sed command to change an app's label
- AppConfig.ready()
- autodiscover()
- Apps restrictions (custom user model)

The Check Framework section is now also very clear, and it
seems to be very complete as well.  I like the iterative
approach you use: describing a simple way, then a problem,
then an enhancement that solves the problem, then another
problem, etc.  Nice job!  This will be very useful when I start
using the Check Framework to enforce coding standards for
my team when we move to Django 1.7 or 1.8.

--Fred

---

Fred Stluka -- mailto:fr...@bristle.com -- http://bristle.com/~fred/
Bristle Software, Inc -- http://bristle.com -- Glad to be of service!
Open Source: Without walls and fences, we need no Windows or Gates.

---

[Fred Stluka]

Andrew,

Part 4 looks great!

Thanks for the shoutout as a reviewer of Part 4.  Like with
Part 3, it would be better to link it to my one-man company's

corporate site than to just my links page.  Use: 
http://bristle.com

I'm re-posting my privately emailed comments on the latest
version here so people can find them as part of this thread.

Lots of good advice on how to upgrade, including:
- Tools/resources to use
- Value of having a test suite
- Use of DEV/TEST/PROD and virtualenv
- Importance of going to 1.5, then 1.6, then 1.7, as separate
  steps
- Upgrading packages you depend on
- Upgrading Python itself
- Release notes for each version
  -- Backwards-incompatible changes
  -- Deprecated features
  -- New features
- Use new Check Framework to enforce coding standards
- etc.

See also:
- http://andrewsforge.com/article/upgrading-django-to-17/checklist

Thanks!

--Fred

---

Fred Stluka -- mailto:fr...@bristle.com -- http://bristle.com/~fred/
Bristle Software, Inc -- http://bristle.com -- Glad to be of service!
Open Source: Without walls and fences, we need no Windows or Gates.

---