Question (potential feature request) on dropping database column in migration

[martin_x]

Hi there,

Thanks for making Django a great framework for starters :D

Recently my team have run into problems when trying to remove a database column from the model without stopping our Django server, below is the timeline:

1. Before migration happens, we have column_A we want to remove, and running on old release_1
2. Now we run migration to remove column_A, our webserver is still running on old release_1 and serving requests
3. After migration, we ship the new release_2

However, between step 2 and 3, there are requests coming and referencing column_A we want to remove, and throws a exception of course.

So is there anyway I can mark a database column a special state (deprecated, purged in memory, etc), so it has the following affect:
1. Won’t generate migration file, which means database wise, nothing has changed
2. However, when Django loads the model into memory, it will ignore column_A completely. So query, create, update, etc won’t try to access column_A.

The reason we want to do it that way is so we can achieve no downtime, seamless migration for our application. I believe more people will benefit from this.

Please let me know if more information is needed. Looking forward to hearing from you.

Martin

[Tobias McNulty]

Hello,

I don't think this is something Django can or should handle. You need to make your change in smaller increments so it doesn't break.

The proper forum for discussing this is the Django users list (or IRC or Google...).

Good luck!

Tobias

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/50dbe737-0b6c-42d2-9f76-99c188c916e7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Curtis Maloney]

On 10/02/2018 09:42 AM, martin_x wrote:
> Hi there,
>
> Thanks for making Django a great framework for starters :D
>
> Recently my team have run into problems when trying to remove a database
> column from the model without stopping our Django server, below is the
> timeline:

There have been processes developed for addressing this, but they're
quite involved.

You need to ensure any field the current running codebase knows about
never disappears. So, the process of removing a field must be spread
across multiple code deployments

The first setting it nullable [or with a "harmless" default] so when the
new code inserts new rows the DBMS doesn't complain.

The next deploy can then drop the field, as Django no longer interacts
with it.

Either way, you're still likely to encounter a full-table exclusive lock
whilst the DBMS rewrites the table to no longer include that field.

Hope this helps some.

--
Curtis

[Todor Velichkov]

Won’t generate migration file, which means database wise, nothing has changed

You are the one generating migration files, Django is not forcing them.

The problem is not in Django, the problem is that this cannot be solved with a single deploy, there is not way for the old code to know whats coming in the future, which means you need to make a deploy just for this, and after that make a second deploy with the real changes. But if you are gonna make two deploys then you don't need Django to help you, you can solve this alone. Here is an example: Lets say we want to migrate from a full_name column to first_name and last_name columns. Here is what you can do:

1) Add first_name and last_name into the model and create a migration (which adds two columns and eventually populates them with data from full_name).

2) Change the code to make use of the new fields, remove full_name from the code, BUT do not make migration yet!

3) Deploy.

Now, during the deployment your old code will still use full_name which is still in the database, and when the new code gets live it will start using the new columns.

Note: during the deployment you need to run migrations first, and reload the server after that, so your new code won't fail because new columns don't exist yet.

Note2: since you are running the migrations first, and restarting the server later, in theory its possible for a new data to appear into the database from the old code! Thus you wont have it migrated yet. e.g. A new user registered right after the migration ended and before the new code gets live, his name will be stored into the old full_name column! But you can fix this on the next step!

4) Now after the new code is live, and no one is using full_name anymore create a new migrations which will remove it from the database. In this migration you can double check that all data from full_name is properly migrated to the new columns.

5) Deploy again. (no code changes, just removing the old unused column from the database).

[vishvajit pathak]

Well this is what we call as the downtime.