Deprecating model field (Deleting model field, but keeping DB column)

3,442 views
Skip to first unread message

tay...@cedar.com

unread,
Jul 5, 2017, 6:55:58 AM7/5/17
to Django users
I am having some trouble figuring out the best way to remove model fields in Django. If I remove a field from a model and then run makemigrations, it creates a RemoveField operation in a migration. That is great, but if I decide to run the migration before releasing the new code, the existing code will break (for a short time between running the migration and releasing the new code) because the old code is still querying for the removed column (Django queries for all columns by default). I could run the migration after the release, but that won't work if I also have an AddField operation because the new code needs the new column, so it needs to be run before. I am wondering if anyone has solved this issue?

My best solution (I don't think Django supports this) would be to have a special type of field called a DeprecatedField. It would delete the field from Django's perspective, but keep the column in the DB. Django would no longer query for the column, but the column would still be in the DB. On the next release, I could remove the column completely (with a RemoveField operation) and the existing code would not error because it has no knowledge of the column.

I noticed Django has an idea of a private field, which is on a model but not in the DB. Is there a way to create a field that is in the DB, but Django model doesn't query for it or allow it to be used in creates and updates? Very similar to the managed=False on the Model, but on the Field level. If anyone has other approaches to the problem, I would be very excited to find alternative methods.

Thanks,
Taylor

Avraham Serour

unread,
Jul 5, 2017, 7:27:22 AM7/5/17
to django-users
you can remove the field and don't run migrations until you are ready to actually remove the column, or you may run migrations fake and leave the column there forever


--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscribe@googlegroups.com.
To post to this group, send email to django...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/a52ae01a-1a7d-43ce-a94f-fb00c4e1b7d1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

tay...@cedar.com

unread,
Jul 5, 2017, 8:09:29 AM7/5/17
to Django users
Thanks for responding Avraham.

That would be a good option if I was developing by myself, but I am working with a team of 20 developers. The process needs to be the same whether there are deprecated fields or not. I can't realistically expect 20 people to not apply one migration (or a few specific ones). There may be other migrations after the deprecation that need to be applied, so it's very difficult to apply certain ones and ignore others. It would be similarly difficult to get everyone to apply one of the migrations with --fake, but do a different process with every other migration. Also, --fake applies to the entire migration (not just specific operations), so people would be forced to make separate migrations for other model changes and hopefully remember not to run --fake on those.

My current best attempt was to create a custom DB migration operation https://docs.djangoproject.com/en/1.11/ref/migration-operations/#writing-your-own , that removes the field in state_forwards, but doesn't do anything to the db in database_forwards. That works, but the person that deprecates the field need to remember to change the RemoveField operation into the custom DeprecateField operation. It would be great if makemigrations created the correct operations automatically. Is there a way to do that?

On Wednesday, July 5, 2017 at 7:27:22 AM UTC-4, Avraham Serour wrote:
you can remove the field and don't run migrations until you are ready to actually remove the column, or you may run migrations fake and leave the column there forever

On Wed, Jul 5, 2017 at 6:39 AM, <tay...@cedar.com> wrote:
I am having some trouble figuring out the best way to remove model fields in Django. If I remove a field from a model and then run makemigrations, it creates a RemoveField operation in a migration. That is great, but if I decide to run the migration before releasing the new code, the existing code will break (for a short time between running the migration and releasing the new code) because the old code is still querying for the removed column (Django queries for all columns by default). I could run the migration after the release, but that won't work if I also have an AddField operation because the new code needs the new column, so it needs to be run before. I am wondering if anyone has solved this issue?

My best solution (I don't think Django supports this) would be to have a special type of field called a DeprecatedField. It would delete the field from Django's perspective, but keep the column in the DB. Django would no longer query for the column, but the column would still be in the DB. On the next release, I could remove the column completely (with a RemoveField operation) and the existing code would not error because it has no knowledge of the column.

I noticed Django has an idea of a private field, which is on a model but not in the DB. Is there a way to create a field that is in the DB, but Django model doesn't query for it or allow it to be used in creates and updates? Very similar to the managed=False on the Model, but on the Field level. If anyone has other approaches to the problem, I would be very excited to find alternative methods.

Thanks,
Taylor

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.

Jani Tiainen

unread,
Jul 5, 2017, 8:19:56 AM7/5/17
to django...@googlegroups.com

Hi,

Sounds like your all developers do use same database if you have such a problems.

It's usually good practice to have per developer development database. That will allow individual developers to do changes to database and migrate others as they please. Also it doesn't "matter" if one developer breaks their database for example by accidentally running migrations that are not in the repo yet.

Of course, it requires that you have either database creation script, or like we do, we clone our staging database for development basis.


For more options, visit https://groups.google.com/d/optout.

-- 
Jani Tiainen

Avraham Serour

unread,
Jul 5, 2017, 8:41:11 AM7/5/17
to django-users
From what you are describing it seems that you are planning for the long term. which I don't believe is ideal
You may mark the whole model as not managed, but I don't think you can mark just one field and unmanaged.

Another simple approach would be to transform the attribute to a property and print deprecation warnings or raise an exception whenever someone tries to write to it (or read)

I hope that if you have tests it will be enough to remove all use of the deprecated field

Is that temporary? That would influence the approach
From what I understood from your original post this is just a matter of deploying the new app version that removes a field, so you would just need to run migrations after deploying


To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscribe@googlegroups.com.

-- 
Jani Tiainen

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscribe@googlegroups.com.

To post to this group, send email to django...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/d4ba64c5-0dac-bcb4-520f-78835d049ad4%40gmail.com.

tay...@cedar.com

unread,
Jul 5, 2017, 10:19:55 AM7/5/17
to Django users
Avraham,
Thanks for your answer again. When you say planning for the long term is not ideal, what do you mean? I am interested in learning better processes. I do currently work with 20 developers, if that is what you meant.

Your approach with a property was my first attempt. That didn't work because Django still queries for the column by default, unless every single queryset in the project we put .only(<fields>) excluding the deprecated columns, which can introduce other issues if we add new fields. I tried overriding get_queryset to do that globally on the model, but Django doesn't respect .only when doing inserts, only updates and retrieves.

We do have very good test coverage, so warnings would be a great option if I could find a way to have Django not try to write or query that field by default.

In regards to running migrations after deploying, that would be very difficult. Some migrations need to be run before a code release (adding fields), so people would have to be very careful not to put an AddField and a RemoveField in the same code release. Even if that was possible, we don't release manually (we use continuous deployment), so we would have to write some kind of migration reader that could determine whether to run a migration before or after the release.

With your question about temporary, it may or may not be temporary, it depends on the field. Certain fields can later be removed safely, but there will be cases where we keep deprecated fields forever for reporting reasons, etc.


-- 
Jani Tiainen

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.
To post to this group, send email to django...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.

Thomas Lockhart

unread,
Jul 5, 2017, 10:41:02 AM7/5/17
to django...@googlegroups.com

> On Jul 4, 2017, at 8:39 PM, tay...@cedar.com wrote:
>
> I am having some trouble figuring out the best way to remove model fields in Django. If I remove a field from a model and then run makemigrations, it creates a RemoveField operation in a migration. That is great, but if I decide to run the migration before releasing the new code, the existing code will break (for a short time between running the migration and releasing the new code) because the old code is still querying for the removed column (Django queries for all columns by default). I could run the migration after the release, but that won't work if I also have an AddField operation because the new code needs the new column, so it needs to be run before. I am wondering if anyone has solved this issue?


A common practice is to do this in several upgrade steps:

1) Add new field as nullable.
2) Add code to populate and use new field, and migrate.
3) Remove code using old field.
4) Remove old field, and migrate.

With a large development team the work flow is perhaps not much different that managing production releases, since you may not want to have broken installations. It also gives needed practice at managing deployed releases for upgrades.

hth

- Tom

Alceu Rodrigues de Freitas Junior

unread,
Jul 5, 2017, 3:11:17 PM7/5/17
to django...@googlegroups.com

It seems to me more a organization/project management isssue that something that should be handled by Django itself. Well, if I got the problem correctly, that is.

It seems you can't remove the field from the Model class because that would remove the column from the underline table itself as soon as the migration is executed.

If your developers are doing development on their own computers (like they should), it would be a matter to leave them working on a different branch of the SCM where you're not applying that. Later is just a matter to provide comunication that they should update their own local repositories after you finished the merge process.

In the case you need to keep the column around for a while, you might want to add some abstraction, like a view with a harcoded value for that column.

tay...@cedar.com

unread,
Jul 5, 2017, 9:14:54 PM7/5/17
to Django users
I am sorry everyone. I guess I am not very good at explaining the problem. I will try explain it better. Please ignore everything I have written before because it seems to be confusing.

I have perfectly working code in production. Later, I want to make a model change to remove a field and update the code to not use the field. I use makemigrations to create a migration with a RemoveField operation. When it comes time to release my new code, if I run the migration first in production (before pushing the code), for a few seconds I will get errors in queries. The old code will try to query the deleted field and will error.

Your immediate response may be "just run the migration after the code release". Well that could work if there are only RemoveField operations. If there is a AddField operation somewhere, the migrations must be run before the code release. So there is the dilemma. Depending upon the operation in the migration, a migration
may need to be run before or after.

My solution would be to always run migrations before a release, but all migrations must be backwards compatible. I am looking for a way to remove fields from the model, but keep the field in the DB, so the old code doesn't break.

Jani Tiainen

unread,
Jul 6, 2017, 1:46:23 AM7/6/17
to django...@googlegroups.com

Hi,

Well, seems that your deployment process needs fixing.

Simplest option is to stop serving site (put it to maintenance mode to show for example static maintenance page), update codebase and run migrations. Restart servers and put django serving back.

If you're looking zero downtime migrations you have to plan model modifications very carefully.

Here is pretty nice article how it could be done:
http://lucasroesler.com/2017/02/zero-downtime-deploys-a-tale-of-django-migrations/

For more options, visit https://groups.google.com/d/optout.

-- 
Jani Tiainen

Melvyn Sopacua

unread,
Jul 6, 2017, 5:38:56 AM7/6/17
to django...@googlegroups.com

On Tuesday 04 July 2017 20:39:37 tay...@cedar.com wrote:

> I am having some trouble figuring out the best way to remove model

> fields in Django. If I remove a field from a model and then run

> makemigrations, it creates a RemoveField operation in a migration.

> That is great, but if I decide to run the migration before releasing

> the new code, the existing code will break (for a short time between

> running the migration and releasing the new code) because the old

> code is still querying for the removed column (Django queries for all

> columns by default). I could run the migration after the release, but

> that won't work if I also have an AddField operation because the new

> code needs the new column, so it needs to be run before. I am

> wondering if anyone has solved this issue?

 

You're fixing this in the wrong place. This is for the project manager and technical lead to fix. If you have dependencies between releases that break the product if either one of them is not applied, then you need to move components of those releases so that it doesn't happen.

 

Also, migrations exist to have a consistent code versus database.

 

A developer's task to "remove field A", isn't simply to remove the field from the model and makemigrations. Her task isn't done till all querysets that reference that field by name are refactored.

 

--

Melvyn Sopacua

Reply all
Reply to author
Forward
0 new messages