[Feature Request][Model, ORM] Disabling a field before removal to support continuous delivery

[Matthieu Rudelle]

Hi there, 

I can't find any previous ticket proposing a solution to this problem so here are my findings: 

**Use case**:

When using continuous delivery several versions of the code can be running in parallel on se same DB. Say for instance that release 2.42 is in production, 2.43 is about to be rolled out and in this release one field (say ''MyModel.my_unused_field'') is not used anymore and was removed. Before rolling out 2.43 the DB is migrated and column ''my_unused_field'' of ''MyModel'' is removed. This makes 2.42 crash saying that one column is not found even though 2.42 does not use the field anywhere in the code.

**Temporary solution**:

Do not makemigrations until de 2.44 release, but it does not scale well with many contributors and CI tools (doing their awesome job of making sure migrations and models are in sync) will complain.

**Proposed solution**:

Have a ''disabled'' param on Field. When activated this field is not fetched from the DB but replaced by a hardcoded value. 

In our use case, ''disabled'' is added at the 2.42 release, then when 2.43 rolls out and migrates the DB no error is thrown.

**Refs**:

- django-users discussion: https://groups.google.com/forum/#!topic/django-users/HY_6rZZ0Kk8

- the same problem discussed in this article, but with a slightly different solution: https://pankrat.github.io/2015/django-migrations-without-downtimes/#django-wishlist (third item in the wishlist)

- Previous django-developpers discussion: https://groups.google.com/d/msg/django-developers/5ofrxeLT95E/XhppgA_KAQAJ

What do you guys think?

[Ryan Hiebert]

I'm not sure about the solution you mentioned, but the problem you mention is one that I definitely do deal with. At my work we have been happy with using a "safe" migrate command that only runs migrations that are marked as safe to run before the deployment happens, to address exactly this kind of scenario. You still need to make sure you don't mix stuff in the same migration that delete and add a column, for example, but it makes the process much easier, and we've been able to add the safemigrate command to our regular automated deployments. The full migrate then only runs when manually triggered, to deal with removing columns, etc.

You can check out our repository. We've been pretty happy with how it's working for us. https://github.com/aspiredu/django-safemigrate

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/637c3542-c5bd-4493-8b12-44eb61f34a68%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Dan Davis]

I'd agree that it is a definite use case.  In the dev-ops world, it is evidently called a "blue torquoise green deployment".   It could be done as long as the code is not adding a table/field/etc.   

My discussion on django-users, https://groups.google.com/forum/#!topic/django-users/QCmy9reH8cI, also included this issue, but I was also concerned about:

* Removing a model

* Renaming a field

* etc.

The django-safemigrate repository/app seems to be easier to generalize, as such an option could be added to makemigrations.

Some questions:

• How does the "safe" field of migrations work with other migrations related commands, such as squashmigrations?   It seems to me that only migrations that share the same value of "safe" could be squashed.
• Can makemigrations figure out which migrations are "safe", such as:
• It is always safe to add a column or table.
• It is never safe to drop or rename a column or table.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CABpHFHSa4hXVrp7M2MWLcsPXSRnCnvhMQ%3D8D3q1Rg8AsAXU4Cw%40mail.gmail.com.

[Ryan Hiebert]

On Mon, Jun 24, 2019, 21:29 Dan Davis <dans...@gmail.com> wrote:

Some questions:

• How does the "safe" field of migrations work with other migrations related commands, such as squashmigrations?   It seems to me that only migrations that share the same value of "safe" could be squashed.

If does not affect squashing. I think this is roughly reasonable, at least for my uses, although I haven't had any real world use for squash myself. I recently had to remake the migrations in my repo for $WORK project, and it made the most sense to keep them as the default safe after deploy, to match what other apps would get. I think in most cases it won't matter when the first migration is safe for practical use.

• Can makemigrations figure out which migrations are "safe", such as:
• It is always safe to add a column or table.
• It is never safe to drop or rename a column or table.

This is a feature that has sounded neat to me, approximately feasible, but hasn't yet been worth the time to implement. We've been happy adding the safety annotations manually.

[Dan Davis]

If does not affect squashing. I think this is roughly reasonable, at least for my uses, although I haven't had any real world use for squash myself. I recently had to remake the migrations in my repo for $WORK project, and it made the most sense to keep them as the default safe after deploy, to match what other apps would get. I think in most cases it won't matter when the first migration is safe for practical use.

I don't usually use squashmigrations either, being more likely to simply rewrite or make them initial.   However, if the functionality of your module were to be added to Django itself, or become more productized, then these are the issues to think about, right?

I certainly think the use case (support for something like blue/green deployments with shared database) is very important.

I'm not a super experienced Django contributor, so I'd love to hear what more experienced hands have to say about it.

--

You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CABpHFHSb2AHKb-XhZcAk6gZ3%2B%2Bt5nz7UkBUNyQ_FC%3DsYGgO_Yw%40mail.gmail.com.

[Paveł Tyślacki]

there are similar to `alter table drop column`  issue:  `alter table rename column`, `drop table`, `rename table`. (honestly `alter table drop column` and `drop table` a bit different wiith `alter table remane column` and `rename table`)

Look like you general flow for migration:

1. change code and create migrations

2. apply migrations

3. apply code for all your instances

the issue in this cases I can describe: we have code that we still used.

for `alter table drop column`, `drop table` next scenario works fine (as your temp solution too):

1. remove column/table usage from codebase

2. apply this code for all of your instances

3. apply migration with field/table removal

But if you mix migrations for `alter table add column` and `alter table drop column` - you cannot safely apply both migrations simultaneously, and your proposal sounds pretty reasonable there.

However if you add disabled option, then make migration, remove field, then make migration, then apply this changes to your environment - you will get same issue. So to automate deployment process and to get achievable result you want I think you need something more complex: deploying atomic changes, but for me it sounds more about deployment than django itself.

[Matthieu Rudelle]

How does the "safe" field of migrations work with other migrations related commands, such as squashmigrations? 

Squashmigrations typically targets and produces migrations that are old enough to be assumed safe.

 

You can check out our repository. We've been pretty happy with how it's working for us. https://github.com/aspiredu/django-safemigrate

safe-migrate is a neat solution. Although it lacks support for rollbacks: as soon as "migrate" is run the app is stuck in the current release

If migrations are seen as a rolling window, you want a window of X releases (2 in our case) to play well together (minus discrepancies due to updates to the application logic) so being able to postpone a column removal to a later release is important. The "disabled" feature (or anything similar to a flag written in the codebase) has this added benefit.

Squash migration would be used to squash up to but excluding the current sliding window. 

there are similar to `alter table drop column`  issue:  `alter table rename column`, `drop table`, `rename table`. (honestly `alter table drop column` and `drop table` a bit different wiith `alter table remane column` and `rename table`)

• `alter table rename column` can be reduced to `alter table add column` and later `alter table drop column` plus some temporary update replication implemented in the app (idem for `rename table`) 
  
• `drop table` follow the same logic with the SQL actions delayed to a later release

But if you mix migrations for `alter table add column` and `alter table drop column` - you cannot safely apply both migrations simultaneously, and your proposal sounds pretty reasonable there. 

 

However if you add disabled option, then make migration, remove field, then make migration, then apply this changes to your environment - you will get same issue. So to automate deployment process and to get achievable result you want I think you need something more complex: deploying atomic changes, but for me it sounds more about deployment than django itself.

As long as the columns are different these actions can happen in parallel. The flag will just delay the DB migration to a later release and the deployment needs this kind of flexibility from django to avoid direct access to the DB by the deployment process.

[Matthieu Rudelle]

Does that sound like feature-request material? It seems to me django is the good place to add this flexibility as it has exclusive access to the underlying DB.

--
You received this message because you are subscribed to a topic in the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/django-developers/Gr9x2OlWYN4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to django-develop...@googlegroups.com.

To post to this group, send email to django-d...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/59bd29b1-5de3-4239-b534-fc5146995f72%40googlegroups.com.

[Pkl]

Just for reference, here is a package I crossed recently and which has an approach for that : https://github.com/3YOURMIND/django-deprecate-fields

[Matthieu Rudelle]

Nice find, It's an interesting way to solve it. We might give it a try.

[Pkl]

This society also has linters and such, so that DB migrations are safer regarding deployment rollouts - https://medium.com/3yourmind/keeping-django-database-migrations-backward-compatible-727820260dbb