Do people actually squash migrations?

[Mike Lissner]

I have a pretty big django project, and since I created the 100th migration within one of its apps today, I thought I'd finally do some squashing. It hasn't gone well, but I eventually got the data migrations cleaned up.

Finally, I run it, and it runs smack into a CircularDependencyError, as described here:

https://code.djangoproject.com/ticket/23337

Basically, from what I understand, after the squash you have one migration that depends on various others from your other apps. Naturally, that totally falls over, because can't go from this series of migrations:

app1: migration 1

app2: migration 1

app2: migration 2

app1: migration 2

To, well...any series of migrations in which migration 1&2 from app1 or app2 have been squashed. The docs have something to say about this*, but it feels like this must affect practically any biggish project.

Stackoverflow also has a variety of dubious (and very complex) advice (read it and weep):

https://stackoverflow.com/questions/37711402/circular-dependency-when-squashing-django-migrations

So, my question is: Do people actually use squashmigrations with success? And if not, is it reasonable to consider deprecating it or fixing the bug, or updating the docs to loudly say it largely doesn't work? I'm surprised the issue above has so little movement since it was created seven years ago.

Maybe it's just me? If not, it'd be nice to do something to help future people with ambitions of a simple squash.

Thanks,

Mike

 

* Note that model interdependencies in Django can get very complex, and squashing may result in migrations that do not run; either mis-optimized (in which case you can try again with --no-optimize, though you should also report an issue), or with a CircularDependencyError, in which case you can manually resolve it.

To manually resolve a CircularDependencyError, break out one of the ForeignKeys in the circular dependency loop into a separate migration, and move the dependency on the other app with it. If you’re unsure, see how makemigrations deals with the problem when asked to create brand new migrations from your models. In a future release of Django, squashmigrations will be updated to attempt to resolve these errors itself. [Author's note: These sentences really leave me blowing in the wind...maybe I can figure out what they mean, I guess? I thought squashing was supposed to be easy.]

[Matthew Pava]

I've had similar issues. I just avoid squashing anymore. It's just not with the pain, and having so many little files that get looked at a minimal amount of time isn't worth fretting over. Saying that, I'd love to get it fixed...

Sent from my Verizon, Samsung Galaxy smartphone

Get Outlook for Android

---

From: 'Mike Lissner' via Django developers (Contributions to Django itself) <django-d...@googlegroups.com>
Sent: Tuesday, May 11, 2021 7:50:31 PM
To: Django developers (Contributions to Django itself) <django-d...@googlegroups.com>
Subject: Do people actually squash migrations?

 

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/87f449bc-d653-427a-ac28-879ee0701c8bn%40googlegroups.com.

[Kye Russell]

I’ve never successfully squashed my migrations to any material degree, but I’ve chalked that up to lack of doing it with any regularity. I suspect that squashing works a lot better if you aren’t trying to clean up a mess of hundreds of migrations files over 5 years, which is where I find myself! 

Kye

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/SN6PR05MB3951135EF4ECFE249F51ED4BF1529%40SN6PR05MB3951.namprd05.prod.outlook.com.

[Benny]

Sorry in advance of this isn’t helpful - We’ve done it successfully a few times since 1.11… but not without a sacrifice of the virgins-in-volcano variety. Just about any kind of RunPython command seems to get in the way. And sometimes it’s easier to blow them all away and create fresh migrations.

I have faith in migration squashing. It feels like more art than science, and to be sure - dealing with a database is more art than science in general… especially with all the creative minds that have access to such a logical system. I believe squashing should be used with care, and mostly on systems with ideal schema. Aside of that, I’ve worked on projects with migrations in the several-hundreds which took long enough to stand up that I could get a coffee and continue working.

All that said, I think it’s a nice-to-have feature, but ultimately isn’t necessary for the vast majority of projects. 

Benny

On May 11, 2021, at 7:50 PM, 'Mike Lissner' via Django developers (Contributions to Django itself) <django-d...@googlegroups.com> wrote:



[Andrew Godwin]

Migration squashing was always meant to be something that was useful in a rapid development environment where you can't control all the installs (since it was a feature developed alongside a CMS run by many clients at the time).

If you have control of all the places your project is installed and can have everyone developing on it do something at once, you can just do it via a complete migration reset and be happy. Squashing is for a very specific backwards-compatibility scenario that, I suspect, many Django projects developed internally are not in.

Andrew

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/76351f53-d185-4dd1-9979-6a7c73015ef5%40Spark.

[Mike Lissner]

So sort of sounds like an update to the squash migration docs is needed if this is representative of the general sentiment. Looking at the section on this, the general outline is:

1. Overview

2. How it works

3. The commands

4. Gotchas

5. A bunch of wonky stuff you have to do ("Update all migrations that depend on the deleted migrations", "Remove the 'replaces' attribute in the Migration class ")

6. Another gotcha in an info box

Would it be a bad idea to update the docs to bifurcate this section so it has an intro that says something like:

As you work on your project you will create more and more migrations. When they get to be too many, there are two approaches to trimming them down. The first is to use the squashmigrations command and process to create a merged migration file, however this approach comes with a number of caveats and gotchas that often make it impractical. The second way is to coordinate with your team to ensure that all installations of your app are up to date, then to have a coordinated day when migrations are removed and recreated from scratch. Which one is best for your organization will depend on the complexity of your project and the flexibility of your team.

From there, the docs could go on to explain first how to do this manually, then move onto the squashmigrations docs. This disfavors squashmigrations by putting it after the manual approach, but after this conversation (and my experience) that seems right to me. 

I haven't done the manual approach but I imagine it's something like:

1. Check your migrations across all apps with interdependencies for RunPython or RunSQL code.

2. If found, make a decision about keeping or deleting that code.

3. Delete all migrations across all apps that have interdependencies.

4. Run the makemigrations command

5. Add your custom RunPython or RunSQL code back

That'd be a big demotion for the squashmigrations code. I don't know how married we are to it, but it seems like there's not much energy for making it better and that lots of people have already demoted it in their minds and workflows.

Thanks,

Mike

 

[Mike Lissner]

Oh, I guess there's also a step in the manual process to reset the migrations table in the DB, but I don't know how to do that. Tricky stuff!

[Ryan Hiebert]

You’d run the migrations that you manually created with --fake. My experience also corroborates the idea that squashmigrations may be unsuitable for many situation that are similar to mine, where I am able to fully control the full set of places that the code is deployed.

Ryan

-- 
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/ce1cb4d6-905f-4411-93df-4449796558a8n%40googlegroups.com.

[Raffaele Salmaso]

On Wed, May 12, 2021 at 2:50 AM 'Mike Lissner' via Django developers (Contributions to Django itself) <django-d...@googlegroups.com> wrote:

So, my question is: Do people actually use squashmigrations with success?

For what is worth: yes. The only problem I have is a pbcak problem (I like to do esoteric things just because).

--

| Raffaele Salmaso
| https://salmaso.org
| https://bitbucket.org/rsalmaso
| https://github.com/rsalmaso

[René Fleschenberg]

Hi,

I agree that it would be good to extend the docs and to describe how to
reset a project's migrations. Some prior art on this:

https://geekchick77.dreamwidth.org/5560.html

https://simpleisbetterthancomplex.com/tutorial/2016/07/26/how-to-reset-migrations.html


Regards,
René

[Mike Lissner]

Thanks for those links. I went ahead and filed a PR that describes a new "Migration Trimming" process. I'd love comments and suggestions if folks are interested and able:

https://github.com/django/django/pull/14408

Mike

--
You received this message because you are subscribed to a topic in the Google Groups "Django developers  (Contributions to Django itself)" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/django-developers/xpeFRpMTBZw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to django-develop...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/9d103a6d-1301-1a37-ef35-29b6f1521f51%40fleschenberg.net.

--

Mike Lissner

Executive Director

Free Law Project

https://free.law

[Hanne Moa]

On Wed, 12 May 2021 at 18:40, 'Mike Lissner' via Django developers

(Contributions to Django itself) <django-d...@googlegroups.com>

wrote:

> Oh, I guess there's also a step in the manual process to reset the migrations table in the DB, but I don't know how to do that. Tricky stuff!

I've made a management command for that: resetmigrationhistory, which
nukes just the migrations of *my* apps. Then I run migrate --fake
manually afterwards.

I squash as part of any major release (semver). Rewrite them a lot by
hand to get rid of any datamigrations and have lots of extra
mini-releseas to make sure everything works etc.

There's https://github.com/kingbuzzman/django-squash, which I have
considered using, their "elidable" concept is very good.


HM

[Shai Berger]

On Wed, 12 May 2021 09:37:53 -0700 (PDT)

"'Mike Lissner' via Django developers (Contributions to Django

itself)" <django-d...@googlegroups.com> wrote:

>
> I haven't done the manual approach but I imagine it's something like:
>
> 1. Check your migrations across all apps with interdependencies for
> RunPython or RunSQL code.
> 2. If found, make a decision about keeping or deleting that code.
> 3. Delete all migrations across all apps that have interdependencies.
> 4. Run the makemigrations command
> 5. Add your custom RunPython or RunSQL code back
>

I've tried it. In my experience, it isn't as easy as described either.

When you have circular dependencies between the migration of apps, in
many (most?) cases it means you also have circular dependencies between
the models (FKs etc going in both directions). When that happens,
makemigrations from scratch doesn't do the right thing either --
basically, you need to recreate the interdependencies by having each
app's models created in two steps: One for the "core" of the models,
so they all exist, and one for the relations, which can only be added
once all the referenced models are present.

The makemigrations command will only add one migration per app, and so
it cannot do this. I haven't tried this in a while, so I don't remember
if the command actually fails, or just produces non-working migrations
which you then need to edit by hand.

This all ties in to a discussion we had on the forum, about the
separation to apps in large projects (linking to my own post there, but
the whole discussion is worth a read if you haven't):
https://forum.djangoproject.com/t/why-do-we-need-apps/827/20

Have fun,
Shai.