[FEATURE] Allow squashmigrations to squash the whole project

[Javier Buzzi]

This comes from a place of headaches: we're a large team, and our sprint are a month long, in that month we generate a lot of migrations, after a couple of sprints we see our tests take a real toll when running migrations, this is due to constantly adding columns/deleting columns, moving data around -- what normally happens with an ever changing app.

The way quashing migrations currently works is by an app by app, I'm proposing that if no arguments are supplied (no app_label, start_migration_name, migration_name) to squash the entire project, who's apps belong to the project of course.

I'm hopping that I can get all the modifications already done to this project: https://github.com/kingbuzzman/django-squash and simply integrate it directly into the django source. I don't want to take such an undertaking and just have the PR be rejected.

Is there any interest for this?

Thanks,

Javier Buzzi

[Andrew Godwin]

I definitely think this is worth pursuing - the reason I didn't do it back in the day was that squashing the entire project involved solving some rather nasty dependency graph issues if there were any cycles of apps depending on models from other apps.

If we can overcome that, though, and squash down to just a couple of migrations each app, it would be nice to have that be a shortcut.

Andrew

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/ad3e7507-01af-49a6-ab15-e1b6d7d32bb7%40googlegroups.com.

[Javier Buzzi]

Sounds great Andrew, i'm going to say this up front:

1. You're awesome, thank you for making south and migrations.
2. I'm inevitably need your help on this one, there were a lot of hacks I did because I couldn't come up with a better way. I'll get started, as a v1 to just port all the work done from the other project, and from there I will REALLY welcome your insight, guidance, and specially your criticism.

Thank you!

Javier Buzzi

[Adam Johnson]

One concern here is to avoid squashing third party apps' migrations. Django doesn't currently distinguish between third party and project apps. It might be possible to use the heuristic "is site-packages in the apps' module's absolute path?" But this is not 100%  accurate since third party packages can be installed other ways.

--

You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/220584c4-a0bc-4535-906e-f120b7080add%40googlegroups.com.

--

Adam

[Javier Buzzi]

Correct. Im adding that functionality currently -- im taking the VERY naive approach of looking at BASE_DIR and if the app is not in there, its foreign to the project. Some insight or better ideas to this would be appreciated.

[Adam Johnson]

That's good, but unfortunately BASE_DIR is only a "pseudo setting." It's something in the startproject template that's useful for building file paths but nothing inside Django actually relies on it or checks that it's set. Perhaps we should consider "promoting" it to a real setting though, to make this easier. Still, that might be naive for projects that bundle their dependencies in some way :/

(Slightly tangential - my post on BASE_DIR and its move to pathlib in 3.1: https://adamj.eu/tech/2020/03/16/use-pathlib-in-your-django-project/ )

On Tue, 28 Apr 2020 at 13:36, Javier Buzzi <buzzi....@gmail.com> wrote:

Correct. Im adding that functionality currently -- im taking the VERY naive approach of looking at BASE_DIR and if the app is not in there, its foreign to the project. Some insight or better ideas to this would be appreciated.

--

You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/f202a4bb-1d7b-4c85-9695-f4ef3b996c3c%40googlegroups.com.

--

Adam

[Markus Holtermann]

Have you considered to allow for multiple app_labels in the `squashmigrations` command, maybe together with a `--cross-app` flag, to specify the apps from which migrations should be squashed? That way we don't need to rely on paths at all, but can log up all migrations in question, based on the provided app labels.

Cheers,

Markus

> > To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/f202a4bb-1d7b-4c85-9695-f4ef3b996c3c%40googlegroups.com <https://groups.google.com/d/msgid/django-developers/f202a4bb-1d7b-4c85-9695-f4ef3b996c3c%40googlegroups.com?utm_medium=email&utm_source=footer>.
>
>
> --
> Adam

>
> --
> You received this message because you are subscribed to the Google
> Groups "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to django-develop...@googlegroups.com.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/django-developers/CAMyDDM2%3DSTi9pe3LyaYL4%2BLEu8ZcWCgcjynR5MvV30PJiTsoDQ%40mail.gmail.com <https://groups.google.com/d/msgid/django-developers/CAMyDDM2%3DSTi9pe3LyaYL4%2BLEu8ZcWCgcjynR5MvV30PJiTsoDQ%40mail.gmail.com?utm_medium=email&utm_source=footer>.

[Adam Johnson]

That's a good suggestion Markus, and more flexible.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/eaa8b320-15b9-43ab-b50c-3bb6a704898f%40www.fastmail.com.

--

Adam

[Javier Buzzi]

And then do?

$ ./manage.py squashmigrations --crossapp=$(find . -name apps.py | sed 's|/apps.py$||; s|^./||; s|/|.|g' | xargs printf '%s,')

We have 54 apps currently.. more to come. I'm sure there are projects out there with more, this can get frighting, specially for windows users -- not sure what the equivalent powershell/batch command is -- do they have limitations in terms of command length? Perhaps I'm only looking at this from my narrow point of view where we squash constantly, every app after each release, maybe this is not for everyone. If you guys think this is the way to go, let's do that then.

[charettes]

I guess we could allow an "__all__" or "all" sentinel to mean _all apps_ like we use "zero" to revert the initial migration.

[Javier Buzzi]

To do that we need to know which apps are part of your project, and which are not. Otherwise you'll be squashing "django.contrib.auth" as an example, and you're not going to have a fun time at that..

[charettes]

Don't we need to prevent that from happening anyway? I guess we'd want to prevent --crossapp=auth from working as well.

[Javier Buzzi]

To be kind to the user, absolutely! .. any idea on how to implement it O:)

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/ca814e82-9258-40eb-afed-82edb0bc29c8%40googlegroups.com.

[charettes]

It's notoriously hard to distinguish between first-party and third-party apps but I know some package manage to do it relatively well (e.g. isort).

Le mardi 28 avril 2020 14:32:13 UTC-4, Javier Buzzi a écrit :

To be kind to the user, absolutely! .. any idea on how to implement it O:)

On Apr 28, 2020, at 2:29 PM, charettes <char...@gmail.com> wrote:

Don't we need to prevent that from happening anyway? I guess we'd want to prevent --crossapp=auth from working as well.

Le mardi 28 avril 2020 13:50:54 UTC-4, Javier Buzzi a écrit :

To do that we need to know which apps are part of your project, and which are not. Otherwise you'll be squashing "django.contrib.auth" as an example, and you're not going to have a fun time at that..

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.

To unsubscribe from this group and stop receiving emails from it, send an email to django-d...@googlegroups.com.

[Markus Holtermann]

But for isort one specifies the `known_first_party` and `known_third_party` packages. https://github.com/timothycrosley/isort#configuring-isort

At least I was under the impression that that's the only way how it decides where to place imports.

Cheers,

Markus

> >> To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/ca814e82-9258-40eb-afed-82edb0bc29c8%40googlegroups.com <https://groups.google.com/d/msgid/django-developers/ca814e82-9258-40eb-afed-82edb0bc29c8%40googlegroups.com?utm_medium=email&utm_source=footer>.

> >
>
> --
> You received this message because you are subscribed to the Google
> Groups "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send

> an email to django-develop...@googlegroups.com.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/django-developers/8af08a8f-4f4d-4fef-8daa-e3743a1f8468%40googlegroups.com <https://groups.google.com/d/msgid/django-developers/8af08a8f-4f4d-4fef-8daa-e3743a1f8468%40googlegroups.com?utm_medium=email&utm_source=footer>.

[charettes]

You don't have to explicitly configure them as long as you run isort within a Python environment that has all dependency installed properly. If you want to run isort without installing all of the deps of your package you have to explicitly specify which module is first or third party.

Here's the routine I was referring to https://github.com/timothycrosley/isort/blob/0ee43b6a4f3c40876c3c7714846ecf8e584dc314/isort/finders.py#L165-L191

Simon

> https://groups.google.com/d/msgid/django-developers/8af08a8f-4f4d-4fef-8daa-e3743a1f8468%40googlegroups.com <https://groups.google.com/d/msgid/django-developers/8af08a8f-4f4d-4fef-8daa-e3743a1f8468%40googlegroups.com?utm_medium=email&utm_source=footer>.

[Javier Buzzi]

How terrible would something like this be:

class AppConfig:

    """Class representing a Django application and its configuration."""

    .....

   @cached_property

    def is_squashable(self):

        if hasattr(self, 'squashable'):

            return self.squashable

        .... code from isort

in your local apps.py you can chose to do:

class MyAppConfig(AppConfig):

    squashable = False  # or omit it, if you want django to figure it out

This plays back into a requirement i had, where we have apps just for testing inside our application that we don't want squashed. We would also need to have a little more binding power from Migration to the AppConfig, since we would need to pick this value up. From my naive research, doesn't seem like an issue since Apps should always exist in order to make the migration anyway, i dont see a big risk in coupling the two.

What does everyone think?

Buzzi

To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/2a1d8093-8447-4a1d-a434-53d03cc6680d%40googlegroups.com.