Improve migration conflict detection

[Caio Ariede]

Hey folks,

I was looking at the code used to detect conflicts in migrations [1]. It seems to use a very safe approach, by avoiding with multiple node leafs in the migration graph.

While this is safe, I’ve been having some problems when it comes to scalability when having multiple migrations created in a short period of time (for the same app).

Questions:

1. Are there any obvious impediments on improving the conflicts detection?

2. Does anyone have ideas on how to improve the conflict detection? (eg. going down from app-level to model-level detection)

Thanks!

[1] https://github.com/django/django/blob/e3f6e18513224c8ad081e5a19da641f49b0b43da/django/db/migrations/loader.py#L301-L313

--

Caio Ariede

caio....@gmail.com

[jackotonye]

Definitely a plus one on auto resolving migrations a test package still in planning aims to solve this https://github.com/jackton1/django-migration-resolver-hook

On Feb 11, 2020, at 1:42 PM, Caio Ariede <caio....@gmail.com> wrote:

Hey folks,

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/FE717E60-7B66-4050-B233-20C47FBF6038%40gmail.com.

[caio]

Cool. If I'm understanding this correctly, it auto-resolves during makemigrations?

I'm looking for something that could handle conflicts during the migrate command, but I'm not sure if that's really possible. I guess it depends on how intrinsic the single-leaf-node restriction is to the whole migration system

Thanks!

Em terça-feira, 11 de fevereiro de 2020 16:22:16 UTC-3, jackotonye escreveu:

Definitely a plus one on auto resolving migrations a test package still in planning aims to solve this https://github.com/jackton1/django-migration-resolver-hook

On Feb 11, 2020, at 1:42 PM, Caio Ariede <caio....@gmail.com> wrote:

Hey folks,

I was looking at the code used to detect conflicts in migrations [1]. It seems to use a very safe approach, by avoiding with multiple node leafs in the migration graph.

While this is safe, I’ve been having some problems when it comes to scalability when having multiple migrations created in a short period of time (for the same app).

Questions:

1. Are there any obvious impediments on improving the conflicts detection?

2. Does anyone have ideas on how to improve the conflict detection? (eg. going down from app-level to model-level detection)

Thanks!

[1] https://github.com/django/django/blob/e3f6e18513224c8ad081e5a19da641f49b0b43da/django/db/migrations/loader.py#L301-L313

--

Caio Ariede

caio....@gmail.com

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.

To unsubscribe from this group and stop receiving emails from it, send an email to django-d...@googlegroups.com.

[Adam Johnson]

I don’t think many people can answer this off the top of their heads. I certainly can’t and I have contributed a couple things to migrations.

It’s probably quite necessary there’s only one leaf node but I can’t say for sure.

To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/31d78cfc-05bc-4bee-897b-3d9e2e502a3d%40googlegroups.com.

--

Adam

[Dave Vernon]

If I had to guess, it would be that with more than one leaf node, you would end up a substantial challenge resolving dependancies which would create an Order(N) problem (i.e. there's a chance of excessive time to complete the resolution).

I certainly worked on some migration logic that took a similar approach to this.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/31d78cfc-05bc-4bee-897b-3d9e2e502a3d%40googlegroups.com.

--

Adam

[Caio Ariede]

I’m working on this as a standalone PoC app for now, I may be able to share a repository with the code soon in order to get some feedback

Here’s in simple words where I’m at:

* I’ve replaced the restriction of only-one-leaf-node-per-app from the Migration Loader [1] to only-one-leaf-node-per-app-per-model. It means that with this, you can have more than one person changing different models in the same app, without introducing conflicts. Migrations referring to the same app and model would still cause a conflict

* I’m now trying to get the migration graph’s forwards/backwards plan to handle cases where there are more than one leaf node in the graph.

I may be doing the wrong assumption here, but it seems to me that migrations changing different models shouldn’t introduce a real conflict? If I have the following migration graph:

[ 0001_initial ] => [ 0002_auto ] => [ 0003_abc ] => [ 0003_def ] => [ 0004_auto ]

Where 0003_abc and 0003_def don’t have any model changes in common: no FK references, nothing related at all

If this assumption is correct, I could have the two migrations in ANY order in the graph, that it would still give me the same result, for example:

[ 0001_initial ] => [ 0002_auto ] => [ 0003_def ] => [ 0003_abc ] => [ 0004_auto ]

Please let me know if I’m missing something, ideas and thoughts are really welcome :)

[1] https://github.com/django/django/blob/2a038521c4eabdc5f6d5026d3dd6d22868e329cd/django/db/migrations/loader.py#L301-L313

--

Caio Ariede

caio....@gmail.com

To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/bd688186-752b-4bf5-9340-5d9607d17038%40googlegroups.com.

[Tim Graham]

Have you considered what would happen if a migration has a RunPython or RunSQL? They may require a certain model state.

--

Caio Ariede

caio....@gmail.com

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/bd688186-752b-4bf5-9340-5d9607d17038%40googlegroups.com.

[charettes]

+1 to what Tim said.

Also what about migration leaf migrations with models that refer to each others? What about a model rename in a leaf node that a model in the other leaf references to for some attribute names (e.g. m2m field intermediary table name).

I don't want to discourage you from trying because you'll likely learn a ton of stuff trying but I'm worried this might be a dead end or cause the migration framework code to be even more complex than it already is.

[Dave Vernon]

What if the migrations aren't merged before 0004 is created and you get:

[ 0001_initial ] => [ 0002_auto ] => [ 0003_abc ] => [ 0004(a) ]

and

[ 0001_initial ] => [ 0002_auto ] => [ 0003_def ] => [ 0004(b) ]

where 0004(a) != 0004(b) and 0004(a) modifies the model schema in a way that overwrites 0003_def's change but then 0004(b) relies on 0003_def.

To be sure, I think at that point you need to check for all conflicts downstream and their inter-dependancy not just down their own branch, but all migrations down each diverged branch; also it wouldn't be enough to compare each equivalent step number, e.g in our example above you now need to check 0004(a) against 0003_def as well as against 0004(b), etc...

[Dave Vernon]

+1 charettes too; to add to the factors to consider I suspect the issue of keeping it performant too (especially for unit tests) could become a challenge.