Ability to migrate other applications

[Emmanuelle Delescolle]

Hi everyone,

I'd like to talk about giving the ability to migrate models from other applications and I am addressing this mailing list as suggested by Tim Graham (https://code.djangoproject.com/ticket/25288).

Although I think this might be dangerous if mis-used, this is not the first time I come across the need for migrating models from other apps.

One simple use case is when using modeltranslations. When creating a translation.py in your own application to add multi-lingual capabilities to models from other (third party) apps, when running manage.py makemigations the migrations for those third-party apps will be created in the original app's directory which isn't (shouldn't be) part of your repository.

Therefore, when deploying the project on another machine, the migrations will not be available (as the original third party app will probably be deployed from pypi) and the whole project will then crash.

In this example, it is just a trivial case of running makemigrations on the server (which is not always possible depending on the deployment solution in use) but there are some more complex cases where the actual migration file is needed.

Using MIGRATION_MODULES is not a suitable solutions as we still want to use the original app's other migrations and also keep the possibility to upgrade the said application.

This is not the only use case, but it's probably one of the most frequent ones. Another use-case is when using contribute_to_class.

A quite easy solution to these use-cases is to add (maybe with a warning) a "migrated_app" parameter to the Migration class. If this parameter is not None it will be used to determine the application the migration has to be applied to. If it is None, normal behavior is resumed with the app to which that migration belongs to.

With this feature, manage.py makemigrations would still generate the migrations in the original thirs party app's directory, then the user would need to move and rename those newly generated files to their own (the user's) application (with appropriate dependencies). This isn't more complicated than creating a data migration.

You can find mention of this feature here: ​http://stackoverflow.com/questions/24311993/how-to-add-a-new-field-to-a-model-with-new-django-migrations/27450550#27450550 and you can find an implementation on https://github.com/nanuxbe/django/commit/30c6d5beadb7ae989f9b19d4c2f94c0ee2302519 .

The migration file would then be part of the project's repo and would be easily deployable

I'm looking forward to your feedback.

Thanks,

Emma

[Andrew Godwin]

The main problem I forsee here is just making sure that the MigrationLoader knows to take these located-in-the-wrong-place migrations and insert them into the migration tree right - in particular, you have issues with the dependencies, because say you made a third-party migration that depends on app_a.0003, and then they ship a new version that has app_a.0004, then suddenly app_a.0003 has two different children and the migration solver is going to really hate you.

Do you have a proposed workflow or tooling that would get around this?

Andrew

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/d7f57adc-2669-4176-8ced-2bc63c4adac3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Emmanuelle Delescolle]

I've been using this solution on a couple projects (including ones with several children to the same migration) and have never had troubles with the migration solver, I guess I have been lucky.

I'll throw a sample application today and try to reproduce the problem and see how it can be fixed/worked around. And write the appropriate tests.

Emma

[Shai Berger]

On Tuesday 25 August 2015 05:13:14 Andrew Godwin wrote:
> The main problem I forsee here is just making sure that the MigrationLoader
> knows to take these located-in-the-wrong-place migrations and insert them
> into the migration tree right - in particular, you have issues with the
> dependencies, because say you made a third-party migration that depends on
> app_a.0003, and then they ship a new version that has app_a.0004, then
> suddenly app_a.0003 has two different children and the migration solver is
> going to really hate you.
>

It's not going to hate her very badly, it will just ask her to add an empty
"merge" migration. If the feature is supported, that migration will be added
in the the external folder, and all will work fine. I still don't like the
idea.

The main problem I see with this is that it allows one app to mess with other
apps' models (yes, it means I think modeltranslations is Doing It Wrong(tm),
and the current issue is just pointing that out more loudly).

Shai.

[Andrew Godwin]

On Mon, Aug 24, 2015 at 10:19 PM, Emmanuelle Delescolle <deles...@gmail.com> wrote:

I've been using this solution on a couple projects (including ones with several children to the same migration) and have never had troubles with the migration solver, I guess I have been lucky.

I'll throw a sample application today and try to reproduce the problem and see how it can be fixed/worked around. And write the appropriate tests.

That would be great - I've not actually tried it, so you're ahead of me there, just trying to think of things that might muck it up!

I don't think it's too onerous to ask people to make merge migrations (as Shai says), or to say you need to manually jiggle round dependencies if you start using migrations like this, just want to work out what our options are in this department.

I'm generally not very pro having apps mess with other apps' migrations - because down that path madness lies, at least in some form - but I've also come to accept that in larger projects it becomes a necessity, be it modeltranslations (which I also disagree with the approach of), or apps that encourage extension with models, etc. I still think Django should make it quite hard to do this and screamingly obvious that you're going down a dark path, but at the same time we should at least have it be possible.

Andrew

[Emmanuelle Delescolle]

The main problem I see with this is that it allows one app to mess with other 
apps' models (yes, it means I think modeltranslations is Doing It Wrong(tm), 
and the current issue is just pointing that out more loudly). 

I agree that allowing apps to mess with other apps' models might be a problem that's why I believe this feature should be highlighted as dangerous or for "people who know what they are doing" but it's already possible to do that today anyways (just write raw sql in you migrations and you'll be able to mess with all the models you want).

Whether modeltranslations is doing right or wrong isn't the point, I took modeltranslations as a use-case as it is a widely used application. I've also had to resort to that feature with other applications or simply when trying to extract an application from a particular project to move it into a reusable app.

I'll throw a sample application today and try to reproduce the problem and see how it can be fixed/worked around. And write the appropriate tests.

That would be great - I've not actually tried it, so you're ahead of me there, just trying to think of things that might muck it up!

The project is available at https://bitbucket.org/emma_nuelle/migrate_othe_app_sample (I'll give a detailed explanation of what has been done at the bottom of this post). I have simply used "contribute_to_class" to trigger the desired behavior (modeltranslations isn't compatible with master anyways)

 I still think Django should make it quite hard to do this and screamingly obvious that you're going down a dark path, but at the same time we should at least have it be possible.

Yes, I'm not sure what's the best way to do that, either inside the documentation or systematically throw a warning when running such migrations? 

The sample app:

• up until https://bitbucket.org/emma_nuelle/migrate_othe_app_sample/commits/1ecb71e8d428638bc8dffe8a0e9aae75fb20cd7f : setting things up
  
• https://bitbucket.org/emma_nuelle/migrate_othe_app_sample/commits/18986dd7384f103f0cefeed94b0021998c3a21d0 : I've used contribute_to_class to create a new field from the "unruly" application on one of the models from the "pizzas" application and the ran makmigrations.
  
• https://bitbucket.org/emma_nuelle/migrate_othe_app_sample/commits/b70976ca4c953c0738ea250c731fefac2e7dff1e : the new migration initially created in "pizzas" has been moved to "unruly" and the "migrated_app" property has been added to it. So far, no problem
  
• https://bitbucket.org/emma_nuelle/migrate_othe_app_sample/commits/fcce7491f6efe90d3a28cea471d6951ed756941b : the maintainer of "pizzas" has added a new migration. Running migrate both on an empty db or on a pre-existing db work fine without having to run --merge
  

• up until https://bitbucket.org/emma_nuelle/migrate_othe_app_sample/commits/668d6877de1f0b2efb6cecb24bf7fee93fc2f9c1 : same scenario for animals -> unruly triggers the need for a migration which is then moved to unruly
• https://bitbucket.org/emma_nuelle/migrate_othe_app_sample/commits/ef14b382eccf463548c25ad19127f8fea9154f54 : a "conflicting" migration has been created by the maintainer of animals. When I say conflicting, it means that it's a migration on the same field as the one created by unruly, actually Django runs both migrations without any trouble (no --merge required). The actual troubles are:
  
• the migrations are not run in the same order depending whether the migrations are run on an empty db or on a pre-existing db.
  
• in the case of running the migration on a pre-existing db, the db field and the Django field are different
• https://bitbucket.org/emma_nuelle/migrate_othe_app_sample/commits/e35f1f4ca4c3e1ddeb6304fe97c43003bf7c2f59 : solution to both problems created by the previous commit -> create a migration manually which will update the db field and will make sure the order in which the previous migrations are run is not important
  

During the creation of the described process, the only time I had to message from Django telling me to run --merge was because I had made a mistake in the order of the steps of "pretending to be the maintainer of the other app".

I believe the second scenario is probably an edge-case not representative of what might happen with migrations created by modeltranslations or app extensions (as long as they are created by sensible people).

Emma

[Emmanuelle Delescolle]

Quote from Russ Magee's Twitter 

Django 1.9 is coming - do you have something on your wishlist? DSF Fellow Tim Graham has an offer for you: groups.google.com/d/msgid/django…

Well, actually, I'd  really like this for 1.9 ;-)

Has anyone +1's for this or is it more like -1's ?

Sample project is available at https://bitbucket.org/emma_nuelle/migrate_othe_app_sample. Django branch is at https://github.com/nanuxbe/django/tree/migrate_other_app <-- current test suite seems to pass fine (at least on my computer) just missing new tests if actual PR has to be made

Cheers,

Emma 

[Tim Graham]

I think if I were doing this, I would go the route of using MIGRATION_MODULES and copying the reusable app's migrations into it, then adding my own migrations. Yes, this would require copying any new migrations from an updated version of the app, but I am a bit nervous about adding more complexity in Django to support other ways of doing it. Too often, these edge case scenarios aren't considered when adding new features or making changes and they inadvertently get broken in some edge cases. Modifying models in other apps using contribute_to_class() isn't a blessed solution as far as I know, so adding features to support it doesn't feel right to me unless we make that blessing first.

[Emmanuelle Delescolle]

I think if I were doing this, I would go the route of using MIGRATION_MODULES and copying the reusable app's migrations into it, then adding my own migrations. Yes, this would require copying any new migrations from an updated version of the app,

Yes that sounds like a  lot of work compared to 3 actual lines of code (Yes, the patch is basically 3 lines + property name change where relevant).

but I am a bit nervous about adding more complexity in Django to support other ways of doing it. Too often, these edge case scenarios aren't considered when adding new features or making changes and they inadvertently get broken in some edge cases.

 

I'm not sure where the complexity is in this case. It's just adding a property to a class, if the property is set, use that as "app to migrate", if not just do as usual. All current tests do pass with the patch (there is no reason why they shouldn't as _nothing changes_ in the logic of the migration process if you don't set the new property), new tests should of course be written (which I'm willing to do if this feature gets accepted).
As I said before, this feature should be labeled as dangerous, so I think this also means people should find another way if it doesn't work for their edge case.

Modifying models in other apps using contribute_to_class() isn't a blessed solution as far as I know, so adding features to support it doesn't feel right to me unless we make that blessing first.

contribute_to_class is one of the many  examples, on the top of my head, here is another one: email fields in application that support several versions of Django but haven't set a fixed length for e-mail fields and yet another one is https://github.com/yourlabs/django-cities-light/issues/96 (I haven't digged int how this happens, but it does), I'm sure there are many more. I've used contribute_to_class in the demo app as it seemed the easiest way to replicate a situation where the feature might be useful but it's clearly not the only case.

[Tim Graham]

My main concern is not with the complexity of your proposed patch but with committing to supporting a new API that solves a problem that can already be solved a different way (albeit maybe less elegantly). Anyway, I don't want to speak definitely about the issue, but it seems like if we keep thinking about the problems maybe we could come up with a more elegant solution that doesn't need to marked as "dangerous".

[Emmanuelle Delescolle]

My main concern is not with the complexity of your proposed patch but with committing to supporting a new API that solves a problem that can already be solved a different way (albeit maybe less elegantly). Anyway, I don't want to speak definitely about the issue, but it seems like if we keep thinking about the problems maybe we could come up with a more elegant solution that doesn't need to marked as "dangerous".

Well, my first idea was to have a re-usable app deal with this case, this is actually how I've been handling it thus far. The problem with my current "reusable" app is that it's not very re-usable since there is no easy way of doing hat the path does in a re-usable app without copy-pasting most of the Django file because the migrated app is "hard-coded" to self.app_label and I now have 3 versions of that code depending on which version of Django is being used.

So here is what I'm thinking: here is another patch https://github.com/django/django/compare/master...nanuxbe:migrate_other_app which would allow for a third party tool to be written while not adding any feature to Django.
Then Django doesn't have to actually support the feature (and therefore isn't responsible for it) while still allowing it.

What do you think?

[Markus Holtermann]

Thank you for your patches and ideas of how to solve this feature, Emmanuelle.

However, I strongly agree with Andrew that, if we allow this -- I'm on Tim's side here, it feels odd encouriging something that's not considered to be used for that purpose: `contribute_to_class` -- the solution should be in the MigrationLoader. But as Andrew already said, you *will have* problems when you have your own migrations and a third party app adds a new one. Suddenly there are two leafs which is an invalid state. You will have to run `manage.py makemigrations --merge theapp` to fix this. The only sane solution I see right now is allowing MIGRATION_MODULES to take a string or list as values for each app. At which point Django needs to know where to place the merge-migration. I'd go with the first item in the list.

However, I still don't like the idea either and would and will continue to recommend to copy the migration files as proposed by Tim earlier.

/Markus

[Emmanuelle Delescolle]

First of all, I have to appologize, I just noticed that the sample application repository was missing some files (the models which is a shame for a sample app demonstrating migrations) which made the app useless. This has been fixed.

However, I strongly agree with Andrew that, if we allow this -- I'm on Tim's side here, it feels odd encouriging something that's not considered to be used for that purpose: `contribute_to_class` --

 

I am not trying to encourage contribute_to_class, I've used it in the sample app as it was the most direct approach. 

the solution should be in the MigrationLoader.

Can you elaborate a bit more on this?

But as Andrew already said, you *will have* problems when you have your own migrations and a third party app adds a new one. Suddenly there are two leafs which is an invalid state. You will have to run `manage.py makemigrations --merge theapp` to fix this.

I am sorry but this assumption is wrong, please take a look at the attached screenshot (the commit hashes are the ones from the sample app at https://bitbucket.org/emma_nuelle/migrate_othe_app_sample/commits/all if you want to run it yourself and the commit messages explain what operations have been done).
The only way I was able to generate the need for a merged migration (in this scenario) was by not updating migration dependencies correctly upon moving migration files to the "unruly" app.
On the left of the screenshot, migrations are applied step by step as they would be during the course of development, on the right they are applied all at once as they would be on CI server (or during a deploy to production depending on your workflow).

 

The only sane solution I see right now is allowing MIGRATION_MODULES to take a string or list as values for each app. At which point Django needs to know where to place the merge-migration. I'd go with the first item in the list.

I like the idea of being able to feed a list as value for a specific app in MIGRATION_MODULES.
As far as where to put the merge-migration (once again I believe this would only happen if one doesn't set migration dependencies correctly). I think we can have some pretty messy situations with people having more than 2 migration modules for the same app. Anyway, I guess we can assume merge conflict would be caused by the "non-default" migration, so I see 2 different case:
- the merge conflict was generated between a migration in the "default module" (the one which would have been loaded if MIGRATION_MODULES hadn't been set for that app) and another module => place the migration in the "non-default" involved in the conflict
- the merge conflict was generated bewteen 2 migrations in 2 "non-default" modules => default to the first one of those modules.

So both cases can be solved at once:
put the merge-migration in the first "non-default" module involved in the conflict.

Would that be something acceptable?

However, I still don't like the idea either and would and will continue to recommend to copy the migration files as proposed by Tim earlier.

I still believe the second proposed patch - which doesn't add any functionality nor does it endorse anything, it is just a separation of concerns (app_label being a label used in naming the app, _migrated_app which designates the migrated app and is set to app_label in __init__) which make Migration easier to extend - is worth considering.

Cheers,
Emma

[Emmanuelle Delescolle]

Looks like the screenshot was thumbed and thus unreadable so I'm attaching it here.

[Shai Berger]

Hi,

On Thursday 03 September 2015 12:44:08 Emmanuelle Delescolle wrote:

[Markus Holtermann wrote:]

> > But as Andrew already said, you *will have* problems when you have your
> > own migrations and a third party app adds a new one. Suddenly there are
> > two leafs which is an invalid state. You will have to run `manage.py
> > makemigrations --merge theapp` to fix this.
>
> I am sorry but this assumption is wrong, please take a look at the attached
> screenshot (the commit hashes are the ones from the sample app at
> https://bitbucket.org/emma_nuelle/migrate_othe_app_sample/commits/all if
> you want to run it yourself and the commit messages explain what operations
> have been done).

Right.

What Markus (and myself) assumed is that when you add the "migrated_app"
attribute, you make the migration belong to the migrated app. What you did is
different: The migration still belongs to the app it is defined in, it is only
the operations in it that "belong" to the migrated app.

This is evidenced both by the printing ("Applying unruly.0001_initial" etc)
and by the handling of leaf migrations:

> The only way I was able to generate the need for a merged migration (in
> this scenario) was by not updating migration dependencies correctly upon
> moving migration files to the "unruly" app.

When you failed to update the dependencies, you had two leaf migrations in
"unruly".

What this all means is, your patch *subverts* the dependency checks, allowing
more than one leaf migration on an app (and forcing a single leaf migration
where it shouldn't -- there should be no requirement for dependencies among
the "unruly" migrations). This makes things even more fragile than you
implied: I think after you move your migration from the original app (say,
"pizzas") to your "unruly", if you "makemigrations pizzas" again it will be
generated again. Which means, if you have two "unruly" apps, the interactions
between them become, let's say, interesting.

Assuming the above description is correct, I'm strongly -1 on the current
implementation. From a framework point of view, it is broken.

Shai.

[Emmanuelle Delescolle]

On Friday, 4 September 2015 00:29:28 UTC+2, Shai Berger wrote:

What Markus (and myself) assumed is that when you add the "migrated_app"
attribute, you make the migration belong to the migrated app. What you did is
different: The migration still belongs to the app it is defined in, it is only
the operations in it that "belong" to the migrated app.

People can actually do that today out of the box by adding those 2 lines to the generated migration when moving it:

def __init__(self, name, app_label):
    super(Migration, self).__init__(name, 'other_third_party_app')

But that would break everything (It would counter-intuitively work on that migration but would break right after that), It wouldn't even generate a merge migration. Because the models the migration is working on would not be loaded.

This is evidenced both by the printing ("Applying unruly.0001_initial" etc)
and by the handling of leaf migrations:

 

No need to show me evidence, that's exactly what I am trying to do (but thanks for pointing out it does work as intended)

What this all means is, your patch *subverts* the dependency checks, allowing
more than one leaf migration on an app (and forcing a single leaf migration
where it shouldn't -- there should be no requirement for dependencies among
the "unruly" migrations). This makes things even more fragile than you
implied: I think after you move your migration from the original app (say,
"pizzas") to your "unruly",

You can call it perverting if you like what, it does is just say "hey this migration belongs to unruly" because, that's true, this migration should belong to the app that created it. But it's working on the models from pizzas (because that's also what it's doing) that way, the migration framework knows not to load the models from unruly but from pizzas instead.

 

if you "makemigrations pizzas" again it will be
generated again. Which means, if you have two "unruly" apps, the interactions
between them become, let's say, interesting.

 

Interesting indeed, since screenshots work better than words or code samples, please take a look at the attached screenshot.
The reason it doesn't create a new migration is the same reason why it created a migration for an app to which you didn't change a line of code in the first place: the state of the model is the same state as the one computed by the migration system and whether it's the maintainer of pizzas (who doesn't have the unruly app in their project) or the owner of the current project (who's current model's state correspond to the one generated by the unruly migration) who runs that command, the model state is still the same as the state generated by the migration framework, so no new migration will be generated.

Assuming the above description is correct, I'm strongly -1 on the current
implementation. From a framework point of view, it is broken.

And what do you think about Markus' idea? 

Cheers,

Emma

[Marten Kenbeek]

Hi Emmanuelle,

I think the main problem is the dependencies between migrations. The order in which migrations 

are run is not fixed. It depends on which migrations have already run, and we can't guarantee the 

same order for each deployment. If each chain of migrations that could act on a single model has 

a fixed order, it's not much of a problem. Once each app can act on each model, it becomes a big, 

and possibly unsolvable problem. I think the scenario you present as an edge-case, where two 

migrations from different apps affect the same field, is the exact case we have to watch for, which can 

cause significant problems in how we handle migrations. 

If we are to "solve" this, I think Markus's idea it the best solution. If you can make all the migrations in

a single app affect the same app (which doesn't have to be the app that contains them), you can make 

consistent rules on the order in which all migrations that affect the target app are run. Once you have 

that, it's not that different from the current situation: each migrated app has a single chain of migrations 

that have to be applied, with several inter-app dependencies that are resolved at runtime. 

The inter-app dependencies might be difficult, but at least they are solvable. I have reasons to believe that if 

you allow each migration file to target a different app, it becomes an unsolvable dependency problem. If you'd 

like I can do some more research to come to a definite conclusion. 

If anything, I think Tim's solution is still the best solution; if you have to manually copy the migrations to an app 

within your project, it is very clear that you need to do something when the third-party app publishes some 

new migrations that could conflict with the order in which the migrations would currently be applied. Once you 

have that in mind, it's not that much trouble to copy and merge any new migraitons, and I think it will save a lot of

headaches if we only allow inter-app migrations through the MIGRATION_MODULES settting. 

Marten

Op vrijdag 4 september 2015 08:28:48 UTC+2 schreef Emmanuelle Delescolle:

[Shai Berger]

Ok, two things:

1) Markus' idea of having more than one folder for migration modules seems
reasonable enough. I disagree with his comment about the placement of merge
migrations --

> Django needs to know where to
> place the merge-migration. I'd go with the first item in the list

I'd require an explicit selection by the user; I'd still want to make sure the
selected path is one of those specified for migration modules, so a dialog for
selection may be more appropriate than a command-line parameter.

BUT

2) Emma's experiment, essentially, proves that the migration system can live,
migrate, and generate new migrations with two leaf-migrations present. Which
begs the question -- are merge migrations really necessary at all? I know why
they were necessary for South, where migrations in an app were ordered
linearly, but we have grown past that.

If, as I now suspect, we actually don't need them, then the whole idea sounds
much more reasonable. I still feel funny about a migration which belongs to
one app and works on the models of another, and would prefer some better-
looking solution -- e.g. "project migrations" (there are other reasons to
think of them, like, special migrations to change swappable models); but
unless some such idea gets some backing, I'd be only -0 on this.

Shai.

[Andrew Godwin]

I still feel like merge migrations are necessary, if not only because they're a way of reducing the potential collision area of migration ordering from different branches, but it might be that the solver is predictable enough that it's not a problem. Two separate branches in migrations still don't have an order relative to each other, so if we change the code in Django that does order resolution even slightly it could result in different operation orders or even different "final" rendered models.

Andrew

[Marten Kenbeek]

Say A.0002 and B.0001 are different leaf nodes for A.0001, and both alter the same field. 

Now, on a clean database, the order is pretty stable. If that order is A.0001, A.0002, B.0001,

you'll always end up with the field from B.0001. This is also what the autodetector sees. 

If you first create and apply the migrations A.0001 and B.0001, and then an update to A adds

the new migration A.0002, the database will end up with the field from A.0002. The autodetector 

will still see the field from B.0001, and won't  create a new migration, so if the final model has 

the field from B.0001, you have a mismatching database table. 

I think the fact that the order depends on which migrations have been applied is enough to require 

a merge migration. 

Marten

Op dinsdag 8 september 2015 21:33:10 UTC+2 schreef Andrew Godwin:

[Emmanuelle Delescolle]

if we change the code in Django that does order resolution even slightly it could result in different operation orders or even different "final" rendered models.

The order of migrations isn't something which is local to this feature, it is something which isn't fixed (probably by design, correct me if I'm wrong) and if it is not "the right way" to do it, maybe it should be part of another issue.

Once again, correct me if I'm wrong but the only way to have different final rendered models would be the original third-party app defining/altering a field which has already been defined/altered by your app, resulting, in regards of the current proposal, in clashing fields.

Creating a field which clashes with another app's field is something bad, there are nine chances out of ten that fixed migration order will not solve your overall problem: you created a field named the same as another field, except if you have a psychic connection with the original library's author you probably have a different usage for the field (however slight the difference may be) and, if you really need to upgrade that third party library, you have a whole other mess to fix than conflicting migrations so I am really wondering if this is worth the hassle.

If you create two models with the same meta db_table, django doesn't come to your rescue and propose a solution, it is something you shouldn't have done and now you have to deal with it. I think this "edge case" is in the same area.

would prefer some better- 
looking solution -- e.g. "project migrations" (there are other reasons to 
think of them, like, special migrations to change swappable models); but 
unless some such idea gets some backing, I'd be only -0 on this. 

A "project" is just a module, so if you create a "migrations" directory, add an __init__.py inside it and add it to your INSTALLED_APPS, you can start placing migrations inside it. That's actually where I put those "migrations for other apps" at the moment.
And BTW, changing a swappable model is another "legitimate" example of a migration being created in a third party app (django.contrib.admin) while strictly using "recommended" features of Django.

Cheers

[Marten Kenbeek]

The order of migrations isn't something which is local to this feature, it is something which isn't fixed (probably by design, correct me if I'm wrong) and if it is not "the right way" to do it, maybe it should be part of another issue.

The order of migrations within a single app is fixed. That means the order of the operations on a single model or table is also fixed. 

This proposition introduces a variable order to migrations that alter the same model, so yeah, this issue is local to this feature. 

Once again, correct me if I'm wrong but the only way to have different final rendered models would be the original third-party app defining/altering a field which has already been defined/altered by your app, resulting, in regards of the current proposal, in clashing fields.
Creating a field which clashes with another app's field is something bad, there are nine chances out of ten that fixed migration order will not solve your overall problem: you created a field named the same as another field, except if you have a psychic connection with the original library's author you probably have a different usage for the field (however slight the difference may be) and, if you really need to upgrade that third party library, you have a whole other mess to fix than conflicting migrations so I am really wondering if this is worth the hassle.
If you create two models with the same meta db_table, django doesn't come to your rescue and propose a solution, it is something you shouldn't have done and now you have to deal with it. I think this "edge case" is in the same area.

Since we're talking about unsupported, undocumented ways to change models anyway (like contribute_to_class), I think altering an existing 

field is a very legitimate reason to create a new migration. For instance, many people want to increase the length of the username field on the 

default User model, and the upcoming 1.9 release will also alter the username field again, resulting in the scenario I described above. So unless 

we explicitly warn against this in the documentation, we should find a suitable solution for this. 

Marten

Op woensdag 9 september 2015 12:02:57 UTC+2 schreef Emmanuelle Delescolle:

[Emmanuelle Delescolle]

On Wednesday, 9 September 2015 12:29:59 UTC+2, Marten Kenbeek wrote:

The order of migrations isn't something which is local to this feature, it is something which isn't fixed (probably by design, correct me if I'm wrong) and if it is not "the right way" to do it, maybe it should be part of another issue.

The order of migrations within a single app is fixed. That means the order of the operations on a single model or table is also fixed. 

This proposition introduces a variable order to migrations that alter the same model, so yeah, this issue is local to this feature. 

I've had the case where migration order between apps was different between initialized and non-initialized database which crashed the migration process when running on a non-initialized database when using another "strongly advised against" behavior in a data migration (namely importing a model directly).

So, since the only case where an fixed order between an "official" and a "non-official" leaf node is important is when using a a "strongly advised against" behavior, this is debatable.

Once again, correct me if I'm wrong but the only way to have different final rendered models would be the original third-party app defining/altering a field which has already been defined/altered by your app, resulting, in regards of the current proposal, in clashing fields.
Creating a field which clashes with another app's field is something bad, there are nine chances out of ten that fixed migration order will not solve your overall problem: you created a field named the same as another field, except if you have a psychic connection with the original library's author you probably have a different usage for the field (however slight the difference may be) and, if you really need to upgrade that third party library, you have a whole other mess to fix than conflicting migrations so I am really wondering if this is worth the hassle.
If you create two models with the same meta db_table, django doesn't come to your rescue and propose a solution, it is something you shouldn't have done and now you have to deal with it. I think this "edge case" is in the same area.

Since we're talking about unsupported, undocumented ways to change models anyway (like contribute_to_class), I think altering an existing 

field is a very legitimate reason to create a new migration. For instance, many people want to increase the length of the username field on the 

default User model, and the upcoming 1.9 release will also alter the username field again, resulting in the scenario I described above. So unless 

we explicitly warn against this in the documentation, we should find a suitable solution for this. 

Personnaly I've been talking about the case which, when running makemigrations, creates a migration in a third party app. There are several legitimate use-cases outside of contribute_to_class. Contribute_to_class is only one case and you have been warned against using it so a warning should be sufficient, shouldn't it?

I would argue that if you need a longer username, you should use a custom user model extending AbstractUser (which would lead to one of the legitimate use-cases I just mentioned), which is the recommended way of doing it.

To go further with your example, a fixed migration order will no solve your problem: to keep your existing data safe. If Django's default length for a username is still smaller than yours you won't be able to run the migration at all on an existing database (or it will truncate your existing usernames), so you'll have to manually fake run it on any production database.

To have the field length set to your custom value with a fixed order, this would mean you have to run your migration after the new django.contrib.auth one, which is not possible since your migration has probably already run on the production database.

(If your migration hasn't run yet on production database, simply delete it and re-run makmigrations, it will automatically depend on the latest migration for that app ;-) )

And if Django's default length for the username field is larger than your custom value, then your data will be safe unrelated to the order in which the migrations run. You may then either remove your change or re-create a new migration (which depends on the new django.contrib.auth migration) to reduce the length to whatever you want it to be if you feel it's necessary.

So, yes, I still believe fixed migration doesn't bring much to this feature.

Cheers,

Emma

 

[Shai Berger]

On 9 בספטמבר 2015 13:29:58 GMT+03:00, Marten Kenbeek <marte...@gmail.com> wrote:
>
>>
>> The order of migrations isn't something which is local to this
>feature, it
>> is something which isn't fixed (probably by design, correct me if I'm
>
>> wrong) and if it is not "the right way" to do it, maybe it should be
>part
>> of another issue.
>
>
>The order of migrations within a single app is fixed.

No, actually it isn't. It is only constrained by dependencies. Merge migrations do not fix it either. The only thing they do is tell the warning-happy migration system : "yes, I have considered the situation here and it's OK to run these two migrations in either order ".

Which raises two points :

First, can we (do we want to ) handle this validation some other way, and would it buy us anything?

Second, how common is Marten's misconception? Do we need to address it?


Shai
--
Sent from my Android device with K-9 Mail. Please excuse my brevity.

[Markus Holtermann]

On Wednesday, September 9, 2015 at 9:21:43 PM UTC+10, Shai Berger wrote:

On 9 בספטמבר 2015 13:29:58 GMT+03:00, Marten Kenbeek <marte...@gmail.com> wrote:
>
>>
>> The order of migrations isn't something which is local to this
>feature, it
>> is something which isn't fixed (probably by design, correct me if I'm
>
>> wrong) and if it is not "the right way" to do it, maybe it should be
>part
>> of another issue.
>
>
>The order of migrations within a single app is fixed.

No, actually it isn't. It is only constrained by dependencies. Merge migrations do not fix it either. The only thing they do is tell the warning-happy migration system : "yes, I have considered the situation here and it's OK to run these two migrations in either order ".

No, they do ensure some sort of deterministic behavior. Migrations are sorted by their fully qualified name (tuple of (app_label, migration_name)):

https://github.com/django/django/blob/master/django/db/migrations/graph.py#L64

This behavior is enough for a 3rd party app to destroy the results of your own migration or the other way round.

Consider a.1 <-- a.2 <-- b.1 where b.1 makes changes to the app a. When the author of app a add another migration a.3 (a.1 <-- a.2 <-- (a.3, b.1)) the resulting order in which migrations are applied will be a.1, a.2, a.3, b.1. However, on an existing database, a.3 will be run after b.1, resulting in a non-deterministic order in which migrations for the same app are being applied.

Consider me a strengthened -1 looking at the discussion.

If we want to have something like "project level migrations" we should rethink a bunch of things, but that's out of scope of the initial proposal of this discussion, I think.

Which raises two points :

First, can we (do we want to ) handle this validation some other way, and would it buy us anything?

Second, how common is Marten's misconception? Do we need to address it?


Shai
--
Sent from my Android device with K-9 Mail. Please excuse my brevity.

/Markus 

[Shai Berger]

On Wednesday 09 September 2015 16:35:16 Markus Holtermann wrote:
> On Wednesday, September 9, 2015 at 9:21:43 PM UTC+10, Shai Berger wrote:
> > On 9 בספטמבר 2015 13:29:58 GMT+03:00, Marten Kenbeek <marte...@gmail.com
> > >

> > >The order of migrations within a single app is fixed.
> >
> > No, actually it isn't. It is only constrained by dependencies.
>

> No, they do ensure some sort of deterministic behavior. Migrations are
> sorted by their fully qualified name (tuple of (app_label,
> migration_name)):
> https://github.com/django/django/blob/master/django/db/migrations/graph.py
> #L64
>

A) This is an implementation detail, not part of the contract AFAIK. It was
added because determinism is generally good. It was never intended to affect
correctness, and
B) More importantly -- it does not add dependencies, meaning, the code treats
the situations where migrations were executed out of this ordering as valid
(as long as no explicit dependencies are violated).

Two migrations affecting the same field, without an explicit ordering between
them (directly or indirectly), are a bug, whether they are in the same app or
not in the same app.

> This behavior is enough for a 3rd party app to destroy the results of your
> own migration or the other way round.
>
> Consider a.1 <-- a.2 <-- b.1 where b.1 makes changes to the app a. When the
> author of app a add another migration a.3 (a.1 <-- a.2 <-- (a.3, b.1)) the
> resulting order in which migrations are applied will be a.1, a.2, a.3, b.1.
> However, on an existing database, a.3 will be run after b.1, resulting in a
> non-deterministic order in which migrations for the same app are being
> applied.
>

Right. Now consider the same setting, except that b.1 is named a.2a (because
more than one developer is working on this app). So we have:
a.1 <-- a.2 <-- (a.2a, a.3)

Nothing bad will happen, because as soon as the merge is done which gets a.2a
and a.3 together in the same codebase, validation will cry foul and you'll
need to sort things out manually and possibly add a merge migrartion. However,
if you do add that merge migration (a,4) without realizing the problem, you
get exactly where you were -- the actual resulting database depends on the
order in which migrations happened historically.

This is all just strengthening my earlier point: Merge migrations' main
contribution, currently, is to help with validation of possibly-conflicting
migrations (their existence also probably allows some simplifications in the
autogenerator). This is a very good mechanism, identifying the problems very
early. However, they have a limitation: Validation only happens for migrations
within the app. I think if model a.A has a fk to b.B and app b suddenly adds
migrations changing B's pk from int to uuid we're already in trouble, for
example. So, an improvement of the validations may be in order, and if we use
a different mechanism (e.g. checking affected fields), perhaps this feature will
become more palatable.

On Wednesday 09 September 2015 13:02:56 Emmanuelle Delescolle wrote:
> A "project" is just a module, so if you create a "migrations" directory,
> add an __init__.py inside it and add it to your INSTALLED_APPS, you can
> start placing migrations inside it.

Right. But, as Markus hinted, project-wide migrations should not be treated as
just migrations in another module. As a trivial example -- for your sanity, if
you're adding a project-level migration, you probably want it to depend on the
last migration in every app at that time (other apps, as well as, of course,
the project app itself). And there are many other similar considerations.

Shai.

[Emmanuelle Delescolle]

As this thread is most likely leading to a dead-end, let me try to close it by summarizing it:

Original proposal:

- adding a property to the current migration class to make it more extendable

Idea behind the proposal:

- being able to move the migrations generated in my venv when running makemigrations to a location under my repository without having to create my own Migration class which is mostly a copy-paste from Django's

Tim:

Thanks for suggesting copying existing migrations to another directory and using MIGRATION_MODULES. Although I really don't believe this is the answer for me as it sounds very error-prone and not very DRY (even if in this case I would be repeating somebody else and not myself).

Shai:

You feel strongly against this code as it subverts the whole idea behind migrations.
However you would enjoy this feature if it was called "project migrations" instead and, maybe, eventually, (if I understand the name of the feature correctly) if the feature was automated rather than manual (project migrations created directly inside the project instead of having to move them yourself and add the extra line in it)

Shai and Markus:

You are both very concerned about multiple leaf migration trees and about the fact that those leaf migrations might clash with each other.
After deeper thinking, this concern is probably more than legitimate but has, also probably, very little to do with the current proposal. If running makemigrations creates a new migration for an app you didn't write, it will end-up being a leaf migration independent of whether you leave it in your venv or are able to move it to your project and exposes you to the "clash" risks all the same.

Markus:

The idea of having several migration modules for the same app might be a good idea, although as I am currently living in a country being roughly slightly smaller than the largest cities on the global but, at the same time, supporting 3 official languages, creating an extra migration module for each app I install which's data needs translation doesn't sound very appealing to me. (more on how modeltranslation does it the wrong way here under)

Marten:

I am sorry but this proposal or any of this derivatives is unlikely to solve your username field length migration problem. If this field is your only trouble, you are probably better off using Tim's solution

About modeltranslation:

I, personally, am not too found of apps using contribute_to_class either so if anyone knows of an app which allows me to have translations for user-generated content living in my database (as those are translations for content from the database) which is pluggable on third party apps and doesn't require an extra join on (almost) every query, please let me know, I've been looking for such an app for a while. In the meantime, I'll keep using modeltranslation however "wrong" it might do things as it's pretty good at filling my personal expectations.

Thank you all for your time, I'll go back to copy-pasting now.

Cheers,

Emma

[Shai Berger]

Hi,

On Thursday 10 September 2015 17:00:07 Emmanuelle Delescolle wrote:
> As this thread is most likely leading to a dead-end, let me try to close it
> by summarizing it:

Not trying to re-open the thread, just that...

>
> Shai:

This seemed a little far from what I said (or, at least, intended):

> You feel strongly against this code as it subverts the whole idea behind
> migrations.

No, it only subverts the validation of conflicts introduced by parallel lines
of development. Essentially the same concern as you attributed (correctly) to
Markus and myself together.

> However you would enjoy this feature if it was called "project migrations"
> instead and, maybe, eventually, (if I understand the name of the feature
> correctly) if the feature was automated rather than manual (project
> migrations created directly inside the project instead of having to move
> them yourself and add the extra line in it)
>

Naming is important, but there's a little more to that too: Current
migrations, and you do not propose to change that, are managed as if each only
affects its own app. Project migrations would have tooling around them --
generation, validation and execution -- that takes into account that they
affect more.

>
> Shai and Markus:
> You are both very concerned about multiple leaf migration trees and about
> the fact that those leaf migrations might clash with each other.
> After deeper thinking, this concern is probably more than legitimate but
> has, also probably, very little to do with the current proposal. If running
> makemigrations creates a new migration for an app you didn't write, it will
> end-up being a leaf migration independent of whether you leave it in your
> venv or are able to move it to your project and exposes you to the "clash"
> risks all the same.
>

... except that when all the migrations are in the same app, DJango already
has mechanisms to detect and warn users against these clashes. True, the tools
are blunt and require the user to make the decision about actual clash-or-no-
clash, with almost no aid beyond pointing out the possibility of the clash.
But this is still a lot better than the completely silent clashes your
suggestion enables.

>
> About modeltranslation:
> I, personally, am not too found of apps using contribute_to_class either so
> if anyone knows of an app which allows me to have translations for
> user-generated content living in my database (as those are translations for
> content from the database) which is pluggable on third party apps and
> doesn't require an extra join on (almost) every query, please let me know,

I'm not sure if it works with modern Django -- last time I looked it was
updated to Django 1.2, and I haven't had to deal with translations in the
database for several years now -- but I always thought django-dbgettext had it
right. Its strategy was to pull specified database strings into .po files so you
can translate them just like you translate strings from your source.
(Oh, upon looking I see that it did get some attention last year. Good).

HTH,
Shai.