Why Django is making migrations for external apps?

[Marcin Nowak]

Hello,

I'm working on a project which is based on Django`s internal migrations.

Sometimes when switching between branches (not sure for 100%), Django complains about changes in my models.

When `makemigrations` command is executed Django generates migrations in external (reusable) app placed in egg. And this is *strange*, I think.

I don't want to modify any external app, and I don't want to depend on temporary migrations created only on my local machine.

Why Django is creating migrations for external apps, especially placed in eggs? 

Migrations should be handled and created only for apps within the project, and read-only for others. 

What do you think?

How I can workaround this issue?

(Django 1.7.8)

Best Regards,

Marcin

[Andrew Godwin]

Hi Marcin,

Django will only make migrations for external apps that already have "migrations" directories and have model changes - it shouldn't make new migrations unless you've actually changed the models in those reuseable apps. Are you sure you're not specifying the names of those apps on the makemigrations command line? Are you using something like --noinput?

Andrew

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/77b46084-ffa1-43a0-ae2d-206f868cbaa5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Marcin Nowak]

Hi Andrew,

I think I've found it. This reusable app uses a kind of dynamic choices (based on settings or somethin' else) and Django detetcs field definition change (choices attr).

So the question is - should Django detect change of choices attribute even if database schema is not really altered?

And the original question modified - should Django generate migration file inside of an egg when runtime configuration or settings are changing?

BR,

Marcin

[Andrew Godwin]

Hi Marcin,

Django persists every single field attribute change, no matter what it is - this was a deliberate decision as we didn't want to declare certain attributes as "not important for schema" (for example, a subclass of ChoiceField might use an enum column type for the choices and so need that change).

That said, this is definitely a common problem and I'm starting to think we should revisit it - specifically, have field classes themselves say which attributes are not worth making changes for (perhaps combined with greater control by fields over how their schema gets made). This isn't currently possible though.

Also, yes, Django doesn't see unpacked .egg files as any different than anything else (they're just directories after all) - we have no way of telling which apps are yours and which are external, really. This should only happen if the external app already has a migrations directory, so the app in question should probably try and present a consistent choices list to the migration system in the meantime if you want to avoid this. There's no other real fix for this right now.

Andrew

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/81e00a02-95fa-451d-aa24-1e4bd592f511%40googlegroups.com.

[Collin Anderson]

For choices, I've sometimes had my migrations import the correct choices from the live model in order to avoid needing to create extra migrations every time they change.

[Marcin Nowak]

Thanks for clarification, Andrew. I understand it. 

The real solution (for me) is not to do anything "automagically". In my "dream" I see migrations whose can be "picked" manually (or semi-manually via "pickmigrations <appname>") into my project, whenever I want. No autochecks, no automigration for anything, only bunch of helpers (i.e. change-detector. picker). Real/valid/selected/picked migrations should be "imported" as a project-wide changesets without splitting by apps (no dependency problem, easier conflicts resolve), and migrations in reusable apps should be provided as a resource(s) to pick/import. 

I would like to spent some time while creating stable and solid migrations in my project instead of searching workarounds for runtime issues.

But this means a big architecture change and I know that is improssible to do for years...

Thanks again.

Marcin

[Andrew Godwin]

Hi Marcin,

Having migrations per-app is kind of a key part of the way the design works, but as for the rest of it, you're free to just write migrations yourself and ignore makemigrations - the format is intended to be human-writable.

We also deliberately separated the schema editing backends from the migration code itself, so you're free to develop your own migration solution that overrides some of the logic of how Django works without having to re-implement all of the SQL abstraction, but I will tell you that in my experience the solution we have now is the one that works the best for most people (South did indeed use to have a form of semi-manual picking).

Dependencies also make things more complex - my attempts to try and communicate interdependent changes to human in textual/graphic form were unsuccessful, hence the current method of just doing everything for you as best we can.

As for a short-term fix, Collin's idea of importing the choices into the migration to stop it detecting more changes sounds like a decent temporary fix.

Andrew

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/4c57141a-7ab0-40fe-ba6a-27d054b745df%40googlegroups.com.

[Tim Graham]

Andrew, do you think having the autodetector not "resolve" settings to their values and instead include that as references in auto-generated migrations is something we should try to implement?

If so, let's suggest that solution on https://code.djangoproject.com/ticket/24648

[Marcin Nowak]

Please also review that case: https://github.com/pennersr/django-allauth/blob/master/allauth/socialaccount/models.py#L40

[Collin Anderson]

Ahh, yes, that's a good example of what I was thinking of:

https://github.com/pennersr/django-allauth/blob/fb2851fe333f303faab28e9633b6cf1b0ddbbe98/allauth/socialaccount/migrations/0001_initial.py#L39

[Andrew Godwin]

The problem is that migrations are designed to work even when you've changed the original models, so we can't have the design include something where we reference the original models file (if I remove that field, all previous migrations that refer to it start having ImportErrors).

I'd rather we work towards something where fields could just say "choices/label/etc. don't matter" and ship default ignores with all the core fields.

Andrew

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/877defe4-6fe4-4ab1-8509-17d4464d67ed%40googlegroups.com.

[Carl Meyer]

Hi Andrew and Marcin,

On 07/15/2015 05:10 PM, Andrew Godwin wrote:
> Django persists every single field attribute change, no matter what it
> is - this was a deliberate decision as we didn't want to declare certain
> attributes as "not important for schema" (for example, a subclass of
> ChoiceField might use an enum column type for the choices and so need
> that change).

There's another reason for this choice: accuracy of the "historical" or
"frozen" ORM used in data migrations. Even if a field attribute has no
effect on the database schema, it could have a significant effect on the
Python-level behavior in a data migration. For instance, it's entirely
possible that a data migration could use a `get_FOO_display` method,
which could certainly break if the choices on that field at that point
in "migration time" are not correctly preserved in the migrations.

> That said, this is definitely a common problem and I'm starting to think
> we should revisit it - specifically, have field classes themselves say
> which attributes are not worth making changes for (perhaps combined with
> greater control by fields over how their schema gets made). This isn't
> currently possible though.

If we do this, I think it should be opt-in and used only by the end
developer for unusual cases (like dynamic choices). The default should
continue to be to preserve all field attributes, including choices, for
frozen-ORM reasons.

Carl

[charettes]

Hi everyone,

Here's just an idea from the top of my head.

What if we exposed a `django.db.migrations.Reference`
that would be a deconstructible proxy to a top level module object.
It would be initialized with a module name string and the referenced
object name as a second argument. We could document
such an object should be returned by fields deconstruct()
method to opt-in the desired behavior described by Carl.

We already do something similar with the string setting case with
`django.db.migration.writer.SettingReference` which could be
a simple subclass of `Reference` that takes a single setting
name argument instead and pass `django.conf.settings` to it's
parent initializer.

Thoughts?

Simon

[Markus Holtermann]

Hi all,

While your proposal could solve the repeated creation of migrations due to dynamic choices, Simon, I don't see how it would solve or prevent the issue Carl mentioned earlier. Data migrations may need to have the exact choices at that particular state. Importing the current choices may fail the migration process on a new database. Furthermore, I think exposing such a feature would encourage people to use it for the wrong reasons, ending up with hard to debug bugs.

I favor Andrew's proposal to revisit the white listing of field attributes on a per-field-instance level, eg:

class MyModel(models.Model):
name = models.CharField(…)
name.migrations_exclude_attributes = {'choices'}

That would be 100% backwards compatible and the autodetector would only need to check if the changed attribute is in that particular set and ignore the change in case it is. Since this property only takes effect when used with the current model, I don't see a need to maintain its presence or value in migrations.

/Markus

[charettes]

Hi Markus,

My proposed solution simply offers a way to opt-in the attribute exclusion behavior, I didn't mean to solve the whole data migration access to a particular state issue.

While I see merit in your proposal I cannot understand how it's not also encouraging people to use it for the wrong reasons, it even looks easier to do. Correct me if I'm wrong but I don't think it's solving the issue Carl mentioned either. I also doubt this won't introduce subtle bugs, fields can behave quite differently if they're missing an attribute. Think of `default`, `limit_choices_to`, `related_name` and friends.

One last point, even if your proposal is backward compatible it doesn't help people on 1.7.x and 1.8.x (or third party application that will support those version for the years to come). If we document that third party application exposing fields with attribute they want to ignore should simply return a deconstructible object it would also work with 1.7.x+.

Simon

[Shai Berger]

On Thursday 16 July 2015 02:10:12 Andrew Godwin wrote:
>
> Also, yes, Django doesn't see unpacked .egg files as any different than
> anything else (they're just directories after all) - we have no way of
> telling which apps are yours and which are external, really.

In my experience, most projects differentiate "our" apps from "external or 3rd
party" apps based on directory structure -- "our" apps go in the main project
directory (and version control) and "external" ones go into virtualenv and/or
system folders.

The issues of choices and other setting-dependent-model-attributes
notwithstanding, can we solve the "external apps" issue by adding a parameter
to the makemigrations command (and maybe even a setting) to tell it to add
migrations only within a given directory?

[Marcin Nowak]

I would like to set something like MIGRATION_MANAGED_APPS = (...) in my project`s settings.

Every not listed app should not be inspected and should be excluded when calling `makemigrations` command. 

Setting MIGRATION_MANAGED_APPS to None (default) should enable backward compatibility.

[Andrew Godwin]

If we're going to introduce another setting, which I'm not super-keen on, then let's made it exclusionary, something like MIGRATION_UNMANAGED_APPS.

Or, we could do something with the existing MIGRATION_MODULES setting, but migrate needs that too so it might be confusing things.

Andrew

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAB2m7CxFSR8sg7kVxPYihrPrsrZx3sa9650k_mhfLoFgGNGgBA%40mail.gmail.com.

[James Bennett]

One option for declaring an app "unmanaged" could be via AppConfig. That's fairly clean, mirrors the way you can already set a model as unmanaged in its Meta declaration, and allows for the end user of the app to override by supplying an AppConfig which will generate migrations (should they, for some use-case reason, want that).

[Andrew Godwin]

On Fri, Jul 17, 2015 at 11:19 AM, James Bennett <ubern...@gmail.com> wrote:

One option for declaring an app "unmanaged" could be via AppConfig. That's fairly clean, mirrors the way you can already set a model as unmanaged in its Meta declaration, and allows for the end user of the app to override by supplying an AppConfig which will generate migrations (should they, for some use-case reason, want that).

Now this I like.

Andrew